Dynamic Shared Libraries (dylib)

For use cases that require maximum performance and unrestricted system access (such as database connections, direct socket I/O, or local file access) but must avoid rebuilding Pyroxide itself, Pyroxide supports Dynamic Shared Library execution.

With this architecture:

• Workloads are executed completely GIL-free on background OS threads.
• You can compile source code on-the-fly at runtime, or load pre-compiled binaries directly.

When to Use What

The ABI Contract

Any dynamic shared library loaded by Pyroxide (whether compiled on-the-fly or pre-compiled) must export exactly two C-compatible functions:

#![allow(unused)]
fn main() {
/// Executes the plugin logic. Receives input bytes, returns output bytes.
/// The output buffer MUST be allocated by the plugin's own allocator.
#[no_mangle]
pub unsafe extern "C" fn pyroxide_plugin_run(
    ptr: *const u8,    // Pointer to input bytes
    len: usize,        // Length of input bytes
    out_len: *mut usize // Write output length here
) -> *mut u8;          // Return pointer to output bytes

/// Deallocates the output buffer returned by pyroxide_plugin_run.
/// Required because host and plugin may use different allocators.
#[no_mangle]
pub unsafe extern "C" fn pyroxide_plugin_free(
    ptr: *mut u8,      // Pointer to free
    len: usize         // Length of allocation
);
}

Why two functions? Memory allocated inside a dynamic library cannot be safely freed by the host process (they may use different allocators). The _free function ensures deallocation happens on the correct allocator.

1. On-the-Fly Compilation

On-the-fly compilation compiles source code strings into shared libraries at runtime using your local toolchain, then registers them with the engine.

Rust (compile_dylib)

Uses your local cargo toolchain under the hood to compile Rust source code. You can also specify Cargo dependencies.

from pyroxide import compile_dylib, dylib_task

RUST_LOGGER = """
use std::fs::OpenOptions;
use std::io::Write;

#[no_mangle]
pub unsafe extern "C" fn pyroxide_plugin_run(ptr: *const u8, len: usize, out_len: *mut usize) -> *mut u8 {
    let input = std::slice::from_raw_parts(ptr, len);
    let message = std::str::from_utf8(input).unwrap_or("invalid utf8");

    if let Ok(mut file) = OpenOptions::new().create(true).append(true).open("app.log") {
        let _ = writeln!(file, "[Log] {}", message);
    }

    let response = format!("Logged: {}", message).into_bytes();
    *out_len = response.len();
    let boxed = response.into_boxed_slice();
    Box::into_raw(boxed) as *mut u8
}

#[no_mangle]
pub unsafe extern "C" fn pyroxide_plugin_free(ptr: *mut u8, len: usize) {
    let _ = Box::from_raw(std::slice::from_raw_parts_mut(ptr, len));
}
"""

# Compile and register. Optional Cargo dependencies can be provided.
compile_dylib("file_logger", RUST_LOGGER)

@dylib_task("file_logger")
def log_event(message: str) -> str:
    pass

# Execute GIL-free
print(log_event("User login").result())  # "Logged: User login"

C (compile_c)

Uses your local C compiler (clang or gcc via CC environment variable) to compile a C source string.

from pyroxide import compile_c, dylib_task

C_SRC = """
#include <stdint.h>
#include <stdlib.h>

uint8_t* pyroxide_plugin_run(const uint8_t* ptr, size_t len, size_t* out_len) {
    uint8_t* res = (uint8_t*)malloc(len);
    for (size_t i = 0; i < len; i++) {
        if (ptr[i] >= 'a' && ptr[i] <= 'z') {
            res[i] = ptr[i] - 32;
        } else {
            res[i] = ptr[i];
        }
    }
    *out_len = len;
    return res;
}

void pyroxide_plugin_free(uint8_t* ptr, size_t len) {
    free(ptr);
}
"""

compile_c("c_upper", C_SRC)

@dylib_task("c_upper")
def to_upper_c(payload: str) -> str:
    pass

print(to_upper_c("hello from c").result())  # "HELLO FROM C"

Zig (compile_zig)

Uses your local zig build-lib toolchain to compile a Zig source string.

from pyroxide import compile_zig, dylib_task

ZIG_SRC = """
const std = @import("std");

export fn pyroxide_plugin_run(ptr: [*]const u8, len: usize, out_len: *usize) [*]u8 {
    const allocator = std.heap.page_allocator;
    const output = allocator.alloc(u8, len) catch unreachable;
    @memcpy(output, ptr[0..len]);
    for (output) |*char| {
        if (char.* >= 'a' and char.* <= 'z') {
            char.* -= 32;
        }
    }
    out_len.* = len;
    return output.ptr;
}

export fn pyroxide_plugin_free(ptr: [*]u8, len: usize) void {
    const allocator = std.heap.page_allocator;
    allocator.free(ptr[0..len]);
}
"""

compile_zig("zig_upper", ZIG_SRC)

@dylib_task("zig_upper")
def to_upper_zig(payload: str) -> str:
    pass

print(to_upper_zig("hello from zig").result())  # "HELLO FROM ZIG"

2. Using Pre-Compiled Shared Libraries

If you already have a compiled shared library file (.so / .dylib / .dll), you can bypass the compilation phase entirely and load it directly using register_dylib.

Supported Languages

Because register_dylib expects a standard C-ABI shared library, you can write the library in any language that supports compiling to a C-compatible shared library (including Rust, C/C++, Zig, Go via -buildmode=c-shared, Nim, Fortran, and others).

Pre-Compiled Example (C Language)

Here is a C library example (my_math.c):

#include <stdint.h>
#include <stdlib.h>
#include <string.h>

// Required run symbol matching Pyroxide's expectations
uint8_t* pyroxide_plugin_run(const uint8_t* ptr, size_t len, size_t* out_len) {
    uint8_t* result = (uint8_t*)malloc(len);
    memcpy(result, ptr, len);
    *out_len = len;
    return result;
}

// Required free symbol
void pyroxide_plugin_free(uint8_t* ptr, size_t len) {
    free(ptr);
}

Compile it to a shared library:

gcc -shared -o libmy_math.so -fPIC my_math.c

Load and execute it in Python:

from pyroxide import register_dylib, dylib_task

# Load the pre-compiled C library directly
register_dylib("c_math", "./libmy_math.so")

@dylib_task("c_math")
def process_data(payload: bytes) -> bytes:
    pass

handle = process_data(b"hello C-ABI")
print(handle.result())  # b"hello C-ABI"

Security Warning

Caution

Dynamically loaded shared libraries run directly inside CPython’s process memory. An unhandled segfault, null pointer dereference, or buffer overflow will crash the entire Python process. Only load trusted code via compilation helpers or register_dylib().