Isolated Worker Processes

By default, Pyroxide runs tasks in background OS threads. In v0.5.0, you can use isolated=True to run tasks in isolated OS processes with high-performance cross-platform IPC and Zero-Copy Shared Memory (SHM) routing.

Why use isolated processes?

1. Crash Safety: If an unstable C extension or dynamic library triggers a Segmentation Fault (SIGSEGV), it only kills the worker process. The main Python app survives, and the task handle returns a RuntimeError.
2. True Python GIL Bypass: Pure Python @tasks normally hold the GIL. With isolated=True, the task runs in a separate Python interpreter process, bypassing the GIL completely.

Usage

Pass isolated=True to any Pyroxide decorator:

from pyroxide import task, wasm_task, dylib_task

# 1. Pure Python (GIL bypass & crash safety)
@task(isolated=True)
def heavy_computation(data: list) -> list:
    return [x * 2 for x in data]

# 2. WASM
@wasm_task("my_module", "process_data", isolated=True)
def process_wasm(data: str) -> str:
    pass

# 3. Dynamic Library
@dylib_task("unsafe_c_plugin", isolated=True)
def process_unsafe_c(data: bytes) -> bytes:
    pass

# Same API as in-process tasks
handle = heavy_computation([1, 2, 3])
print(handle.result())

Internals & Optimizations

1. Warm Worker Pool & Scale-to-Zero: Pyroxide pre-spawns worker processes so there is no execution-time process startup latency. To minimize memory usage, an idle reaper thread automatically scales the pool to zero by terminating workers idle for more than 60 seconds (non-blocking “Steal and Kill” pattern).
2. Cross-Platform Local Sockets: IPC uses Unix Domain Sockets on Linux/macOS and Named Pipes on Windows (backed by the interprocess crate), avoiding slow TCP loopback overhead.
3. Hybrid Zero-Copy Shared Memory (SHM): For small payloads (< 1MB), data is sent directly over the local socket. For large payloads (>= 1MB), Pyroxide utilizes OS-level Shared Memory (shared_memory crate) for zero-copy transfers, avoiding serialization bottleneck.
4. Lifecycle: Workers are single-threaded. When a task completes, the worker is reused. If a worker crashes, Pyroxide drops it and spawns a replacement.

When to use isolated=True

Limitations

• Memory Copying: Although SHM routing provides zero-copy across the process boundary, there is still serialization/deserialization overhead for complex Python objects.
• Resource Isolation: Ensure your system supports shared memory mapping (standard on modern macOS, Linux, and Windows). If SHM creation fails, Pyroxide gracefully falls back to socket transmission.