Library Comparison

Choosing the right concurrency model or task broker is critical for your application’s performance, stability, and developer velocity. This guide compares Pyroxide with other common Python concurrency patterns, backed by the empirical measurements detailed in the Performance & Benchmarks chapter.

1. Pyroxide vs. Python multiprocessing

Python’s built-in multiprocessing module (and ProcessPoolExecutor) runs tasks in separate Python interpreter processes to bypass the Global Interpreter Lock (GIL).

Comparison

• Memory Overhead: multiprocessing forks or spawns brand new OS processes, which duplicate Python interpreter memory and increase startup latency. Pyroxide runs lightweight background OS threads in the same process with negligible overhead.
• IPC (Inter-Process Communication): Data sent to a subprocess must be serialized (using pickle) and sent over OS pipes/sockets. For large payloads (>=1MB), this creates severe CPU and memory copying bottlenecks. Pyroxide uses a hybrid routing approach: small payloads go over sockets, while large payloads (>=1MB) are written to OS Shared Memory (SHM) using a zero-copy mapping mechanism, bypassing serialization.

Hardware: Apple M1 Pro (8 Cores)

Large-Scale Concurrency: Odoo Ledger Audit Benchmark (9.62 MB Arrow Table)

To evaluate performance under heavy database/recordset serialization workloads (e.g. Odoo Ledger Audits), we compared the execution times of 10 concurrent requests processing a 9.62 MB Apache Arrow Table:

Analysis of Results

1. ProcessPoolExecutor is Slow: Spawning processes and using standard multiprocessing IPC is heavy. While it bypasses the GIL, the pickle serialization overhead limits the speedup to only 1.17x.
2. Threads hit the GIL: Both ThreadPoolExecutor and Pyroxide’s default @task hit the GIL ceiling around 0.32s-0.38s.
3. Pyroxide SHM Isolated Bypasses GIL Safely: Pyroxide’s isolated=True runs the task inside warm workers using cross-platform local sockets (Unix Domain Sockets / Named Pipes) and maps the 9.62MB payload via zero-copy shared memory, keeping latency low.
4. Native Code + SHM is the Ultimate Speedup: By writing the audit loop in a compiled C/Rust dynamic plugin, Pyroxide completely bypasses the Python interpreter and the GIL, finishing the entire 9.62MB workload in just 9 milliseconds—achieving a 35.3x speedup over standard Python threads.

Decision Matrix

Tip

Use Python Multiprocessing when:

• You want to stick strictly to the Python standard library with zero external dependencies.
• You have simple payloads that serialize (pickle) quickly.

Use Pyroxide when:

• You want to run CPU-heavy pure Python code GIL-free but want to avoid standard multiprocessing’s heavy startup overhead (using isolated=True with its warm worker pool).
• You have large byte/string payloads and need zero-copy memory performance (using standard isolated=False threads).
• You want to offload I/O-bound or blocking Python callbacks using a low-overhead, in-process thread pool.
• You want to run Rust, C, Zig, or WebAssembly sandbox plugins GIL-free with microsecond-level dispatch latencies.

2. Pyroxide vs. Celery / RQ

Celery and RQ are distributed task queues designed to run jobs on separate worker machines.

Comparison

• Complexity: Celery requires setting up a message broker (like Redis or RabbitMQ) and running separate worker daemon processes. Pyroxide is embedded inside your Python process; it has zero infrastructure dependencies.
• Latency: Celery tasks suffer from TCP round-trips, broker serialization, and polling delay, taking 4.8 ms to 12.5 ms even on localhost, as shown in Benchmark Scenario F. Pyroxide task dispatch and completion signaling takes 25 microseconds (200x to 500x faster) because it executes entirely in-process using OS futex signaling.

Decision Matrix

Tip

Use Celery / RQ when:

• You need distributed, multi-server horizontal scaling.
• You need persistence (saving tasks to disk in case the server crashes).
• You need advanced workflows (task chains, chords, scheduling cron jobs).

Use Pyroxide when:

• You need high-performance, in-process background pipelining.
• You want zero external dependencies (no Redis/RabbitMQ to configure or maintain).
• Microsecond latency is required for high-frequency task offloading.

3. Pyroxide vs. Raw PyO3 / Maturin C-Extensions

Writing a custom Rust C-Extension using PyO3 is the standard way to speed up Python with Rust.

Comparison

• Development Speed: With raw PyO3, any change to your native code requires compiling a new wheel, rebuilding the Python environment, and re-deploying. Pyroxide compiles and loads dynamic code on-the-fly (compile_dylib()), providing rapid iteration directly in Python script files.
• GIL Offloading: PyO3 requires manually calling py.allow_threads(...) to release the GIL, and managing cross-thread safety. Pyroxide abstracts all thread dispatching, lock-free status monitoring, and GIL-release boundaries automatically.
• Call Overhead: Statically compiled raw PyO3 function calls have an overhead of 0.2 µs - 0.8 µs. As measured in Benchmark Scenario G, Pyroxide @dylib_task runs at 1.0 µs call overhead, meaning Pyroxide matches raw PyO3 speeds with zero runtime penalty while gaining dynamic compilation.

Decision Matrix

Tip

Use Raw PyO3 / Maturin when:

• You are shipping a statically packaged Python library to PyPI for other developers to use.
• You need zero-copy sharing of complex data structures (like sharing ndarray objects using the buffer protocol).

Use Pyroxide when:

• You are building application code and want to rapidly prototype compiled native logic without complex build/distribution pipelines.
• You want to run safe, sandboxed calculations using WebAssembly (@wasm_task).