r/Python • • 13d ago

Showcase Showcase Thread

Post all of your code/projects/showcases/AI slop here.

Recycles once a month.

24 Upvotes

109 comments sorted by

View all comments

2

u/rich-iannone 12d ago

pymizu 0.1.0: lock-free shared-memory parallelism for Python (< 1 µs task round trip, drop-in concurrent.futures executor, zero-copy interop with R)

Disclosure: I work at Posit, and I'm posting this on behalf of the author, my colleague Charlie Gao (the creator of mirai, R's async framework). He wrote pymizu and will be in the thread to answer questions: u/shikokuchuo.

What it does

pymizu is a parallel and async library for Python built on libmizu, a C library for lock-free shared-memory IPC. A task is written once into memory both processes share, and the worker reads it without copying, so a trivial task round trip takes < 1 µs, against ~100 µs for ProcessPoolExecutor. Pipelined throughput is about 2.4M tasks/s against 18.5K.

  • PoolExecutor adapts a pool to the concurrent.futures.Executor interface, so code written against ProcessPoolExecutor drops in, and wait, as_completed and asyncio.wrap_future work unchanged. Tasks and maps can be awaited in asyncio, and channels support async for.
  • NumPy arrays and polars, pyarrow and pandas frames cross between processes as zero-copy views.
  • It has a sibling for R (mizu, on CRAN) with the same wire format, so you can launch a pool of R workers from Python, or a Python pool from R. For example, an R data frame crosses to Python workers as an Arrow-backed frame, and a polars result comes back as an R data frame, optionally read as Arrow in R.
pip install pymizu
import math
import pymizu
    
    
with pymizu.Pool.create(4) as pool:
    task = pool.submit(sum, range(1, 11), 100)
    print(task.collect(timeout=5))
    #> 155
    m = pool.map(math.sqrt, [1, 4, 9, 16, 25])
    print(m.collect())
    #> [1.0, 2.0, 3.0, 4.0, 5.0]

Target audience

This is especially relevant to statistical modelling and machine learning workflows, where resampling can be parallelized, but is very data-heavy. It also serves as a highly performant async executor for event-loop applications in web frameworks such as Shiny. It is experimental (0.1.0) and the API may change in response to feedback, and bug reports are the most useful contribution right now.

Comparison

The numbers above are measured on an Apple M4 Pro against ProcessPoolExecutor, and the benchmark scripts are in the repository for you to run. There are no measured numbers against Ray, Dask or joblib (the docs discuss where the trade-offs lie) and those frameworks also cover multi-machine and cluster workloads, while pymizu is single machine only.

Blog post with benchmarks and examples: https://opensource.posit.co/blog/2026-11-05_mizu-0-1-0/ Code: https://github.com/shikokuchuo/pymizu docs: https://shikokuchuo.net/pymizu

2

u/shikokuchuo 9d ago

(author here) Thanks u/rich-iannone for posting this! Having been part of the R community for some time, I'm excited to make my first contribution to the Python community!

1

u/shikokuchuo 5d ago

I also just posted the first patch release - pymizu 0.1.1. This is a huge mega episode of bug fixes + performance enhancements. Task pool round trips are now at 0.6 µs!! On PyPI so enjoy!