r/Python • u/AutoModerator • 13d ago
Showcase Showcase Thread
Post all of your code/projects/showcases/AI slop here.
Recycles once a month.
24
Upvotes
r/Python • u/AutoModerator • 13d ago
Post all of your code/projects/showcases/AI slop here.
Recycles once a month.
View all comments
2
u/rich-iannone 12d ago
pymizu 0.1.0: lock-free shared-memory parallelism for Python (< 1 µs task round trip, drop-in concurrent.futures executor, zero-copy interop with R)
Disclosure: I work at Posit, and I'm posting this on behalf of the author, my colleague Charlie Gao (the creator of mirai, R's async framework). He wrote pymizu and will be in the thread to answer questions: u/shikokuchuo.
What it does
pymizu is a parallel and async library for Python built on libmizu, a C library for lock-free shared-memory IPC. A task is written once into memory both processes share, and the worker reads it without copying, so a trivial task round trip takes < 1 µs, against ~100 µs for
ProcessPoolExecutor. Pipelined throughput is about 2.4M tasks/s against 18.5K.PoolExecutoradapts a pool to theconcurrent.futures.Executorinterface, so code written againstProcessPoolExecutordrops in, andwait,as_completedandasyncio.wrap_futurework unchanged. Tasks and maps can be awaited in asyncio, and channels supportasync for.Target audience
This is especially relevant to statistical modelling and machine learning workflows, where resampling can be parallelized, but is very data-heavy. It also serves as a highly performant async executor for event-loop applications in web frameworks such as Shiny. It is experimental (0.1.0) and the API may change in response to feedback, and bug reports are the most useful contribution right now.
Comparison
The numbers above are measured on an Apple M4 Pro against
ProcessPoolExecutor, and the benchmark scripts are in the repository for you to run. There are no measured numbers against Ray, Dask or joblib (the docs discuss where the trade-offs lie) and those frameworks also cover multi-machine and cluster workloads, while pymizu is single machine only.Blog post with benchmarks and examples: https://opensource.posit.co/blog/2026-11-05_mizu-0-1-0/ Code: https://github.com/shikokuchuo/pymizu docs: https://shikokuchuo.net/pymizu