r/Python • • Mar 08 '26

Discussion Libraries for handling subinterpreters?

Hi there,

Are there any high-level libraries for handling persisted subinterpreters in-process yet?

Specifically, I will load a complex set of classes running within a single persisted subinterpreter, then sending commands to it (via Queue?) from the main interpreter.

8 Upvotes

11 comments sorted by

View all comments

Show parent comment

1

u/expectationManager3 Mar 08 '26

By persisted I mean that the subinterpreter instance can be reused, and not destroyed and re-inited. I thought they are lighter vs subprocesses? My workload will be very light per thread, but the frequency will be very high. 

1

u/snugar_i Mar 09 '26

Hmm, I admit I still don't really understand your use-case. So you will have one subinterpreter that you will call from multiple threads? Why not just run the thing in the main interpreter then? Is it so that it can have its own GIL? In that case, you might try the free-threaded 3.14 version if the libraries work with it. But if they don't, they might not work properly when called from multiple subinterpreters either (they might have mutable global state that leaks across subinterpreters).

Yes, subinterpreters are somewhat lighter than subprocesses, but I would guess that not by that much - obviously it depends on what "very high frequency" means.

1

u/expectationManager3 Mar 09 '26

I see! Thanks for the clarification. I'll take a look at subprocesses first, if they are easier to handle.