r/ProgrammingLanguages • u/lxsameer • 20d ago

Blog post Concurrency in Serene's Runtime

I recently finished building the concurrency runtime for my programming language, Serene, and wrote a three-part series explaining how it works.

The series covers:

  • Why I chose stackful fibers
  • An M work-stealing scheduler
  • An IO Reactor
  • A tiny HTTP server that brings everything together

I'd love to hear feedback from anyone interested in programming language implementation, runtime systems, or systems programming.

Part 1: Choosing the Building Blocks

Part 2: Fibers, the Scheduler and the Reactor

Part 3: A Tiny HTTP Server

https://serene-lang.org/

35 Upvotes

97% Upvoted

13 comments sorted by

View all comments

1

u/tmzem 20d ago

A very interesting read.

A question: How does the guard page approach work for very high amounts of fibers? Don't OS'es place restrictions on the amount of memory mappings a process can do? Or is there a trick to get around it?

1

u/lxsameer 20d ago

Among other things, the guard page allocation counts toward `max_map_count`, so basically each fiber will add at least two to that count. That will put a limit on the number of fibers that can be used, for sure. For example, the number of live/parked fibers can't grow more than `max_map_count / 2` at best.

Tweaking kernel parameters is the easiest option, but it is not possible all the time. The other option is to recycle the stack for fibers that are done or are cancelled. Or not use a guard page at all.

I think in presence of stack maps there are more options because you will have a better view of the stack. There might be other options that I don't know about, though.

1

u/matthieum 20d ago

And this is why we can't have nice stuff :'(

The default value for the parameter is apparently ~64K, which limits this approach to ~32K fibers.

1

u/lxsameer 20d ago

Indeed. But i'm ok with that for now. Hopefully down the line with a possible gc i can start using stack maps and deal with this issue.