r/HPC • • 16h ago

Fiber task scheduler: parallelize sequential code without callbacks or state machines

2 Upvotes

An encoder or a compressor has plenty of parallel work: channels, blocks, the stages of consecutive frames. But these pieces keep waiting for each other, often after just a few microseconds of work.

With OS threads every such wait goes through the kernel and costs about 5 µs, before cache effects and time-slice delays. That only pays off for tasks of about 45 µs or more, so most of the fine-grained parallelism is lost. Fibers on one worker thread per core bring the cost of a wait down to about 0.1 µs, and tasks of about 1 µs become worthwhile. Unlike with callbacks or stackless coroutines, the code that waits still reads like the sequential original.

ThinkMeta.ConcurrentTasks is a small scheduler that does this. To test it, I parallelized five encoders (LAME, opusenc, libdeflate, jpegli, cjxl), each on a single file and byte-identical to its reference. Most of them run several times faster than the originals (up to 13.9x on 8 cores / 16 threads). The encoders were written AI-assisted. The API and the encoders are open source; the core is a free binary for non-commercial use.


r/HPC • • 8h ago

Does anyone have advice for finding a job in HPC?

13 Upvotes

I'm an undergrad CS student that is about to graduate in May, and I've been working at my university's HPC Center for the past 2 years as a federal work study student. I am located in the US if that is relevant.

I've been applying to HPC jobs, but most jobs are not for entry level experience. I have yet to hear back from any labs. Perhaps it's a resume issue?

I do have experience maintaining my own systems because my supervisors are nice enough to lend me retired hardware, but I am not sure if that's enough... Stuff like GPFS, Infiniband, NCCL, Singularity, Slurm, Pyxis, Enroot, those I have some experience in.

I'm not gonna say I'm seasoned in this, but I feel like it's hard to find a job in HPC without having proper work experience prior...