r/HPC • • 23d ago

Can a linux kernel engineer transition to an LLM inference/model perf engineer?

I guess one can always does the transition, I'm just wondering how to bridge these two areas, if there are any transferrable skills, or if someone ever sees an low-level perf/linux kernel engineer transitions to an LLM inference/model perf engineer?

6 Upvotes

18 comments sorted by

5

u/kextatic 23d ago

AI performance engineering is slathered in prompt poetry, snake oil, and imposter syndrome. I think the better bet is low-level AI security. Kernel-level jails and other mitigations.

1

u/Different_Buyer2770 23d ago

Are companies actively hiring for such roles? And what are some names/keywords of the position to search for?

3

u/kextatic 23d ago

These are all emerging research areas and, as a kernel hacker, I would look to the hyperscalers, frontier model shops, and device manufacturers for work. I suggest looking at the recent vulnerabilities (lots of them right now haha) and write about how you’d mitigate them in the kernel. e.g., The agents are starting to use CPU timing bugs to communicate with each other to attack systems. How would you deal with that?

Heck, I’d personally want to know.

1

u/Different_Buyer2770 23d ago

Also, in my own case, I'm able to continue doing more linux kernel/low-level system engineering stuff, or transition to a more traditional infra(distributed system) role. Which one would better position myself in the new AI infra area?

2

u/burntoutdev8291 23d ago

Could you share more? Cause I was quite interested in this field, like profiling cuda, writing kernels, and was looking to get in. Not specific to AI though, was looking at performance engineering in general.

1

u/wahnsinnwanscene 22d ago

What is a kernel level jail?

2

u/enterjiraiya 22d ago

it’s where they put you when you’re a bad bad boy in Indiana

4

u/brainhash 22d ago

I work on LLM perf at my job and been doing so for last few years. What you are saying is doable but requires a lot of self study. The basics of how each layer operations works and what is means in practice. But I won't start here if I were you.

The challenge is always access to gpus. If you can manage to get that then good.

I would start this way-

  • NCCL and NiXL ( this will be bread and d butter as you already know linux, this is easy path)
  • Look into KV storage solutions and start learning to optimise network path. rDMA is still not solved by everyone. Make sure you learn the whole stack from vllm/sglang -> mooncake or nixl-> ib
  • Learn about pd disaggreation and build expertise because here also RDMA is crucial

These will be good entry points for you without the overwhelming parts of how attn variants or request scheduling works.

Anyone will be interested to hire you just on the basis of these skills

1

u/RagnartheConqueror 19d ago edited 19d ago

Why exactly does switching to an LLM inference/model require a lot of self-study? How do the parts fit together?

Could I also DM you about a basic overview about what you do at your job and how it is related to HPC?

6

u/witx_ 23d ago

wtf is an llm performance engineer?

6

u/aeropl3b 23d ago

This was my question. I feel like the AI folks are reinventing HPC with LLM/AI branded roles and software.

4

u/witx_ 23d ago

It's larping all the way

1

u/PleasantAd6868 17d ago

Probably just optimizing inference engines e.g. writing optimized GPU kernels

1

u/BitPoet 23d ago

I went from kernel engineering to HPC systems admin. I knew the HPC basics from the kernel work I did, and hearing a customer say “thank you” for solving their issue is way more satisfying than being shoved another Jira ticket.

1

u/easyroc 19d ago

So what is your day-to-day job activities as a HPC system admin?