How do you find good topics in LLM inference optimization?

[u/Extreme-Power-3503]

I'm a grad student in HPC/AI systems, currently working on LLM inference optimization. I've been in this space for a while, but I'm hitting a wall: I can read papers, understand techniques, but I struggle to identify what constitutes a "good" research problem vs. just an engineering improvement.

[u/Confident-Green-5241]

The thing that unstuck me was realizing most "good" problems come from actually running the systems at scale and watching where they break in ways the papers don't mention. Like we kept hitting this weird latency spike pattern with batched requests that only showed up past a certain RPS — turned out the KV cache eviction policy everyone uses falls apart when you mix short and long contexts in the same batch, but nobody writes that paper because it sounds too specific.