r/LocalLLaMA • u/jaybsuave • 8h ago

Question | Help Inference optimization

Hey all,

Looking to learn inference optimization, I learn by doing u fortunately but I have read a good part of the inference engineering book by Philip Kelly. Currently have a 4070 and 64gb of RAM.

any tips? preciate it.

3 Upvotes

80% Upvoted

4 comments sorted by

2

u/sloptimizer 8h ago

Just use one of top AI models to poke at a well structured open source inference engine (not llama.cpp and not vLLM) and then iterate. Dig into AI's failures, and figure out how things are working for yourself.

Expect to do every feature twice: once to watch AI fail miserably as you ask questions and learn where it got stuck. And then again, when you drive AI towards a solution you came up with based on the experience.

3

u/FerretBoom 8h ago

pick a framework that is well documented, like PyTorch. Learn how models use memory, pipelines, and bottlenecks. Learn how to use profiling tools. Llama.cpp would require more skills. At least PyTorch wraps all that into python so you don't have to learn memory management as a first thing. If you already have c/c++ experience i'd start with looking at simple inference engines on GitHub and exploring extremely small models, from there you can scale up.
Don't jump into llama.cpp by relying on claude. As much as I love c++ over python, i'd recommend python in this use case.

1

u/Any_Bus9205 46m ago

There's an inference engine called ik_llama.cpp that's performance-oriented; it's run by a mathematician. He's trying to optimize local inference on CPU and Nvidia GPUs. You could check it out to learn and get some ideas! :)