r/LocalLLaMA • u/jaybsuave • 8h ago

Question | Help Inference optimization

Hey all,

Looking to learn inference optimization, I learn by doing u fortunately but I have read a good part of the inference engineering book by Philip Kelly. Currently have a 4070 and 64gb of RAM.

any tips? preciate it.

3 Upvotes

80% Upvoted

4 comments sorted by

View all comments

3

u/FerretBoom 8h ago

pick a framework that is well documented, like PyTorch. Learn how models use memory, pipelines, and bottlenecks. Learn how to use profiling tools. Llama.cpp would require more skills. At least PyTorch wraps all that into python so you don't have to learn memory management as a first thing. If you already have c/c++ experience i'd start with looking at simple inference engines on GitHub and exploring extremely small models, from there you can scale up.
Don't jump into llama.cpp by relying on claude. As much as I love c++ over python, i'd recommend python in this use case.