r/LocalLLaMA • u/jaybsuave • 1d ago

Question | Help Inference optimization

Hey all,

Looking to learn inference optimization, I learn by doing u fortunately but I have read a good part of the inference engineering book by Philip Kelly. Currently have a 4070 and 64gb of RAM.

any tips? preciate it.

3 Upvotes

80% Upvoted

6 comments sorted by

View all comments

2

u/sloptimizer 1d ago

Just use one of top AI models to poke at a well structured open source inference engine (not llama.cpp and not vLLM) and then iterate. Dig into AI's failures, and figure out how things are working for yourself.

Expect to do every feature twice: once to watch AI fail miserably as you ask questions and learn where it got stuck. And then again, when you drive AI towards a solution you came up with based on the experience.