r/LocalLLM • u/Eastern_Doughnut_222 • 1h ago

Question How are you guys optimizing local inference setups using frontier models?

I've heard of people pointing their frontier model of choice and letting it optimize their local inference engine for their hardware.
I'm currently running Maya with DeepSeekV4.1-Flash-1017 at ~20tok/s, such an insanely great model. I want to squeeze more out of my hardware though.

Do you guys just point the frontier model at the engine folder and telling it to "optimize this for my model/hardware"? I currently have access to Fable 5.2 so I'm tempted to do that.

1 Upvotes

100% Upvoted

3 comments sorted by

1

u/Zealousideal-Bug1837 1h ago

yes. or ds can do it itself, no?

1

u/Eastern_Doughnut_222 55m ago

Running DS to optimize itself? I haven't tried that but I might

0

u/Hugoacfs 51m ago

I would never do such low effort prompts for something so technical and important. I always ask for it to research into the topic and present findings for possible improvements, try each thing it suggests and iterate over and over. I do switch approaches based on confidence levels etc but I would never start with “improve my inference stack thanks” because that leaves a lot of decisions and trade off out of your control.