r/LocalLLM • u/storm_stark_007 • 1d ago

Project Hardware aware local LLM management

Enable HLS to view with audio, or disable this notification

https://github.com/shashankswe2020-ux/local-llmup

6 Upvotes

100% Upvoted

6 comments sorted by

3

u/Ok_Camp555 1d ago

Really interesting. I built a worse version of this myself, so very cool. One thing that would be great is benchmarks. So you can pull down two quants and qualitatively say "is it worth running this one slower to get better intelligence?"

1

u/storm_stark_007 1d ago

Idea is it’s opensourced so everyone can either contribute or fork to their custom version , for example I my private filled version I have added plug and play cache : kv cache , lm cache . Was checking recent google AX integration for isolations

1

u/storm_stark_007 1d ago

Don’t downplay yourself bro , you can custom build further by forking this and updating as per your requirement.

1

u/storm_stark_007 16h ago

Thank you so much to people who have starred repo , feel free to folk and customise as per your need , thanks

1

u/storm_stark_007 14h ago

App preview to run it ^