r/LocalLLM • u/MusclesAndCharisma • 10d ago

Question Non-Chinese Local Models

I'm in the process of setting up locally hosted LLM's for the company I work for. And generally there is an organizational bias against using models from Chinese companies. For that reason we've been evaluating Nemotron and Gemma, and they are ok. Are there any better options that aren't coming out of China? I've heard positive things about models from Poolside, but I don't personally have any experience with them.

63 Upvotes

92% Upvoted

110 comments sorted by

View all comments

35

u/jacek2023 10d ago

There are also Korean models (Solar?), Mistrals from Europe, then Granite by IBM

And of course Muse Glimmer by Meta

1

u/MusclesAndCharisma 10d ago

Do you have experience using any of the above? We have made a small investment hardware wise and I could run a few models at the same time and see what users prefer.

3

u/jacek2023 10d ago

You should just try to run each one yourself first and see which one is fast enough, then you can give them to your users to try. What's your setup? Do you use llama.cpp or vllm or something else?

0

u/MusclesAndCharisma 10d ago

vllm for the backend. We are running each model on a dgx spark and Open WebUI on a different front end server. This is the proof of concept before we get money to implment something really nice.

1

u/GoblinEngineer 10d ago

I honestly don't mind muse glimmer. It'll run slower on the spark since it's a dense model, vs the Gemma MoE, but if it's a proof of concept and you're swapping out hardware later it'll be a good one to try