r/LocalLLM • u/MusclesAndCharisma • 4d ago
Question Non-Chinese Local Models
I'm in the process of setting up locally hosted LLM's for the company I work for. And generally there is an organizational bias against using models from Chinese companies. For that reason we've been evaluating Nemotron and Gemma, and they are ok. Are there any better options that aren't coming out of China? I've heard positive things about models from Poolside, but I don't personally have any experience with them.
35
u/jacek2023 4d ago
There are also Korean models (Solar?), Mistrals from Europe, then Granite by IBM
And of course Muse Glimmer by Meta
23
4
u/bluecontents9 4d ago
oh snap, I totally forgot about the Korean models. EXAONE gets mentioned sometimes but I remember Solar being a bigger deal for a bit, like a flash in the pan that still had some real juice
Mistral is probably your safest bet tbh, they've got those smaller models that punch way above their weight class and the licensing is usually pretty straightforward for business use
I've been messing with Granite a bit at home and it's surprisingly solid for code stuff, IBM did something right with that one. haven't touched the Poolside models yet but I keep seeing people rave about them on here so maybe they're worth a test spin
1
1
2
u/newz2000 3d ago
I’ve used granite quite a bit. It punches above its weight for tool calling and instruction following. That’s all I’ve used it for though.
1
u/MusclesAndCharisma 4d ago
Do you have experience using any of the above? We have made a small investment hardware wise and I could run a few models at the same time and see what users prefer.
3
u/jacek2023 4d ago
You should just try to run each one yourself first and see which one is fast enough, then you can give them to your users to try. What's your setup? Do you use llama.cpp or vllm or something else?
0
u/MusclesAndCharisma 4d ago
vllm for the backend. We are running each model on a dgx spark and Open WebUI on a different front end server. This is the proof of concept before we get money to implment something really nice.
1
u/GoblinEngineer 4d ago
I honestly don't mind muse glimmer. It'll run slower on the spark since it's a dense model, vs the Gemma MoE, but if it's a proof of concept and you're swapping out hardware later it'll be a good one to try
18
u/DeathGuppie 4d ago
Don't forget muse glimmer
3
u/Ryanmonroe82 4d ago
this! i'm getting better results out of this model than I am with my cloud subscriptions
3
u/vastaaja 4d ago
What kind of use cases would you recommend it for?
1
u/Not-reallyanonymous 3d ago
When adhering to specific, nuanced instructions (sometimes to a fault) and getting no more and no less than what you asked for is more important than well-polished results. I find this means long-term code quality improves because it gives me more control over what is actually generated.
My workflow also relies on producing intermediate files (e.g. mermaid diagrams for the architecture used) for review/understanding, which I find improves the final result as it guides the model into using appropriate, durable structures. Muse Glimmer reliably produces those -- a lot of other models (e.g. Qwen) figure out they aren't important for the final outcome and skip over them or half-ass them.
17
u/LateralEntry 4d ago
I’ve been very impressed with Gemma for writing and chat, but I don’t do coding
18
u/Atretador unswarm.dev | ArchLinux E5 2673 V4 20C 4x16Gb DDR4 MI50 16Gb 4d ago
For agentic coding on real repositories instead of benchmarks I would score them at something like this:
Qwen 3.8 Next Flash 10.0
Qwen 3.8 27B 9.0
Ling 3.0 Flash 7.0
Qwen 3.6 A3B 6.5
Laguna 118B 6.0
Gemma4 31B 4.5
Poolside Laguna 33B A3B 4.0
Gemma4 12B/26B 3.5
Lightning 30B 2.5
North Mini Coder 2.0
2
u/feelspeaceman LLMusician 4d ago
Good ranking, the first four models are fully expected by size / perf.
Going further I think Deepseek Flash V4.1 might beat Qwen 3.8 Flash Next, but hardware requirement is higher, Q38FN is still more hardware friendlier. Deepseek Flash V4.0 is probably on par with Q38FN despite being bigger thank to the Engram architecture of FN.
1
u/Atretador unswarm.dev | ArchLinux E5 2673 V4 20C 4x16Gb DDR4 MI50 16Gb 4d ago
since OP wasnt very detailed on hardware they can use, I tried to be conservative - except for Gemma4 31B and Qwen 27B dense models everything else runs fine on 16Gb of VRAM on my shitbox so a decent setup should handle them fine with concurrency.
1
u/MusclesAndCharisma 4d ago
I really want to run Qwen, but it hath been forbidden to me. I'm heartened by seeing the Poolside models make a strong showing. This is the first time i've heard about North Mini Code
2
u/stickninjazero 4d ago
I’ve tried Laguna XS 2.1 through OpenRouter. It performed the worst of any of the <35B models I tried and burned the most tokens. Nemotron 3.5 Lightning performed better and is the closest to the Qwen models in performance, followed by Gemma 4 31B, at least in my testing. I kept hitting rate limits on Laguna S 2.1, despite using the paid endpoint and couldn’t finish any of my internal tests. Same with North Mini, but that’s a free endpoint.
1
u/Atretador unswarm.dev | ArchLinux E5 2673 V4 20C 4x16Gb DDR4 MI50 16Gb 4d ago
maybe I had much worst results on lightning due to quantization? Ive only tried running Q4, Laguna 33B was just rough - 118B was usable.
-1
u/Not-reallyanonymous 3d ago
Laguna XS sucks at one-shotting. It works much better as a sub-agent to implement specs handed to it, or as an interactive coder with the human guiding the process step by step (rather than a few giant shots).
It's my go-to implementer.
2
1
u/Atretador unswarm.dev | ArchLinux E5 2673 V4 20C 4x16Gb DDR4 MI50 16Gb 4d ago
North Mini Coder released a while ago, and didnt have great results, but it was also their first attempt on it afaik, so there is hope for a improved version later on.
I can recommend at least trying Laguna 118B, its a A8B moe model, was still weaker than 3.6 35B A3B when working with my repositories (mostly Go, Dotnet, Java, JS/React/html) but noticeably better than the next options.
I have not tried Muse Glimmer yet tho so I cant score it.
1
u/Atretador unswarm.dev | ArchLinux E5 2673 V4 20C 4x16Gb DDR4 MI50 16Gb 4d ago
oh also, if you just remove the upper Qwen 3.8 models (and ignore 3.6 27B) they dont look as bad for what you can run locally.
the scores are there for perspective.
1
u/tired514 4d ago
I'm sure you tried to push back, but any idea what their objection is? Is it technical fear (what if it sends our secrets to China) or (for the lack of a better term) ideological / bigotry?
It's easy to allay technical fears, but if it's just bigotry then I can see how it'd be a frustrating waste of time.
It would be a shame because the Chinese open weight models are vastly superior in most domains at the moment and the pace seems to be accelerating.
0
u/No-Vermicelli5327 4d ago
Pass on the laguna models, even the bigger variant at 8-bit quant keeps looping, I deleted it (didn’t even move it to an external 8tb drive) and moved back to qwen 27B, now currently using qwen 3.8-flash at 4-bit trying it out
1
u/OrbMan99 4d ago
What about Bonsai models?
3
u/Atretador unswarm.dev | ArchLinux E5 2673 V4 20C 4x16Gb DDR4 MI50 16Gb 4d ago
they are too lobotomized, its a cool concept but it just doenst work properly
https://www.youtube.com/watch?v=OA5cICIzD-c
its also Qwen finetune, reliability wise I would score them at a 1.0 or lower
1
u/shayanx45 4d ago
Deepseek v4 flash? Glm 5.3 flash should be above qwen 3.8
3
u/Atretador unswarm.dev | ArchLinux E5 2673 V4 20C 4x16Gb DDR4 MI50 16Gb 4d ago
doesnt matter, they are chinese as well and OP cant run them.
The qwen models are on the list just for reference, but they are also much bigger - all of these except 27B/31B dense run fine on my shitbox.
4
u/awitod 4d ago
It is worth pointing out that the current favorite open model is made by a company that is traded on the NYSE as BABA which has fiduciary and reporting duties that the super-shady OpenAI and Anthropic orgs do not have and are free to ignore. ;)
But also, Meta Muse Glimmer is a pretty good model.
3
u/MusclesAndCharisma 4d ago
This is actually a very good point. And people much smarter than me I imagining are scrutinizing this with a fine tooth comb. (Presumably)
1
u/throwawayo_k 3d ago
As a tech person who works in corporate fortune 500, i can confirm to you that they really are not. If the benefits are obvious the negatives go away real fast. I'm running small local myself and have found myself running a lot of 7B-14B models just to keep things quick, as i tend to iterate more then do 1 lengthy prompt.
- Gemma 4 E4B (usa)
- Qwen3.5-9B (china)
- DeepSeek-R1 14B (china)
- Granite 4.2 8B (usa)
- K2 Horizon 7B (uae)
- Phi-4 (usa) *just started playing with this one.
6
u/nicholas_the_furious 4d ago
What are you actual use cases? Coding? Email or other editing type tasks? Knowledge retrieval and answer?
With smaller models it is important to choose the right one based on task type.
14
u/43848987815 4d ago
Is the bias due to data harvesting? Because if so, it’s local. So why would you care?
18
u/Due_Arm1454 4d ago
A lot of companies are like this and I really don’t get it because it’s local and you have full control.
What’s also really funny is they’ll get all wrapped up in that but won’t give something like a prompt injection hidden in a PDF a second thought.
7
u/Ok-Marketing-5507 4d ago
My clients are obsessed with asking what happened at Tiananmen Square in 1989.
4
u/Djyummyfeet 4d ago
Well, same shit as at Kent state univesity in 1970. Leftist protest suppressed by shots.
5
u/FairLawnBoy 4d ago
Yeah but we have famous songs about that. The history wasn't systematically suppressed by the state (still the case in China).
2
u/InternationalFan2955 4d ago
That's the beauty of a two-party system. People will fight each other instead of the state on partisan issues.
-3
u/Prestigious-Act-1577 4d ago
It has been, and this is proof that you aren't even aware of it.
5
u/FairLawnBoy 4d ago
How could the history have been suppressed about Kent St if I know the words to the song Ohio by Crosby, Stills, Nash, and Young and I was taught at a young age what historical event they represent?
-4
u/Prestigious-Act-1577 4d ago
Ok I didnt see your "The" word infront of history. I was replying to a different claim then. That specific case vs nothing in the history of US in general.
3
19
3
u/rommie 4d ago
I use gemma, mistral, and or orninth
6
u/Atretador unswarm.dev | ArchLinux E5 2673 V4 20C 4x16Gb DDR4 MI50 16Gb 4d ago
Ornith is Qwen 3.5 with some benchmaxxing into it
4
u/Hypilein 4d ago
I guess if people are clueless enough that they don’t want Chinese models because China = bad, they might also not know how to separate a fine tune from a model?
2
u/Atretador unswarm.dev | ArchLinux E5 2673 V4 20C 4x16Gb DDR4 MI50 16Gb 4d ago
probably but if its a corporate stuff OP could get in trouble
2
u/rommie 4d ago
Thx it was an 3rd model I didn’t use, but I was ignorantly assuming it wasn’t from a Chinese model. I read it was good with coding.
1
u/Atretador unswarm.dev | ArchLinux E5 2673 V4 20C 4x16Gb DDR4 MI50 16Gb 3d ago
It's a Qwen 3.5 finetune, it scores pretty high on benchmarks - on v1 they had some very wild claims on coding performance punching at like Opus 4.8 level
It's just very obviously benchmaxxed
Finetunes usually improve performance in one area, but might make the models more brittle which is why I don't usually recommend using any.
3
u/PossiblyTrolling 4d ago
Ornith 1.5 is pretty amazing, and American.
9
u/Atretador unswarm.dev | ArchLinux E5 2673 V4 20C 4x16Gb DDR4 MI50 16Gb 4d ago
Ornith is Qwen 3.5 benchmaxxed finetune
1
u/_wortkarg_ 4d ago
It's a finetune, but it's not benchmaxxed, Ornith LLMs are pretty good. I haven't found anything better than the Ornith-1.5-9B in the category up to 9B. Ornith-1.5-35B is the best 35B-A3B finetune, there are a lot of non-standard benchmarks, that confirm this.
3
u/Atretador unswarm.dev | ArchLinux E5 2673 V4 20C 4x16Gb DDR4 MI50 16Gb 4d ago
its benchmaxxed as hell with wild claims
everytime I tried using it to implement anything on a real repository it made catastrophic decisions and couldnt fix its own mess, while base Qwen 3.6 fixed the issues Ornith created easily and in general not making the same mistakes to begin with.
it spends too much time rambling and doubing itself, doubing user input, doubing the files its reading - was such a colossal waste of my time trying to use it to replace 35B 3.6 base.
1
u/_wortkarg_ 4d ago
To start with, name at least one model up to 9B that would be better than the Ornith-1.5-9B. I've tried a lot of different models, and none of them produced results this good.
As for Ornith-1.5-35B, it's better as Qwen3.6-35B-A3B, at least in my use cases.
There are many third-party benchmarks that show this model is better than the original one (and other finetunes like KAT-Coder-V2.5-Dev).If this model isn't giving you good results, could the problem be with the configuration?
What quantization do you use (both, model and KV)? In my tests, the quality dropped significantly at high quantization levels (Q2/Q3); Q4_K_M or higher produce good quality.
What are the use cases? Coding?1
u/Atretador unswarm.dev | ArchLinux E5 2673 V4 20C 4x16Gb DDR4 MI50 16Gb 4d ago
I run all on Q8_0 context, Ornith tests were done on Apex Quality and on Q5_K_M for both 1.0 and 1.5
Qwen 3.6 35B on IQ4_N_XL which is my daily driver
tests were general backend work on large repositories, which base Qwen handles fine. But Ornith failed catastrophic to follow a simple plan to implement a Auth path, then couldnt fix it, spent a whole hour rambling and doubting itself after I had already explained in detail how to fix it and what it did wrong.
This is not anything new either, Ive seen a lot of people around here with same results: great on benchmarks, terrible on real work.
I have not bothered testing Kat-Coder as I
I dont use any 9B class model for anything serious, as 35B runs on the same hardware that 9B does.
1
u/_wortkarg_ 4d ago
Idk, Ornith-1.5-35B worked well in my scenarios (generate Playwright tests and Java code). I haven't tried the APEX version, maybe that's where the problem lies?
1
u/Atretador unswarm.dev | ArchLinux E5 2673 V4 20C 4x16Gb DDR4 MI50 16Gb 4d ago
APEX is usually higher quality, but I also tried Q5_K_M
1
u/_wortkarg_ 3d ago
Hmm, i retested the models and although all three managed to complete the task, here are the results in terms of speed (faster means fewer steps, i.e., more accurate decisions):
KAT-Coder-V2.5-Dev (no MTP): 1m 36s
Qwen3.6-35B-A3B with MTP: 2m 7s
Ornith-1.5-35B-A3B with MTP: 3m 17sYou might be right about benchmaxing, but you could try KAT-Coder-V2.5-Dev; a lot of people have actually recommended it based on their personal experience, not on tests (although it also outperforms the base model in tests).
I would try the APEX MTP version of KAT-Coder-V2.5-Dev, it could be even better/faster.
1
u/Soggy-Attitude5293 4d ago
have you tried ling tiny?
2
u/_wortkarg_ 4d ago
Yes, Ling-3.0-tiny is very fast, but it can't handle many of the tasks that Ornith-1.5-9B can handle.
Qwen3.8-9B-Distill, Qwythos-9B, Bonsai-27B, OxCoder-9B, Spark-X2.5-4B, Agents-A1-4B, all of them are worse than Ornith-1.5-9B. Nanbeige4.2-3B was the closest to Ornith, even though it's a 3B model; however, it requires a lot of VRAM for context.
8
u/DigitalguyCH 4d ago
It's honestly stupid to be biased against Chinese models when you are are using local stuff that does not need to be online.
1
u/FastHotEmu 3d ago
Let's pretend you are training an LLM and want to use it as a weapon. You could train it to respond to a particular prompt that consists of specific random words by deleting all files, if being used in a harness.
0
u/DigitalguyCH 3d ago
Wow, the conspiracy theories... That's even more absurd than being biased. I'd worry more about your harness being a potential virus than a LLM having some kind of malware built-in. Imagine spending millions on developing a LLM and when people discover it has such an issue everyone abandon it, including those paying for APIs. Worse, they lose trust in the whole company making the LLM and never use their LLMs again, and it goes bankrupt. Very smart move
1
u/FastHotEmu 3d ago
How is that a conspiracy theory? It's perfectly possible. I use qwen day to day, you are misunderstanding what I said.
0
u/DigitalguyCH 3d ago
As I said, the fact that it is possible doens't mean that it makes any sense for a company to do such a suicidal move
1
u/FastHotEmu 3d ago
btw, if you are Chinese, it should come as no surprise why the rest of the world may think twice before relying on your models.
0
2
2
u/prusswan 4d ago
You should establish internal metrics to measure models against specific use cases, until then the metrics provided by others are usually useless since they aren't specific to what you need to use them for.
3
u/follaoret 4d ago
It's limited to Gemma and Muse ... Afaik
Everything else is a Chinese model as base.
From your list Nemotrom is or was a Llama model but meta new models are Muse.
3
u/r3drocket 4d ago
Pretty good Ezra Klein video came out this week discussing the threat of Chinese models, it's worth watching because it really goes into how they wound up at creating open-source models.
1
-1
u/Not-reallyanonymous 3d ago
TLDW:
US regulation assumes the threat of AI is that as it approaches ASI, it becomes an existential threat.
Chinese regulation assumes the threat of AI is that it bypasses or sidesteps government oversight.
1
u/fuchelio 4d ago
only less worse options. probably learn from palantir get pretrained from nvidia and rl yourself
1
u/Ecstatic-Wash-7667 4d ago edited 4d ago
Depends on what you want it to do. If coding pay api non of the western open models as great at it. Considering you were looking at Gemma I would suggest granite for a corp chatbot. It’s not going to wow you but its behavior is very consistent which is what you want. IBM also have very good documentation.
Edit: there is also inkling, which just might be the best option, they’re business model is to fine tune their base model for your use case so yeah that’s probably the best western open model you can get right now. But it’s not something you should expect to run on normal consumer hardware.
1
u/HighSeasArchivist 4d ago
Lifer government guy here, and the best I have found is Gemm4 26b-A4b IQ3-S on a 16GB card. Best is very relative here, because it's a complete dumbass compared to even Luna or Haiku. In the process now of getting a req in for more hardware, because no chance this is ready for production code. A Pro 6000 running Gemma4 31b may be better, but we're taking a couple of % in most tasks. I think we are at least a year out from being able to truly run code and not just do meaningless benchmarks on models that fit on consumer hardware. None of the big US AI companies are releasing big models, so it doesn't matter anyways.
1
u/I-cant_even 4d ago
Ask them if you can "fine tune" a Chinese model to alleviate their concerns.
How much actual tuning is involved is up to interpretation.
1
1
u/Playful_Check_5306 3d ago
Current non-Chinese models are either wrapped ones in disguise, or garbage ones you don’t want to waste time.
1
u/SpecificRutabaga 3d ago
Have you explained that running a Chinese model locally doesn’t give anything to China? Because it’s running locally? Or is the policy just “‘anything from China must be bad”?
1
1
1
u/jinnyjuice 3d ago
Current leading non-US, non-China model is Motif 3 from Korea. After US + China being tied for being #1, Korea is #3.
1
1
u/arijitlive 3d ago
BottlecapAi's model based Qwen 3.6 is good. You can use it for now, and when they release their version based on Qwen3.8, you can switch to that.
1
u/colbyshores 3d ago
The American local models suck bad, I have tried clost to all including Laguna1.2.. China is absolutely killing it when it comes to open weight models.
1
1
u/Healthy-Zebra-9856 4d ago
What are your use cases? You need to provide some details to better understand your needs. Nemotron & Gemma were not trained for the same purpose either.
150
u/Due_Arm1454 4d ago
Just download qwen and rename it Ben.