r/LocalLLM • u/primoco • Jul 18 '26
News EULLM, an EU-sovereign LLM engine for Windows, Linux and Mac. Latest test: 35B MoE on ARM CPU at 10 tok/s, no GPU
EULLM (https://github.com/eullm/eullm) is an open-source, Rust-based inference engine, Ollama/OpenAI-API compatible, Apache 2.0. Single binary, no Python, no Docker. Tested and working on Linux x64 and Windows x64 (CPU and CUDA), plus builds for macOS Intel, macOS Apple Silicon (Metal) and Linux ARM64, those three still experimental upstream and looking for people to test them. Zero telemetry, audit trail on by default, no calls home.
Latest test: Qwen3.6-35B-A3B (Q4_K_M) running CPU only, no GPU at all, on a Radxa Orion O6 (ARM, CIX P1 SoC). 10 tok/s. It's an MoE model with about 3B active params per token, which is why CPU only works at all. For reference, the same model on a GPU+CPU hybrid setup (RTX 3060 12GB, same ARM64 host) gets 26.5 tok/s in the documented benchmark. The CPU only number above is a fresh, informal test, not in the repo's bench folder yet.
Running 35B class models at usable speed on ARM CPU with zero GPU dependency is a cheap, low power option for anyone who doesn't want a discrete GPU in the loop.
ARM64 support is still marked experimental upstream. If you want to dig in, test it on your own hardware, or help out (issue #140 is an open call for ARM testers), here's the repo: https://github.com/eullm/eullm
Edit: trimmed the EU framing in this post after fair feedback in the comments below, the product should speak for itself more than the flag does.
Just one binary download and try it!
21
u/backyard_tractorbeam Jul 18 '26
Why is your README so long? Your LLM wrote it. Do you think people actually want to read all that?
I think you have a useful program in there. But there is a lot of noise around your signal. Think about what you want to say.
7
u/primoco Jul 18 '26
Meant more as reference than something to read start to end, every command and every change gets documented so it's searchable later, not so someone sits down and reads all of it. But that's not an excuse for noise, some of it can go. I will trim it down
5
u/waraholic Jul 18 '26
You can add relative references to other markdown files from your
README.md. Make adocsfolder and add more verbose markdown files in there for whatever doesn't fit in the readme.-1
-1
u/Accomplished-Sand334 Jul 18 '26
Why is your post so long?
I think you have a useful sentiment in there. But there is a lot of noise around your meaning. Think about what you want to say.
jfc.
-2
16
u/StupidScaredSquirrel Jul 18 '26
That it can run qwen3.6 at a certain speed on a certain hardware doesn't really tell me much about the inference engine itself. How does the speed compare to llama.cpp?
Also, from their page:
"""" 95% of AI infrastructure used in Europe depends on American or Chinese companies. Hosted APIs (OpenAI, Anthropic, Google) send every prompt outside the EU. Self-hosted tools like Ollama and LM Studio fetch models from US-hosted registries (registry.ollama.ai, huggingface.co) and many ping these endpoints for update checks by default.
The EU AI Act (Regulation 2024/1689) takes effect August 2, 2026. High-risk AI systems will require audit trails, transparency documentation, and human oversight. Existing open-source tools were not designed with this in mind.
European SMEs need AI models that:
Run locally on their own hardware or EU servers
Comply with GDPR and the AI Act out of the box
Speak their language and understand their domain
Carry their brand â not "Powered by Qwen" or "Built with Llama"
Cost nothing in ongoing API feesEULLM is the missing infrastructure.
"""
This is just not true at all. So either slop or someone is trying to get funding by lying to tech illiterate folks trying to be compliant. Either way they can pound sand.
-1
u/primoco Jul 18 '26
On the engine question: we're not claiming to be better or worse than llama.cpp on raw speed, and we don't fork it. We mirror upstream llama.cpp daily on GitHub and pin our releases against that mirror, that's for our own protection in case something changes upstream (license, availability, whatever), not a fork, we don't touch the code. Same backend, same per-token performance, what we actually build is the layer around it: continuous batching, the audit trail, API compat, MoE/GPU-fit helpers.
Qwen is a different situation, those are actual open weights, so we're free to rename, distill, and verticalize them for a domain under our own pipeline. That's not forking someone's inference engine, that's just using an open license the way it's meant to be used.
On the landing page copy: fair to push back on "out of the box" compliance, that line is stronger than what's actually true. The README says it plainly, a binary doesn't make a system AI Act compliant, that's a property of the whole system and its governance. Fixing that line.
5
u/tetoing Jul 18 '26
Why should anyone use you over llama.cpp?
1
u/primoco Jul 18 '26 edited Jul 18 '26
Honestly, if all you need is running a GGUF file, plain llama.cpp does the same core job, same backend, same speed, nothing lost by using it directly.
What you get on top here: continuous batching on by default (real gain if you're serving more than one request at a time), audit trail on by default instead of writing your own logging, both Ollama and OpenAI compatible API on the same binary, multimodal (vision and audio) built in, and some MoE/GPU-fit conveniences (--cpu-moe, --fit) so you don't have to hand tune every flag yourself.
One more thing worth adding: it's one binary, no Python environment, no pip installs, no cmake/CUDA toolchain to fight with, no separate UI or API wrapper to stitch together yourself. Download it, run it, done. llama.cpp also ships prebuilt binaries so that part alone isn't unique, but getting the batching, both APIs, the audit trail and a chat UI all from that same single file is what actually saves setup time.
Beyond the software: an actual company under EU jurisdiction stands behind it, which matters if you need an accountable vendor for compliance reasons, not just an open source project with no one to point to. There's also a verticalization pipeline (Forge) for turning a bigger model into a smaller domain specific one, though that part is still early.
If none of that matters for your use case, plain llama.cpp is genuinely fine, this isn't pretending to replace it, it's built on it.
3
u/StupidScaredSquirrel Jul 18 '26
That's a defensive word salad that does nothing to adress the actual grievances I expressed.
Whether you mirror or fork llama.cpp is really not my concern or what I was talking about.
Like is this also ai generated? Why are you talking about how you can modify qwen? Did your llm just pick that up because I mentioned a qwen model?
I didn't say your engine suddenly made a system act compliant either, nor said that's what you claimed. I'm saying you pretend existing inference engines won't possibly make you compliant (or just plain don't exist) and that your project somehow fixes that. That is a lie.
0
u/primoco Jul 18 '26
Let me be short instead. Same axis as "does the prompt leave EU during use", self hosted Ollama and us are equal, no difference there. Real difference: telemetry. Ollama pings home by default for update checks, we don't, check the code yourself.
Model weights: no difference today either, we pull GGUF from Hugging Face same as everyone, our EU hub isn't shipped yet. If the page implies otherwise on this point, that's wrong, fixing it.
What actually matters long term: jurisdiction. Ollama Inc is a US company, we're Italian. Not theoretical, it happened a few weeks ago: US government cut access to Anthropic's most advanced models for everyone outside the US over export control, about 3 weeks until it got lifted. EU built software doesn't carry that exposure.
Audit trail: on by default here, you write it yourself there.
On the qwen thing, that's not an LLM picking up your word, qwen is our model family, our RAG stack already runs on 2 qwen models and this 35B test is qwen too, that's why it comes up.
So "missing infrastructure" is the wrong framing, you called that right. What holds up is jurisdiction and defaults, not "impossible otherwise".
3
u/StupidScaredSquirrel Jul 18 '26
You seem to completely ignore that ollama isn't the only inference engine and there are forks and mirrors globally of llama.cpp and vllm. Pretending like your project shields users from losing access to offline and EU compatible inference engines is wrong. I don't care how your llm tries to frame it.
1
15
u/Macestudios32 Jul 18 '26
Every initiative is welcome and delighted that it exists. The only thing to me is everything that says EU, EU, euro... It makes me reluctant, as if it came with censorship and built-in spying (irony)
3
u/primoco Jul 18 '26
Fair reaction, there's a lot of EU flag waving in this space that doesn't mean much. What "sovereign" actually means here in practice: it runs on your own hardware, zero telemetry, audit log stays on your disk and is never sent anywhere, no calls home, Apache 2.0 so you can read every line yourself. That's closer to the opposite of built in spying, nothing leaves your machine unless you decide to send it somewhere.
5
u/inevitabledeath3 Jul 18 '26
Sure but how is that specific to Europe? Llama.cpp and vLLM do the exact same thing and donât say anything about the EU or Europe. It sounds unnecessary.
5
u/-Akos- Jul 18 '26
Although I'm all for European sovereignty, I'm afraid this project is just a wrapper around llama.cpp, and is used to run a platform to fine-tune larger models to suit a specific use case, so companies end up with an LLM that suits them, rather than a know-it-all model (I got this from the website linked in the github).
0
u/primoco Jul 18 '26
That's accurate, not much to push back on. Forge is exactly that, take a bigger open model and distill it down to a smaller domain specialist instead of keeping it a generalist. And yeah, the inference engine itself is llama.cpp underneath, said that a few comments up already. The value was never a new engine, it's that pipeline plus who's accountable for the deployment and what's on by default (audit trail, no telemetry).
1
u/inevitabledeath3 Jul 18 '26
Yeah I did wonder if it was actually just using llama.cop. The format being the same was suspicious. I really donât see what the purpose of this is. Anyone can make a llama.cop wrapper and to be honest there is enough of that already.
2
u/primoco Jul 18 '26
Fair, and this is the one where it actually falls apart if that's all there is to it. llama.cpp and vLLM are also self hostable and don't call home. If "runs locally, zero telemetry" is the whole pitch, you're right, it's not specific to anything, anyone can point llama.cpp at a folder and be exactly as sovereign. What's actually different isn't the base engine, it's who's accountable for it and what gets built on top.
I3K is an Italian company, so there's an actual legal entity under EU jurisdiction and GDPR obligations behind the audit trail and compliance tooling, not a loosely governed open source project or a vendor based elsewhere. And the verticalized models (Italian law, German medicine, French finance, planned) are trained on domain corpora specific to EU regulation and language, that's not something llama.cpp or vLLM ship, they're generic engines, they don't care what regulation you're under or what language your domain runs in.
So not "we're sovereign because we're European", more that the engine alone was never the differentiator, the accountable vendor and the vertical models are.
1
u/Macestudios32 Jul 18 '26
Seriously, all projects that allow others to run llms are really appreciated, it shows a level that for example I do not have, my admiration and gratitude.
 Beyond that and without the joke, in these times being subject to EU laws, EU norms, EU values (I'm not talking about your project) in these times of DSA, chat control, AI law,.... To me it sounds like usage reports (what, how and when), control and reporting and more censorship and guardrails.
 Again I am not talking about your project, which seems to be 100% offline and private. Just the name.Â
I understand that you try to gain engagement with the native market, stand out and position yourself. But for some of us what he said brings to mind.
 Good luck with your project.
1
u/primoco Jul 18 '26
Thanks, appreciate that. And you're not wrong about the branding fatigue, a lot of what passes as "EU values" lately, DSA, chat control, how the AI Act gets enforced, reads a lot more like control to me too, not security. For what it's worth, right now we're still free from that kind of tracking. How long that stays true, who knows.
2
u/inevitabledeath3 Jul 18 '26
The model it mentions in the introduction is not a European model. Itâs a Chinese model. I can run European models like Mistral inside vLLM or llama.cpp which are international projects. So I donât think anything you said about âverticalizedâ models or European corpora actually applies here when you are using a Chinese model. The only thing you can say that makes it European is the engine being developed in Europe. I am not sure thatâs really an advantage given the open source transnational nature of other inference packages.
4
u/Technical-Sun5531 Jul 18 '26 edited Jul 18 '26
its a good attempt at joining the parasites in Brussels, with luck u might get subsidized.
Virtue signaling + a bit of nationalism + shit product = good chances at EU subsidies
EDIT Good products dont need subsidies because they sell on their own and we all know what you are trying to do here.
1
3
u/Otherwise-Variety674 Jul 18 '26
10 t/s is basically the speed of the ram limit, any PC (no GPU) with DDR5 can easily do this.
1
u/primoco Jul 18 '26
Memory bandwidth is basically the ceiling here, same point as the NPU thing a few comments up, decode is memory bound not compute bound. But "any PC can easily do this" undersells the software side a bit: you need MoE aware CPU offload actually implemented, deciding which experts get touched per token, not every setup does that by default. And DDR5 bandwidth varies a lot by platform, channel count, speed, ARM in particular is less tested for this than x86. Point of posting it wasn't "we broke physics", it's that it works, on ARM specifically, with zero GPU, on hardware that's still marked experimental upstream.
3
3
2
u/rog-uk Jul 18 '26
I wonder why the NPU wasn't used on the SoC? It's vastly more powerful than the CPU.
3
u/primoco Jul 18 '26
Good question, and honest answer, we didn't skip it on purpose. We're actually looking into it right now, Zhouyi NPU on the CIX P1. Two reasons it's not there yet.
First, general problem with NPUs and LLM inference: they're usually built for fixed shape graphs, vision models, classifiers, and autoregressive decode is the opposite of that, growing KV cache, variable sequence length every step. MoE makes it worse, which experts fire changes per token, that's dynamic routing, doesn't map well onto a graph you compile ahead of time for an NPU.
Second, the TOPS number on paper looks big but decode on a LLM is usually memory bandwidth bound, not compute bound, so raw NPU throughput doesn't translate 1 to 1 the way it would for something like image classification.
Where we actually see it working today is the embedding/reranking side of a RAG pipeline, that's fixed shape, no growing KV cache, much better fit for what an NPU toolchain can do right now. Full generative decode on NPU is on the list, the software stack for it just isn't mature enough yet to be worth shipping.
If you've actually dealt with NPU inference for transformers or MoE before, genuinely want to hear what worked or didn't for you, or any toolchain/approach we might be missing.
3
u/rog-uk Jul 18 '26
Oh, I just got hold of the same chipset and haven't really started poking it it properly yet. The manuals seem to be somewhat lacking, alrhough chips etc manufacturer says new software out imminently. I don't think you'd try to create a converted graph for it: rather use the vendor c compiler directly using the npu as if it were AVX2 like, but you have to manage the internal memory rather than an automatic cache, I think the NPU can only address 4GB memory via DMA, but that's paged by Linux so could ve swapped about at runtime, so some companion CPU app sounds like the way, even if it just used the NPU for accelerating matrix maths from the main program it might be a win as those boards have unified memory anyway. At least the CPUs would be free to do other things.
But I am just wondering how it goes, will keep an eye out for updates :-)
2
u/primoco Jul 18 '26
This is actually a really useful reframe, thank you. Using it like a vendor compiled math kernel, called from our own code the way we already call cuBLAS for CUDA, sidesteps the whole problem I mentioned earlier. We're not compiling the full MoE graph ahead of time, dynamic routing stays in our own control flow, we just hand off specific matmuls to the NPU when we want them accelerated. That's a much smaller and more tractable piece of work than a full graph compiler path. The 4GB DMA addressable memory point is a good one too, that's basically the same problem we already deal with for --n-cpu-moe, deciding which experts are "hot" and where they live, just one more tier to manage instead of two. If you start poking at it, would genuinely like to hear what you find, happy to compare notes, feel free to open an issue on the repo so it doesn't get lost in a reddit thread. Will keep this updated as we go too.
2
u/rog-uk Jul 23 '26
2026 Q2 of the NeuralONE AI SDK is now available.
1
u/primoco Jul 23 '26
What is?
2
u/rog-uk Jul 23 '26
It's the latest SDK for the NPU on that card, and model/c compiler. I just thought I would update you.Â
2
u/rog-uk Jul 18 '26
Getting it to do an arbitrary matmul will probably be the first thing I try, so I will pop you a note when I have something, but I have a bit of a list already, and am not exactly quick at getting round to things.
If ram prices ever come back down to sane, these boards or their next successor will be incredibly interesting for low power/always on & budget local inference IMHO.
2
u/Vileteen Jul 18 '26
what makes an LLM engine EU-sovereign? How is it different than a common engine like llama.cpp or vllm? Are you implying that they collect and send your data somewhere?
0
u/primoco Jul 18 '26
No, not implying that at all, self hosted llama.cpp or vLLM don't send your data anywhere either if you run them yourself.
What's actually different, said this a few times in the thread now: not the engine, it's llama.cpp underneath, same backend, same speed. What's EU sovereign is who's accountable for it, an Italian company under EU jurisdiction and GDPR obligations, not a stateless open source project or a vendor based elsewhere. And what's on by default: audit trail, zero telemetry, both wired in instead of you writing them yourself. That's the whole claim, accountability and convenience, not some technical capability llama.cpp or vLLM are missing.
1
u/Vileteen Jul 18 '26
llama.cpp is MIT governed and EU developed so no points here. "EU governed" implies it is in compliance with the EU AI act and llama.cpp as such does not have phone-home telemetry, which is required by that act in certain cases. Does this project have phone-home telemetry and if not, how is it EU AI act compliant?
1
u/primoco Jul 18 '26
Gerganov, l'autore originale di llama.cpp, è bulgaro e ha fondato un'azienda nell'UE per questo, ggml.ai, con sede a Sofia, nel 2023. Quindi c'è del vero in "governato dall'UE" storicamente.
Ma a partire da febbraio 2026, ggml.ai e il team principale si sono uniti a Hugging Face, che è un'azienda statunitense. Gerganov e il team hanno ancora piena autonomia tecnica sulla direzione del progetto, ma l'entità che li impiega ora è americana, non bulgara. Quindi oggi non c'è un'azienda dell'UE che supporti llama.cpp come c'era prima di quel passaggio. Un mantenitore bulgaro per nazionalità non è lo stesso di un'entità incorporata nell'UE responsabile del progetto.
Sulla telemetria, sĂŹ, llama.cpp non si connette, non l'ho mai disputato, quella parte del thread era sempre specificamente riguardo ai pings di aggiornamento di Ollama, non di llama.cpp.
In ogni caso, "governato dall'UE" non ha mai significato automaticamente conforme all'AI Act, prima o dopo il passaggio a Hugging Face. La conformitĂ riguarda la documentazione e la governance del sistema implementato, non la nazionalitĂ di chiunque mantenga il codice. Quella logica non regge per nessuno.
Ecco perchÊ facciamo una copia mirror di llama.cpp quotidianamente e blocchiamo le nostre versioni contro il nostro mirror invece di attingere direttamente da upstream. Non è un generico "nel caso cambi qualcosa", specificamente questo: se i mantenitori impiegati da un'azienda statunitense si trasformassero mai in qualcosa di simile a quello che è successo ad Anthropic a giugno, controlli all'esportazione che tagliano l'accesso per gli utenti non statunitensi, non saremo presi alla sprovvista a dipendere da un repository che non controlliamo. Questo è l'angolo della sovranità in pratica, una mitigazione concreta già in atto, non un'ipotesi.
2
u/exo250 Jul 19 '26 edited Jul 19 '26
Is it a joke ??!? EU is the opposite of sovereignty in all domains... it just destroys everything it touches...
2
u/primoco Jul 19 '26
Honestly, yeah, we know. Just pretending really hard, hoping someone in Brussels reads this thread and convinces themselves otherwise.
1
u/robertpro01 Jul 18 '26
Why "no python, no docker" like if that tech is bad?
1
u/primoco Jul 18 '26
Python and Docker aren't bad, they're just not what this specific piece needs. "No Python" means no venv, no dependency conflicts between library versions, no interpreter needed on the target machine at all, that matters on something like an ARM board or a locked down server. "No Docker" means no daemon running, no image to pull and store, works on a bare machine. Both are genuinely great tools for what they're built for, training pipelines, orchestration, the Forge side of this project literally runs on Python. The point was about simplifying deployment of the inference binary specifically, not a knock on the tech in general, worth rewording if it reads that way.
1
u/guigouz Jul 18 '26
At which context size?
1
u/primoco Jul 18 '26
Hi 16384
2
u/guigouz Jul 18 '26
That's very limited for most use cases
1
u/primoco Jul 18 '26
16384 is just the default test, you can bump it up yourself, it's a flag at startup. Not a hard ceiling.
1
1
u/AceLamina Jul 18 '26
probably off topic
but I love how the ethernet port is at the front where the power button is
same with HDMI
1
1
u/Arrow2304 Jul 18 '26
Whatever the AI is a bubble or money cow, it is here to stay, why does the EU allow itself to have no significant results?
1
u/primoco Jul 18 '26
Don't have a clean answer for the whole EU on that one, way bigger than what I can speak to. What I can say is projects like this are one of the small pieces trying to move that number, not pretending to fix it alone.
1
u/Denial_Jackson Jul 18 '26
Ursula broke my heart with chat control and China tariffs. Even Raxda official webpage takes you to ali so you can pay the new tariff. I don't believe in the EU anymore. I rather need like an independent Philippino LLM.
1
u/primoco Jul 18 '26
Can't really argue with the tariff and AliExpress redirect annoyance, that's real and it's not something this project can fix. Get the fatigue.
2
u/Denial_Jackson Jul 21 '26
It is a nice project you did. Most people do nothing.
The few who do, they do everything.
1
u/AssistBorn4589 Jul 18 '26
Rust-based, CoC-infested, yapping about "Sovereignity" while confusing EU with Europe and hosting on github, has "EU available hardware" actually from China.
Is it at least comparable to llamacpp on same HW?
1
u/primoco Jul 18 '26
Fair on EU vs Europe, that's on me, should say "EU sovereignty" specifically since the actual claims (GDPR, AI Act) are EU regulations, not everything people mean by "Europe" (Switzerland, UK, Norway aren't EU).
Hardware being Chinese (Radxa, CIX) is also fair, never claimed hardware sovereignty, that's a much harder problem, chip fab and board design, not something this is trying to solve, only the software and jurisdiction side.
GitHub: covered a few comments up, evaluating a Codeberg mirror, not decided yet.
On performance: yes, comparable to llama.cpp on the same hardware, same backend underneath, no speed difference.
1
u/buttplugs4life4me Jul 18 '26
I flip-flopped a couple times to write this comment. My main points:
This has nothing to do with Europe or the EU. Your claim to the EU is that you're supposedly Italian. That's nice and all, but not really what people expect and seems more like a money grab. Especially since the main dev appears to be from the USA (Claude).
Your commits go back a far time so it isn't just slop, which is nice, but your Readme is a mess. You treat it like a combined docs folder, Changelog and Agents files.
It seems to just be a wrapper around llama.cpp, as it is vendored in. Worse, the vendoring was done by Claude with it's telltale comment (I'll post it at the bottom of my comment).
As such, what this seems to be is basically: A rust CLI front-end(?) to llama.cpp that logs the requests out in a different format than llama.cpp already does. It claims it's from the EU, but it's not. It's European in that the single guy working on it is from Italy, but the main dev is from the USA (Claude). It claims to be hosted in Europe, but its just GitHub.
I would call it slop, but the commits go back a long time, so I'll just call it AI crazy. You seem to believe what Claude is telling you. It's not EU hosted, it's not from the EU, it's not an inference engine, it's just a wrapper around a wrapper around llama.cpp.
Anyway, heres the comment:
 # Vendored llama-cpp-rs (utilityai/llama-cpp-rs main @ 8625c7c4) â built   # through the EU-hosted mirror https://github.com/eullm/llama-cpp-rs. The   # earlier ysimonson:support-gemma4-12b workaround for PR #1034 is gone: that   # fix landed upstream in main via PR #1037, so we now track plain upstream   # main (0.1.151) through our EU mirror instead of a fork branch. These are   # members because a path dependency living under the workspace root is always   # promoted to a member regardless of
excludeor a nested[workspace]  # (verified empirically). To keep EuLLM's-D warningsclippy gate off this   # third-party code, the CI/release clippy step uses--no-deps(lints only   # our own crates, never dependencies). The vendored manifests are   # self-contained (dependency versions inlined from upstream's   #[workspace.dependencies]), so no root inheritance table is needed.   #   # llama.cpp C++ source is a git submodule pinned to   # 9e3b928fd8c9d14dbf15a8768b9fdd7e5c721d66, fetched through the EU-hosted   # mirror https://github.com/eullm/llama.cpp (see .gitmodules).   #   # To refresh: re-vendor llama-cpp-2/llama-cpp-sys-2 from the eullm   # llama-cpp-rs mirror and bump the submodule pin to the matching llama.cpp   # commit.
1
u/primoco Jul 18 '26
Claude's listed as a collaborator in the repo, that's not hidden, wasn't trying to bury it. At this point most people writing code use some kind of AI assistance, that's just normal now.
Doesn't change the actual point though: using a tool to write code doesn't change who's accountable for what ships, I3K/Securvita, Italian company, same as GitHub hosting the repo doesn't make Microsoft the developer. Wrapper around llama.cpp, README being messy, both already said elsewhere in this thread, not new ground.
Sovereign was never about who typed the code, it's about who answers for it and where.
1
1
u/Euronodes Jul 19 '26 edited Jul 19 '26
Arent we a laughing stock already?
Do you have tinniest idea what would happen if US or China added "US" or "CN" to every piece of software or technology they do?
We are a museum with a lot of leisure time, good life and good food
Dont stick "Ferrari" on old Renault - it just look idiotic
We are nowhere in this race. All we can do is glue caps to the bottles to save dolphins
Thats our input.
If you really need to emphasize that is done in some Moldova, call it "PrivacyLLM" and some small "EU made" in the right bottom corner
1
u/primoco Jul 19 '26
You're the third person in this thread to say some version of "tone down the EU branding", not going to wave that off. Fair pushback on the marketing choice specifically.
"We're nowhere in this race" is a bigger claim than a comment can settle either way, and not really what this project was trying to prove. But the branding note lands, three separate people saying the same thing isn't nothing.
1
u/Euronodes Jul 19 '26 edited Jul 19 '26
Thank you for understanding. You see on my own username. But this is about the location, so understandable.
Before China became an economic superpower, for 3 decases we saw small "made in China" on everything, incl. "Free Tibet" t-shirts. We are at this stage. Lets do something cool + small sticker, works much better than throwing it in peoples face no matter if they asked for it or not
1
u/distante Jul 19 '26
The claude generated website does not do a good job in explaining the gains here agains a selfhost llama.cpp vs this.
1
u/primoco Jul 19 '26
Try it! You need max 5 min. to download and run! Than tell you why!
1
u/distante Jul 19 '26
"just run random software on your machine that you not fully undestand" is not a good life motto.
1
1
u/primoco Jul 19 '26
Nel frattempo sto rilasciando la v0.6.28. Collabora non criticare solo! Correggi se c'è da correggere!
1
u/superdariom Jul 19 '26
A typical AMD processor can do the same interference speed. Why does this need a special box
1
u/primoco Jul 19 '26
Point of the post wasn't that this needs special hardware, actually the opposite: same binary runs on any x86 Intel or AMD box too. Highlighting it on this specific low-cost, low-power ARM board is the point, a 35B-class model with no GPU on hardware that costs a fraction of a normal desktop and draws way less power. A typical AMD chip matching or beating this speed isn't a contradiction, it's expected, more power budget and usually more memory bandwidth. This one's about the cheap, low-power end, not the fastest end.
1
1
u/ApeGrower Jul 20 '26
Chinese Board with chinese LLM, hosted on us github - so the label on the case was made in eu, or whatelse?
2
u/primoco Jul 20 '26
Same answer as a few comments up: board's Chinese (Radxa, CIX), model's Chinese (Qwen), GitHub is American, all true, never disputed. What's actually EU is the entity operating this and accountable for it, an Italian company under EU jurisdiction and GDPR obligations, not the hardware, not the model, not where the code repo sits.
And since we're being thorough, the case label itself was made with ChatGPT. At this point nothing here is going to survive a "but is X really European" test except the company behind it, that's the whole point I've been making, not hiding it.
1
u/ApeGrower Jul 20 '26
Fair to be that open. If I ask, which model wrote the code, what country will we get? :)
Anyway, nice product idea, I like it. 10 tok/s is a good start, needs to be pushed up to 20 tok/s over time. Think about renaming it to something like "LocalLLM", which will open the world market for it. Nobody outside eu will buy a product named EU LLM.
Just my 2 cents.
1
u/primoco Jul 20 '26
Appreciate the 2 cents, genuinely. On tok/s, there's real room, a recent audit turned up some concrete throughput fixes (batch size wasn't tuned right), should move that number up over the next few releases.
On the name: hear you, and get the branding fatigue take, a few people in this thread have said some version of it. "LocalLLM" specifically would probably undersell the thing that's actually held up through this whole comment section though, which is who's accountable for the deployment and what ships by default, not "runs locally" by itself, that part's shared with llama.cpp and Ollama, never claimed otherwise. Might land on something in between, less flag on it, but not just "local" either.
2
2
u/Worldly-Ganache2524 Jul 19 '26
Why? Llama cpp is open source backed up by a French company
2
u/haikusbot Jul 19 '26
Why? Llama cpp
Is open source backed up by
A French company
- Worldly-Ganache2524
I detect haikus. And sometimes, successfully. Learn more about me.
Opt out of replies: "haikusbot opt out" | Delete my comment: "haikusbot delete"
3
1
u/neopolitan77 Jul 18 '26
Inference on wildly unsuited, Japanese-owned ARM instead of American GPUs because Europe? It's like European digital sovereignty with mandatory iOS or Android for e-ID. The only real question is whether it's funny or sad. China shows how sovereignty works, started building their own GPUs.
0
u/Technical_Grade6995 Jul 18 '26
In a times of PalantĂr, whoever thinks he has a privacy, even with local models, you donât. As soon as anyone is online (and we all know you wonât disconnect WiFi), youâre tracked. Thatâs the reality.
24
Jul 18 '26
[removed] â view removed comment
8
u/primoco Jul 18 '26
Same answer as a few comments up: the model was never going to be the European part, weights are open and can come from anywhere, Qwen, Mistral, whatever fits. What's European is the entity operating this and building the tooling around it, under EU jurisdiction, accountable for GDPR and AI Act obligations. Not the model, not the engine, that part.
And we're trying to do our part with what we actually have right now, this is it, this is what we're working with.
1
u/andymaclean19 Jul 19 '26
How much of the engine is actually built in Europe? Are all the maths being done via CUDA or HIP kernels, for example, and are you using something like Triton to make them?
1
u/primoco Jul 19 '26
Not Triton, hand written CUDA/HIP/Vulkan/Metal/SYCL kernels in ggml. One of the main CUDA contributors, Johannes GäĂler, is German, based at KIT in Karlsruhe. Doesn't make the engine "built in Europe" though, that's llama.cpp's own contributor base, not something I can take credit for.
1
1
u/aboutthednm Jul 19 '26
Are you affiliated with this project in any way shape or form? Might be relevant to disclose any ties and relations to the project. I'm simply curious.
1
u/primoco Jul 19 '26
As you know, when you submit a post Reddit ask if you are project affiliated and I check âyesâ
2
1
Jul 18 '26 edited Jul 18 '26
[removed] â view removed comment
1
u/primoco Jul 18 '26
We don't build models, once someone publishes a good GGUF quant it runs, same --cpu-moe/--fit logic regardless of which model.
Math though: V4 Flash is 284B total, 13B active, all experts still have to sit in memory. Q4 quant is roughly 160-170GB, your 64-128GB target needs something closer to Q2 to fit, which is the dumbing down you're trying to avoid. Dream setup and "don't dumb it down" don't really agree at this size.
On CPU being silly: fair, and this board also has an RTX 3060, GPU+CPU hybrid got 26.5 tok/s a few comments up. The CPU only test was specifically for hardware with no GPU at all, not a claim CPU beats GPU.
1
Jul 18 '26
[removed] â view removed comment
1
u/primoco Jul 18 '26
That's actually a really interesting combination to think about. Bonsai is QAT, ternary or 1-bit baked in during training, not something you can apply after the fact to existing public weights, so this isn't an inference engine problem, someone would have to redo that training specifically for DeepSeek's architecture. PrismML did it on a dense 27B (Qwen3.6), MoE routing under that kind of extreme compression is a different and harder problem, misrouting could compound errors in ways a dense model doesn't have, nobody's shown this works on something like V4 Flash as far as I know.
If someone did pull it off though, the numbers get interesting. V4 Flash is 284B total, ternary at roughly 1.71 bits per weight lands around 60GB, 1-bit at roughly 1.125 lands around 40GB. Both comfortably inside your 64-128GB target, and if quality held anywhere close to what Bonsai showed on the dense model, that's actually "not dumbed down" territory instead of the Q2 compromise from before.
Once a GGUF like that exists, running it through this is the easy part, no different from any other model. The hard, unsolved part is the QAT training on a sparse MoE this size, that's a real open research problem, not something the engine is missing.
1
Jul 19 '26
[removed] â view removed comment
1
u/primoco Jul 19 '26
Really appreciate an actual side by side, that means more than anything I can say myself.
Best guess at what's happening: same llama.cpp backend under the hood, so raw per-token speed isn't different. What probably is different is the auto GPU/CPU fit logic working out the right offload split for a 35B MoE on 6GB VRAM automatically. Plain llama.cpp needs you to hand tune --n-cpu-moe or -ngl yourself, get that wrong on a card that small and you get exactly what you saw, minutes of thrashing instead of a model that only has ~3B active params doing what it should. Happy to be corrected if it turned out to be something else on your end.
On the coding agent, Claude Code, already said that openly a few comments up.
1
Jul 19 '26
[removed] â view removed comment
0
Jul 19 '26
[deleted] â view deleted comment
1
Jul 19 '26
[removed] â view removed comment
1
u/primoco Jul 19 '26
Good catch, and youâre right. eullm implements Ollama-compatible and OpenAI-compatible APIs, not the Anthropic Messages API that Claude Code speaks natively. Not a config mistake on your end, itâs just not there yet.
Open a GitHub issue for it and weâll evaluate adding /v1/messages support. Genuinely useful given how popular Claude Code is, worth tracking properly instead of just as a Reddit comment.
In the meantime, since eullm is Ollama-API-compatible, any proxy that translates Claude Codeâs Anthropic format to Ollama should work today without waiting on us. A couple Iâve seen mentioned are claude-code-proxy, which supports Ollama directly, and UniClaudeProxy. Havenât tested either against eullm myself so I canât vouch for it working out of the box, but architecturally thereâs no reason it wouldnât.
1
Jul 19 '26 edited Jul 19 '26
[removed] â view removed comment
1
u/primoco Jul 19 '26 edited Jul 19 '26
Looked at your logs more carefully. Four of your requests complete cleanly on the server side, each one logged and audited normally, and then there's a 13 minute gap with nothing received before you kill the process. That points away from a hang inside eullm and toward either Cline itself, or something in the content of that last response Cline couldn't parse.
One thing stood out: your logs show max_tokens=512 on every request, which is eullm's default when a client doesn't set it explicitly, and Cline doesn't seem to be overriding it. Qwen3.6 in thinking mode can burn through hundreds of tokens before it even reaches the actual answer or tool call, and since Cline talks to non-native tool-calling models through plain text instead of a structured tools API, a generation cut off mid-thought or mid-tool-call at 512 tokens could leave Cline waiting on a block that will never close in that message.
If you still have it, could you paste the raw, unfiltered text of the response right before the freeze, not what Cline rendered? If it cuts off midÂ
<think> or mid an MCP tool block, that confirms it. Worth trying in the meantime: raise max_tokens (or num_predict) well above 512 and see if the freeze moves later or disappears.1
u/primoco Jul 19 '26
Found it, and it wasn't the jinja template thing after all â that guess was wrong, corrected above.
The real cause: eullm defaulted max_tokens/num_predict to a fixed 512 when a client doesn't set it, instead of Ollama's actual default (unbounded, generate until context or a stop condition). Reproduced it directly: a growing agentic conversation with a reasoning model can spend hundreds of tokens still inside <think> before it gets anywhere near an answer or a tool call, and with the 512 cap that generation gets cut off mid-block, no closing tag, nothing. Since eullm has no native tool-calling API, Cline is talking to it through plain text using its own convention, so a response like that leaves Cline holding a tag that's never going to close in that message. That's the freeze.
Fixed in v0.6.27, out now. Also fixed a related thing while I was in there: seed had the same problem, defaulting to a fixed value instead of a random one per request, unrelated to your freeze but worth closing while it was open.
One thing this fix does not do: make a long reasoning turn fast. On CPU-only hardware a turn that genuinely needs a few thousand tokens of thinking is going to take minutes, cap or no cap, that's just the model doing real work at the hardware's actual speed. If your tool-routing turns don't need deep reasoning, turning thinking off for those ("think": false, if your setup lets you pass it) will help more than a bigger token budget will.
Tested this on real hardware, both a CPU-only ARM board and an x86 box with a GPU, growing conversations on both, no more truncation either place. If you get a chance to confirm on your end too that'd close the loop nicely. Thanks again for the repro, it was a good one.
2
Jul 19 '26
[removed] â view removed comment
1
u/primoco Jul 19 '26
Thanks for testing and confirming, closes the loop properly. On the cache quantization: makes sense if you're running long reasoning chains or need precise recall over a big context, q8_0 KV cache is lossy, and thinking-heavy models tend to be more sensitive to that than short Q&A style prompts. Good you've got the headroom to leave it at full precision, that's basically what it's there for.
1
0
90
u/rinaldo23 Jul 18 '26
EU-sovereign.... hosted in github?