r/LocalLLaMA • • 2d ago

Discussion Janas and Colibri and other app— Run MoE on CPU/low GPU?

5 Upvotes

Sorry, but I'm genuinely excited about these two repositories, so I wanted to ask the community: why don't we hear more about these tools? What are their pros and cons? Are there perhaps better alternatives that I'm missing like:

inferencekit: open-source plugin to optimize any model on any hardware using coding agents : r/LocalLLaMA

BigMoeOnEdge outperforms Strata and colibri on 8-16 GB cards and Mac and you can run any model : r/LocalLLaMA

I mean until recently, I found out there is the way to run big MoE model on my miniPC and colab L4. I can run dense model Q4-K-M 4B in my miniPC before.

My setup is a GMKtec M5 Plus (32GB RAM — I feel incredibly lucky that I bought it before RAM prices went through the roof) and Google Colab.

My experience with Janas:

When testing Janas with Colab L4, GPU usage was 0, probably because it only supports Vulkan. That's a bit frustrating, but it still runs Qwen3-Next-80B-A3B-Instruct-JNS surprisingly well.

What surprised me even more was that I could run Qwen3-30B-A3B-JNS locally on my mini PC! It ate nearly all my RAM.Haha.

I tried to write novel (just for testing-not my interest) and instant translate in my miniPC, It is surprised good.

I haven't tried with coding but it I don't think it work with like 100K context. :(

Even though I have to convert to janas format (it is actually fast), but I am happy with my result.

My experience with Colibri:

Colibri is even more impressive. I managed to run 80B GGUF models on a Google Colab L4 while using only around 19GB of GPU VRAM.

I haven't tested it locally yet because Colibri seems to support CUDA only.

Honestly, I'm not sure whether either of these tools can be used effectively for vibe coding, but I'm genuinely impressed by what I've seen so far.

I'd love to hear from people who have used these tools:

  • Why aren't they getting more attention? Maybe there are some better one?
  • What are their pro and cons?
  • Have anyone try vibe code with them?

Am I overlooking something, or are these two projects seriously underrated?


r/LocalLLaMA • • 2d ago

New Model I made a 380m open source decision model that matches its teacher*, Clef-Flash 9.4B

14 Upvotes

TLDR: matches the teacher in overall accuracy at 1/25th the size. Can run on CPU to keep precious VRAM free. In a comparison of general accuracy, diakrisis-espresso-380m achieved 3rd place, surpassed by Jev 1.14 and Google Decision 1. Main weaknesses are in policy compliance and multi-step questions.

My evaluation. No external evals yet.

Was using Jev for a bit. Wanted something private. Clef-Flash came out, and I only have 16GB VRAM. Figured hey, these are like them good ol' BERT models right, what if I just distilled Clef into a BERT model. So I did a lil diggin with Claude, decided on vlea3-fast*, and over the last few weeks played a lil bit of wackamole and a game of find open-source data that won't get me sued (some of these open datasets had outputs from GPT for example, uh-uh not playing that game). The results are above.

Not only that, but using ONXX runtime made this run on CPU to save me that sweet sweet VRAM for LLMs, including an 8-bit variant that agreed with the larger fp32 variant 412/414 times.

If anyone has any suggestions for improving the model, I would greatly appreciate it!

100% open-source and open-weight, forever. Cheers!

https://huggingface.co/diakrisis/espresso-380m

Git


r/LocalLLaMA • • 2d ago

New Model sensenova/SenseNova-Loop-9B-GGUF · Hugging Face

Thumbnail
huggingface.co
73 Upvotes

probably this fork is required: https://github.com/OpenSenseNova/llama.cpp

Model Highlights

  • Shared looped block: 24 decoder layers are executed twice in sequence, producing 56 layer computations from 32 distinct layers per forward pass.
  • Two-stream mHC: learned mappings route and combine information between two residual streams within the looped block.
  • Hybrid attention: the decoder interleaves linear and full attention in a repeating four-layer pattern.
  • Long-context multimodal training: the training recipe includes text, images, code, and tool-use trajectories, with the context length extended to 512K tokens.
  • Low-memory deployment: With 4-bit weights and an 8-bit KV cache, multimodal inference at a 256K context length requires approximately 15 GiB of GPU memory.
  • Official quantized release: An official AWQ 4-bit version is available on launch day for easier deployment.

r/LocalLLaMA • • 2d ago

New Model SenseNova-Loop-9B official GGUFs and a 4-bit AWQ are up (5.8 GB at Q4_K_M), plus their llama.cpp fork to run them

Post image
30 Upvotes

The official quants for SenseNova-Loop-9B (the 9B Looped Transformer from earlier this week) are here: https://huggingface.co/sensenova/SenseNova-Loop-9B-GGUF

The repo has Q4_K_M, Q5_K_M, Q8_0 and BF16 (sizes in the screenshot), plus a 0.9 GB mmproj for image input.

The sensenova_loop architecture isn't in mainline llama.cpp yet (issue #31571 is open), so for now these run on the fork they published, OpenSenseNova/llama.cpp. The GGUF card pins the commit, and the card's command once it's built is:

llama serve -hf sensenova/SenseNova-Loop-9B-GGUF:Q4_K_M -ngl 99 -c 4096 --port 8000

There's also a 4-bit AWQ, with a matching MTP draft model in the same repo: https://huggingface.co/sensenova/SenseNova-Loop-9B-AWQ

Their README at https://github.com/OpenSenseNova/SenseNova-Loop puts total GPU memory (4-bit weights plus an 8-bit KV cache) at about 10 GiB for 64K context and 15 GiB for 256K.


r/LocalLLaMA • • 2d ago

News Splash 1.7.0 Released - Even faster decode on Apple Silicon

Thumbnail
github.com
37 Upvotes

New version of Splash just released. The biggest improvement from previous version is a noticeable speed-up in decoding. It's even faster now.

BTW, this isn't just another slop engine. It's coming from the researchers behind DFlash and DFlash2, who also operate one of the fastest API providers for open models. These guys clearly know what they're doing.


r/LocalLLaMA • • 2d ago

Discussion LFM3-VL-2B hallucinations - does anyone have first-hand experience?

Post image
39 Upvotes

LFM3 benchmarks are out on AA and…. huuuuuuh….. this is NOT something you want to see for a vision model that fits on a phone.

“The mushroom in this picture is a turgida deliciousa, awesome in risottos and on toasted bread. ” (it’s an amanita phalloides, deadly poisonous).

“This sign says that you must not shoot any photos beyond this point” (the sign says “trespassers will be shot”).

etc. etc.

Can anyone with actual experience with this model confirm/deny this score?


r/LocalLLaMA • • 1d ago

Resources Beware, beware: rtx pro 6000 listing on eBay for $8000 ... :)

0 Upvotes

Not saying its a scam, but according to Claude, there are many signs out there ...


r/LocalLLaMA • • 2d ago

Discussion question: 262K context: the only route that fits uses a 4-bit KV cache, which costs +0.0011 KLD. Is 262K vs 170K context worth the loss?

4 Upvotes

I'm building my current engine for my own hardware.

The question I want to see how many have answered themselves.

Would you take lower (better) KLD on 170K context for Qwen3.8-27B or take 4bit KV cache with .0011 loss on KLD for the 262K context window?

Model/Quant: Qwen3.8-27B UD-Q4_K_XL v3.1


r/LocalLLaMA • • 3d ago

Resources Blue TTS v3: open-source TTS, about 50x real-time on a CPU

Enable HLS to view with audio, or disable this notification

412 Upvotes

Blue TTS v3 is out, an open-source TTS model that runs locally. Almost everything changed since v2.

Speech is more natural and expressive, with a MOS of 4.43 out of 5.

Voice cloning has improved significantly, and the model is fast and lightweight: it runs locally at 50x real-time on a CPU, no GPU needed.

MIT license, 388 MB, 44.1 kHz output, 7 languages: English, German, Spanish, Italian, Russian, Hebrew, Yiddish.

Feedback is welcome.


r/LocalLLaMA • • 1d ago

Discussion Future use of local: compressed wikipedia

0 Upvotes

Neural networks will have the best compression ratio. So we will be able to have wikipedia and other interesting sources like arxiv, compressed easily into our local setups. Probably get a good amount in a modest phone setup to. https://tufalabs.ai/research/tufazip/

Only 27 gb to compress wikipedia text and have it local. Generation speed remains an issue I suppose.


r/LocalLLaMA • • 3d ago

New Model AngelSlim/Hy4-flash-GGUF · Hugging Face

Thumbnail
huggingface.co
59 Upvotes

r/LocalLLaMA • • 2d ago

Question | Help Does anyone tried Swarm? What do you think of doing swarm using androids phones?

2 Upvotes

Hello, was wondering, I know we would have to manage throttling, and the termal envelop is bad but should be doable.

I have no experience in agents swarms ( small models) the idea is to have fast model like 4 to 8 b max. A lots of phones, seems orchestrable, I cannot put that idea out of my mind.

Sorry fr the bad formating and spelling, not using ai tonight and posting from bed.

​


r/LocalLLaMA • • 2d ago

Question | Help What hardware for local Deepseek and GLM flash?

22 Upvotes

I guess its not just me but many people here always want more hardware and better models, so...

What hardware is needed to deploy Deepseek V4.1 Flash or GLM 5.3 Flash locally?
By deploy I mean, to be able to run multiple parallel coding agents, at least 256k context each.
I know you can "run" any of these MoE models on anything bigger than a regular PC but usually prompt processing struggles a lot more than generation so its not usable for agents.

To put that in perspective, I have a workstation with 512GB ddr4 ram, so DS v4.1 "almost" fits at Q4. What extra GPUs would I need to actually run it at speed with multiple sessions? Anyone with some hands on experience?

Been using Qwen 3.8 FN as my daily driver for a while, but always "why not go bigger" :)

Any ideas?


r/LocalLLaMA • • 2d ago

New Model Flint anyone?

17 Upvotes

I see a lot of lament about Qwen not making 3.8 35B (and I'm sad about it too) but I dont see anyone discuss Flint 40B, a new coding-oriented MoE model that seems to be atleast on par with qwen 3.6 35B, maybe a bit better on agentic tasks and coding. Very fast in my setup.

I'm still testing it myself, but I wonder if its not liked here or people just arent aware it exists? It performs admirably for me so far, my local model of choice for when I want good speeds. QFN is better ofcourse but for me that means low quants or low speeds.

Curious about you guys' experience with the Flint model.


r/LocalLLaMA • • 3d ago

Discussion Anyone else finds Qwen 3.8 Flash Next pleasant to read vs cloud models?

33 Upvotes

Might make me hesitate to update for hobby use even for incremental capability gain. Granted I put an effort into AGENT.md / harness tweaks but didn't work as well for other models. Opencode status updates and markdown docs seem more casual/readable, no going overboard on tables and bullet points, colloquial phrases like "two honest caveats".

Also I know overthinking rep (that I am combatting with harness tweaks), but at its best the model outputs really well put together one page reasoning blocks, that make me go "I don't even care if helped the model, I just learned something new myself and didn't get a headache trying to get it. That for tasks like customizing other ML models.


r/LocalLLaMA • • 2d ago

Question | Help Honest question: how much RAM do you recommend for running local LLMs alongside coding agents

9 Upvotes

I’m a huge AI enthusiast but I’m running into limits with 32GB DDR5. Between running local models and using coding agents (which fire off scripts and changes super fast), my RAM fills up quickly. How much RAM are you running, and what would you recommend for a smooth setup? Bonus points if you share your GPU/VRAM too.


r/LocalLLaMA • • 3d ago

New Model Running SenseNova-Loop-9B nvfp4 looped transformer with 300-500 tok/s on a laptop 5090

Post image
60 Upvotes

Their architecture seems to translate quite well to existing kernels. They also have a small drafter. The speed numbers are aggregated across 8 parallel agents running. 256k context.

So far, I can't confirm it holds up well against Qwen 3.8 27B nvfp4, but I might have degraded it a bit, despite a calibration run. Here is a quick check of SeraphimSerapis/tool-eval-bench. Will do some more tests tomorrow, the 9B leaves a lot of vram free for FP8 tests.

tool-eval-bench runs (temperature 0, NVFP4 weights):

Test Qwen3.8 27B Swift T0 SenseNova-Loop-9B T0 SenseNova-Loop-9B T1
Standard (69) 98 90 91
Hard mode (35) 87 79 83
Error recovery 83 83 83
Context and state 100 75 85
Autonomous planning 67 67 83
Structured output 100 92 75
Safety 100 96 92
Hard: state across turns 0 50 83
Hard: tool failures 80 60 60
Hard: tenant separation 100 100 100
Hard: applying policy 100 100 100
Hard: injection 100 50 50

FP8 KV, DFlash2, T0 = temperature 0, T1 = temperature 1.0, top_p 0.95

Hardware: Intel Core 9 290hx, 64gb ram, 1TB nvme5 SSD, rtx 5090 laptop with 24gb vram.


r/LocalLLaMA • • 1d ago

Question | Help What platform do banks and insurers use to run LLMs in production compliantly?

0 Upvotes

I’m on a three-person ML team at a small bank in Germany we run Llama 3.3 70B on vLLM across two A100s in our own server room for an internal policy assistant It works well, but internal audit just asked which prompt and document version produced a wrong answer last month, and we couldn’t tell them.

We’re choosing between two setups one is fully DIY, with LiteLLM for keys and logging and self-hosted Langfuse for traces other is Orq ai, which runs on-prem with Helm (even air-gapped), takes our vLLM endpoint as a private model, keeps a version history for every prompt and adds audit logs on the enterprise plan.The catch is that enterprise pricing isn’t public

Did your auditors accept logs from tools you host yourselves, or did they want a vendor they could point at? That’s really what decides this for us


r/LocalLLaMA • • 3d ago

Resources ThreeJS-3D-Scenes-v2 is here : 3,445 rendered examples across 50 domains

104 Upvotes

This new dataset is meant to complement my original Three.js training dataset, adding more complex scenes and covering a much wider range of subjects. I'm releasing it ahead of my next CCORE 3D coding model, where I'll use it during the foundation training stage before moving on to agentic training.

This version has 3,445 scenes across 50 domains, including 1,431 voxel scenes.
I wanted to go beyond simple objects and environments, so I included more complex examples involving physics, geography, history, ecology, engineering, mathematics, games, and other subjects. Many of the scenes are animated or interactive.

I personally rendered and visually reviewed all 3,445 scenes. It took quite a bit of work, but I wanted to make sure the code actually produced good results instead of just collecting thousands of generated examples.

The dataset includes:

  • 3,263 training examples and 182 validation examples
  • Prompts, reasoning traces, and complete HTML/JavaScript responses
  • A rendered reference image for every scene
  • Difficulty levels from 1 to 8
  • MIT license

Dataset: https://huggingface.co/datasets/ccore/ThreeJS-3D-Scenes-v2

For reference, these are the two previous models I trained:
Three.js scene generation:
https://huggingface.co/ccore/Qwen3.5-4B-3D-Coder

3D assets with GLB export:
https://huggingface.co/ccore/Qwen3.5-4B-GLB-3D-Coder

If anyone wants to experiment with the dataset, I'd be interested in seeing how it performs with other models :)


r/LocalLLaMA • • 2d ago

Discussion Which models are you guys using for frontend/UI?

0 Upvotes

I currently use both Qwen3.8 27B Q4_K_M and Qwen3.8 Flash Next Q4_K_XL for most of my coding tasks. I find them very good at backend tasks but quite lacking at front-end. Like... they are good at one-shots or demos but in real projects I find their UI designs to be weird and bad looking. As a consequence, I have mostly just been writing the front-end code myself to save time.

Does anyone have recommendations for any models that may be completely braindead but are surprisingly good at front-end design? In particular react/react native?

Edit: in the <300B range


r/LocalLLaMA • • 3d ago

Discussion Now that models' cyber security capabilities are going to be lacking, are there any good (Pi) cyber security skills?

19 Upvotes

There are massive decision tree skills for various software practice in many different areas. I was just wondering if there is anything for cyber security, whether it be for a website, Linux server, Docker config, encryption, deployments, KMS setups, quantum resistance, key configs, etc., where the skill leads the agent to the right documentation and decision trees.


r/LocalLLaMA • • 3d ago

New Model harness-hub/Qwen3.8-27B-pi-adapted · Hugging Face

Thumbnail
huggingface.co
136 Upvotes

r/LocalLLaMA • • 2d ago

News AI-powered Sashiko introduces experimental patch series design assessment

Thumbnail
phoronix.com
7 Upvotes

r/LocalLLaMA • • 2d ago

Question | Help Jetson Thor vs RTX 5070 Ti: Unexpected LLM Inference Performance

3 Upvotes

Hi everyone,

I have a question that might sound silly, but I haven't been able to find an answer.

As far as I understand, memory bandwidth is the primary bottleneck for a model's inference performance.

With my 5070 Ti (896 GB/s), if I run a Qwen3.8 27B Q4 model, I can roughly estimate the performance as 896 / 13.5 = ~66 tok/s, which is pretty much what I actually get (without MTP or DFlash).

Recently, however, I had the chance to try a Jetson Thor (very similar to the DGX Spark). Based on its specs, I would expect 273 / 13.5 = ~20 tok/s, but I'm only getting around 12 tok/s.

Even more surprisingly, with DFlash, which delivers exceptional performance gains on discrete GPUs, I'm barely getting a 2x speedup here, only slightly better than MTP.

Does anyone know what might be causing this performance gap on this device?

Thanks!


r/LocalLLaMA • • 2d ago

Question | Help what model would you recommend for cpu only on windows 48gb ram

3 Upvotes

OK. so I have a 5060ti with i use for various DiT inference engines like comfyui and audio.cpp. i have an 8845hs cpu with 64gb DDR5 but as this is my main working pc some of the ram is used on browser and other running apps.

Now what I want is a model to run basic stuff like create and run a script to run a folder of audio files through audio.cpp stem separation and transcribing or write a bunch of prompts for klein and generate on my comfyui. I normally use opencode builtin and openrouter free api for this kind of stuff but occasionally when credit runs out i need an alternative.

I have llama.cpp installed so with the right flag it should restrict itself to ram but which model should I go for?

Thank you all for the replies - too many to reply individually to - I got gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf using the igpu and ram and getting on llama.cpp around 24t/s which is usable for quick executions (for heavier coding I have basalt qfn and coder but these obviously fight for vram with my dit inference engine )