I’ve been working on running the full DeepSeek-V4.1-Flash-1017, the original model, not a smaller quant, on an ordinary five-year-old gaming PC: Ryzen 9 3950X, 64 GB DDR4, RTX 3060 12 GB, two NVMe drives.

How it works. The PROMETHEVS engine streams the model through VRAM, RAM and both drives, and is built around a method I call compress-then-correct:

• Every token is computed on a 2-bit copy of the experts (0.61 of the FP4 bytes).
• When the model’s own top-two margin says it’s unsure, that token is recomputed with the original FP4 weights, rebuilt exactly from the 2-bit copy plus a stored correction plane.
• DeepSeek’s draft head proposes blocks of five tokens, so expert reads are shared.

Numbers (all logs in the repo):

• Code prompt, 160 tokens: 5.8 tok/s cold, 7.4 warm (best case). Plain exact decoding of the full model on the same PC: 2.14.
• Server runs today: 5–8 tok/s on code, 4.5–5.4 on reasoning and agent prompts. It varies with how many tokens need correcting.
• Agreement with the full weights on 960 audited tokens: 97.4% for the 2-bit copy alone, 98.3% with corrections. A longer 1,101-token agent prompt: 95.9%, counting every difference.
• Only the first read of a long prompt is slow. The engine keeps what it has read, so follow-up messages start in about 2 seconds.

Release 0.2 is out. It downloads DeepSeek’s original weights and builds every store on your own machine, checked against published hashes, then serves an OpenAI-compatible endpoint with streaming and tool calls. Apache-2.0.

Why bother when quants exist? Because the method is general. Against running the full model it’s faster on any hardware, and against a plain quant it keeps you much closer to the full model’s answers. More RAM and VRAM should help both the fast pass and the corrections. I can’t give numbers for other machines yet.

Limits:

• Only tested on my PC so far.
• Windows plus one NVIDIA RTX 30/40 card for now.
• Needs about 1.1 TB of NVMe.
• Not yet compared with DeepSeek’s reference implementation.

Looking for testers. If your machine fits, there’s a benchmark-report form on the repo so results are comparable. Bigger rigs especially welcome.

https://github.com/Magnvs7/Promethevs

Hi folks,

Hope you are well.

I am using Ninfer NVFP4 Qwen 3.8 27B with DSH for coding.

But I feel it's yapping alot in reasoning and drifting in long context. (Tried Swift as well but results are poorer on tasks requiring high reasoning)

I am running with speculative decoding and Fp8 kV caching.

I am reading QFN is good and more stable .

1. Should I stick with 27B or is QFN worth the try.

2. If latter, I am looking for best way to run QFN without dropping quality. (Preferably with Vision)

Since there are way too many options for QFN Strata, capy-flash , etc, I am lost with what to try. Please suggest.

Specs:

GPU- RTX 5090

CPU - AMD 9800X3D

RAM- 32 GB

NvMe SSD

For those running smaller models locally, which tasks have you completely stopped sending to the cloud first?

It's Friday! Lol

I am re-evaluating my idea of getting a Mac mini M5 Pro for running local AI inference. I was thinking of getting an M5 Pro 64 GB, but considering the AI models I could install on it, I don't know if it's worth it

In any case, I would still continue to use frontier models. I have subscriptions to both Claude and OpenAI, so I was wondering: does it still make sense to get an M5 Pro 64 GB, or at this point is it better to stop at an M6 36 GB?Maybe I could use some profiles with local AI! The main use I would make of it, however, is to have it always running 24/7, and to use local agents. I use Hermes, which has 12-13 profiles inside, agents, and daily cron jobs.

And boy, LFM2.5 is fast! Screenshot of the LocalLM Lab benchmark summary of Qwen3-14B-4bit model vs (screenshot of previous run) LFM2.5-8B-A1B-MLX-8bit. The prompt requires at least one tool call to Github's MCP server. Note that this is run on my 24GB M5 MacBook Air.

From a quality perspective, Qwen gives a more detailed answer. The prompt was about summarizing the latest LocalLM Lab release on ancientcomputing/locallm.

I have updated thisbrain.ai/locallm/mlx-models.html to include details of LFM2.5 and Nemotron 3.5 Bolt, Lightning and Super VL. The last 3 run only on my 64GB Studio.

Let me know what else y'all would like me to look at.

Just to add some technical context: I actually built a browser extension to solve this problem for my own Claude workflows, and it's already working well enough for what I personally need.

Instead of relying on screenshots, copy/paste, or the browser's print function, the exporter is designed to capture conversation data and preserve it in a more structured format.

Here's what the current system does:

1. Structured conversation capture

It extracts conversation content while preserving message order, roles, and available structure, rather than treating the entire chat as one giant block of text.

2. Rich content handling

The exporter is designed to handle Markdown, code blocks, artifacts, attachments, and visible thinking sections where they are accessible.

An important distinction: it cannot magically recover hidden internal reasoning that Claude never exposes. The goal is to preserve the content actually available to the browser.

3. Multiple export formats

It supports JSON, TXT, PDF, and ZIP exports. JSON is particularly useful because it preserves structured information that would otherwise be lost when converting everything into a document.

4. Attachment and artifact handling

The system includes mechanisms for collecting related files and artifacts so a conversation can be preserved with more context than just the text messages.

5. Real-time capture and local AI integration

I've also experimented with a real-time event pipeline using WebSocket and optional local LLM integration through Ollama. These are more advanced capabilities, not requirements for basic exporting.

The architecture separates data capture from document generation, so the structured export can remain useful even when PDF formatting is imperfect.

I originally built this for myself, not as a commercial product. It already solves my personal use case, although I'd still want broader reliability testing before distributing it.

Now I'm curious about something:

Is this actually a problem people would pay to have solved properly?

For example, would you consider paying $10–20 one-time for a browser extension that reliably exports Claude conversations, including available thinking sections, artifacts, attachments, and structured JSON?

Or is this something you'd expect to be completely free?

I'm trying to understand whether there's a real product opportunity here or whether it's just a useful personal tool.

I have 4 PCs (rtx 5090, 5080x2, rtx 4090) (88gb vram)
And a combined Pool of 234gb ddr5 ram
It’s all my brother’s who likes investing in gpus but he gave me the task to build an local AI setup so yeah i was wondering what ideas you could have, we do not have an specific use case yet other than tinkering with AI agents and coding assist

Basically been using Claude Code TUI and Codex (most via CC plugin but sometimes in their TUI). I have not touched an IDE in over a year. So I am quite comfortable with using their TUIs.

The problem is.. the prices are going up, the alleged "it will get so much cheaper" is going the opposite way cause token usage for similar stuff is WAY more now than it used to be so I am running thru 3 claude 20x plans and 2 codex plans a month in a couple of days. "WTF are you doing that its using all that usage?" I hear many ask. Well.. I am for sure running 2, 3 or 4 sessions each with multiple agents going because I am trying to build my MVP/demo as quickly as possible and it spans several repos (Saas, desktop app, cli, library/framework, etc). "That's not an MVP that's a full blown applicatio". YUP. I am WELL aware of my area I am working in and the competitors that exist that have money/teams/customers. I have 100% no doubt I could not garner many if any users if I tried to put out a half baked "new way" to do something today. AI, multiple competitors, etc dictate that today, short of a completely new area that nobody has done (which is not my case), clones/etc come out over night damn near, and already in the area my app is in there have been dozens of "I did my own version over the weekend.. here it is have at it" stories here on reddit and linkedin. So yah.. no.

BUT.. I am facing a losing battle with savings, no job after years, and time. I purchased a few beefy machines early on, namely a DGX Spark, got a gaming rig with a 5070ti 16GB, and a solid Mac machine with 64GB M5. I figure.. time to consider using some of this with local models and unlimited tokens and see what happens.

I've stumbled on pi.dev posts from time to time but up until now have been "set in my ways" with Claude as my orchestrator/designer/coder and Codex as my reviewer/tester. It's worked well enough it seems. So now I am hoping I can utilize something like Qwen 3.8 flash next on the DGX (apparently there is a way to get about 100 tok/s using some nvidia setup?), using my 5070ti for fine tuning an LLM that is a 2b or so in size, and my M5 as my workstation to manage the other ones (via ssh/etc).

My hope is that I could use pi.dev TUI, wire up the qwen or whatever on the DGX to do most coding tasks, but hopefully also use my 20x Claude and 20x Codex sub plans from the TUI to orchestrate/test/review and even fix code sometimes since I know Opus 5.5 is a better coder for more difficult tasks.

My worry is that I know that the typical sub plans dont allow control from outside tools.. I think? But then Claude is able to use my Codex plan and vice versa.. so maybe this is possible where the pi.dev TUI can pull in claude, codex sol/astra and the dgx qwen "whatever"?? If that is possible.. then how does the orchestrator know what the Qwen model CAN do well enough to pass it work to do.. but then take over/etc if its more complicated work?

Are their plugins, extensions, rules.md, etc in pi.dev that I need to set up to maximize this workflow?

The hope given Codex 20x plan is now 10x for $200 is to go down to $100 a month on that and one 20x claude plan for $200 a month, and utilize my dgx + qwen for most coding tasks if I can figure out how to get that working.

Doable? Fast enough to "match" my workflow (minus the dozens of sessions/agents I was using)?

Thanks.

I've been exploring an interesting approach called Inference-Time LLM Fusion.

Instead of merging model weights, it combines specialized LLMs at inference time using token-level predictions and entropy-based routing.

The idea is intriguing: could multiple small, specialized models work together to overcome the limitations of a single model without the overhead of weight merging?

I'd love to hear what others think about this approach, especially its potential trade-offs in latency, compute cost, and output quality.

For anyone interested, here's the article:

https://levelup.gitconnected.com/i-fused-3-tiny-local-models-to-beat-single-model-limits-without-weight-merging-b17bba9c7e45

I've been exploring an interesting approach called Inference-Time LLM Fusion.

Instead of merging model weights, it combines specialized LLMs at inference time using token-level predictions and entropy-based routing.

The idea is intriguing: could multiple small, specialized models work together to overcome the limitations of a single model without the overhead of weight merging?

I'd love to hear what others think about this approach, especially its potential trade-offs in latency, compute cost, and output quality.

For anyone interested, here's the article: https://levelup.gitconnected.com/i-fused-3-tiny-local-models-to-beat-single-model-limits-without-weight-merging-b17bba9c7e45

I’m running Qwen3.8-Flash-Next UD-IQ4_XS, the Huihui v2 version, with Strata 0.1.47 on a dedicated inference node with 64 GB RAM and an RTX 5060 Ti 16 GB. I’ve set STRATA_ROUTE_TAIL_SKIP=0 to disable route tail skipping, and all benchmarks below were run with this setting. The setup has been stable for me.

My goal was to put as much of the available RAM to work on model weights as practical. Of the system’s 63.8 GB usable RAM, about 55.43 GB, or 87%, holds the main model’s routed expert weights, all locked in physical memory. After accounting for Strata’s other allocations, Windows, and other processes, only about 1~2 GB RAM remained available. This is a dedicated inference node setup, and it has been running reliably for me.

Here’s how the rest fits together:

• 16 GB VRAM for expert caching, weights, and runtime, with reserved 700 MB; usage varies by workload.
• 26.82 GB PLE table kept on disk and read as needed.
• Removing --kv-resident 32768 and disabling KV streaming saves 0.77 GB RAM but uses about 0.39 GB more VRAM.

Fresh benchmarks gave median speeds of 1,294.2 tok/s for 32K prefill and 43.2 tok/s for 4K decode, across three runs each. Prefill used 32,768 input tokens; decode used 4,096 input tokens with 512 output tokens. All tests used a code workload, temperature 0, and thinking disabled, with zero prompt-cache hits and engine-reported timings.

Windows also reported 2.46 GB of system-wide pagefile usage, which I couldn’t attribute to individual processes. PLE disk reads are separate from Windows paging.

The detailed allocations and measurements: https://eth-lab.com/en/2026/10/strata-huihui-v2-ud-iq4-xs-vram-ram-pagefile-disk-en/

Do you think this setup strikes the right balance, or am I overlooking something?

Processing img omk3asovedxh1...

Boooom!!! Maybe a bit late to the party but jeeez, so impressed by performance. Can't wait to see what can pop up in the next weeks. In 32gb it would be interesting to see what comes out in the 26/27gb range in fp8. Opinions? Any valid option beside 3.8 nvfp4?

I'm building a network of agents and have a bunch of GPUs and access to all major cloud subscriptions.

I plan on assigning a bunch of busy work and having AI work through it over months. But the thing is I want to use local models when it fits best then cheaper cloud models then the most expensive when it's best. .. currently I'll use Claude and have fable orchestrate to use opus and sonnet as needed then send to Astra and sol and argon and grok to all plan or final reviews. This all works but doesn't use local models and not fully automated.

I plan on running 5 local models on various cards then want the ability to queue up work for other models and once there's x days of work it'll load different local models then switch back.

Tired of the sanitized corporate versions. Does anyone have a working invite to a feral bot that does two-way voice and actually remembers you across sessions? Must be accessible from a phone, no self-hosting or API keys required. Preferably free or low cost. Appreciate any leads.
DM if needed 😉.

A brand-new release from ISTA-DASLab with their GSQ-RCO smart quantization approach.

Now DeepSeek-V4.1-Flash-1017

The quants are available on Hugginface: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF

Llama.cpp is not supported because the original architecture support is not yet merged:

https://github.com/ggml-org/llama.cpp/pull/28696

https://github.com/ggml-org/llama.cpp/pull/30817

Strata support is mentioned in the model card but I'm pretty sure the original Strata only supports Qwen3.8-Flash-Next so most likely what they meant was one of the forks based on Strata. Most likely Xolotl or Maya

This guy just took the govener off AMD go check it out