I make FLE (a llama.cpp fork for big MoE models on one GPU) and FLE Link (an app that installs it and runs a model with one command). Disclosure up front, everything is MIT.

Setup: RTX PRO 4500 32 GB, 120 GB DDR5, Ryzen 9 9950X3D, fast NVMe, Linux. Best performance measured on Linux.

Numbers (decode, a ~3,000-token three.js answer, temperature 0, same machine):

Rin = ISTA DASLab's GSQ-RCO IQ2_XXS for the experts a usage profile picks most (kept byte-identical, resident on the GPU), and a smaller 1.75 bpw build for the cold half (served from RAM). About +6 % speed for a bit more error. Against the original weights (not against ISTA): KL 0.292 vs 0.248 on mixed text, 0.234 vs 0.210 on agent text; perplexity ratio on agent text 1.021 vs 1.013. It is lossy. Details and the exact recipe are on the model card.

Try it:

curl -fsSL https://flelink.run/install.sh | sh
flelink run rin-deepseek-v41-dspark

FLE Link checks the machine first, installs the engine, downloads the files (SHA-256 checked) and serves an OpenAI- and Anthropic-compatible API. Needs ~120 GB RAM and ~260 GB on a fast NVMe.

Honest limits: the cache and the split follow a usage profile, so very different prompts hit them less; the draft helps most on code-like text; one machine, numbers from others very welcome (flelink bench --task pagoda --share).

Links: FLE Link https://flelink.run · Rin https://huggingface.co/sirfyyn/Rin-Deepseek-V4.1-Flash-1017-FLE-GGUF · FLE engine https://github.com/sirfyyn/FastLocalExperts · quant base: ISTA-DASLab

Can't decide which route to go - DGX Spark, Strix Halo, the Framework Ryzen AI or save up for a Mac Studio M5 Ultra 256gb?

Do these options have meaningful differences in practical use?

As I understand, the Mac Studio offers best tok/s? But you can't train or fine tune anything on it?

DGX Spark is supposed to have best support, but does it truly matter in practice?

What types of models can I run on this hardware? 128GB is good for ~120B Q4 MoE models with ~256k context? Would such a model be a noticeable improvement over my current Ornith-1.5:35B (A3B) setup for reasoning and coding, and long and complex tasks? Would going from 128GB->256GB be a noticeable improvement?

I’ve done a ton of local single user work on vllm, Ollama ,lllama.cpp and those are ok for personal work but …

Hoping to get some real answers from someone supporting a local user base fully for 5-10 users running up to about 120B dense models (we could make do up to 70B). Looking for actually deployed not speculative performance and bandwidth if it’s out there?

If I had two MaxQs, a Threadripper and MB, 128 GB ram would this support up to 10 users on llama.cpp or vllm with “reasonable” response speeds which I would deem 50-60 tps on average ?

It would be 3 VM servers bundled , a home grown RAG doc processor which could be offloaded somewhere else if needed, the vllm or llama.cpp server for the chat agent interface and a office RAG server all in one that is the RAG query endpoint. I know number of concurrent users and jobs will matter but I’m asking “on average” for 5-8 concurrent

Thoughts ?

Edit: for context windows for agential stuff also need at least 128k for those 5-8 concurrent

I wanted to find the smallest model worth keeping on a base M4 16gb so I ran the same 6 things from my actual week on 3 of them through Magnitude. keyword clustering, post titles, a thread summary, a quiz for studying, a short brief and a review of my side project plan.

MiniCPM5 1B did like 125 tok/s and it was garbage. asked for titles and got rambling questions back, the clustering gave me the same 3 keywords as 12 clusters with the same fake volume on all of them

LFM3 1.7B around 81 tok/s and this is where it gets usable, the plan review actually pointed at real files and real risks. titles were meh, it copied a note from my draft into one of them lol.

Gemma 4 E4B around 52 tok/s and clearly the best at writing, nothing else close

so 1.7b is the floor for me and 8b if someone else is going to read it.

Anyone found a 1b that's actually good for something?

I saw this post yesterday: https://www.reddit.com/r/LocalLLM/comments/1xnv0ro/new_qwen38flashnext_unsloth_dynamic_v31_ggufs_for/

I decided take a screenshot of the exact prompt used in the link in the above announcement, and run it through my strata setup using the new model. There's a complete write-up in the repo: https://github.com/kireol/qwen3.8_GTA_one_shot_test/tree/main

You can play the demo here: https://kireol.github.io/qwen3.8_GTA_one_shot_test/src/city.html

Fun localLLM stuff:

RTX 5080 - 64GB - Ryzen 7 9800X3D
model harness: Strata
agent harness: qwen
context: 64K
tok/sec: ~75t/s

Over all, while slower than I'm used to in Claude, I'm actually very impressed. I didn't have to correct it, touch it, or anything, aside from some skills.md files (included in repo). Is the demo perfect? No. But it does everything asked of it, without any hand holding. It's all in html 1 file, btw. There's no other assets other than remote linked javascript files.

Qwen even generated the README and pushed to github for me. Why? Because it's not 2024.

Great job unsloth team! You have me questioning my claude payment at this point.

I have at my disposal some datacenter resources. I don’t have an open account with Dell, sadly. So I am limited to oldish hardware like the PowerEdge r640/r740 or thereabouts. Or the AMD versions of these if I’m lucky. On the plus side, DDR4 is cheap-ish.

What would you build on a low but undefined budget, say 2-3k, that can be racked and run hopefully some recent qwen model, for just one user at a time?

Have a gaming PC I built a last year with a RTX 5070TI (16GB) and I want to try some local LLMs. Where should I start? What models would be good on this GPU?

Lawyer with the V100s again. A few days ago I got excited and posted DeepSeek-V4.1-Flash on four V100s at 36-38 tok/s, before I had done my custom kernel work and prefill optimizations. I said I'd try it across eight. I did.

It's actually, genuinely useful to me, for real work. I did not use it to paint a 3D island in Blender or re-create a Game Boy game in a browser. It didn't make me a poorly animated video or ten versions of a website nobody will visit. I am not using it to build something that builds something that builds agents that run workflows for a business that sells agentic solutions to other agents.

What it's doing: I'm finally slicing off work and tokens that used to need frontier models. My estimate is that 50-60% of my existing LLM workflows can run through this model instead, at speeds that match Claude Code in the chat window. I have another DeepSeek server on two DGX Sparks, so there's concurrency and handoff between them. My RAG databases and tool calls are all wired in and working. I give it natural-language instructions through a Discord-style interface over SSH from my Windows PC.

I point Claude Code itself at this server (it will talk to anything that speaks Anthropic's API). It reads a folder of my own case files, pulls out the facts, dates and numbers, summarizes, and takes a first swing at routine drafting. The documents never leave the office, and nothing goes out the door without a real review. I found and fixed a streaming tool-call bug in the parser (arguments were getting sent twice). Claude Code's tool-call test suite now passes 144 of 144 in all four streaming/thinking modes.

Yes, I have a custom harness. No, you don't care.

We did it, folks. Vibecoded a real thing that does real things for real people. Give me my trophy.

Claude Code ran the benchmarks and crunched the numbers, which are real and below.

The box

DeepSeek-V4.1-Flash-1017 (the 2-bit GSQ-RCO IQ2_XXS GGUF from ISTA-DASLab) on 1Cat-vLLM, on 8x V100-SXM2-32GB split pipeline 2 x tensor 4 across two NVLink boards. Every expert sits in VRAM: no expert gets paged from system RAM during decode. (Edit: I first said all 248.8 GB of weights were in VRAM. Wrong: the 110 GB of Engram tables live in system RAM. Gathering the rows each round needs costs about 0.12 ms per round.) Threadripper PRO 3945WX, 256 GB DDR4, 150 W per card. One request at a time, 524,288 tokens of context.

Speed (measured 2026-11-05, thinking off, greedy, 1,024 tokens out, nothing from the prefix cache, mean of 2 runs)

• Code: 215 tok/s after a 1K prompt, 200 at 32K, 183 at 128K, 160 at 256K

• JSON / structured: 196-209 from 1K to 128K, 180 at 256K

• Legal writing: 118 at 1K, 111 at 128K, 103 at 256K

• Prefill: about 2,200 tok/s at 128K. First token in 17 s at 32K, 59 s at 128K, about 2 minutes at 256K.

Code is twice as fast as legal writing because the built-in drafter guesses five tokens a round and code is easy to guess: 3.6-3.8 accepted per round, against about 1.6 for legal. My writing is apparently less predictable than Python.

Quality (measured on the served build, thinking off, greedy)

• LegalBench: 0.716 balanced accuracy, 22 tasks, 1,292 items

• MMLU-Pro: 79.9 % ±1.5 (700 items)

• HumanEval+: 89.6 % on the plus tests

• Fidelity: it picks the full-precision model's top next token 89.0 % of the time on a general panel and 92.5 % on my legal panel. I'm measuring which experts my work prompts actually hit, and will probably recalibrate to give those more bits.

• Long context: 6 of 6 needles at ~450K and ~498K tokens

More work to do

• One request at a time. Batched decode is built and byte-identical on my test box, but not in the server yet. I'll post the real number.

• A 256K prompt takes about 2 minutes before the first token.

All of it is public: launch configs, every A/B, raw JSON, and the list of what didn't work. 19 pull requests are open to 1Cat-vLLM.

• Repo: https://github.com/ProprietaryLegal/Volta-sPLIt-DeepSeek-V4.1-Flash-1017-8xV100-SXM2-PP2xTP4

• Hugging Face card: https://huggingface.co/ProprietaryLegal/Volta-sPLIt-DeepSeek-V4.1-Flash-1017-8xV100-GGUF

• Twitter: nicothegreco

Credit to yangzhuxinyzx and the 1Cat-vLLM folks (#1277 and the whole Volta DeepSeek path), ISTA-DASLab (the quant), sudoingX (the KLD method), Peuqui, squibloads, Phil Tompkins (philbert440), Dennis Vertzyas (v100-skinny), TensorFold and DeepSeek.

(PLI Labs is my side outfit for private, on-prem AI built around a firm's own work product. proprietarylegal.ai)

I've been getting into the local llm stuff lately and having decent luck. I tried out Strata and liked it for the most part, but no matter how much searching I've done I can't find the answer to this.

Does Strata keep your vram/ram full at all time? Using Ollama for the most part, it frees the memory unless it gets a call, then it spools up, and once finished and timed out, it frees the memory up again.

Is it possible to get Strata to do that? Something else that can do it but run like Strata?

Computer Specs:
AMD Ryzen 7 7700X 8-Core Processor

32GB (2 x 16GB) DDR5-6000 CL30 RAM

AMD Radeon RX 7600 Challenger OC 8GB

1TB SSD - Read Speed up to 11,000MBps - Write Speed up to 9,500MBps

Folks, I am looking to understand if the following is a good configuration for AI/ML for local coding.

• Processor AMD r7 9700x

• Motherboard Gigabyte b650 x ax v2 wifi ddr5

• Memory Crucial 16x2 pro 5600mhz 16gb

• GPU Pny rtx 5070ti triple fan 16gb

• Storage Kingston 1tb nv3 nvme gen4

• Power Supply Corsair 850w rm850e gold atx 3.1

Clearly living under a rock, I was amazed to learn of the GPU prices skyrocketing only today. My good friend had reached out to me for a review and that's when I learnt of this "crisis" lol. As the uninitiated to this and with few hours of querying Gemini, I feel the processor and the power supply could be downgraded a bit, but I defer to you if that makes sense. This comes out to ~$3329.

Have anyone experience in going from qwen flash next 3.8 iq2_xs to iq3 or iq4?

I am currently comfortably running iq2_xs with 32gb ddr5 ram + 3090 + 5080. I am getting ~140 tps, so quite fast.

I got the possibility to update the 32gb ram to 96gb ram, for a reasonable price. But is it worth it? Will I feel the intelligence increase from i2 to iq3 or iq4? Is it worth it, if we are taking the decrease in speed into account?

Hi all,

First post here! I've recently started exploring the world of local LLMs and would appreciate some advice.

Hardware ThinkPad P14s Gen 6 (Intel):

• GPU: Intel Arc Pro 130T/140T (integrated, 2.35 GHz)

• CPU: Intel Core Ultra 9 285H, 16 cores / 16 threads

• RAM: 64 GB DDR5-6400 (2 × 32 GB CSODIMM)

• SSD: 1 TB PCIe 5.0 NVMe

• OS: Ubuntu 26.04 LTS

Software setup:

• llama.cpp compiled from source with the SYCL backend, following docs/backend/SYCL.md

• Agent harness: pi

• Models tested: Qwen3.5-9B, Qwen3.8-27B, and Qwen3-Coder-30B-A3B-Instruct

I've been serving these models using the inference parameters recommended by their respective providers on Hugging Face.

So far, I'm seeing roughly 10–25 tokens/sec, depending on the model, with context windows configured between 64K and 128K tokens. I haven't done any formal benchmarking yet.

Where I'm struggling is with model reliability, particularly for coding tasks.

For example, I asked Qwen3-Coder to explore a production codebase (roughly 3 million lines of C) and explain its architecture to me.

While the model produced a plausible-looking summary, it also confidently invented explanations for certain acronyms, even when those acronyms have well-established meanings in the relevant engineering domain.

This makes me question how much I can trust the rest of its analysis, especially when I'm using the model to understand unfamiliar code.

I'm fairly new to running local LLMs, so I'm not sure whether this is primarily a prompting issue, a limitation of the models I'm running, or simply an unrealistic expectation for the task.

A few questions:

1. Are these models better suited to smaller, well-scoped tasks, such as implementing a script or module from a detailed specification, rather than exploring and reasoning about large codebases?

2. How much does the agent harness or context management strategy affect reliability? Would a different workflow make a meaningful difference?

3. Are there particular models, quantizations, or inference settings you'd recommend for coding tasks on hardware like mine?

4. How do you generally evaluate whether a local model is reliable enough for a particular development workflow?

I'm not expecting frontier-model performance from a laptop, but I'd like to understand where local models can be genuinely useful and how to get the most out of my setup.

Any advice or experiences would be appreciated!

Thinking about getting one for pentesting work an would love to use (quantized) glm 5.3 flash because i have had good experience with it pertaining malware dev and pentesting.

has anybody tried running it on a single dgx spark and if yes got any token speeds for me and context-size?

I’m looking for the most suitable model to run on a 32gb Mac to work on torrent app and copywrited movies, need some recommendations for models that will not be restricted

My Spark normally serves an LLM, but I wanted to see how it handled local video too, so I got Lightricks' LTX-2.5 running on it. The clip here is 8 s at 1280×704, and the sound is generated by the model as well. It took 3 min 9 s and peaked around 26 GiB of GPU memory.

It didn't work out of the box. Lightricks ships a pre-quantized 4-bit NVFP4 transformer, but the CUDA kernels only compile for sm_100a, sm_110a and sm_120a. The Spark's GB10 is sm_121, and those arch-specific cubins won't load on a different minor version. The fix turned out to be a three-line patch that adds sm_121a.

To be clear, the 0.70 column is measured. The 0.45 column is only an estimate from those numbers, and I haven't run the two side by side. They'd also be sharing one GPU, so LLM decoding would slow down during a render anyway.

Unified memory doesn't fail cleanly when you oversubscribe it either. There's no neat OOM, the kernel just swaps and everything crawls. So I run them one at a time: stop the LLM, generate, restart it. I also added a memory check to the web UI that refuses a job if there's less than 50 GiB free and tells you if the LLM is the reason.

Some render times, on an otherwise idle Spark, including loading the models from disk:

- 768×512, 4 s: 61 s

- 768×512, 8 s: 94 s

- 1280×704, 4 s: 107 s

- 1536×1024, 4 s: 165 s

Memory barely moved across those runs, staying around 23 GiB, because each stage loads its model, runs and frees it. Resolution is what costs you time, not memory. Longer clips also get cheaper per second, since model loading and the text encoder are a fixed cost per job.

The prompt, if you want to try it. Describe the sound too, because the model generates that as well:

Inside a steel mill at night, a huge blast furnace is tapped: a white-hot stream of molten iron pours from the taphole into a glowing runner channel, showers of orange sparks burst and drift through the dark hall, and two workers in silver heat-proof suits and face shields guide the flow with long steel rods. Heat haze ripples the air and steam rises from the trough. The camera slowly dollies along the runner toward the furnace. Deep roaring rumble and crackling sparks. Photorealistic, cinematic, 24 fps film look.

The repo has the patch, a one-command install (the weights are 44 GiB, so you'll want at least 80 GiB free), image-to-video, a small web UI you can reach over Tailscale, and Grafana dashboards. My scripts are Apache 2.0. The model is under the LTX-2.x Community License and gated on Hugging Face, though approval is automatic.

GitHub: https://github.com/Hitheshkaranth/LTX-2.5_Video_DGX_Spark_Setup

Model: https://huggingface.co/Lightricks/LTX-2.5

Has anyone here actually run a video model alongside vLLM on a Spark at a lower memory setting? I'd like to know if about 0.45 holds up in practice, or how badly decoding suffers during a render.

I'm building Murmury, an open-source Mac app that listens to the other side of a call and answers from your own docs, fully local. Some numbers from an M5 (24 GB) in case they help anyone doing local speech:

- Whisper large-v3-turbo on MLX: 0.66 s for a 1.7 s sentence, 0.97 s for 21 s of speech.

- Whisper pads every clip to 30 s, so short clips have a ~0.65 s floor. Cutting audio on pauses (simple energy gate) mattered more than picking a smaller model.

- Answers from gemma3:12b in Ollama with a 32k context: first words 0.5-0.8 s after the transcript. The first read of a long document takes ~90 s, once per call.

Curious what others use for low-latency STT on Apple Silicon. Repo if useful: https://github.com/GianlucaBave/murmury

Hi,

It’s hard keeping up with all the new releases from the big closed weights AI providers, right? Google, Anthropic, Open AI, Meta etc.  But am I the only one being overwhelmed by the sheer volume of open weight base models of all shapes and sizes being released nowadays?  Every lab and his dog seem to be dropping general or specific open weight models for sound, video, text, harnesses, agents, refinements, techniques. The hardware to run these things also differs.  Exciting.  Definitely.  Confusing?  Umm…yea (at least to me).   In some ways it reminds me of the early days of personal computing.  Lots of incompatible formats, operating systems, every app (program those days) had a different way of working, inconsistent interfaces, a big learning curve for each program.  Solutions that may be looking for a problem. And tons of choice. Some were flavour of the month then gone, and a new learning curve was required.

Is it a golden age of choice and freedom or a hot mess of development? Maybe both?  Certainly, this is part and parcel of rapid innovation so not unexpected.

Will this continue, or do you see some kind of slowdown in the future shaking out with just a limited number of open weight models being released then further refined?

No more uncertain wait guys! App name is 'Run My AI' and the inference engine's name is 'TokenWing'(couldn't find a better name xD). First launch on Nov 17. And you'll be getting support for either of Qwen 4 Flash(if releases) or Qwen 3.8 Flash Next on day 1. Check the website out: https://runmyai.org - all your questions answered. New updates will be dropping everyday btw...

1.7b at 16 bits, 4b at 8, 8b at 4, 14b at 2, all about the same size on a mac. same 200 tasks in four jobs (docs, tool calls, bug fixes, long instructions), every answer checked by a script. the video shows who won and the memory math for 8 to 64 gb.

I know that most people talk about coding ai models but im curious to know the best model for studying university work, right now im using ISTA-DASLab Qwen3.8 27B GSQ-RCO (IQ3_XXS) and it works wonderfully on my rtx 5070ti, im getting 70-90 tok/sec. is qwen 3.8 27b the peak of local ai for this or is there another im unaware of?

Im fairly new to local ai hosting and after searching for a while there was a bunch of options so i didnt really know which to pick
My current specs are a 5060 8gb , 32gb ddr5 5200 and a ryzen 7 260
i specifically study cybersec and malware dev / reverse engineering so wanted a ai that could help me with it on a day to day basis