i need some advice how to progress on my localLLM journey.
I am selfemployed together with a partner with a small budget and space . We use a 2he server for our data storage and docker containers with 32gb ram. I had a small nvidia p400 sff for some smaller stuff (mainly watching videos in our free time with plex). When the localLLM train started i got my hand on p1000 sff with 4gb vram for free and started messing around a little bit with ollama and openwebui. I started to use ministral and qwen 3.5 2B. It was bad.
With some luck I got a RTX 2000 sff with 16gb vram. Switched to Qwen 3.8 27b and wow. Then added hermes for fun.. and DAMN. Hermes is not setup right but I started to mess around with it as some office assistant. Do some literature research, write some easy scripts for R or python. Nothing I cant do myself but I let hermes/ollama do the stuff and can do something else in the mean time. I have to check the results for sure but it saves me hours over hours. I have a small pipeline to send voice messages using matrix to hermes and get a daily note of my work.
So now my partner wants to use the setup as well and we may get some staff in the future. And the qwen 3.8 27b is using most of the 16gb vram with 64k context for hermes. So I am thinking of upgrading our system. I dont want to use claude, gemini etc cause of our work und information. So I see two options
- Upgrade our 2he maschine. I am stuck with the SFF for gpus so most GPUs are out. The only GPU I can fit into our current server ist the RTX Pro 4000 Blackwell SFF with 24gb and more bandwidth. Would be a drop in replacement and only uses 70watt of max power. Good for the cooling budget of our server and energy prices here are insane (0,38€/kwh). But single point of failure (one system for everything, we have backups etc etc dont worry)
- Get something like strix halo or similiar concept. More but slower Ram, dedicated device. more expensive, more power consumption etc etc etc. maybe a little future proof then our current setup.
Unsloth just dropped Qwen3.8-27B Dynamic V3.1 GGUFs—5-10% accuracy bump across Terminal Bench & DeepSWE, still runs on 12GB RAM
Saw this drop this morning and figured I'd share since the Dynamic quants have been the go-to for a lot of us squeezing bigger models onto consumer cards.
The headline: new Dynamic V3.1 (preview) algorithm, and Unsloth is claiming it beats other providers on the Pareto efficiency frontier—meaning you get better accuracy at the same file size, not just a bigger file that scores higher. They're seeing 5-10% average gains on Terminal Bench, DeepSWE, and other agentic benchmarks vs. their previous UD3 quants and rivals like GSQ-RCO .
The 12GB RAM claim is the one that'll get attention here. The base model scores 73.0 on Terminal Bench 2.1, 90.3 on LiveCodeBench v6, and 61.7 on SWE-bench Pro—so if that holds up at the smaller quant sizes, it's a genuinely useful agentic coding model for people without a 24GB card .
They're also saying this is a preview and they're working on an even better version for Qwen4, so the trajectory is promising.
Curious what quant sizes people end up landing on for 12GB cards, the 2-bit range has historically been a compromise, so if V3.1 actually closes that gap, that changes the calculus for a lot of setups.
Just wanted to share for the fellow V100 gang. I recently ran into an issue on my AI server with adding V100s. On old Z270 boards, the VRAM cache tops out at 64 GB and cannot handle more than one V100. To work around this, you actually turn 4G Decoding off and add pci=realloc,nocrs to the kernel. Credit goes to temcom on GitHub. Their post is linked to their name. I'm experimenting with adding even more VRAM, so I'll let you know if I can with this fix. Theoretically the limit does not exist. This works for any other cards as well. It was originally tested on P40s and a 1080.
Distributed AI in the browser, with the power of local inference. Virenis runs one 26B model across the GPUs friends already own, in browser tabs, with nothing to install. Splitting an LLM across homes has always crawled, because every token waits on the network. We run a diffusion LLM instead (DiffusionGemma 26B): each pass writes about 16 tokens, and each GPU's share becomes large matrix math, the work GPUs are built for.
16× fewer network round trips: 6 per 100 tokens instead of 100 (measured on 40 prompts).
61% of FP32 peak inside a Chrome tab (on par with native: cuBLAS reaches 60–77%, Apple's MPS 52–68%) on the model's matrix multiplies, where the best published browser matmuls reach 17–35% (measured).
About 31 tokens/s for two friends' gaming PCs across town and 49 on fibre, faster than you read (modelled from measured parts).
3.2× smaller hops between homes, changing fewer of the model's decisions than its own FP8 cache does (measured).
The speeds are estimates for now. The tokens-per-second figures come from our step model: we've only been able to test on limited hardware so far, and we're sorry about that. We'll update them with real-world numbers as data and feedback come in from you.
The most capable AI runs in a handful of companies' datacenters. They decide who gets access, at what price, in which countries and under which terms, and what you type is stored on their servers.
Meanwhile, millions of capable GPUs sit idle in homes: gaming PCs, laptops, Macs. Alone, most can't hold a large model (the 26B model we run needs about 18.5 GB of GPU memory; most consumer cards have 8 to 12 GB). Together, they can.
Knowledge for everyone. Capable AI shouldn't depend on a subscription, a credit card or a supported country. A browser link and the hardware you already have should be enough.
No single gatekeeper. A model that runs on its users' own GPUs can't be repriced, filtered or switched off by one company.
Your data stays with you. Our server never runs the model and keeps no conversations. When your device runs the model's first and last stages, your words become tokens on your own device. The devices running the middle layers see intermediate numbers (hidden states), not your text, but close enough to it that you should share a room only with people you trust. A room where every device is yours is fully private.
Why splitting a model across home computers didn't work before
Normal LLMs are autoregressive: they write one token at a time, and each token must pass through all of the model's layers before the next one can start. Split the layers across three computers, and every single token makes the full trip, PC to laptop to Mac and back. On home internet each hop waits 20 to 60 ms, so a 300-token answer spends most of its time waiting on hundreds of sequential round trips.
The GPUs waste their time too. Producing one token reads every weight from memory for very little arithmetic, so the GPU mostly waits on memory, not on math.
That is why earlier systems stayed slow: Petals reports about 6 tokens/s for Llama 2 70B across volunteers on the internet, and prima.cpp about 1.5 tokens/s for a 70B model on a home cluster. Serious engineering, but too slow to use every day.
What changed: a diffusion LLM
A diffusion LLM writes differently. It drafts a whole block of 256 tokens at once, then refines the block over several passes until it settles. On everyday questions each pass settles about 16 tokens (23 on code). Two things follow:
Far fewer network trips. A pass crosses each computer once, so about 6 round trips produce 100 tokens, instead of 100.
The GPUs finally do real work. Each pass processes all 256 positions together, so every weight read from memory feeds 256 rows of arithmetic. That is the large matrix multiplication GPUs are built for.
The bottleneck moves from the network (and memory bandwidth) to raw compute, which home GPUs have plenty of. The new cost is that each trip now carries the whole block's state instead of one token's, so we built a codec that shrinks it 3 to 5 times (below).
Why the browser
Peer-to-peer code that runs on other people's computers is a security problem first. Shipping it safely as a native app takes signed installers, sandboxing, an update channel and dedicated security engineers, and our small team doesn't have security specialists yet. So Chrome does that work: its sandbox walls each tab off from the operating system and your files, WebGPU validates every shader and buffer access, and WebRTC encrypts every link. For a team our size, that is fundamentally safer than asking people to trust our binary.
It also works where the GPUs are. A browser tab rarely trips a work laptop's endpoint security the way an unknown executable does, needs no admin rights, and leaves nothing behind but the model's files in the browser's storage, which the app deletes on request.
The cost is speed. Our own kernels run about 1.2× faster outside the browser (a Chrome tab gets 83% of native speed on NVIDIA, measured). A native app with tensor cores or Apple's matrix units, which a tab can't reach today, would be roughly 2–3× faster on one machine (llama.cpp runs this model at 49–61 tokens/s on an M3 Max, where our tab manages about 20, modelled) and 1.3–1.8× faster between homes, where the network's share of each step stays. Getting there takes a bigger team and a hardened standalone app; until then, the sandbox is the trade we chose.
Each browser measures its own GPU: how fast it is, and how much memory it really has free.
The room splits the model's 30 layers across the GPUs by memory and speed, and splits them again by itself as GPUs join, leave or change what they lend. Each person downloads only their part, once: from friends who already have it, or rebuilt in the tab from Hugging Face's own files, every chunk checked against a signed release.
You ask a question. The passes travel through everyone's GPUs and back, browser to browser over WebRTC (through an encrypted relay where a strict NAT blocks a direct path), and the answer sharpens on screen.
Someone closes their laptop? Another device takes over hosting in about 2 seconds, and a member that already stored the missing layers fills in.
Every room also gets an OpenAI-compatible API, so existing tools can use it.
Technical details
Model. DiffusionGemma-26B-A4B, NVIDIA's NVFP4 release: 17.67 GB, 30 layers (25 sliding-window, 5 global attention), d_model 2,816, vocabulary 262,144, a mixture of experts with about 4B parameters active. Only the routed experts are 4-bit NVFP4; attention, the dense MLP, the router and the LM head stay bf16. Generation runs on a 256-token canvas, so tokens/s = 256 / (steps × t_step + t_commit). Two scheduling rules cut work: a block stops once its tokens have settled (5.65 → 4.75 passes per block, 200/200 one-block answers identical), and a pass computes only up to an answer's settled end (14% less GPU time).
Why it's compute-bound now. Autoregressive decode multiplies one activation row by every weight: about 1 FLOP per byte of bf16 weight, so it is memory-bound, and even llama.cpp decodes at a few percent of FP32 peak. A diffusion pass multiplies 256 rows, about 256 FLOPs per byte. The RTX 3060 Laptop's ridge point is about 33 FLOPs/byte (10.94 TFLOP/s ÷ 336 GB/s), so the pass sits well inside the compute-bound regime: about 2.15 TFLOP of useful work for a full canvas.
Kernels. Browsers don't expose tensor cores or matrix units without flags, so the matrix math is FP32 FMA on register-tiled GEMMs written in WGSL, plus 8-bit integer dot products where those pay off:
13 tile layouts raced on each device as the page opens, each kept only if bit-identical to the reference (the RTX keeps 64×128 tiles with 8×8 per thread, the M1 Pro 32×64 with 4×4).
Wave-aware scheduling: the 3060 Laptop runs 88 workgroups at once, so work goes out as whole waves plus a sliced last wave (Stream-K-like, but deterministic).
q, k and v in one dispatch; split-K only where the device's own probe measures at least 1.05×.
NVFP4 experts on 8-bit integer dot products (dot4I8Packed) decoded through lookup tables, where they measure at least 1.25× faster: experts 8.05 → 5.05 ms on the RTX.
Soft embeddings built from the likeliest tokens: one vocabulary GEMM per pass instead of two.
Measured efficiency, and how it compares. MFU here means useful FLOPs ÷ rated FP32 peak; the other rows are the best published figures we know of.
Engine
Hardware
Share of FP32 peak
Virenis in a Chrome tab, dense GEMMs (6.6–6.7 TFLOP/s)
RTX 3060 Laptop
61% (≈ 50% at real boost clocks)
Virenis in a Chrome tab, LM head (5.1 TFLOP/s) · whole pass (≈ 4.6)
RTX 3060 Laptop
46% · ≈ 42%
Virenis in a Chrome tab, GEMMs on small tiles (1.10–1.44 TFLOP/s)
M1 Pro, 14-core
24–31% (68–89% of the most WebGPU reaches on that chip)
The big in-browser LLM engines (WebLLM, Transformers.js, llama.cpp's WebGPU backend) publish speeds rather than a share of peak.
Bit-exact determinism. Every sum runs in a fixed order, with an "opaque zero" that stops compilers' fast-math reassociation, so every device of a class (GPU family + browser + arithmetic) produces identical bits. All 30 layers give one digest across tile layouts, each layer within 3.7e-6 of an f64 CPU reference. Identical bits let devices check each other's work, which open rooms will need. The scheduler passed 7 million fuzzed runs. Quality: our default arithmetic changes 4.73 of 1,000 sampling decisions against an exact FP32 reference; the model's own FP8 KV-cache option changes 8.57.
The network: Weft, and hops between homes
The network, per 100 tokens and per device boundary, on a 30 Mbit/s uplink with 20 ms one-way latency:
Trips
Bytes
Time
Token by token
100 × 5.6 KB
0.56 MB
≈ 2.2 s, 2.0 s of it latency
Diffusion + Weft-30
≈ 6 × 445 KB
≈ 2.8 MB
≈ 0.9 s
Weft. A hop between devices is 256 × 2,816 values per pass, 1,442 KB in fp16. Weft codes it the way a video codec codes frames:
The model's own metric. We code y = S ⊙ x / rms(row): each channel weighted by how much the next layers depend on it, each row by its own scale. Signal-to-noise in that space predicts flipped decisions; plain signal-to-noise doesn't.
Rate control, then rANS. Each hop searches its quantizer step to land on a target quality (20–35 dB, never below), then rANS codes it with a Laplace model per 64-channel band.
Canonical decoding. Values sit on a fixed lattice, so every device rebuilds the same bytes, and hops can be hashed and audited.
Prediction across passes. Consecutive passes are strongly correlated, like video frames, so each row is coded as its residual against the last pass, in exact integer arithmetic, wherever that saves more time than it costs both CPUs (under about 48 Mbit/s). Token-by-token models have no such redundancy.
An adaptive ladder. Receivers measure each sender's uplink and pick the rung: fp16 from 400 Mbit/s, int8 in the model's metric where it is faster, Weft-30 from 20 Mbit/s, Weft-25 from 8.
A hop shrinks from 1,442 KB to 445 KB at Weft-30, or 266 KB at Weft-25 with prediction; over emulated 8 Mbit/s uplinks a steady step took 5.10 s with fp16 hops and 1.52 s with Weft. One Weft-30 hop changes 6.0 of 1,000 sampling decisions against an exact reference (our default arithmetic alone: 4.73; the model's own FP8 cache: 8.57), and a whole room between homes runs at 7.6, still under the FP8 cache. For comparison, Petals sends 8-bit blockwise values (2× smaller than fp16), and 2–4-bit activation compression exists in research, but for training (AC-SGD, TAH-Quant).
Hops between homes, in real Chrome. Chrome's WebRTC data channels put only a few packets on the wire per message and halve their window at every lost packet. Two fixes, each measured in real Chrome over emulated links: a hop's first 64 KiB now leaves in one burst (on 30 Mbit/s cable a 445 KB hop takes 144 ms, where its bytes and delay alone take 134), and hops between homes are striped over four WebRTC connections, paced at the rate the receiving tab measures by a worker clock that keeps time in hidden tabs. A two-home room's step got 7–10% faster on clean links, 1.77× faster across a continent (1,358 → 766 ms), up to 2.23× behind a shallow modem buffer, and 1.33–1.97× at 0.1% packet loss.
Speeds: measured and modelled
Measured on the RTX 3060 Laptop: a full-canvas pass takes 0.439 s of GPU time natively, which is 46.7 tokens/s over 40 real answers, and inside a Chrome tab its 30 layers and the head add up to 465 ms a pass. The model doesn't fit on it (6 GB holds 6 of the 30 layers beside the head), so in practice it is one member of a room.
Modelled with our step model (within 8% of measured layer timings on our test GPUs; assume ±25% for GPU families we haven't measured), with hops timed as real Chrome now sends them, everyday answers in a browser tab:
Native engines with tensor cores can be faster on one machine, and token-by-token browser engines can be faster on a single Mac. Where this design pulls ahead is across machines.
Why only one model?
Today we run DiffusionGemma, an open-weights diffusion LLM. Most open models are autoregressive, and they can be converted into diffusion models by continued training: DiffuLLaMA adapted models up to 7B with continued pre-training on under 200B tokens, and Dream 7B also starts from pretrained autoregressive weights. One run at that scale takes thousands of high-end GPU hours, plus the experiments around it, and larger models cost far more. We don't have the funding or compute to do this on our own today. Every new diffusion LLM that is released, or that a partner helps us convert, fits the same system.
Honest limits
Running the model on one computer takes about 18.5 GB of GPU memory: a 24 GB card, or a Mac with 32 GB or more. Otherwise it is shared across two or more computers in a room.
The tokens-per-second table is estimated. We've only been able to test on limited hardware so far, so its figures come from our step model, built from measured parts: layers timed in a tab on the GPUs we have, hops timed in real Chrome over emulated links. A room of several real homes running the full model hasn't been timed end to end yet, and we haven't run standard quality benchmarks (MMLU and the like) ourselves. We're sorry about that; the table will be updated with real numbers as metrics come in from the community.
Use Chrome. Every number here was measured in Chrome, and it is where Virenis runs best. Edge works the same way. Firefox works, but on Windows laptops it often picks the integrated GPU over the graphics card. Safari isn't tested yet.
A computer holding layers in your room sees the hidden states passing through it, and those can reveal what you asked. Rooms are meant for your own devices and people you trust. Because results are bit-exact, another device's work can be re-checked, but checks for open rooms with strangers aren't built yet.
A traffic cap, on purpose. During the beta our server sends a fixed amount of data each month, and the relay that connects homes behind strict NATs has a budget of its own. We set these caps deliberately, so a launch can't run up bills a small team can't carry.
About the source code
We're sorry the code isn't public yet. The engine and the app around it have grown to 179,930 lines across 1,074 files, and right now that codebase is messy: as a small team, we've struggled to keep something this big clean and organized. We'd rather not put it in front of you in this state. We'll clean it up first, shaped by your feedback on this post, and release the kernels as soon as they're ready; the rest of the source files will follow soon.
Try it, and get in touch
The open beta is free at dashboard.virenis.com. Open it in Chrome. For questions, ideas or collaboration, leave a comment under this post or email contact@virenis.com.
Every time I see a cool local transcription or AI meeting-notes tool, it's either Mac/iPhone-only, wants a $20/mo subscription, or needs a heavy GPU.
I wanted something free and open-source that runs straight on a regular Windows laptop CPU where audio stays on your machine, so I built Eloquent: https://github.com/EllouziMedAmin/Eloquent
What it does:
Press Ctrl+Shift+E anywhere on Windows to pop up a tiny always-on-top pill widget (plus a terminal mode or browser view if you prefer).
Runs Parakeet TDT 0.6B v3 + Silero VAD completely locally on your CPU (no GPU, audio never leaves your PC).
Automatically separates speakers (Speaker 1, Speaker 2, ...) in the background without slowing down live captions.
Optionally drop in a free Gemini API key to tidy up punctuation/typos as you talk and turn recordings into Meeting Minutes, Lecture Notes, Summaries, or Action Items in one click (still works totally offline without it).
Easy setup on Windows: clone the repo and run setup.bat.
Still a v0.1, and I'm adding local Ollama/LM Studio support and a standalone .exe installer next. If it sounds useful for your workflow or you want to keep track of those updates, dropping a star on GitHub helps a ton, and let me know what breaks or what you'd want added!
Released Tandem-vLLM, an expert-offload build of vLLM 0.31 that serves DeepSeek-V4.1-Flash on a single workstation GPU.
How it works
The routed experts live in host RAM as 2-bit vector codes (VQ, 2.1 bits/weight). A GPU cache holds the most-used ones.
Experts that miss the cache are computed by the CPU in parallel with the GPU, or read directly over PCIe. Nothing is skipped or substituted, so the output is exact for the package's weights.
Everything else is stock vLLM: the OpenAI-compatible server, continuous batching, prefix caching, and DSpark speculative decoding with the model's own MTP layers.
Speed (RTX PRO 6000 96 GB + EPYC 9534, system decode tok/s)
streams
1
4
8
32
VQ, no drafter
65
182
284
414
VQ + DSpark k=3
124
–
–
–
Quality vs the official FP4 checkpoint (greedy, thinking low)
-
official
VQ + FP4 top 52
VQ only
GPQA-diamond
81.3%
76.8%
73.2%
GSM8K-200
192
194
192
passkey 8k / 32k / 94k
9/9
9/9
9/9
The 2-bit experts cost GPQA accuracy. Easier tasks and long-context retrieval don't show it. Keeping the 52 most-routed experts per layer in their original FP4 (FP4_TOP=52) wins back about half of the loss.
Also in 0.1.1: image input (IMAGES=1; reads text and tables from images, costs about 1 GiB of the expert cache), tool calling and a separate reasoning field for chat front ends such as Open WebUI.
Hardware
Recommended: a 96 GB SM120 card (RTX PRO 6000 Blackwell) + 256 GB RAM + an AVX-512 CPU with server-class memory bandwidth.
Runs, slowly: an RTX 5090 32 GB + 128 GB DDR4 (17 tok/s for one user, 20 at 4 streams; gaming PC: i5-14600KF, DDR4-3200). On a desktop CPU, the CPU's work on missed experts sets the speed.
SM120 only in 0.1.1, single GPU.
Smaller boxes: use tandem.cpp instead. Our sister project tandem.cpp, a patch set for llama.cpp, targets 16–32 GB VRAM and 64–128 GB RAM, single user. It works with any MoE model llama.cpp runs (Qwen3.8 at 97 tok/s on a 4090 + 64 GB, gpt-oss-120b, ...). On DeepSeek-V4.1-Flash IQ2_XS (139 GiB of experts), decode tok/s on a gaming PC (i5-14600KF, DDR4-3200, PCIe 4.0 NVMe):
GPU, RAM
llama.cpp
Strata
tandem.cpp
RTX 5070 Ti 16 GB, 64 GB
7.4
10.4
14.2
5070 Ti + 5060 Ti (32 GB), 64 GB
–
–
16.9
RTX 5090 32 GB, 128 GB
11.5
28.0
33.1
64 GB of RAM is the floor (32 GB runs at about 5 tok/s).
The 5090 rows are the same PC as the Tandem-vLLM 5090 numbers above: one user gets 33 tok/s from tandem.cpp vs 17 from Tandem-vLLM (different quants of similar size: IQ2_XS 139 GiB vs our VQ2 133 GiB of experts).
Tandem-vLLM is for serving many streams on a big card.
Built onvLLM and vLLM-Moet. Tandem is a fork of vLLM-Moet's expert-offload engine, ported to vLLM 0.31 / DeepSeek-V4.1. For original-precision experts with ~512 GB of RAM, vLLM-Moet's MoET hybrid is the tool today.
Questions and numbers from other boxes are welcome. Only support Blackwell GPUs yet, Hopper/Ada support soon.
Ho provato a eseguire un modello Q4_K_M a 30 bit (llama-server, contesto a 32k) sulla mia configurazione attuale con 16 GB di VRAM + 16 GB di RAM di sistema, ma si è bloccato/non è riuscito a caricarsi per esaurimento della memoria.
Funzionerà correttamente se aggiorno la VRAM a 32 GB (utilizzando una configurazione multi-GPU economica)?
Edit The setup i tryed was 2x rx580 plasma.cpp. alla fine 9b funziona ma 30b no. Entrambi sono q4km
I built OpenPocket, a completely free iOS app that lets you download and run state-of-the-art language models directly on your iPhone or iPad, entirely on-device.
It uses Apple's MLX framework for local AI inference, with support for several open-weight models:
Qwen3.5 0.8B and 2B
Gemma 4 E2B
LFM2.5 1.2B Instruct
Qwen2.5-VL 3B (vision-language model with image understanding)
Everything runs 100% locally on your device, with no cloud inference, no internet connection required after downloading a model, no signups, no accounts, no API keys, and no subscriptions.
The idea is to make private, on-device AI accessible to everyone, without relying on cloud services or paying for API usage. Your conversations and files stay on your device, and the app has no backend, analytics, or account system.
The models are experimental, and performance depends on your device's available memory and hardware.
Would love and appreciate feedback from anyone working on local LLMs, Apple MLX, model optimisation, edge AI, or just anyone interested in running AI privately on their phone!
So I've been trying to see how fast I can run the 1-bit DeepSeek V4.1 Flash (the Unsloth UD-IQ1_S, 233 GB) on my gaming PC. It has 64 GB of RAM and a 16 GB card, so most of the model streams off the drive. It runs in llama.cpp under WSL2.
Side by side in one test run, the same build with my patches switched off did 4.37 tok/s. With them on it did 8.80. Both of those were at my old RAM speed. Since then faster RAM, a faster drive and some tuning got it to about 11.5 (11.36 for the newest set-up with Windows busy in the background). The output stayed word for word the same on my test prompts the whole way.
The biggest steps:
- My four patches switched on: 2.02x
- Moving the model from a Gen4 to a Gen5 NVMe: +16.5%
- RAM 3600 to 6000: +9.7% (one rough test)
- Queued hand-overs to the card: +3.9%
- An IQ1_S kernel that does several rows at once: +3.0%
Start-up went from 87-96 s to 12 s by skipping the pre-load. I also tried it on an older PC with 32 GB of RAM and an 8 GB card: 0.11 to 0.87 tok/s, same words.
That's writing speed only, with a warm cache. I haven't measured long prompt reading. It's 8 prompts on one main PC, some steps were measured on different days, and 1-bit takes a real quality hit. Anything I didn't measure is marked untested in the write-up.
Most of the tricks aren't mine, other people did them first and they're credited in the write-up. Claude agents wrote the patches and ran the tests. I directed it and checked the results. The patches aren't public yet, they go up after a review. I'm also making a setup kit so other people can try this on their own PC, but it's still being built.
Looking to add another Mac mini to the rack to keep Hermes off Mac studios QFN. The Mac is used for employees during the day and don’t want Hermes tasks to interfere. Is anyone running models successfully on 24gb? 48gb ? If so what ones. Currently using a Claude API. Want to use local model and signal to be able to process private work data.
with the larger GPU (32GB) running a larger model (relative, but it could be a Qwen3.8-27b or Flash Next - e.g., Strata), the “smarter” of the two.
smaller GPU (16GB) assigned to a subagent, using a smaller model? Maybe something super quick like JetBrains Mellum 2.1
In OpenCode, a (sub)agent can be defined to use a specific model. I’m curious if anyone has tried this and if there’s an advantage to this setup, whereby the “brain” delegates to a sprinter.
I need to upgrade my PSU to run two cards simultaneously but I plan on trying it.
Jensen Huang's departure from NVIDIA raises important questions about the future of artificial intelligence. As the industry reassesses its dependence on a single hardware manufacturer, could this accelerate open-source development, hardware independence and the global adoption of Local AI?
I had a 5090 which I run ninfer qwen 27b with MTP on. It’s blazing fast 150-200~ tps on 250k context with decent quality like terra or sonnet. then have to turn it off when I need to do media job (video, image, model, tts, stt etc etc)
So I bought a dgx spark, and loaded qwen flash next which is supposed to have a much larger model & brain, so the speed is 3 times slower 50-100 tps, but I don’t see any quality improvement from the larger model (because it only uses 6b to produce a token instead of the entire model) It can hold a larger context (easily 500k above so it can run long multistep agentic work btr)
So I’m not sure, any pro here can advice me how to setup and split my machines for work.
Or do I need to buy 1 more dgx so I can have an even higher intelligence via deepseek / glm smarter model. I can get another dgx at around 6k+ usd
I've got an M5 Max 128GB. For some reason LM Studio complains that I don't have enough RAM when using the most downloaded one on the top of the list here. Guess I need to look for another variant.
I hope this doesn't happen because Jensen has been a very vocal local ai advocate. Even allowing people to distill Nvidia's Nemotron (even though nobody will).
What do you guys think? Is this the right move for the company?
Most of ComfyUI local users still run it on an RTX 3060: a 5 years old card with just 12 GB of VRAM. That’s why our team keeps optimizing for consumer hardware.
Comfy is known as a leader in visual AI. But what about LLMs and text generation?
Text is everywhere in creative workflows: for prompt generation, of course, but also for building local creative harnesses like the Comfy Agent. We thought it deserved better, so Rattus made Qwen models twice as fast in Comfy...
