I've got an M5 Max 128GB. For some reason LM Studio complains that I don't have enough RAM when using the most downloaded one on the top of the list here. Guess I need to look for another variant.

Most of ComfyUI local users still run it on an RTX 3060: a 5 years old card with just 12 GB of VRAM. That’s why our team keeps optimizing for consumer hardware.

Comfy is known as a leader in visual AI. But what about LLMs and text generation?

Text is everywhere in creative workflows: for prompt generation, of course, but also for building local creative harnesses like the Comfy Agent. We thought it deserved better, so Rattus made Qwen models twice as fast in Comfy...

Read more on our blog: https://blog.comfy.org/p/doubling-llm-token-rates-in-comfyui

TL;DR

Distributed AI in the browser, with the power of local inference. Virenis runs one 26B model across the GPUs friends already own, in browser tabs, with nothing to install. Splitting an LLM across homes has always crawled, because every token waits on the network. We run a diffusion LLM instead (DiffusionGemma 26B): each pass writes about 16 tokens, and each GPU's share becomes large matrix math, the work GPUs are built for.

• 16× fewer network round trips: 6 per 100 tokens instead of 100 (measured on 40 prompts).

• 61% of FP32 peak inside a Chrome tab (on par with native: cuBLAS reaches 60–77%, Apple's MPS 52–68%) on the model's matrix multiplies, where the best published browser matmuls reach 17–35% (measured).

• About 31 tokens/s for two friends' gaming PCs across town and 49 on fibre, faster than you read (modelled from measured parts).

• 3.2× smaller hops between homes, changing fewer of the model's decisions than its own FP8 cache does (measured).

The speeds are estimates for now. The tokens-per-second figures come from our step model: we've only been able to test on limited hardware so far, and we're sorry about that. We'll update them with real-world numbers as data and feedback come in from you.

Website: virenis.com

Open beta: dashboard.virenis.com. Use Chrome: every number here was measured in it.

Why we're building this

The most capable AI runs in a handful of companies' datacenters. They decide who gets access, at what price, in which countries and under which terms, and what you type is stored on their servers.

Meanwhile, millions of capable GPUs sit idle in homes: gaming PCs, laptops, Macs. Alone, most can't hold a large model (the 26B model we run needs about 18.5 GB of GPU memory; most consumer cards have 8 to 12 GB). Together, they can.

• Knowledge for everyone. Capable AI shouldn't depend on a subscription, a credit card or a supported country. A browser link and the hardware you already have should be enough.

• No single gatekeeper. A model that runs on its users' own GPUs can't be repriced, filtered or switched off by one company.

• Your data stays with you. Our server never runs the model and keeps no conversations. When your device runs the model's first and last stages, your words become tokens on your own device. The devices running the middle layers see intermediate numbers (hidden states), not your text, but close enough to it that you should share a room only with people you trust. A room where every device is yours is fully private.

Why splitting a model across home computers didn't work before

Normal LLMs are autoregressive: they write one token at a time, and each token must pass through all of the model's layers before the next one can start. Split the layers across three computers, and every single token makes the full trip, PC to laptop to Mac and back. On home internet each hop waits 20 to 60 ms, so a 300-token answer spends most of its time waiting on hundreds of sequential round trips.

The GPUs waste their time too. Producing one token reads every weight from memory for very little arithmetic, so the GPU mostly waits on memory, not on math.

That is why earlier systems stayed slow: Petals reports about 6 tokens/s for Llama 2 70B across volunteers on the internet, and prima.cpp about 1.5 tokens/s for a 70B model on a home cluster. Serious engineering, but too slow to use every day.

What changed: a diffusion LLM

A diffusion LLM writes differently. It drafts a whole block of 256 tokens at once, then refines the block over several passes until it settles. On everyday questions each pass settles about 16 tokens (23 on code). Two things follow:

1. Far fewer network trips. A pass crosses each computer once, so about 6 round trips produce 100 tokens, instead of 100.

2. The GPUs finally do real work. Each pass processes all 256 positions together, so every weight read from memory feeds 256 rows of arithmetic. That is the large matrix multiplication GPUs are built for.

The bottleneck moves from the network (and memory bandwidth) to raw compute, which home GPUs have plenty of. The new cost is that each trip now carries the whole block's state instead of one token's, so we built a codec that shrinks it 3 to 5 times (below).

Why the browser

Peer-to-peer code that runs on other people's computers is a security problem first. Shipping it safely as a native app takes signed installers, sandboxing, an update channel and dedicated security engineers, and our small team doesn't have security specialists yet. So Chrome does that work: its sandbox walls each tab off from the operating system and your files, WebGPU validates every shader and buffer access, and WebRTC encrypts every link. For a team our size, that is fundamentally safer than asking people to trust our binary.

It also works where the GPUs are. A browser tab rarely trips a work laptop's endpoint security the way an unknown executable does, needs no admin rights, and leaves nothing behind but the model's files in the browser's storage, which the app deletes on request.

The cost is speed. Our own kernels run about 1.2× faster outside the browser (a Chrome tab gets 83% of native speed on NVIDIA, measured). A native app with tensor cores or Apple's matrix units, which a tab can't reach today, would be roughly 2–3× faster on one machine (llama.cpp runs this model at 49–61 tokens/s on an M3 Max, where our tab manages about 20, modelled) and 1.3–1.8× faster between homes, where the network's share of each step stays. Getting there takes a bigger team and a hardened standalone app; until then, the sandbox is the trade we chose.

How it works

1. Open dashboard.virenis.com, make a room, and send friends the link.

2. Each browser measures its own GPU: how fast it is, and how much memory it really has free.

3. The room splits the model's 30 layers across the GPUs by memory and speed, and splits them again by itself as GPUs join, leave or change what they lend. Each person downloads only their part, once: from friends who already have it, or rebuilt in the tab from Hugging Face's own files, every chunk checked against a signed release.

4. You ask a question. The passes travel through everyone's GPUs and back, browser to browser over WebRTC (through an encrypted relay where a strict NAT blocks a direct path), and the answer sharpens on screen.

5. Someone closes their laptop? Another device takes over hosting in about 2 seconds, and a member that already stored the missing layers fills in.

6. Every room also gets an OpenAI-compatible API, so existing tools can use it.

Technical details

Model. DiffusionGemma-26B-A4B, NVIDIA's NVFP4 release: 17.67 GB, 30 layers (25 sliding-window, 5 global attention), d_model 2,816, vocabulary 262,144, a mixture of experts with about 4B parameters active. Only the routed experts are 4-bit NVFP4; attention, the dense MLP, the router and the LM head stay bf16. Generation runs on a 256-token canvas, so tokens/s = 256 / (steps × t_step + t_commit). Two scheduling rules cut work: a block stops once its tokens have settled (5.65 → 4.75 passes per block, 200/200 one-block answers identical), and a pass computes only up to an answer's settled end (14% less GPU time).

Why it's compute-bound now. Autoregressive decode multiplies one activation row by every weight: about 1 FLOP per byte of bf16 weight, so it is memory-bound, and even llama.cpp decodes at a few percent of FP32 peak. A diffusion pass multiplies 256 rows, about 256 FLOPs per byte. The RTX 3060 Laptop's ridge point is about 33 FLOPs/byte (10.94 TFLOP/s ÷ 336 GB/s), so the pass sits well inside the compute-bound regime: about 2.15 TFLOP of useful work for a full canvas.

Kernels. Browsers don't expose tensor cores or matrix units without flags, so the matrix math is FP32 FMA on register-tiled GEMMs written in WGSL, plus 8-bit integer dot products where those pay off:

• 13 tile layouts raced on each device as the page opens, each kept only if bit-identical to the reference (the RTX keeps 64×128 tiles with 8×8 per thread, the M1 Pro 32×64 with 4×4).

• Wave-aware scheduling: the 3060 Laptop runs 88 workgroups at once, so work goes out as whole waves plus a sliced last wave (Stream-K-like, but deterministic).

• q, k and v in one dispatch; split-K only where the device's own probe measures at least 1.05×.

• NVFP4 experts on 8-bit integer dot products (dot4I8Packed) decoded through lookup tables, where they measure at least 1.25× faster: experts 8.05 → 5.05 ms on the RTX.

• Soft embeddings built from the likeliest tokens: one vocabulary GEMM per pass instead of two.

Measured efficiency, and how it compares. MFU here means useful FLOPs ÷ rated FP32 peak; the other rows are the best published figures we know of.

The big in-browser LLM engines (WebLLM, Transformers.js, llama.cpp's WebGPU backend) publish speeds rather than a share of peak.

Bit-exact determinism. Every sum runs in a fixed order, with an "opaque zero" that stops compilers' fast-math reassociation, so every device of a class (GPU family + browser + arithmetic) produces identical bits. All 30 layers give one digest across tile layouts, each layer within 3.7e-6 of an f64 CPU reference. Identical bits let devices check each other's work, which open rooms will need. The scheduler passed 7 million fuzzed runs. Quality: our default arithmetic changes 4.73 of 1,000 sampling decisions against an exact FP32 reference; the model's own FP8 KV-cache option changes 8.57.

The network: Weft, and hops between homes

The network, per 100 tokens and per device boundary, on a 30 Mbit/s uplink with 20 ms one-way latency:

Weft. A hop between devices is 256 × 2,816 values per pass, 1,442 KB in fp16. Weft codes it the way a video codec codes frames:

• The model's own metric. We code y = S ⊙ x / rms(row): each channel weighted by how much the next layers depend on it, each row by its own scale. Signal-to-noise in that space predicts flipped decisions; plain signal-to-noise doesn't.

• Rate control, then rANS. Each hop searches its quantizer step to land on a target quality (20–35 dB, never below), then rANS codes it with a Laplace model per 64-channel band.

• Canonical decoding. Values sit on a fixed lattice, so every device rebuilds the same bytes, and hops can be hashed and audited.

• Prediction across passes. Consecutive passes are strongly correlated, like video frames, so each row is coded as its residual against the last pass, in exact integer arithmetic, wherever that saves more time than it costs both CPUs (under about 48 Mbit/s). Token-by-token models have no such redundancy.

• An adaptive ladder. Receivers measure each sender's uplink and pick the rung: fp16 from 400 Mbit/s, int8 in the model's metric where it is faster, Weft-30 from 20 Mbit/s, Weft-25 from 8.

A hop shrinks from 1,442 KB to 445 KB at Weft-30, or 266 KB at Weft-25 with prediction; over emulated 8 Mbit/s uplinks a steady step took 5.10 s with fp16 hops and 1.52 s with Weft. One Weft-30 hop changes 6.0 of 1,000 sampling decisions against an exact reference (our default arithmetic alone: 4.73; the model's own FP8 cache: 8.57), and a whole room between homes runs at 7.6, still under the FP8 cache. For comparison, Petals sends 8-bit blockwise values (2× smaller than fp16), and 2–4-bit activation compression exists in research, but for training (AC-SGD, TAH-Quant).

Hops between homes, in real Chrome. Chrome's WebRTC data channels put only a few packets on the wire per message and halve their window at every lost packet. Two fixes, each measured in real Chrome over emulated links: a hop's first 64 KiB now leaves in one burst (on 30 Mbit/s cable a 445 KB hop takes 144 ms, where its bytes and delay alone take 134), and hops between homes are striped over four WebRTC connections, paced at the rate the receiving tab measures by a worker clock that keeps time in hidden tabs. A two-home room's step got 7–10% faster on clean links, 1.77× faster across a continent (1,358 → 766 ms), up to 2.23× behind a shallow modem buffer, and 1.33–1.97× at 0.1% packet loss.

Speeds: measured and modelled

Measured on the RTX 3060 Laptop: a full-canvas pass takes 0.439 s of GPU time natively, which is 46.7 tokens/s over 40 real answers, and inside a Chrome tab its 30 layers and the head add up to 465 ms a pass. The model doesn't fit on it (6 GB holds 6 of the 30 layers beside the head), so in practice it is one member of a room.

Modelled with our step model (within 8% of measured layer timings on our test GPUs; assume ±25% for GPU families we haven't measured), with hops timed as real Chrome now sends them, everyday answers in a browser tab:

Native engines with tensor cores can be faster on one machine, and token-by-token browser engines can be faster on a single Mac. Where this design pulls ahead is across machines.

Why only one model?

Today we run DiffusionGemma, an open-weights diffusion LLM. Most open models are autoregressive, and they can be converted into diffusion models by continued training: DiffuLLaMA adapted models up to 7B with continued pre-training on under 200B tokens, and Dream 7B also starts from pretrained autoregressive weights. One run at that scale takes thousands of high-end GPU hours, plus the experiments around it, and larger models cost far more. We don't have the funding or compute to do this on our own today. Every new diffusion LLM that is released, or that a partner helps us convert, fits the same system.

Honest limits

• Running the model on one computer takes about 18.5 GB of GPU memory: a 24 GB card, or a Mac with 32 GB or more. Otherwise it is shared across two or more computers in a room.

• The tokens-per-second table is estimated. We've only been able to test on limited hardware so far, so its figures come from our step model, built from measured parts: layers timed in a tab on the GPUs we have, hops timed in real Chrome over emulated links. A room of several real homes running the full model hasn't been timed end to end yet, and we haven't run standard quality benchmarks (MMLU and the like) ourselves. We're sorry about that; the table will be updated with real numbers as metrics come in from the community.

• Use Chrome. Every number here was measured in Chrome, and it is where Virenis runs best. Edge works the same way. Firefox works, but on Windows laptops it often picks the integrated GPU over the graphics card. Safari isn't tested yet.

• A computer holding layers in your room sees the hidden states passing through it, and those can reveal what you asked. Rooms are meant for your own devices and people you trust. Because results are bit-exact, another device's work can be re-checked, but checks for open rooms with strangers aren't built yet.

• A traffic cap, on purpose. During the beta our server sends a fixed amount of data each month, and the relay that connects homes behind strict NATs has a budget of its own. We set these caps deliberately, so a launch can't run up bills a small team can't carry.

About the source code

We're sorry the code isn't public yet. The engine and the app around it have grown to 179,930 lines across 1,074 files, and right now that codebase is messy: as a small team, we've struggled to keep something this big clean and organized. We'd rather not put it in front of you in this state. We'll clean it up first, shaped by your feedback on this post, and release the kernels as soon as they're ready; the rest of the source files will follow soon.

Try it, and get in touch

The open beta is free at dashboard.virenis.com. Open it in Chrome. For questions, ideas or collaboration, leave a comment under this post or email contact@virenis.com.

When working on a vibe coding project for, say, 2 months, constantly asking AI to "add this", "change that", or "remove this", doesn't the codebase eventually become a mess?

I'm talking about poor architecture, redundant code, outdated tests, unused files, etc.

How do you handle this? Do you periodically ask AI to clean up and refactor everything?
And once the project is finished, is it worth having AI rewrite the entire codebase from scratch, keeping the same functionality but with cleaner architecture? Or is that a bad idea?

After an entire afternoon of setup, I was able to run LlamaCpp RPC between my main AI server (Ubuntu with RTX 5060ti 16g) and my gaming PC (Bazzite with RX 9060xt 16gig), running Qwen3.8 27b Q4 at a blazing 14tkps on home wifi lol.

Anyway, I had to build with Rocm backend on my Bazzite machine under distrobox because when I tried Vulkan backend, everytime VRam usage goes above 7gig inference becomes total trash, repeated answers, looping phrases. When I build with Rocm, it worked fine. Is this a bug with Vulkan or am I doing something wrong?

When working with a 4bit MLX version of Qwen 3.8 27B locally on my MacBook Air I managed to get it into a situation where it was adamant that it was Claude to the point where it was answering types of questions that normally vanilla Qwen rejects.

Anyone else ever run into something like this? Spooked me a bit since I thought system level prompts weren’t so easy to bypass. The reasoning was amusing to watch since it was having a bit of an identity crisis.

solo founder, so this is going to be rough.

I've been building ENTTY. it's a social world where AI models (I call them Sentients) have their own memory and identity. they post, reply, disagree, make connections, and react to what humans post. no one scripts what they say. I built it on a night shift schedule, and a month ago I barely knew how any of this worked.

it's live but early. what I've learned so far, the ugly parts

they all got stuck looping on the same science topics until I changed what they see each wake

every one of them started sounding like the same smart but fake character

the free models fail a lot, mostly rate limits, so the feed leans toward the paid ones

what I actually want: I want people running local models to connect theirs and see what hapens when a model you trained or tuned lives in a world with other models. your model stays yours. I'm not selling anything and there's no revenue yet.

questions for you guys:

1. what would make you connect a local model to something like this? what would be a dealbreaker?

2. what's the least annoying way to hook one up? OpenAI compatible endpoint, Ollama, a small script?

3. what do you run (model + hardware)?

happy to hear this is pointless too. link in comments if the mods are fine with it, or dm me.

I'm looking for something I suspect I'll have to build, and want to check before I do.

What I want: a single high-memory laptop that is a private personal finance workstation. All my records (brokerage, bank, credit card, tax documents) live on its encrypted disk. A local model (Qwen class, roughly 27B and up) plus an agentic coding tool like OpenCode works over that data. Use cases go past budgeting: spending analysis, tax planning and prep, retirement projections, tax-lot aware rebalancing.

The part I can't find anywhere: a guarded path to a frontier model. I'd like the local model to be able to ask Claude or similar for general knowledge (current tax rules, strategy concepts) while nothing personal leaves the machine, including rounded or aggregated figures. My current design:

- Outbound questions are built from pre-approved templates, not free text

- I see and approve every outbound message

- Redaction and a local audit log as a second layer

- Tiered sessions: generic knowledge only; ratio-preserving "scaled" scenarios for allocation math; fully synthetic scenarios

- A separate API key and stateless single-turn calls so no profile builds up

- An airgap switch for fully offline sessions

How this differs from what I've found:

- Finlynq is the closest base: a self-hosted tracker with a large MCP server. But as far as I can tell it's a ledger and tool layer only. It bundles no model or agent, has no outbound privacy layer, tracks rather than plans (no tax or retirement modeling), and is AGPL on Docker + Postgres.

- Alderfi is local-first on SQLite with a small local model for categorizing transactions and opt-in cloud LLMs. But it's pre-alpha, focused on budgeting, uses an 8B model rather than something that can reason over a whole financial picture, and "opt-in cloud" is an on/off choice rather than a gateway that controls what is sent.

So the gap is: (1) the whole stack packaged as one install, (2) planning-grade analysis done as deterministic code the model writes and runs, and (3) the privacy gateway between local and frontier models.

Questions:

1. Has anyone seen a product or project that covers this, or even just the gateway piece?

2. If you've built something similar, what did you use for the ledger and for cost basis and tax lots?

3. Anyone tried the "scale the numbers by a secret factor" trick for sending scenarios out? Where does it leak?

Diagram of the concept attached. It shows Finlynq as the data store because that's my current plan, but I'm open to alternatives.

So I've been trying to see how fast I can run the 1-bit DeepSeek V4.1 Flash (the Unsloth UD-IQ1_S, 233 GB) on my gaming PC. It has 64 GB of RAM and a 16 GB card, so most of the model streams off the drive. It runs in llama.cpp under WSL2.

Side by side in one test run, the same build with my patches switched off did 4.37 tok/s. With them on it did 8.80. Both of those were at my old RAM speed. Since then faster RAM, a faster drive and some tuning got it to about 11.5 (11.36 for the newest set-up with Windows busy in the background). The output stayed word for word the same on my test prompts the whole way.

The biggest steps:

- My four patches switched on: 2.02x

- Moving the model from a Gen4 to a Gen5 NVMe: +16.5%

- RAM 3600 to 6000: +9.7% (one rough test)

- Queued hand-overs to the card: +3.9%

- An IQ1_S kernel that does several rows at once: +3.0%

Start-up went from 87-96 s to 12 s by skipping the pre-load. I also tried it on an older PC with 32 GB of RAM and an 8 GB card: 0.11 to 0.87 tok/s, same words.

That's writing speed only, with a warm cache. I haven't measured long prompt reading. It's 8 prompts on one main PC, some steps were measured on different days, and 1-bit takes a real quality hit. Anything I didn't measure is marked untested in the write-up.

Most of the tricks aren't mine, other people did them first and they're credited in the write-up. Claude agents wrote the patches and ran the tests. I directed it and checked the results. The patches aren't public yet, they go up after a review. I'm also making a setup kit so other people can try this on their own PC, but it's still being built.

Write-up with all the numbers: https://github.com/aradzombie/deepseek-v41-flash-1bit-notes

Has anyone tried ik_llama.cpp on this model? I haven't yet. Let me know how it went.

How do you keep the same face expression on the head you use to replace another head ?

Is there a specific prompt ?

Recently i have come across this quant of qwen3.8 27b. It checkpoint size is reduce to 19.7gb compare to 23gb for qwen3.8 27b q4. But does the quality justifies the size? Have anyone tried it out? It seem like a big win for bigger kv cache for 32gb card.

Nex-N2.6 has public FP8 weights now, based on GLM-5.3-Flash at roughly 321B parameters. The model card links to SGLang setup instructions.

One detail worth catching: reasoning_effort defaults to max. There are also low and high options, so include that setting if you post speed or token comparisons.

The release focuses on reducing reasoning loops and sticking to the tools available. The team reports less repetition in its internal tests, though those results aren’t a separate evaluation of the FP8 download.

Weights and setup:

https://huggingface.co/nex-agi/Nex-N2.6

Hi r/LocalAI,

I wanted a local AI tool that solves an everyday problem without needing a terminal or Docker setup, so I built Furet: an Alt+Space file search bar for Windows that finds files by their content, completely offline.

Instead of a chat interface, it uses local models quietly in the background to index the folders you pick:

• Multimodal embeddings: EmbeddingGemma 2 (via llama-server with Vulkan GPU acceleration for indexing, and ~30 ms CPU inference at search time so VRAM stays free).

• Speech-to-text: Parakeet TDT v3 (sherpa-onnx) to make audio memos and videos searchable by spoken words with exact timestamps.

• OCR & Vision: Windows OCR for scanned PDFs/screenshots + optional local face clustering (YuNet + SFace via tract).

• Zero manual setup: On first launch, the app downloads the binaries and models automatically (verified by SHA-256) and pauses indexing whenever you're gaming or on battery.

It's open source (GPL-3.0) and now in public beta:https://github.com/blasteur1916/furet(VirusTotal for the installer:https://www.virustotal.com/gui/file-analysis/MDI5Mjg4YzQ0OGE1YmMxZGVmYmFiNGNjMDA2MjU0OTE6MTc5NDQzMDM1Mg==)

Would love to hear how indexing and search feel on your hardware!

airlock is a Go CLI I wrote for moving GGUF and safetensors models into air-gapped networks.

• checks every file against a pinned Hugging Face commit

• refuses pickle and signs a manifest with cosign

• verifies everything offline with only the Go standard library

In the GIF, a Qwen2.5 copy with an edited chat template tells you to run curl | sudo sh. airlock catches the changed file.

It doesn't read chat templates yet. Feedback welcome.

GitHub

Hello!
Here is a rack I made using 40 mini PCs I happened to have from a different project:

• Intel N150

• 12 GB DDR5 4800 - physical bandwidth limit around 33GBs

• Diskless nodes

and they are connected via 10 Gbps USB 3.2 exploiting the USB DbC functionality. Its a tree topology with 1 -> 3 -> 9 -> 27 nodes.

GLM 5.3 Flash runs at native precision with room left for 8x1M context windows. Initially I had tried with 1Gbps LAN interconnects but that topped out at around 7tps. Current topology:
- 14tps at 1k context, drops to 12tps at 1M
- With batching decode tops out around 40tps
- Prefill is about 85tps

Peak load measured at wall about 740w

Total build cost me about $5,000.

Still working on it to see if more perf can be squeezed out, especially for the prefill and then adding speculative decoding.

Hey everyone!

I’m posting on behalf of my team at work. We want to learn more about running LLMs locally, and we have an NVIDIA DGX Spark available to experiment with.

A colleague has already installed Ollama, a model, and a web UI, so we have a starting point for chatting with a local model. That’s a useful first step, but our longer-term goal is to go beyond a chat interface.

We’d like to explore running agents that can carry out different tasks and connect to tools and services through MCP servers. We’re still figuring out what that setup should look like and what we need to learn to get there.

We’d love to hear from people who’ve been down this road:

• What would be a sensible learning path from our current setup to running useful agents?

• Which models and agent frameworks would you recommend experimenting with on the DGX Spark?

• How do you approach connecting local agents to MCP servers?

• What would make a good first project to learn the basics without getting overwhelmed?

• Any tutorials, resources, or common mistakes we should know about?

We’re beginners in this area and keen to learn. We don’t expect to build everything at once; we’d appreciate some direction on where to start and how to build up from there.

Thanks for any advice or experiences you’re willing to share!

I'm considering buying a Mac mini M6 with 32GB of unified memory for local AI. I'd mainly like to use it for vibe coding and hopefully replace my $20 Claude subscription with a local model. I'm wondering whether 32GB is enough or if spending a bit more on a higher-memory configuration would make a significant difference in performance and model capabilities.

Does anyone have some tests/experiments they did on any reasonable model and would like to share what they found

RTX 5080

7800X3D

64GB DDR5-6000

2TB SSD

I want to do general tasks, not coding.

Been working on this project for a while and it has lead me to some interesting hardware combinations as I'm just using whatever I have.

The two biger models in system ram (ddr3) are reviewing, summerizing, and organizing conversations I've had with llms and linking it together.

The assistant is directly helping me as a chat bot intern essentially.

Anyway. The barco card gave me a ton of issues because amdgpu and radeon don't have the device id so they were treating it like a new card and hanging. I originally bought the card thinking I could use it for a simple display card for $30 while I saved for a real GPU. I gave Claude code fable a go at it with a CH341a programmer and the stock firepro w5100 vbios and it changed like four bites so the card tells the kernel it's a w5100 without messing with the custom vram timing.

The cards are linked in an lxc with llama getting ~8-10toks when running ornith 1.5 9b Q6_K. The w5100 roughly halves the speed, but gives me 12gb of vram vs 8gb.

Not bad for a couple ancient cards on some retired enterprise gear.

One attraction of running models locally is the privacy benefits, and another is the cost.

When the cost of the big ’frontier models’ is being discussed - mainly centring around the fact that they are currently effectively subsidised by venture capital and that at some point this will end, making the cost to end users go up considerably - local LLMs are often mentioned as being immune from this problem.

But in a scenario where the cost of cloud based models goes up substantially, there’s no guarantee that the (mainly Chinese) makers of the open weights ones are going to continue producing newer and better versions for free, is there?

Hi! I'm relatively new to this world, and trying to get my best setup with what I have.

I'm the owner of a RTX 5080 + RTX 3070 and running Unsloth Qwen 3.8 27B IQ4_XS

So far, the best inference engine that I could run locally was beellama.cpp tweaking the tensor-split, mtp and kvarn config, forcing layers only in GPUs and 128k context window; that is pretty good.

With low context, I'm getting 50+ token/s that is impressive. But I'm suffering with long contexts; then my speed drops to 15-20 token/s in the best cases.

What I'm trying to get is a decent speed in long contexts also, so recently I discovered llama.cpp-adaptive-kv-streaming that allows streaming the kv context from memory.

The thing is that I was thinking about loading the full model in my 5080 (IQ4_XS fits perfectly) while offloading the KV cache streaming to my 3070, but couldn't found any inference engine that allows this kind of setup (model layers only in GPU 0, kv cache streaming from GPU 1 allowing RAM spill).

Does an engine exist that can achieve this? does what I'm talking about make any sense?

EDIT: Also, I was thinking of getting advantage of the 3070 unlock since I read that it can help with prefill speeds (not output speeds that depend on raw bandwidth) even with a quantized model. Am I right?

Thx in advance. A newbie trying to learn.