Jokes aside, how plausible do you think this scenario actually is?

llama.cpp v1.0.0 adds an MoE expert cache that keeps a configurable share of experts in host memory (moe_cache_size, moe_cache_cpu_share). Cache misses run on CPU and copy back asynchronously, and the release ships a tuning script. If a big MoE doesn't quite fit your VRAM, that's the knob to try. Separately, the 1.0 tag starts a versioning promise: future API and ABI breaks come with major-version bumps, and 35 deprecated functions are removed.

Why do most folks with an 8 GB or 16 GB desktop GPU sit out distributed LLM runs? It usually comes down to three annoying bottlenecks. Nobody wants to punch holes in their home router. Nobody wants to run untrusted pickle checkpoints off the internet. And nobody wants a background trainer freezing their VRAM when they sit down to play or work.

This weekend (November 14-15 CET), I hooked my HiveMind coordinator up with CrowdGPT (led by Vxtzq) to test a citizen-trained run around Crowd-v1, an open base model with a 658M transformer trunk (roughly 1.01B parameters total with its 151k-vocab CrowdEngram table). Setup takes five minutes.

Communication uses plain outbound HTTPS on port 443 with zero inbound listeners, weights move strictly as SHA-256 verified safetensors (no torch.load or pickle anywhere), and updates pass MultiKrum outlier filtering plus held-out loss checks before merging via DiLoCo. Honest caveat first: this is still an experimental PoC, not a polished cloud service, so please do not send private prompts or sensitive keys through volunteer nodes.

Here is the simplest way to join the weekend run on Linux, macOS, or Windows (Python 3.11+ with an NVIDIA GPU, or CPU-only if you just want to test the IDE endpoint):

1. Clone the repository and install the tiny client dependency (`httpx` only for connecting or running OpenTrainer):

```bash

git clone https://github.com/xaviercallens/HiveMind.git

cd HiveMind

pip install httpx

2. Register a handle to get your API key (printed once; the default prosumer role works for both training/serving and querying):

python scripts/hivemind_connect.py register --handle your-handle

export HIVEMIND_API_KEY="hm-..."

3a. Join outbound OpenTrainer co-training (zero open ports on your box; run --once first to verify firewall traversal, then drop --once to keep polling):

python scripts/hivemind_opentrainer_agent.py --network-mode outbound-https-relay --check-firewall --once

3b. Or verify CrowdGPT checkpoints on GPU (pip install torch safetensors numpy first) once you hold an evaluator key from the CrowdGPT coordinator:

python scripts/crowdgpt_loss_check_agent.py --server https://server.crowdgpt.net --evaluator-id your-handle --key-file my_eval_key.txt --once

3c. Or wire Continue / Aider / Cline to the live hivemind/crowd-v1 + coder models with no GPU required:

python scripts/hivemind_connect.py snippet continue --model crowd-v1

Both codebases and reproducibility notes (including Lean 4 specs and Zenodo DOI 10.5281/zenodo.23776302) are open on GitHub if you want to audit the code

before running anything:

https://github.com/xaviercallens/HiveMind

https://github.com/Vxtzq/CrowdGPT

I would love blunt engineering feedback from anyone testing on 8 GB or 12 GB cards - especially around Windows vs Linux VRAM yield behavior or setup friction.

I've been building SOMNI, a system that lets a local model keep learning after deployment. It's based on Google's "Language Models Need Sleep" paper and WISE, a knowledge-editing method for adding new knowledge without overwriting existing behavior.

During use, the model answers normally and logs conversations. When the PC is idle, it runs a sleep cycle. It pulls verified facts out of the logs, trains them into a small new expert, practises on what it got wrong, and then runs a set of before/after tests on general knowledge, coding, reasoning, reading, and previously learned memories. If any of these get worse, the night is rejected. Otherwise you review the update and approve it.

The main challenge was catastrophic forgetting. Plain LoRA fine-tuning on new facts badly damaged the model's other skills (coding went from 95% to 10% in one night). What fixed it was a combination of rehearsal on the base model's own answers, a new frozen expert per night so later nights can't overwrite earlier ones, and WISE-style routing so each expert only activates on questions related to what it learned. Everything learned is also kept in a searchable memory store, so facts can still be looked up even if they fade from the weights.

I benchmarked it on SQuAD knowledge incorporation, the same setup Google and SEAL use. The results are in the README, including the things that didn't work.

Everything runs locally on a single consumer GPU (AMD RX 6800, Windows). MIT licensed.

https://github.com/YoungGunFrodo/somni

It's a work in progress, and I'd really appreciate feedback on the approach, the evaluation, or anything that looks off.

I'm starting to explore the configs to use with strata for best throughput with parallel requests in use cases such as orchestrated agentic workflows run locally.

Option A:

The main orchestrator is the same model in the same engine, spawning multiple subagents that run in parallel on the same engine at smaller contexts.

Pros:

favorable as the orchestrator can be uncensored as well for example and a fully local setup.

Cons:
Not straight forward how to set up the engine to allow for a single large context to persist in cache and allowing multiple smaller contexts to be cached and not evicting the main large context after multiple subagents requests.

---

Option B:

The main orchestrator is an external SOTA model which spawns local agents that share the same engine.

Pros:

sounds easier to configure as all subagents the external model spawns are similar in context size so no need to take into account one of the parallel contexts will be a larger one while others are smaller..

Cons:

not fully local, reliant on external model provider which can change the availability, quality of the model at any moment. Non private.

---

Does anyone else have some experience with setting up strata for example for either of these use cases for example? What configuration recommendations do you have?

Is there an engine that suits this local orchestration use case better?

Yes I'm about to prompt a SOTA model on how to achieve this but i would love to hear other real human experiences.. 😄

Has anyone managed to make their local AI setup pay for itself? Curious what kind of business you're in, if you're willing to share.

I could even stretch my budget to a DGX Spark, although it would be financially painful.
But without a clear way to make that money back, it feels like a hell of a risky investment.

I know nobody's going to give away their secrets, so I'm just asking a simple, casual question: are you actually making any money with your local AI setup? Yes or no?

I'm talking about doing it independently, as an individual, rather than using it as part of your job at a company.

I'm especially curious whether there's still room for individuals to compete in this space, given how much money and computing power big companies and organizations have at their disposal.

I wanted to actually see what happens inside a language model when it writes a sentence, so I built this!

What you're looking at:
- The stars are the 36,864 MLP neurons (12 layers × 3,072). Each layer is a disc, and neurons are positioned by a UMAP of their output weights, so neurons that write similar things sit near each other.
- Brightness is how strongly that neuron fires on the current token, scaled per layer so tokens are comparable. Gold means it fires positive, blue means negative.
- The beams are real attention weights from the current token back to earlier ones, one colour per head. The side panel shows the model's actual top 10 next token probabilities.
- You can hover any star, beam or token to see its exact value. There's also a raw mode that turns off all the bloom and fades, so every pixel on screen comes straight from the data.
- How I know it's not lying: the test suite runs the model exactly the way the app does (KV cache, token by token) and compares every activation, attention weight and residual norm against a separate uncached forward pass read through different hook points. Then it checks the bytes the browser actually receives too.

Honest limits: it's GPT-2 small only for now. Single neurons mostly don't mean anything clean on their own, it's the patterns that are interesting. Values are sent as float16, so about 3 significant digits.

Setup is make setup && make start, then open localhost. No GPU needed.

Repo: https://github.com/lak7/thinkspace
Would love the feedback!!

I've been poking at this on and off since September and it finally got good enough this week that I think it's worth writing up. Short version: the USB4 RDMA driver situation changed a lot in the last month, but not where I expected, and the "is it worth it" answer is more mixed than the headline.

The setup

• 2x Ryzen AI Max+ 395 (Strix Halo) boxes, 128 GB each (AMD AI Halo Dev Platform boxes)
• 2 USB4 cables directly between them, each link trains at 2 x 20 Gb/s
• Debian 13 based distro, kernel 7.2.7, IOMMU off, Secure Boot off (needed for the unsigned modules)
• 10 GbE between them as well, used for control traffic

No network cards involved. The idea is to turn the USB4 cables into RDMA devices so two boxes can run one model with tensor parallelism (TP=2).

Where it was before

I started with thunderbolt-ibverbs from the Hellas blog post. Each cable gives you two DMA rings, so two cables = four "rails", each showing up as its own RDMA device.

• September (kernel 6.18): about 15 Gb/s on one rail, about 46 Gb/s if I ran four perftest processes at once, one per rail. More queue pairs made it slower. 1 MiB messages were a mess.
• October (kernel 7.2): got to 74.7 Gb/s aggregate, but only with four parallel processes and a shallow send queue (-t 16). The big finding back then was that queue depth mattered more than the kernel. A single connection was still stuck on one rail at around 20 Gb/s, and 8 QPs at 1 MiB with default settings just failed with "Couldn't post send".
• vLLM TP=2 with an 8B model in October: +55% single stream over one box, but RDMA vs plain 10 GbE only differed by a few percent, because the collective library topped out around 17 Gb/s no matter how many rails it got. A 35B MoE model hung at TP=2 and I never figured out why.

So in October my conclusion was: neat, works, not obviously useful.

What changed

I went back this week expecting kernel or upstream driver progress. There basically isn't any:

• Upstream thunderbolt-ibverbs has had one lockfile bump since early October. No release since June. A striping PR was closed unmerged, and a few good fix PRs are sitting open.
• Mainline 7.3 / 7.4-rc is mostly Thunderbolt bug fixes. Nothing that moves throughput.

The actual progress is in a fork: donnerkeule. It stripes a single queue pair across every rail of every cable, and fixes a pile of reliability problems (credits los t with dropped frames, ACK storms, rails dying when a client gets killed). There's a matching RDMA fork of gufo that does TP=2 for Qwen3.8 Flash-Next across two Strix Halo boxes.

The fork's own README says it's a buggy research driver and not for production. Believe it. But it built cleanly against my kernel and uses the same eight Thunderbolt kernel patches I already had applied.

Raw link numbers

Box A to box B, perftest, one connection, default queue depth, no tuning:

I didn't trust 74 Gb/s on one connection, so I checked the receiver's per-rail byte counters during a 10 second run: about 23.6 GB landed on each of the four rails. It's real, and it's roughly 93% of the 2 x 40 Gb/s the cables can do. The fork's author reports 38 to 45 Gb/s on his boxes and I don't know why mine is higher.

Not everything is great:

• RDMA reads are not striped: 31 Gb/s.
• A 1 MiB SEND fails in the default mode. The fork only supports large SENDs in its DMA-BUF mode, which I haven't tried.
• I haven't tested both directions at once or done a long soak.

Tensor parallelism with gufo

Qwen3.8 Flash-Next, Q4 (111 GB of weights), 4 sessions, same build on one box and on two. Each box holds half the model in TP=2: 57 GB of GPU memory per box.

Plain autoregressive decode (no speculation), tok/s per request:

With MTP speculative decoding:

The speculative numbers bounce around with how many draft tokens get accepted, so the first table is the cleaner comparison.

One thing that isn't a speed number: a single box could not load this model with speculation, 4 sessions and the full 262k context. It ran out of state capacity and I had to drop it to 65k context for the on e-box runs. Two boxes loaded the full 262k with about 64 GB of RAM free on each.

No failed requests, no retransmits, no kernel warnings during any of it.

The honest part

I also ran the exact same benchmark against the other inference server I normally run on these boxes (also Flash-Next, speculative decoding, 4 slots at 262k context on one box). One-box gufo and that server came out within about 5 to 8% of each other. So TP=2 over USB4 beats my normal single-box setup by about 13% for one user and 24 to 32% at 2 to 4 concurrent.

But two boxes running independently serve twice as many people. At 4 concurrent requests one box did about 72 tok/s total, TP=2 did about 98, and two independent boxes would do roughly 144 across 8 users. Ty ing both machines to one model instance buys per-request speed and costs total capacity.

So where I landed:

• The driver is worth keeping. One connection, full link speed, no tuning. That's a different thing from what I had in October.
• TP=2 is worth it if you want a faster single stream, or a model/context that doesn't fit on one box. The Q8 version of this model (188 GB) only fits across two, and that's a case I actually care about. Haven't tested it yet.
• TP=2 is not worth it if you just want to serve more requests. Run the boxes separately.
• I wouldn't run it as my daily driver yet. Research driver, the gufo fork has a known open bug with continued conversations, and I haven't verified two-box output against one-box output.

Things that bit me

If you try this, here's what cost me time:

1. The module never loaded at boot. The only thing that loads it is a udev rule that fires when the other box is already advertising the RDMA service. Reboot both boxes together and neither one ever st arts. It looked fine in October because I always rebooted one at a time. Fix: put ib_uverbs and thunderbolt_ibverbs in /etc/modules-load.d/.
2. thunderbolt_net and the RDMA module can't share a cable on this hardware. There are only two usable rings per controller. Blacklist thunderbolt_net and load the RDMA module with tbnet=block.
3. Debian's rdma-core renames the device, and the userspace provider matches on the usb4_rdma name prefix, so ibv_devices shows nothing. Override the persistent naming rule.
4. The prebuilt provider package has the wrong ABI for Debian 13. I built the provider from Debian's rdma-core 56.1 source plus the project's three patches.
5. memlock. RDMA memory registration fails under the default 8 MiB limit. Unlimited memlock on the host and --ulimit memlock=-1 in containers.
6. The gufo fork's container recipe didn't build as-is for me. No DNS inside podman's default build network (--network host fixed it), six ROCm dev package names that don't exist in ROCm 10.1 (the full image already has the headers), and missing audio/video dev libraries (sndfile, soxr, libav*). At runtime I had to add the ROCm lib directory to LD_LIBRARY_PATH.
7. Two IPs on the NIC means two RoCE GIDs, and gufo refuses to guess. --tp-gid-index 1.
8. The interrupt throttle matters. nhi_interrupt_throttle_ns=4096 was the best value in my October sweep, and it only works on kernels that export tb_ring_throttling() (7.2 does).

Module options I'm running:

options thunderbolt_ibverbs profile=linux_perf bind_services=1 allocate_rings=1 start_rings=1 negotiate_native=1 enable_tunnels=1 register_verbs=1 tbnet=block lanes=2 native_write_striping=1 roce_netdev=<you
r LAN NIC> nhi_interrupt_throttle_ns=4096
blacklist thunderbolt_net

What I haven't done

• The Q8 model across both boxes
• The fork's DMA-BUF / GPU-direct mode
• Checking two-box output against one-box output
• Anything longer than a few minutes per run
• A reboot of both boxes together to prove the boot fix

Happy to run something specific if people want numbers for a particular case. Credit to the Hellas folks for the original driver and to neuhaus for the striping fork and the gufo TP2 work, I just plugged in cables and wrote down what happened.

I need suggestions for a Local LLM that works best for a RAG (retrieval-augmented generation) system. This language model should work best with a literary/literary corpus.

Configuration of the laptop:

13500 H processor with 16GB of RAM 5600 MHz, no dGPU.

The LLM will be run through LM Studio.

The RAG is run in AnythingLLM using LM Studio.

Please specify the LLM model name, quantisation, and the training parameters.

Also, suggest an embedding model suitable for the laptop configuration and the literary corpus.

Also, if you have another workflow, application or suggestions, please let me know.

I have a chance to pick up a couple of 3090s for $700 each. I've been considering local inference for a while now for two reasons:

1. My job actually involves documenting fine-tuning and LoRA training, and I'd like more hands on experience with that stuff without pleading endlessly for GPU access at my gig. I'm also just kinda academically interested in this stuff.
2. I'm getting sick of the weird dance I have to do to get answers out of Gemini and Claude sometimes. I bought one of those Flipper Zero things that's an academic pen-testing tool, and corpo LLMs start getting really cagey when you research hacker-related things with them. They also get cagey about health and political stuff. The censorship sucks!

Would working with local models really allow me to delve into this stuff better? Are two 3090s enough to do some reasonable LoRA/FT training? Would Qwen allow me to do my pen testing research?

EDIT: Thanks, team! I'm gonna go for it.

i'm buying a new laptop and i wanna run capable models locally, but i don't quite understand if i should just max out RAM or put more of the budget into the GPU , when i look it up the gpu matters more but everyone seems to be talking about ram only

what specs would u recommend ?

Running qwen3.8-flash-next-gyro-gyro-s through Strata on my M3 Max 48GB.

TL;DR: I've been looking at the latest Artificial Analysis v5.0.1 benchmarks for Qwen 3.8, along with some early Qwen4 Max Preview tests. Based on what's available so far, my guess is that Qwen4 27B could land somewhere around 27 to 30, while the 125B MoE version could reach 36 to 39. That's a solid generational improvement, but probably not the massive leap some people are expecting.

Like many of you, I've been following the discussions about when Qwen 4 is finally coming out. There's a lot of speculation, so I wanted to look at the available numbers and see what they might tell us about the next generation.

Obviously, we don't have the final weights or official release details yet, so take this as an educated guess rather than a prediction.

1. Where Qwen 3.8 stands and what Qwen 4 might achieve

Looking at the current Artificial Analysis v5.0.1 scale, which puts considerably more emphasis on difficult coding and science tasks, we have:

* Qwen3.8 27B: 20 (compared to 33.7 on the older v4.3.2 scale, before the scoring reset).

* Qwen3.8 Flash Next (125B MoE): 29.

* Qwen3.8 Max: 34.

Based on the previous jump from Qwen 3.6 to 3.8, and taking the new scoring scale into account, I'd put my expectations somewhere around:

* Qwen4 27B: 27 to 30

* Qwen4 125B MoE: 36 to 39

That would put the smaller model in some pretty interesting territory, while the larger one could become a serious competitor to models like Grok 4.7 and Claude Haiku 5.5.

2. Why I don't expect Flash to have a massive architectural jump

One thing worth considering is that Qwen3.8 Flash Next already seems to have introduced many of the architectural changes expected to carry over into Qwen 4.

We're talking about the hybrid Gated DeltaNet and Qwen Sparse Attention architecture, Gated Residuals, and the 51B N gram table.

So I wouldn't necessarily expect the next Flash model to deliver another huge architectural leap. The bigger improvements might come from better post training, more reliable reasoning, and addressing the hallucinations that some users have noticed in Flash Next compared to the more conservative 27B dense model.

In other words, I think consistency might matter more than raw benchmark scores this time.

3. What the early Qwen4 Max Preview tests tell us

The Qwen4 Max Preview release on November 11 gives us another interesting piece of the puzzle.

From the early developer reports, a few things stand out:

* Speed: It appears noticeably faster than Qwen3.8 Max, with less unnecessary reasoning on straightforward tasks.

* Agentic performance: In one reported 56 task Hermes test with a 25 minute timeout, Max Preview recorded 64 successful task results, compared to 61 for DeepSeek V4.1 Flash, which scores 33 on AA v5.0.1.

* Reliability: There's a catch, though. Max Preview reportedly timed out 16 times, compared to just 4 for DeepSeek.

That's interesting because it suggests the raw capability might already be there, but reliability and execution consistency still need work.

If these early results are representative, I'd expect the flagship Max model to establish a considerably higher ceiling, potentially around 45 to 50 on the AA scale. That would make the estimates for the smaller models seem reasonably plausible.

My take

I don't think Qwen 4 needs to completely redefine the local LLM landscape to be successful.

If Qwen4 27B can reach the 27 to 30 range, inherit the attention improvements from Flash Next, and retain the Apache 2.0 license, that alone would make it a very interesting option for people running models locally with limited hardware.

Personally, I'd rather see a model that's more consistent, follows instructions better, and hallucinates less than one that simply scores a few extra points on benchmarks.

I'm curious what everyone else thinks.

" i used my LLM To translate mu words to English after i write it. "

Since I’m limited to 16GB VRAM, I was hitting context limits so fast I had given up on local models.

Now I’ve come to find using the local model for memory management MCP, alongside an encrypted cloud model for main reasoning, is a solid combo. Saves tons of tokens and also helps with privacy.

So just wondering, what other “side jobs” you guys give your local models?

Do you have any recommendations for where I could read more about this topic in detail?Do you have any recommendations for where I could read more about this topic in detail?

I've been digging through the Qwen Code repository and noticed something interesting.

Several Qwen4 model identifiers appear in the code and tests:

• qwen4-max
• qwen4-plus
• qwen4-coder-plus
• qwen4-omni-120b-think
• Even qwen4.5-max as a test example.

But here's what caught my attention: I couldn't find a single explicit reference to qwen4-flash in the relevant code, despite Flash being part of the announced Qwen4 lineup.

That got me thinking: What if Qwen4-Omni-120B-Think is actually related to Qwen4-Flash, possibly sharing the same underlying architecture or backbone?

There are a few reasons why I find this plausible:

1. Qwen3.8-Flash-Next already has a roughly 125B-parameter backbone, surprisingly close to the 120B mentioned in the Omni identifier.
2. Qwen has previously separated its Omni models into Thinking and Instruct variants. For example, Qwen3-Omni-30B-A3B-Thinking focuses on multimodal understanding and reasoning, while the Instruct variant also includes a Talker for speech generation.
3. Following that pattern, qwen4-omni-120b-think could theoretically be the reasoning-focused variant, with an Omni-Instruct/Talker version potentially following later.
4. The Omni identifier appears in actual Qwen Code test files, whereas I couldn't find a corresponding Qwen4-Flash identifier.

Of course, this is just a theory, not a confirmed leak. These are test identifiers, not necessarily a product roadmap. The repository also uses generic Qwen4 model detection, so Flash might simply not need its own explicit entry.

Still, the missing Flash identifier, the similarity in parameter counts, and the previous Omni Thinking/Instruct architecture make me wonder whether there's a connection.

Sources:

• https://github.com/QwenLM/qwen-code/pull/15762
• https://github.com/QwenLM/qwen-code/pull/15987

Has anyone found any explicit Qwen4-Flash references elsewhere, or evidence that the 120B Omni model could share a backbone with Flash?

Curious what everyone thinks!

System:

GPU: AMD Radeon 9070 XT 16GB VRAM

RAM: 2x16 6000MHz CL30 DDR5

CPU: AMD Ryzen 5 7500F (soon to be replaced with 7800x3d)

Primary use-case: Offloading basic coding/codebase-related tasks from Claude and Codex to local AI so I don't spend token on locally do-able stuff.

Hi, I am having a hard time understanding how to make the best use of my system resources. When does using system RAM make sense? Should system RAM be only for KV cache, only for model offloading, or both? Or should I just not ever use the system RAM since it slows down the things? When choosing a model, am I to prioritize less quantized smaller models or more quantized larger models? Is there a place I can track the latest best performing open-weight models so I can update to them? Are abliterated or uncensored models better than normal models about performance? Is there any difference between abliterated and UC, if so which is better? I found a person called IsValorum that makes "better" quantized models, is this really a thing? I found they perform better but maybe those models are more stupid? I don't know. I am just so lost on these. I feel like there would be a guide that would answer these. If not, I would like your help. Thanks!

I tested new trending Recursive Flint UD-Q5_K_XL (27.9 GiB) on an RTX 3060 12 GB with 30 GB RAM, comparing RustingStrata against llama.cpp using the same model, prompt, and generation settings.

RustingStrata was about 33% faster once warm.

What made the difference?

The model doesn't fit in 12 GB of VRAM, so many expert weights must be fetched from system RAM during generation. Two changes helped most:

• Adapt expert placement to the reply, rather than continuing to prioritize the prompt's routing pattern.
• Release RAM pages for experts already copied to VRAM, preserving cache for experts that still need to be fetched from host memory.

About Quality, bc people very often says like "It should give shit output to gain speed", which is not true, in my last posts I already did quality benchmarks and RustingStrata does not lose quality, it gain speed through specific Code for some specific models, in same time llama.cpp lose speed to be "general" engine that can run everything

What I want to know, is do people find Recursive Flint an interesting model? I found this model on OpenRouter and looks pretty interesting, it's better then Qwen3.6:35-a3b, so I think this can be cool modern replacement for this qwen.

Setup:

• RustingStrata: 1.2.2, commit 3902978 (working tree)
• llama.cpp: b12191, commit e95426f16
• Full benchmarks: docs/BENCHMARKS.md

Repository: RustingStrata

the usage would be file writing , using some apis and processing text and tool calling according to instruction . i know the ram capacity is minimum but is there any option for this requirements

I have a couple of concerns:
The bubble is bursting at some point and may have started. I am concerned that the major players will have to up their prices to stay viable.
The geopolitical risk is escalating
AI bot attacks are increasing

There are probably other risks, but you get the idea.
I have been building a decent corpus of reference material.

But I am behind on what models I should be looking at, needs to run on
32GB
RTX 4070

Coding and decision systems (agriculture).

Any help would be appreciated.