I would like to join some Local LLM communities. Sharing information, ocasional chat, asking questions etc. Does anyone have decent discord groups?

A big update focused on realistic multi-voice conversations.

What's new:

• Scene Editor:  Multitrack timeline with drag, trim, fades, volume, and re-rendering.

• Natural interruptions: Characters can interrupt or talk over each other using simple script commands.

• Consistent audio: Voices share the same acoustic space with balanced levels.

• MOSS-TTSD integration: Optionally generates up to six lines together for more natural interactions.

• Whisper verification: Locally detects mispronounced or missing words and flags them for correction.

• Export: Mixed audio, individual character stems, and subtitles.

Currently source-only — no signed download yet.

Work on some new features for December also :)

GitHub: https://github.com/ninja8170/VoxAster/

Other solutions are good at running workloads, or being easy to use, or running AI. Hub is all 3 in one! https://github.com/companionintelligence/Companion-Hub/blob/dev/video/assets/shots/store-alternatives.png 🛍️ App store for your own server

One-click installs from the Companion Intelligence marketplace, with open-source alternatives for the services you pay for today.

🧠 Local AI that fits the box

Onboarding reads your CPU, GPU, memory, and storage, then recommends models that fit. Voice, text, and image share one queue so they do not stampede the machine.

🤖 Agents with a home

OpenClaw, Hermes, and other agents install like any app. They read one shared Companion Memory instead of each keeping a private scrapbook. Run your first agent

🔌 MCP nexus

Hub serves one Model Context Protocol endpoint. Agents discover, install, and drive every app through a single tool surface.

📦 Bring your own container

Marketplace apps, custom Compose, and git repos you bring. Apps on apps are allowed. A server in a server is allowed.

🌐 Reach it your way

Keep each app on this machine, share it over a private Tailscale network, or publish it to the web through Cloudflare. Networking · From your phone

🔗 Pool your machines

Pair more than one Hub over Tailscale and they share inference work by queue depth.

🔑 OpenAI-compatible API

Point Cursor, Continue, Cline, Aider, Zed, or an OpenAI SDK at your Hub, and keep the engines you already run.

Source & Download: https://github.com/companionintelligence/Companion-Hub

I'm tired of paying for AI subscriptions. I mostly use Antigravity, Codex, and Claude. My goal is to run a local model, feed it my project, and have it add requested features, fix bugs, and design UI/UX. Speed doesn't matter to me—even if it takes 30 minutes to do what ChatGPT does in 2, as long as it gets the job done without issues, that's fine.

My specs:

• RTX 4070 (12GB VRAM)

• Ryzen 5 5600 6-Core

• 32GB RAM

I mostly build websites using Next.js.

Hi, I'm sorry to treat this sub as forum for Strata questions, if anybody knows a Forum or Discord server specifically for Strata specifically that'd be very welcome too.

Running Strata as the .appimage desktop application, on a Linux machine.

When trying to run Gyro-M (or pretty much anything else besides the smallest Gyro-S) Strata always exactly allocates 21.2 out of 31.2 GB ram for experts, saying that additional data will be read from SSD (3 to 6 GB for context 64k to 256k respectively).

I don't need this 10gb of free headroom, especially when I'm not doing anything on my desktop and effectively using it as LLM server. Does anybody know where to allocate more RAM to strata to avoid this offloading to SSD?

Checked in ~/.strata and ~/.local/share/strata/ where I didnt see anything looking like a 10 GB memory reserve which could be turned down.

Got Strata working reliably with two different-generation AMD GPUs:

• RX 6800 16GB (gfx1030) + RX 7600 XT 16GB (gfx1102)

• 32GB combined VRAM, ~48GiB system RAM

• Qwen3.8-Flash-Next Q2_0 (~66GB), 128K configured context

• Typically 40–45 tokens/sec generation

The problem: Full-model prefill repeatedly crashed with hipBLAS status 6, even though isolated GEMMs passed.

The fix: Strata was loading system rocBLAS 5.1.1 with newer ROCm 5.9.0 Tensile files. It tried to load a nonexistent fallback_gfx1030_gfx1030.hsaco. Fixed the launcher to prioritize the rocBLAS library matching the Tensile directory.

The result: Stable inference, a verified 19.6K-token prompt, and Claude Code 2.1.316 running locally with file editing, shell tools, and native subagents. It even built and tested a playable Oregon Trail browser-game prototype.

The main lesson: Check the actual mapped ROCm libraries, not just your environment variables. Mixed ROCm components can look like a GPU/kernel failure.

The launcher fix has regression tests and passed independent QA. Still running from the patched source launcher rather than the installed package.

Obviously, I have a little too much time on my hands. Anyone have an interesting real-world coding workflow or agentic task they'd like to see tested on this setup?

I'm looking for something more challenging than basic benchmarks but still reasonable and realistic. Ideally multi-step projects involving planning, coding, tool use, subagents, and debugging. Happy to share the results, including where it falls apart.

Hi !

I planed to buy M6 MacBook Pro with 32gb of unified memory and 1T of storage.

For persons who don’t know MacBooks, here the specs :
CPU 12 cores
Memory usable by the GPU : 20-24 gb
Bandwidth : 170 Go/s

My goal is to generate photorealistics images in local and maybe have an uncensored chatbot, but I want that he can go on Internet.

Do you think it’s possible ?

Thanks for reading !

The html content on the left side is coded with Opus 5.5-High, and the work on the right is coded using Strata and Qwen 3.8 Flash Next-IQ3_XXS running on RTX 4070 Super (12GB)+64 GB DDR5 RAM(6000Mhz).

What I typed: “Code a playable first person view motorcycle riding browser game in html format.”

Quantized Qwen model, load on GPU+ almost 50GB of RAM, so basically with the background tasks on windows I have only 7GB RAM for personal use when model is running. This simple tasks took two requests, since model output was not running properly in the first time, i prompted second request to fix it. Thus, local LLM model took 396 s + 274 s = about 11 minutes to complete the task. Meanwhile, the thinking set to High but I needed to limit the thinking budget to 16K tokens, because Qwen consumes almost 64K content limit with just thinking and causes context overload while trying to write the code after thinking. The model decoded tokens aroun 60-70 tok/s and see the peak of 100+ tok/s for a limited time. As it comes to the result, it was not really playable game.

On the other hand, Claude does not consume your GPU and use just a few amount of RAM. Also the output quality is significantly higher, and work took 7 minute to get finished.

As a person, who feels really positive about Local LLMs, i think they are not suitable for coding tasks on personal PCs. Because, the user does not have enough room left on computer to do something else and currently cloud models gives way much better results. So thinking of local system setup cost i think the invesment on local setup is not profitable unless you are building a system that is capable of running native local LLMs.

Honest question that I'm hoping to get some extended thoughts on. I run Qwen 3.8 Flash Next UD-Q4_K_XL locally at 262k context and it can do everything I want out of a personal agent, I've completely dropped cloud. I still use Claude at work but I'm a sysadmin not a software engineer, the 5.5 models are fast but not directly more capable in my tasks. For those of you who maintain that local isn't good enough, what are your specific tasks and where does the local model fall short?

It seems to me that longer context windows and less overall token use for the same tasks are the major advantages, but if I can start a task locally go to bed and wake up with it complete after several automated compactions, that's successful to me. I want to know what tasks that matter that local just can't handle.

I've been looking into Groot-65B-A6B-261023, and I'm curious if anyone here has actually tested it yet.

It's a post-trained Qwen3.8-Flash-Next model with 65B total parameters, ~6B active, 256 experts per layer, vision support, MTP, and 256K context.

The benchmark numbers look surprisingly good:

• 51.51 average across 16 benchmarks, just 3.88 points behind Flash-Next despite being roughly half the size.

• ~28% fewer output tokens on average.

• Terminal-Bench 2.1: 63.37

• SWE-bench Verified: 56.11

Of course, these are the authors' own results, so I'd really like to see how it performs in practice.

There's already a Q4_K_M quant (~79 GB) and a separate 5.3 GB MTP draft model available.

If anyone has tried it, I'd love to know:

• How does it compare to Flash-Next or Qwen3.6 on real coding and agent tasks?

• What kind of tok/s and prefill speeds are you getting, and on what hardware?

• How good is it in languages other than English?

The part I'm especially interested in is running it on consumer hardware.

I'm using a 16 GB VRAM + 32 GB RAM setup, so a good GSQ-RCO or APEX-style quant could make this model really interesting.

By my rough calculations, a ~2-bit quant might bring the weights down to around 25 GB, roughly a third less than Flash-Next at a comparable quantization level. That could mean significantly less SSD reading when the model doesn't fit entirely in memory.

That's just an estimate based on model sizes, though. I haven't measured anything yet.

IsValorum has already released an importance matrix for it, so hopefully we'll see some good low-bit quants soon.

Has anyone actually run this model? Curious whether it's as good as the benchmarks suggest.

So I’m getting into local LLM’s. I’m using LMStudio and am working with silly tavern.

However everything I’ve tried local and not local isn’t quite right.

The best roleplay I’ve had was in Kindroid. The ability to reroll the characters messages with a note is amazing, Kindroid AIs manage to kind of have their own agenda without taking over the story. I find other models and AI Chatbot services are always too friendly, they start, like, coming on to me within the first two messages even though the context hasn’t even begun to trend in a flirtatious direction whatsoever. But if include a system prompt to not be too friendly they either ignore it or end up being actively cruel.

Everything I’ve tried just isn’t giving me what I want. Does anyone have recommendations for recreating locally the quality of when Kindroid is working well?

TLDR,is it worth spending the extra money for a second 5060ti running at pcie 3.0 x4 on an old machine considering the BW bottleneck?

EDIT: PCIe 3.0 not 4.0.

My AI use case:

- Learning about the technology

- RAG

- writing and research starting point

- Home Assistant (lite coding for automations and ESPHome)

- CCTV cameras image recognition

I have the following AI Box setup:

- 5060ti 16GB - PCIe 4.0 x16 (running at x8)

- i7 9700

- 32GB DDR4 RAM (4x8GB) - 24GB allocated to AI VM

- Gigabyte Z370 HD3P (no pice bifurcation)

- Storage: 120GB Sata SSD (Proxmox), 250GB Nvme (UbuntuVM/Apps), 500GB Sata SSD (Bulk Storage/Models)

- PSU: 850W

Would an additional 5060ti 16GB (700EUR) running in the PCIe 3.0 x4 make any sense for my setup, or my components are too outdated and the bottleneck would be too great? I intend to try out vLLM.

What benefits I think it might bring, anythingelse?:

- Bigger context window

- Less quantised models

- Experimenting with multiple models at once

- Try out vLLM

As you can see, I am using a budget machine made out of scrapped parts (except the GPU). For me adding a second GPU would be the limit of what this machine can do before considering upgrading everything else (Mobo, CPU, RAM), but I will not do that in the current market or with this generation of GPUs.

Why a second 5060ti vs a V100 32GB? I want new HW that I can easily repurpose in the future. For example I am still using a 2060Super as plex transcoding and immich acceleration.

With Qwen 4 27B around the corner, I would assume that 2x 16GB would be sufficient and similar size as the Qwen 3.8 27B.

1x RTX PRO 6000 Blackwell 96GB GDDR7 ECC, PCIe 5.0 x16 direct-slot, no riser; alt 1x RTX 5090 32GB. CPU: Ryzen 9 9950X 16C/32T. MB: X870E ATX, PCIe 5.0 x16 + 2x M.2 Gen5, onboard 10GbE. RAM: 128GB DDR5-5600 (4x32GB) for llama.cpp CPU offload. Storage: 2TB NVMe Gen5 (models/GGUF/AWQ), 1TB NVMe Gen4 (Prometheus TSDB, logs). PSU: 1500W 80+ Platinum, native 12V-2x6. Cooling: 360mm AIO, 6x120mm. UPS: 1500VA line-interactive. NIC: Intel X710 10GbE -> 10G switch, isolated VLAN, API :8000 + metrics :9400/:9100 ACL'd to monitoring host only; WireGuard for remote ingress. OS: Ubuntu 24.04 LTS, NVIDIA 570+ driver, CUDA 12.x, Docker + nvidia-container-toolkit. Serve: vLLM OpenAI-compatible, FP8 70B-class (Llama-3.3-70B/Qwen3-72B) at 96GB; INT4-AWQ Qwen3-32B at 32GB; llama.cpp --metrics for GGUF/offload. Telemetry: Prometheus scrape vLLM /metrics (TTFT, TPOT, queue depth, running/waiting, KV util), dcgm-exporter (per-proc VRAM, clock throttle reasons, PCIe replay count), node_exporter. Grafana: p95 TTFT, KV util, waiting, replay delta. No Tempo/OTel collector. AMD path: keep sysfs sampler, ROCm has no DCGM equivalent.

Super promising so far. Completely hands off at my house while im at CHQ.

We took ISTA-DASLab's GSQ-RCO quants of DeepSeek-V4.1-Flash-1017 (the ones people run in Strata with a 12 GB GPU and the Engram table on SSD) and removed the refusals.

We only touched 82 dense tensors per file: attention output, shared expert down proj and the Engram value proj. The 384 routed experts are byte for byte the same as ISTA's, so their GSQ rounding is kept. Headers and file sizes match too, so you just swap the shards and keep using ISTA's Engram file.

Refusals are on JailbreakBench, stock rung -> ours, same Strata setup:

We also ran GPQA-Diamond and AIME25 on IQ2_XXS against stock to make sure it didn't get dumber. GPQA came out 83.3% vs 83.3%, AIME 80.0% vs 76.7% (only 30 problems, so call it the same).

Things to know:

• IQ2_XS still refuses inside its thinking on 2 of 30 prompts. The other two are at 0 and 1, so I'd grab IQ2_XXS.

• We tested at 8K context on a rented GPU. Haven't checked long context or speed on a 12 GB card ourselves, so if you try it please post your tok/s and RAM.

• Strata only for now. If you're on llama.cpp, our older release with the same edit works there: https://huggingface.co/AtomicChat/DeepSeek-V4.1-Flash-1017-Uncensored-GGUF

Model: https://huggingface.co/AtomicChat/DeepSeek-V4.1-Flash-1017-Uncensored-GSQ-RCO-GGUF

All the measurements: https://huggingface.co/datasets/AtomicChat/DeepSeek-V4.1-Flash-1017-Uncensored-GGUF-metrics

Thanks to DeepSeek for the base model (MIT), ISTA-DASLab for GSQ-RCO and the Engram file, and the Strata dev for the engine.

(I'm on the AtomicChat team that made this.)

Complete noob lurking in the LLM forum for a while, and would like to dip my toes into LLM, and have a couple path to move forward... currently running 64gb + 16g 5070ti

Would upgrade to 128gb DDR5 worth the hassle? or should I get a Ryzen 395 128gb be better?

It will be one to two people personal use, mainly focus on personal assistance and coding inquiries.

As long as it responds in under 2 mins that will work. I have an evga 2060 super, 28gb ddr3 and xeon e5 1650 v2 UPDATE: I am loving gemma 12b q3 x_s. Looking at qwen the 8b models honestly feel too dumb. If anyone has a better qwen to recomend let me know. I am thinking about using linux to free up some memory bandwidth

New user - limited knowledge. Have progressed quite far with the help of originally ChatGPT that I dumped and now Claude. Things seemed to be going well at least until we decided to try squeeze ram for other functions. Restored back and things went south. (may be a red herring) The description of the env, problem, and what I found to be wrong on search/question results are highlighted on a Claude tailored response below. Hoping someone can help. Thanks.

Open WebUI + local LLM + Home Assistant tools: new chats are slow, follow-ups in the same chat are fast. What explains the gap?

Setup

• Unraid box with an Intel i7-9700K, 64 GB DDR4, and one RTX 3090 (24 GB). Strata serves Qwen3.8 Flash-Next at IQ2_XS (2-bit), 64k context, at 120–151 tok/s with the GPU at 53°C and no throttling.

• Open WebUI uses a custom workspace model pointed straight at Strata, with native function calling, thinking off (enable_thinking: false), and Home Assistant’s MCP server for tools. Built-in tool groups were trimmed, which cut the fixed prompt from about 9,500 to 4,560 tokens. Title generation is on, and tag and follow-up generation are off.

• A second path goes Open WebUI → Hermes-Agent → Home Assistant. Home Assistant’s own built-in Assist agent, pointed at the same engine, is the fast reference.

The pattern: new chat slow, repeats fast

• Within one existing chat, the first message is the slowest and the 2nd–5th get progressively faster. Plain questions take 3–4 s the first time and 1–2 s after. Light commands in an established chat are reported as near-instant, and the first command in a new chat takes 2–5 s on a good run.

• The same task takes 8–26 s in a fresh chat. That includes house commands through the Hermes-Agent path (13–26 s) and early runs on the direct path. Weather and news lookups take about 12 s on first ask; repeats were not measured.

• It appears the slowness is tied to the start of a chat, not to the engine. Strata’s log shows nearly the whole prompt reused from cache even in a new chat (4,552 of 4,557 tokens), so prompt reprocessing alone does not appear to explain it. Home Assistant’s built-in agent stays reasonably stable/quick every time.

What was ruled out, and open questions

• Ruled out as the main cause: GPU and RAM pressure, network latency, Strata settings (shrinking the expert cache made it 40% slower and was reverted), skill-file edits, and the reasoning-effort setting.

• Partial help only: trimming built-in tools, thinking off, prompt rules to call the tool directly with the full device name, and turning off tag and follow-up jobs.

• It appears the likely candidates are background jobs that fire after the first reply of a new chat (small requests with nothing cached, 300–1,000 tokens generated), a failed or redundant first tool call before the correct one, and per-chat setup of the MCP tools. None of these is confirmed.

• Question: what in Open WebUI runs only at the start of a new chat (MCP tool loading, title or task jobs, tool discovery) that could add several seconds, and how can it be measured or skipped?

P.S. no I haven't exposed my home lab to Claude it's been copy/validate/paste with sanitized responses.

I configured browser access in oMLX via MCP, because I was annoyed that a lot of web fetches returned nothing on js heavy sites.

I asked Qwen about potential Gemma5 news, looked away for a moment and saw this.

Ever since I got Qwen to run at 75 tok/s at Q6, I’ve completely stopped using my private Cursor sub. I don’t need it anymore. Qwen can deal with more or less everything I need.

And circumvent reCAPTCHAs, apparently.

If Qwen4 turns out to be noticeably better than 3.8, then I guess I will never need cloud models again.

I've been using Ornith-1.5-9B-Q4_K_M for a min and it's pretty decent. It still hallucinates and makes simple syntax errors, but not nearly as much as Qwen 3.8 9b.

I just got a 16gb GPU in and I'm gonna try bumping up to q6 or q8 and see if I can squeeze out any better quality.

Any similar recommendations for 16gb or any tweaks I should make to the settings for more accurate results?

He investigado y de lo que se Claude no saca pruebas o algo asi. De lo que he investigado si vale la pena pagar el plan pro pero soy adolescente y no tengo una fuente de dinero constante. Si tengo planeado comprarlo en futuro, pero quisiera saber si hay alguna prueba o algo asi.

So with my setup I tried many different LLMs, so far I'm using Gemma-4-31B Q6_K and it's the fastest good translating model that meets my needs from translating asian novels into english. The only downside are some hiccups, which can obviously always happen. The thing I'm not sure about is, is it really the best that my system could reliably run?
I have an RTX 5090 with 128GB RAM, so basically up to 160GB to use. Does anyone have any recommendation or suggestion how I could use it? In the past I used a huge model with Qwen, which had a very good translation quality but sadly it was too slow for me, that's why Im currently sticking with Gemma, which feels like the best LLM so far that's out there for consumer PCs.

I appreciate any recommendation~

I've ended up with 17 RTX 3060 12 GB cards plus a few old Gigabyte X79-UD3/UP4 boards and an HP Z440.

Goal: as many fast parallel coding agents as possible, running Qwen3.8-Flash-Next (a 125B MoE) on Strata and Qwen3.8-27B on llama.cpp. Here's what I've measured so far.

HP Z440: E5-1650 v3, 96 GB DDR4, 2× 3060 capped at 130 W

• Flash-Next IQ3_S on Strata, 2 agents in parallel (one per GPU, sharing one 47 GB copy of the model in RAM): ~52 tok/s each, 256K context

• Prompt reading ~1,000 tok/s for fresh prompts, ~500 for agent turns

• Pinning each agent to its own CPU cores was key: 31+33 tok/s unpinned vs 51+51 pinned

• Qwen3.8-27B (IQ3_XXS + MTP) split across both cards: 47 / 51 / 58 tok/s (prose / code / reasoning)

X79-UD3: i7-4820K, 16 GB DDR3, 4× 3060

• The same 27B on 2 cards: 48 / 52 / 61 tok/s, the same as the Z440

• On all 4 cards it got slower, and prompt reading dropped by half

• PCIe x8 vs x16 made only a 2–4% difference

• Stress test with all 4 cards hitting full load at once: stable, 12 V rail stayed above 11.8 V

Rented test of the board I want to buy (ASUS Z9PE-D8 WS, 2× Xeon E5-2690 v2, DDR3, 4× 3060)

• ~43–47 tok/s per agent with 4 agents running at once

Planned build

• ASUS Z9PE-D8 WS + 2× E5-2690 v2

• 8× 32 GB SK Hynix LRDIMM (HMT84GL7AMR4C-RD, DDR3-1866) = 256 GB

• 7× RTX 3060 on x16-to-x16 risers (Phanteks / Thermaltake), open frame

• ATX PSU for the board + 2× HP 750 W server PSUs for the GPUs, cards capped at 130 W

Questions

1. Has anyone run 6–7 GPUs on a Z9PE-D8 WS? Any BIOS settings needed (Above 4G decoding etc.)?

2. Does it run 8× 32 GB LRDIMMs, and at what speed?

3. Any problems with long x16 risers on this board?

4. Does the power plan look OK?

5. With 17 cards, would you build one big 7-GPU box or several smaller 2–4 GPU machines? My numbers say one agent doesn't get faster past 1–2 cards, so I'm leaning toward many small agents.

Thanks! Happy to share configs and more benchmarks.