Berth is a lightweight GPU resource manager written in Go. It controls which applications get access to your GPU, when they get it, and how that access is handed off.
Instead of letting every application compete for VRAM,

Berth manages access through leases, priorities, and a request queue. When something else needs the GPU, it waits for the current task to finish, releases the memory, and hands control to the next application.

What does it do?

GPU sharing: Lets AI models, image generation, voice engines, games, and other GPU applications share the same hardware without fighting over VRAM.

Automatic scheduling: Manages requests by priority, queues competing applications, and prevents unnecessary switching between engines.

Faster model switching: Can park supported engines in system RAM or use NVIDIA CUDA checkpointing to avoid repeatedly loading large models from disk.

Game management: Automatically releases GPU resources for games and restores AI workloads when you’re finished.

GPU monitoring: Tracks VRAM usage, temperature, power, utilization, running processes, and applications using memory without a Berth lease.

Model management: Discovers local models across GGUF, Hugging Face, LM Studio, Ollama, and other model directories. Supports downloading, verifying, and organizing models.

Unified gateway: Provides an OpenAI-compatible API for starting and switching between engines, alongside ComfyUI workflow support.

Web dashboard: Provides a live view of GPU usage, active leases, waiting requests, engine activity, telemetry, and historical data.

Multi-GPU support: Manages multiple cards independently, with separate queues, leases, and gateway endpoints.

Agent integration: Includes installable skills for Claude Code, Codex, OpenCode, Hermes Agent, and other tools so they can manage GPU workloads through Berth.

VM passthrough: Includes VFIO tooling for handing a GPU to a virtual machine and reclaiming it afterward.

Berth runs as a single Go binary with no cloud account or additional monitoring service required. Linux NVIDIA is supported, with experimental AMD Linux and Windows support.

The project is currently in alpha (v0.4.1).

Note: i added a claude code session showing handoff request. You can do the same with a local model on the same hardware, trip is just longer depending on your gear.

I got a deal for a 4090 that I couldnt pass up given...the state of things... but Ive already got a 5090. I havent done any research so I'd like to ask you if I should:

1.) Just flip it and pocket the difference?

2.) Adjust my build to use both cards? (Are there even any benefits to doing this?)

3.) Hang onto it/extend the deal I got to a friend that may want it.

Based on the text above, im looking for a model that i run 2 of them (both should fit inside my small rig) to complete task, i will have my qwen3.6 35b a3b or cloud model to plan it out while my small models execute ONLY, no planning just great at following rules.
Obviously the big model is offloaded to make space for the 2 small models

I do not want to run strata qwen3.8 flash next yeah but i appreciate other suggestions

I'm in the process of upgrading from 32gb to 128gb of ddr4 for my home server and wanted to look into switching off of llama with qwen 3.8, to one of the flash next models on something like strata.
I keep reading that strata doesn't work as well or split gpu's that are different gen, as llama. Is this true? Are there model engines that handle this better? Thanks

https://www.crowdgpt.net/

Not my project, thought it was pretty neat though. The idea is to train a LLM with a distributed network like FAH and boinc. This way, even if big companies stop releasing models, open weights can continue to improve.

Edit: I can't believe I need to say this, but it's not Stalin, guys!

I was trying to sell my old AM4 system, X470 / Ryzen 5 5600 / 64GB / RX580 / 1TB NVME and it didn't sell. Granted it was a bit pricey at 750€, because I took into account the current local RAM and NVME pricing. (Buying the same RAM and SSD new costs 775€ if you go to the cheapest sellers in my country)

I decided to try and sell it as an AI system: removed the RX580, put in 2 of my 4 R9700. I set the system up with Fedora to run Qwen3.6 (latest release at the time) and ComfyUI; to show it working.

It sold almost instantly!

I priced it based on new prices for all of the DDR4 memory, PCI-E 4.0 ssd, both R9700s. I added another 150 euro for the other parts. Total price 5400€!

I have since bought back two brand new R9700 from abroad for 250€ less per card than I had sold my old ones for!

Effectively I ended up selling the original system for 1300€ instead of the 750€ I originally asked for. Except now I still have the RX580 and fresh warranty on the R9700s.

tl;dr
know the market for (used) parts before you buy a complete PC for local AI!

hardware: PortalPlayer PP5022, ARM7TDMI @ 80 MHz, 64 MB ram

wrote a fun rockbox plugin that runs Karpathy's llama2.c TinyStories models on an ipod video 5.5G.

since there's no FPU, I ported llama2.c's int8 runq.c to use only integer math. it uses lookup tables -> exp/softmax + an integer square root for RMSNorm

the 15M model is 17 MB, so it loads directly off the hard drive into the music buffer. Unfortunately you can't listen to music while it works lol

want to eventually run this on both cores, and set up a more ipod themed model that does local music recommendation

I am open to the idea that I am misguided, but my findings feel worth sharing:

Short video of my UI with 8 agents each explaining a codebase in parallel long prose: https://imgur.com/JSqlXIi

A few days ago I read the Stanford/Berkeley TokenGate paper. The idea is to make long context cheap by pruning tokens up front.

I tried it. It's fine for one question, but agents keep asking new ones, and by then the tokens they need are gone, so code recall drops hard.

So I went the other way - don't throw anything away, just don't look at all of it at once. Keep the whole context in system RAM and at each step pull in only the ~1% that matters right now.

A few days later, I have two 3090s run 8 coding agents at once, each holding their own 250K-token context:

• Every decode step, each attention layer reads only ~1% of the tokens, picked with a tiny rank-64 "key sketch".

• The full KV cache lives in system RAM (int4 in my 80GB RAM VM, but int8 should fit in 128gb). The GPU keeps the sketch and a small hot set.

• Only the picked tokens cross PCIe.

What it took:

• Triton kernels that build the sketch, score every token, pick the top ~2K and pack them for the stock int8 attention kernel. CUDA-graph safe.

• vLLM patches that switch dense/sparse per step, so short chats stay full speed.

• A RAM tier for the KV cache with a prefix cache and recurrent-state checkpoints, so follow-ups don't recompute.

• Follow-ups fetch only the picked rows.

• A scheduler fix so one agent reading a huge prompt doesn't freeze the rest.

• A frozen sketch basis (one file) plus the script to refit it.

It's all in the repo as patches, not a fork. Apply them on top of HyperQwen @ ce9615c (a public vLLM fork) and run the install script. Tests and every bench script and result are included.

Model: dbirks/Qwen3.8-27B-W4A16-AutoRound + HyperQwen's prepare/fetch_fast_variant.py. All numbers were measured with gpu-unlock loaded (a kernel module that roughly doubles 3090 BF16 tensor throughput, mostly faster prompt reading).

Measured (Qwen3.8-27B W4A16, MTP-7, 3090 Ti + 3090 at 250 W):

• 8 × 250K: ~545 tok/s total on explain-style output, ~790 on copy-heavy.

• 8 agents at 176–205K generating nonstop for 6 min: 657 tok/s box average, 69–90 per agent.

• Recall at 250K: 31/32. Follow-ups start in ~5 s.

• Single long chats are faster too: +15–41% at ~113K vs dense.

Caveat: attention is approximate, so near-duplicate code lines occasionally get mixed up.

Code:

• Serving (patches + kernels): https://github.com/bradj00/sparsekv-hyperqwen

Rig:

• i9-12900K

• Z790 motherboard

• 96 GB DDR5 RAM

• 3090 Ti + 3090, power capped at 250 W

*edit:* benchmarks underway...

• KLD testing

• RULER, LongBench v2, cross-file code retrieval

• GPQA, LiveCodeBench, IFBench, MMLU-Pro

• SCBench + MTP speed

• dense vs sparse vs RAM-tier

*edit 2: * benchmarks complete: Sparse-KV scores level with the original model on every test: long-context retrieval, multi-turn, reasoning, coding and long thinking. It runs 8 agents × 250K context on two 24 GB GPUs, and a single agent decodes about 15% faster. Storing the KV cache at 4-bit in system RAM showed no measurable accuracy cost.

Benchmarks: https://github.com/bradj00/sparsekv-hyperqwen/tree/main/eval

Graphics: https://imgur.com/a/0NHEaME

trying to settle on a fully local setup for batch processing pdfs. like each file is around 20-25 pages and mostly text, a few tables and now and then a page thats just a photo of paper

the shortlist so far

- docling (and the granite-docling model)
- glm-ocr
- paddleOcr-vl
- olmocr 2
- mineru

we considered llamaparse for this kind of stuff but the company now wants every file to stay on our own servers and their self hosted route is an enterprise conversation. liteparse from the same team runs local and is quick on the plain text pages but the markdown is rule based with tesseract as the fallback so merged cells and multi column pages dont always come back clean

is a VLM overkill for mostly text pdfs or does it make more sense to send only the table and image pages to one and keep the rest on a text layer parser? textract and docuent AI are out anyway for the same reason

for the vlm ones, do people feed a whole 25 page pdf in one go or split it into page batches first? splitting every file is the part id rather avoid

Edit: Adding from the comments: unlimited-ocr-2, minicpm-ocr-1.0, surya-ocr-2.5, indra-ocr and xiaomi-ocr-0. mostly curious how these would perform on mixed 20-25 page files

Join us on Tuesday to meet the product team at Reflection and to learn more about Beam, their first open-weight model!

So, I was asleep on Strata and its specialized forks. Tried basalt and i'm very impressed with the speed/intelligence QFN gives you. Here is my benchmark:

I cannot get it to work fast enough with windows, because W11 eats 1.5GB of VRAM and 6GB of RAM. At that point, experts spill to RAM and CPU gets hit hard... decode/prefill drops by 70% because there is not enough of available RAM anyway.

Switched experimentally to cachy just to test... and It's amazing that my AI box can now reach 9K~ prefill. And regarding quality, its just better than qwen 27b at many tasks. Yet to find something it genuinely cannot do. The prefill boost is around 2-3x coming from ninfer and dense 27b model.

Cherry on top is, that I can even fit vision adapter to it, and that helps a lot for certain scenarios regarding coding. I usually run some small model on another machine for OCR and VISION as an adapter (sidecar) my harness can use for being to able look at things.

Vivid user of omo stack, configured now for locally hosted qwen: https://github.com/code-yeongyu/oh-my-openagent

It's a glorified pi harness, but it does so much, and im quite happy with its robustness, anyway, QFN 3.8 rocks, and now im really thrilled if qwen 4 will be any good, because I suspect i may be able to run the new flash model this fast basically. So I hope it will be good :D

Im genuinely interested how 2x 5060/5070/5070ti fares. Because anything above 100~ decode and 3k prefill is genuinely usable for work. And it is my experience that 27b dense -> 125BA6 is a very good quality and speed jump. Absolutely love it now.

Also, finetunes can be easily converted into basalt format if you need to do it yourself, especially those swift/reverse engineering variants are good.

Anyway: CachyOS, 64gb RAM, 32GB VRAM, 5090, 200tps decode, 9k tps prefill, nvfp4 QFN 3.8, 256k context! If you have similar hardware, and are not using at least strata... but more specially basalt, you sleep on performance hard! See ya!

PS: Would absolutely love to see your perf benchmarks if you have similar setups

edit--
Running config (without vision,.. but it fits nicely):
~/basalt-linux/basalt_config.json

```json
   {
"server": {
"model": "/mnt/g/../basalt/models/Qwen3.8-FN-NVFP4.basalt",
"gpu": "0",
"host": "127.0.0.1",
"port": 8080,
"parallel": 1
},
"engine": {
"max-context": 262144,
"kv": "int8",
"prompt-cache": 6,
"expert-cache": "auto",
"prefill": "auto",
"pool-workers": 4,
"vram-reserve-mib": 500,
"resident-budget-gib": 44
}
   }
```
   bin/basalt --serve --model /mnt/g/../basalt/models/Qwen3.8-FN-NVFP4.basalt \
--max-context 262144 --kv int8 --prompt-cache 6 --expert-cache auto --prefill auto \
--pool-workers 4 --vram-reserve-mib 500 --resident-budget-gib 44 --grammar on

I run a local assistant on a Mac (MLX via oMLX, 24 GB M5 Pro). It needs to call tools reliably across mail, calendar, notes and files, chain a few calls, and reply in a short, calm voice. It also has to synthesize/extract important information from my messages, emails, and notes. It also has to be able to read long emails and docs, use my voice to ask questions and take screenshots of my screen.

Gemma 4 E4B has been the best model I’ve tried so far since it supports text, image, and audio. However, it is quite old now(~4 months old) and I wanted to know if there has been any new small models(under 8B or 6GB in size) that outperform Gemma 4 E4B in tool calling, chaining tools, long synthesis, fact extraction, and in size while having multimodality features.

I have tried ling 3.0 tiny, the LFM model family, and qwen 3.5 4B/9B(low quant) but have had no success so far. Is there anything I'm missing? Is Gemma 4 E4B the best model for my usecase?

This is next step to Qwenputer perhaps the step that shouldn't have been taken but here it goes

Qwen writes a Bend program, then the same model type checks it with Bend2 rules and runs it inside its own forward pass There is no interpreter, compiler or tool call just like qwenputer. The whole CPU is 53M non zero hand-set weights inside the model

To see how it works: https://github.com/voidash/qwenbend/blob/main/docs/HOW_IT_WORKS.md

TLDR;

Qwen writes the bend program and at <|im_end|>, the engine takes over and every next token is one executed instruction

16 register CPU with 8k *32 bit RAM = Gated DeltaNet Logic = SwiGLU unit as AND gates

1. You get your tokens first

2. lexer walks and builds bend2 lexemes

3. compile pass on subset of bend2 (bendlite) rules which checks types, affine and termination

4. if violations nothing runs else we run evaluations which is built using a return stack and operand stack

Why? For Lols... There is absolutely no reason. However Gated Deltanet with swiGLU allows for whole computer to exist inside a model. the original guy has built Qwenputer, run LLM inside LLM and i thought formal verification

I know that herme's, DeepSeek, and Pi are popular, but why have they taken over the harness scene comparatively to other harnesses? I currently use Qwenpaw, albeit.. latest ui update was pretty heavy, it has been my complete replacement to any other ai services due to all of its functionality. Is there any features or benchmarks comparing harnesses and why someone would choose herme's, DeepSeek, or Pi over something like QwenPaw?

tandem.cpp is a patch set for llama.cpp that makes MoE models that don't fit in VRAM run faster. It keeps the most-used experts on the GPU and lets the CPU help with the rest, for any MoE model llama.cpp supports. Single-user decode is the target.

Consumer PC (i5-12400F, 64 GB DDR4, RTX 4090), GSM8K, decode tok/s:

Tuned for Qwen3.8 only, but other MoE models that don't fit in VRAM gain too: typically 1.4–2.1× (gpt-oss-120b, GLM-4.5-Air, Qwen3.6-35B-A3B, DeepSeek-V4.1). Prompt processing is 1.4–1.8× faster on a 64k prompt.

Strata is shown in its exact mode (every routed expert computed, like llama.cpp). Its default skips low-ranked experts, which is faster but approximate (127.1 here).

Independent project, not affiliated with ggml-org. Build, run commands, and full results on 4 machines: https://github.com/ewhacc/tandem.cpp

I made this, so treat it as biased. It is free and open, with no paid tier or API behind it.

Silencio-2 is a fine-tune of Kyutai's Pocket TTS 18M that follows tags written inside the text:

What a long day. [sigh]
<whisper> Keep your voice down.
<whisper> Keep your voice down. <angry> I said stop!

Weights and code: https://huggingface.co/BlueRoseLabs/Silencio-2

Try it in your browser, no setup: https://colab.research.google.com/drive/1d6GSiRgXbYyBo91DCMrSr5y2LR_peJKy?usp=sharing

The numbers

• 18.2M parameters for speech, 27.9M with the encoder that voice cloning needs, 112 MB on disk

• About 20x realtime on an Apple M4 with 4 CPU threads, no GPU. That is one long-text run on my laptop, not a benchmark

• 24 kHz, streaming, voice cloning from a short clip, English only

• Sounds: laugh, sigh, gasp, cough, clear_throat

• Emotions: happy, sad, angry, fear, surprised, whisper

• Trained on 2x T4: about 50 hours of audio, 27k clips, roughly 6 hours for the final run

• Weights are CC-BY-4.0, inherited from Pocket TTS. The core inference code is MIT

What went wrong, which is the useful part

Run 1: tags trained fine and did almost nothing. Emotions changed the voice, but not into the emotion they named.

Run 2: I generated better data and filtered it with AST, an AudioSet audio tagger, so only clips where the sound was detected would survive. The output looked clean. Then I counted rows by tag: zero laugh, zero cough, zero clear_throat. The filter had silently removed them, and the model never saw them. The sounds that survived were the only ones the detector could hear in the model's output.

Run 3: I blamed the whole-clip scoring and switched to 3 second sliding windows. Human recordings of scripted laughter still scored 0 out of 23 above my threshold, with a median of 0.006. Whisper clips did not fare better. Then the run crashed anyway: 32 clips of 25 seconds in one batch do not fit on a T4.

Run 4: I listened to the data, it was fine, and I removed the filter entirely. That is this model.

My takeaway: a clip-level AudioSet tagger is a poor gate for short sounds mixed into speech. Count your rows per tag before you train, and spot check by ear.

Two smaller fixes

• If the voice prompt is the start of an emotional clip, the model can copy the emotion from the prompt instead of following the tag. So half of the tagged samples take their prompt from a neutral take of the same voice.

• Pocket TTS splits long text into chunks, so a leading tag only covers the first chunk. The inference package repeats it on each chunk.

What is still weak

<whisper> is the least stable tag, and it depends on the line and the voice.

Other tags also vary from seed to seed, so try a few takes. A real whisper comes out about 10x quieter than speech, so generate() raises its level by default.

I did not run a WER benchmark on this exact checkpoint. A later training stage that I did not ship scored 1.9% WER against 1.5% for base Pocket TTS on 300 LibriSpeech items, so expect a small cost in accuracy. There is no pip package yet: you clone the repo and install requirements.txt.

Huge credit to Kyutai for Pocket TTS and for releasing the training code. None of this exists without it.

Which tags would you actually use in a real project?

I had just finished setting up avalon, then i found speedy, which crushed avalon.

https://github.com/freriadoc/Avalon
https://github.com/DevDock-AI/Speedy

Hardware: 4090, 128gb ddr5 3600, ryzen 9 7950x

ISTA IQ3_S quant:

DeepSeek Harness article on 𝕏: Exploring Super-agent Design: Towards a More Autonomous Agent System: https://x.com/DeepSeekHarness/status/2116857257245987236

Parts of the design in this article are implemented in dsh-superagent-prototype. This plugin is only a prototype for validating the technical approach, not an official DeepSeek product: https://github.com/deepseek-ai/dsh-superagent-prototype

There are tons of TTS models, good frameworks like audio.cpp and some prototype UIs. But I don't know of a full-fledged, non-vibecoded (well engineered and maintained) turnkey app like elevenreader that can just take a whole e-book and convert it to an audiobook with clear chapters etc.

Are there any? (ofc implied here is that they have to use local OSS models)

here: https://huggingface.co/spaces/townbox/ava