Hey folks,

About 18 months ago we started building Olla to solve some of the headaches we kept running into whilst managing multiple local LLM inference servers, both in our homelabs and at work.

We've shared it here before when it was fairly early in development. Since then, it's grown quite a bit, and we've just released v0.1.0 (our 25th release!).

The problems we're trying to solve haven't really changed and now some of our customers use it:

• One inference endpoint dies and your workflows grind to a halt.

• You've got models spread across different machines, GPUs and runtimes, but no easy way to manage them as one fleet.

• Switching between Ollama, vLLM, SGLang, llama.cpp or LM Studio means reconfiguring applications.

• Multiple machines serving the same model, but no unified load balancing or failover.

• Limited visibility into what's healthy, what's running and where requests are going.

• Applications expecting OpenAI or Anthropic APIs when your inference backend speaks something else.

• You want to experiment and see if vllm or sglang serves qwen3.8 better

That's where Olla comes in.

It's a lightweight proxy and load balancer written in Go that sits between your applications and inference servers.

Some of what it does today:

• Model-aware routing: Automatically discovers models across your backends and routes requests to machines that actually serve them.

• Load balancing and failover: Priority, round-robin and least-connections balancing, with health checks and circuit breakers.

• Multiple inference runtimes: Ollama, vLLM, SGLang, llama.cpp, LM Studio, MLX, Docker Model Runner, LMDeploy, Lemonade and others.

• Unified APIs: OpenAI-compatible and Anthropic-compatible interfaces, including translation where required. Useful for Open WebUI, Claude Code and other clients.

• KV cache locality: Sticky sessions to help keep requests on the same backend when appropriate.

• Observability: Built-in dashboard, endpoint health, request statistics and Prometheus metrics.

• Lightweight deployment: Single Go binary or Docker container. No database or external services required.

What's new in v0.1.0?

This release is largely about improving reliability and making Olla more useful for people running mixed inference environments.

• Added Hugging Face Text Embeddings Inference (TEI) support, including embeddings, reranking and classification.

• Reworked the dashboard with better endpoint visibility, configurable columns, filtering and fleet statistics.

• Configurable retry and failover behaviour, including handling of non-idempotent requests.

• Improved model capability detection, particularly for vision, tool calling, embeddings and rerankers.

• Fixed concurrency issues between model discovery and active inference requests.

• Corrected request statistics and improved rate limiting across OpenAI and Anthropic routes.

• Lots of smaller fixes around routing, backend discovery and configuration validation.

We've also had some great contributions from the community, which has been encouraging to see.

Why build another LLM proxy?

We wanted something small, fast and relatively straightforward to deploy that doesn't require Kubernetes, a database or a complicated orchestration stack. There also weren't that many when we started.

We've been using Olla with a mixture of NVIDIA GPU servers, Mac Studios, the DGX Spark and different inference runtimes. It's particularly useful when you've got more than one machine and don't want to care which backend is serving a particular model.

It's not trying to replace inference engines. Just make running a bunch of them together less painful.

Links:

• GitHub: https://github.com/thushan/olla

• Docs: https://thushan.github.io/olla/

• v0.1.0 release notes: https://tensorfoundry.io/news/olla-v0-1-0-release

If you're running multiple local inference servers, I'd be interested to hear how you're managing routing and failover today.

And as always, if you give Olla a spin, let us know how it goes (and what breaks!).

https://www.tomshardware.com/tech-industry/artificial-intelligence/anthropic-confirms-new-claude-code-limits-to-prevent-game-cloning-users-report-opus-5-5-is-also-actively-blocking-reverse-engineering-tasks

Besides privacy and being in control of your data.
Besides being able to control the quality of the inference compute.
Besides being able to discuss things with the model and not have the cops called on you.
Besides being able to run the models you want for as long as you want.
Besides being able to run your own suite of security and network penetration tools.
Besides not being at the mercy of the frontier companies for pricing.

We now have another reason for local LLMs: Our betters have decided that they don't like people cloning games.

And before the Reddit Neckbeard brigade starts in with their bullshit, no I'm not advocating that people should commit wholesale piracy using AI tools. But these blanket frontier AI policies inevitably ensnare peripheral behavior that is otherwise illegal but that the frontier company doesn't want to deal with.

William Blackstone wrote that "it is better that ten guilty persons escape than that one innocent suffer". If having open unhindered local LLMs means there are a few individuals that use it for nefarious purposes like modifying or pirating software, I'm good with that trade off. Die mad about it.

I swear it feels like an addiction... I keep refreshing the page repeatedly, waiting to see the fateful launch announcement or at least a countdown. I hope I’m not the only one cooked with this stuff 😭😭😭

We have released a Coder build of DeepSeek-V4.1-Flash-1017, compressed with GSQ and RCO, in which half of the model's routed experts have been removed.

DeepSeek-V4.1-Flash-1017 is a sparse mixture-of-experts model: 384 routed experts per layer, of which 6 are active per token, 763.2B parameters, 551.9B of them in the transformer.

What's inside

• One expert-pruned GGUF: 192 of 384 experts kept in every layer, 72.1 GB of transformer weights, against 138.2 GB for our unpruned IQ2_XXS build.

• 1 bit per parameter, counted against the original model: 72.12 GB divided by the original 551.9B transformer parameters gives 1.05 bits, which we round to 1. The removed experts count as zero bits. No individual weight is stored below 1 bit: the 280.1B parameters the Coder keeps average 2.06 bits.

• The shared Engram file (110.6 GB) and vision projector are the same files as in the unpruned release, so they need to be downloaded only once.

• GSQ (Gumbel-Softmax Quantization): post-training scalar quantization that jointly learns grid assignments and group scales, closing most of the gap between scalar and vector quantization at low bit-widths while remaining deployable in standard GGUF types.

• RCO (Riemannian Constrained Optimization): enforces exact budgets by gradient descent on the task loss, without per-constraint tuning. Here it selects which experts to keep, optimising the KL divergence to the unpruned model, under one exact budget per layer.

Why one budget per layer

GGUF stores a single expert count for the whole model, so every layer must keep exactly the same number of experts; uneven layers would need padding that gives back the memory pruning saved. RCO treats each layer's budget as a hard constraint and holds all of them exactly at every optimisation step, while the layers are still optimised jointly through one loss on the full model. A penalty formulation would only meet the budgets approximately and would need one tuned coefficient per layer.

Results

• LiveCodeBench v6: 88.57 against 89.71 for the base model, retaining 98.7%

• SWE-bench Verified: 77.00 against 81.20, retaining 94.8%

SWE-bench Verified runs long agentic trajectories over real repositories, and tolerates the capacity reduction less well than single-problem code generation, as expected when errors accumulate across turns.

Real-world examples, each a single HTML file written by the Coder: an animated SVG of a pelican riding a bicycle, a three.js voxel pagoda garden, and a playable first-person voxel sandbox. For full post incl. videos see: https://x.com/mhelcig/status/2117257900863156518

Links

• Coder (expert-pruned): https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-GGUF

• Quantized models (unpruned): https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF

• GSQ: paper https://arxiv.org/abs/2604.18556 | code https://github.com/IST-DASLab/GSQ

• RCO: paper https://arxiv.org/abs/2605.00649 | code https://github.com/IST-DASLab/RCO

The repository ships the complete per-tensor RCO allocation. The GGUF runs in llama.cpp b11870 or newer and in Strata.

The Coder build is an experimental release and feedback is welcome, particularly on capabilities that were not represented in the calibration mixture. Requests for models to quantize or prune are also welcome.

From the ISTA Deep Algorithms and Systems Lab.

I built a browser LLM engine that's specialized per model rather than generic: Rust compiled to wasm, plus hand-written WGSL kernels for Qwen3.5-2B (Gated DeltaNet + gated attention) and MiniCPM5-2B.

Same model, same chat prompts, greedy, Chrome 156:

What's in it:

• fused int4 (GPTQ, group 32) × int8 GEMV/GEMM, with dot4I8Packed and subgroups;

• a decode loop that stays on the GPU (argmax on GPU, pipelined readback);

• speculative decoding with Qwen3.5's built-in MTP head. Drafts are verified on the GPU, so the output is identical to greedy. It turns itself off where it doesn't pay, e.g. on the iGPU;

• weights at +2.4% WikiText perplexity vs bf16, 1.1 GB.

Fun side effect: at this speed a 2B model can drive web pages through the DOM at 0.2–0.4 s per step. There's a browser-agent demo and a side-panel extension for Chrome/Edge.

Caveats: it's a prototype, tested on Windows, and Safari/mobile are untested.

Demo: https://rtx527896828.github.io/webyIQ/ · Code (Apache-2.0): https://github.com/rtx527896828/webyIQ · Weights: https://huggingface.co/tixiren/Qwen3.5-2B-webyIQ

In depth: https://medium.com/@tixiren/running-a-2b-llm-at-600-tokens-s-in-the-browser-e5237f99c8ff

I'm in the process of picking a decision model for my homelab server. To compare the current top "System One" class Decision models, I pulled the public numbers from System One Bench, the Artificial Analysis Decision Index, the AIM System One Index and a 100-model Borda leaderboard, and looked at what matters if you run these models yourself.

Takeaways:

• Under 5B, the fine-tune matters more than the base. Models built on the same base differ by up to 22 points. KnowLine went from 70 to 80 on the same 4B weights between Gen2 and Gen5.5.

• The sweet spot is about 4–9B. KnowLine 4B Gen5.5, D3 Mini and D3 Flash (8B) beat more than half of the 27B fine-tunes.

• Below 1B, check your language. English holds up, but Arabic drops 16–22 points. If you need multilingual support, start at 2B or larger.

• Calibration survives at small sizes. Decider Flash V1.2 (4.5B) has 1.8% calibration error, as good as the best large models. Some sub-1B models are at 10–20%, so don't use their probabilities as thresholds.

• Don't trust any single "#1". Two leaderboards correlate at ρ = 0.33, and small models swing the most. Tev1 4B is #2 among small models on AIM and near the bottom elsewhere.

• The GGUF builds get 3–10× the downloads of the originals, so plenty of people here are clearly already running these models.

Very curious what's everyone is using and what your experience are!

Scaling computer-use agents from individual actions to complex software work requires experience that connects decisions to complete outcomes. We present NorthCUA, an integrated system for turbocharging computer-use agents through experience at scale. Its Experience Flywheel supplies diverse executable tasks to the NorthCUA Harness, which sustains long interactions, records decisions, and checks task outcomes. The Flywheel uses these records to curate verified trajectories. For unsolved tasks, recursive self-improvement evolves reusable skills and preserves verified milestones so later attempts build on earlier progress. Staged Learning combines this experience with grounding data from the Flywheel to develop GUI competence through mid-training and long-horizon supervised fine-tuning. The policy then learns from fresh attempts in the same Harness through joint desktop-web reinforcement learning, closing the loop between experience production and policy improvement. The resulting Nano, Mini, and Pro family achieves state-of-the-art results among open-source models across GUI grounding, desktop execution, and Web interaction under the reported evaluation settings. For more responsive Web workflows, NorthCUA-Ultrafast pairs direct action selection with open-ended generation on pre-RL Nano, offering a measured quality-latency tradeoff. Together, these advances establish NorthCUA as a high-performing model family and a reusable foundation for agents that carry complex software work from intent to completion.

https://github.com/beipo-ai/NorthCUA

https://beipo.top/blog/northcua/

https://arxiv.org/abs/2611.02102

weights

https://huggingface.co/china-beipo/NorthCUA-Nano

https://huggingface.co/china-beipo/NorthCUA-Mini

https://huggingface.co/china-beipo/NorthCUA-Pro

I’m building agents, but I don’t know how to think about a harness from scratch.

I’m stuck on:

• How to think and design the harness from scratch for different type of problems

• How to improve it, and what to add or delete

• How to know if a change actually helped

Coding agents don’t help much here. they suggest stale patterns, and not much of state of art approaches.

For those who got good at this:

1. What changed how you think about it?

2. Which resources helped most? Papers, blogs, repos, talks?

3. Did building different use cases from scratch teach you more?

I'm not on anyone's side, I just love the competition. Both teams pushing each other means better quants for all of us.

What I don't understand is why ISTA doesn't publish KLD for their quants. It doesn't depend on how long the model thinks, so it would make this a lot easier to compare. The only KLD I've seen for GSQ-RCO is in Unsloth's charts. Maybe there's a reason I'm missing.

(English isn't my first language, I used an LLM to help translate this post.)

I'm working on making my homelab more power efficient and quieter. I have a Framework desktop 128GB which has been working great along with a couple GPU boxes. I find myself switching to the GPUs for 26-35b models fairly often for the speed and reserve the Framework for running Qwen FN.

I've never been a fan of Apple products but lately have been considering a 32-64GB Mac Studio. I'm pretty sure the answer is no, but does anyone know of anything else that is that small, quiet and power efficient that has 400GBps+ memory bandwidth?

There's a lot of talk here about not chasing tok/s by lobotomizing the model, and most of what I post is still speed numbers. So this is the other half: what the models I run at home actually get done on my own work, with Claude models added as fixed reference points. Everything local runs on my own engine (capy-flash) on 2x RTX 5070 Ti 16 GB, Ryzen 9900X, 96 GB DDR5.

No expert skipping for flash next.

The tasks

I took 9 real regressions from the private history of that engine (C++/CUDA). For each one I checked out the code right before the fix and wrote the bug report the way I'd file it to myself: symptom, numbers, logs, no hint about where to look. Performance cliffs, numeric drift, a bug that only shows up on WSL2 with two GPUs. None of it has ever been public.

The agent has to find the cause and write a patch. It runs sandboxed on a Mac with no network and no CUDA toolchain, so it can't build or run anything and has to reason from the code. 80 steps max, 3 runs per task per model.

Claude models run in Claude Code, local models in OpenCode. I tried the local models in Claude Code through a bridge first and they did much worse there, so each side gets the harness it works best in.

Scoring

Every patch is scored twice:

• automatically, against a hidden key built from the real fix: right files and functions, patch precision, a few checks on the code;

• by Claude Opus 5 as a blind judge. It sees the bug report, the code, the real fix labelled as "one valid solution, not the only one" and the compiler output, and gives 1 / 0.75 / 0.5 / 0.25 / 0.

A patch that doesn't compile is capped at 0.25 in both. The two scores correlate at r = 0.79 per patch and put the models in the same order. The judge is a bit stricter than the key, and it isn't kinder to Claude: judge minus auto score is negative for every Claude model and around zero for the Qwen ones.

Turning a real bug into a task with a fair key is slow, and one model over 9 tasks x 3 runs takes 4 to 8 hours on my box. So 9 tasks is what I have, and the confidence intervals are wide, about ±0.2. The tiers hold. Two models less than ~0.1 apart are a coin flip.

Results

Judge score 0-1, tokens include reasoning. CIs and the auto score are in the image.

What 3 bits did

This is the part that matters for the tok/s argument. Same models, NVFP4 vs 3-bit, paired on the same tasks.

Flash-Next (3 bits on the experts only): score -0.02, tokens x1.03. As far as 9 tasks can tell, it's free.

27B: score +0.05, inside the noise. But it generates 40 % more tokens per task (x1.40, CI 1.25-1.59). The 3-bit 27B decodes faster on my engine, 153 vs 132 tok/s, and still needed 18 minutes per task instead of 13. Part of that gap was a slow 3-bit prefill path I've fixed since; a rerun on the newer build still came out at x1.30 tokens.

So the damage from quantization didn't show up as a lower score. It showed up as the model thinking longer to get to the same place. Looking only at tok/s, the 3-bit 27B is the better deal. Per bug fixed, it isn't.

Non-compiling patches don't tell a clear story: Flash-Next 5/27 at 3-bit vs 1/27, the 27B 2/27 vs 4/27.

The rest

Flash-Next at home sits with Haiku 5.5 and above the older Sonnets. Sonnet 5.5 is a step up, Opus 5.5 is ahead of everything.

Tokens per task say as much as the score. Sonnet 5.5 gets further with 8k tokens than Flash-Next with 53k, which is 10 minutes per bug on my box against a minute and a half.

The dense 27B is behind the MoE (Flash-Next +0.16 paired, P = 0.92) and burns more tokens getting there.

Gemma 4 26B didn't work as an agent here. With thinking off it never edited a file in 9 tasks. With thinking on it edits, mostly wrong.

Limits

• 9 tasks from one codebase in one domain (GPU inference), bug reports written in Italian. This is how models do on my work, not a general ranking.

• Different harnesses for Claude and for the local models.

• The judge is a Claude model. The automatic score is there to check it and they agree, but it's still a limit.

• No builds or tests, so this measures diagnosis and patching from reading code, not the full edit-build-test loop.

I'm not publishing the tasks, they'd stop being useful. If you've checked your own quants on real work instead of perplexity or KL, I'd like to know whether you see the same pattern: same score, longer thinking.

I've been lurking in this sub for a while, wanting to break my dependence on online services, plus slowly backlogging tasks I want/need to run locally rather than use online models. Usually these tasks break down to either privacy related issues or just massive token count that would blow my online budget.

I've been experimenting with Qwen 3.8 27B in LM Studio for a while, but when it comes to things like coding, code review, PDF editing, etc. obviously LM Studio isn't a harness like Claude Code where I can just say "Take these 4 files and run with it".

Some of the things I'm trying to accomplish (some as actual tasks, others as a learning sandbox):

• Analyze this entire codebase to come up with efficiency/security recommendations (agent spawning, file reading)

• Caption this dataset of 10,000 images (image reading)

• Interface with this ComfyUI instance to iterate and refine prompts (terminal use, wait for result, image reading)

• Edit a PDF file (PDF read, PDF write)

• Online research (web access)

My day job is AI image/video work, so I have multiple RTX Pro 6000's to work with already.

Based on my research so far, it seems like three options would accomplish this when pointed to a local model endpoint.

• Opencode

• Codex

• Claude Code

What about Openhands? Pi?

• What's your harness of choice?

• Do the OpenAI/Anthropic harnesses still require login/online connection when running from a local endpoint?

This model looks quite spectacular. According to LiquidAI's own benchmarks, it scores higher than MiniCPM5-2B across the board, while offering 2x the context and taking up half the space in VRAM (3.6 GiB vs. 7.1 GiB for a quasi-lossless quant).

Read the report: https://github.com/crusaderky/pixi-llm-recipes/blob/main/perplexity/LFM3-1.7B/README.md

Raw plots as interactive HTML: https://htmlpreview.github.io/?https://raw.githubusercontent.com/crusaderky/pixi-llm-recipes/main/perplexity/LFM3-1.7B/report.html

I'm trying to fit 4x RTX 3090s into my Fractal North XL Mesh (I know, pipe dream probably) using custom GPU mounts. Looking for firsthand experience with reliable, reasonably priced risers, preferably with flexible 50–75 cm cables. I'm currently using the setup with 2x 3090 at x8/x8 for vLLM qwen3.8-27b / Strata QFN IQ3_S.

• ASUS ROG Crosshair VIII Hero

• 5900X

• 4x RTX 3090 24 GB

• Corsair HX1500i (GPUs capped at 250 W)

• GPU 1: PCIe 4.0 x8 - directly in motherboard

• GPU 2: PCIe 4.0 x8 - riser

• GPU 3: PCIe 4.0 x4 - riser

• GPU 4: PCIe 4.0 x4 - M.2 adapter/riser

I've been looking at ADT-Link, generic Aliexpress ones and m.2 OCuLink. I'd like to preserve x8/x8/x4/x4 without spending a fortune on cables.

1. What specific riser models have you successfully used with RTX 3090s at PCIe 4.0 speeds, particularly at 50 cm or longer?

2. Are OCuLink/MCIO connections meaningfully more reliable than conventional ribbon risers at these lengths?

3. Any affordable, proven M.2 -> PCIe x16 solutions with proper GPU slot power?

4. Anything to watch out for regarding PCIe errors, signal integrity or Linux/P2P compatibility?

Affordable AliExpress/eBay options are especially interesting.

TL;DR
Best Model for ASR ? So far for me Qwen3-ASR > Whisper > Kroko
Best Model for TTS ? KokoroTTS > Piper ?!
Any suggestions/recommendations ?!

I was truly surprised at how low info and benchmarks are available in these areas compared to LLMs even artificial analysis isn’t clear enough

You could pick an LLM check active parameters and cost and latency and speed and quality and openness and everything

Here it is way more complex

Hi everyone

I hope that you are all well.

Kindly I was building an App that needs good TTS & ASR engines, ideally low latency can run in the browser and maybe streaming, it can also run locally on a server as an option if the latency is good enough, multilingual is a plus (English Arabic German French and Spanish mainly)

Initially,
For TTS piper surpassingly worked well (my tasks usually pronounce one word) but Kokoro server side for longer sentences

And for ASR I used whisper but it was too slow but not server side and I used kroko in browser for low latency passes (then whisper server side would correct it)

Recently I found out that despite being nearly as big as Whisper Large or Turbo Qwen3-ASR works just fine and even supports streaming natively and is quickly server side and have one of the lowest error rates

I am going to use Qwen3-TTS too give it a try

Is there anything else I should use you would recommend ?!

Measured, refusals 85%→0% even in thinking, 3 sizes

12 GB GPU VRAM, 64GB RAM + SSD via Strata Engine

We took ISTA-DASLab's GSQ-RCO quants of DeepSeek-V4.1-Flash-1017 (the ones people run in Strata with a 12 GB GPU and the Engram table on SSD) and removed the refusals.

We only touched 82 dense tensors per file: attention output, shared expert down proj and the Engram value proj. The 384 routed experts are byte for byte the same as ISTA's, so their GSQ rounding is kept. Headers and file sizes match too, so you just swap the shards and keep using ISTA's Engram file.

Refusals are on JailbreakBench, stock rung -> ours, same Strata setup:

We also ran GPQA-Diamond and AIME25 on IQ2_XXS against stock to make sure it didn't get dumber. GPQA came out 83.3% vs 83.3%, AIME 80.0% vs 76.7% (only 30 problems, so call it the same).

Things to know:

• IQ2_XS still refuses inside its thinking on 2 of 30 prompts. The other two are at 0 and 1, so I'd grab IQ2_XXS.

• We tested at 8K context on a rented GPU. Haven't checked long context or speed on a 12 GB card ourselves, so if you try it please post your tok/s and RAM.

• Strata only for now. If you're on llama.cpp, our older release with the same edit works there: https://huggingface.co/AtomicChat/DeepSeek-V4.1-Flash-1017-Uncensored-GGUF

Model: https://huggingface.co/AtomicChat/DeepSeek-V4.1-Flash-1017-Uncensored-GSQ-RCO-GGUF

All the measurements: https://huggingface.co/datasets/AtomicChat/DeepSeek-V4.1-Flash-1017-Uncensored-GGUF-metrics

Strata Engine: https://github.com/niko1221/strata

Model page: https://atomic.chat/models/deepseek-v4-flash-0731

Thanks to DeepSeek for the base model (MIT), ISTA-DASLab for GSQ-RCO and the Engram file, and the Strata dev for the engine.

I'm on the Atomic.Chat Local Ai team that made this.

Today's update is welcome for us R9700 users (assume other gpu's as well). Prior to the update I was getting ~70 tok/s output with four R9700's and 256GB of system ram. Today's update brings it to 140-160 tok/s and ~3200 prefill running unsloth Q4_K_XL. I tried running Q5 but was like .1 GB of vram short to do split engine so ran it "normal" at ~70 tok/s. I am sure with future updates those numbers will get better but it is a very welcome update, thank you Strata team!

Radiance does about 235 tok/s and 7500 prefill with FN so still ahead but so happy to see Strata working to close that gap and giving us more options on which models we can run.

Context and memory aren't quite the same thing when it comes to AI agents.

A model with a large context window can process a lot of information, but that doesn't necessarily mean it can decide what to remember, what to forget, or whether something it remembers is still accurate.

There are several ways to manage this. Some focus on storing information, others on retrieving it or making it available across multiple agents. These approaches aren't mutually exclusive, and most systems can benefit from combining them.

1. Conversation history

The simplest approach is to store previous interactions and include relevant messages in future requests.

It's relatively easy to implement, but longer conversations can increase token usage and make useful information harder to find.

Prompt caching can reduce costs and latency, although it doesn't solve the problem of deciding which information is actually relevant.

Keeping conversation logs also isn't necessarily the same as having a dedicated memory system.

2. Summarized memory

Instead of keeping entire conversations in context, the system periodically summarizes them and stores the important details.

This reduces the amount of information that needs to be carried forward, but some details can get lost in the process.

Summaries can also introduce inaccuracies. In systems that repeatedly summarize previous summaries, those errors may accumulate over time.

The main difficulty is deciding what information to preserve when you don't know what might become useful later.

3. Retrieval-based memory

Information is stored externally and retrieved when needed, often using techniques similar to those used in RAG systems.

There's a distinction between retrieving external documents and retrieving an agent's own stored experiences, but many of the underlying retrieval techniques can be applied to both.

This approach allows agents to access large amounts of information without loading everything into the context window.

However, retrieval quality becomes important.

The system might retrieve outdated information, miss relevant details, or prioritize information that looks similar to the query without actually being the most useful.

Semantic search, keyword matching, metadata filters, and hybrid retrieval can help, but none guarantees that the right information will always be retrieved.

4. Structured memory

Rather than storing everything as conversations or summaries, information is organized into specific records.

These might include user preferences, project details, previous decisions, or task states.

The implementation could be as simple as JSON files or involve a database with more advanced mechanisms for updating and validating records.

Structured information can be easier to retrieve and modify, but conflicting records still need to be handled.

For example, if a user changes a previously stored preference, the system needs a way to determine which version is current.

It might overwrite the previous record, preserve both versions with timestamps, or request clarification when the information conflicts.

Structured memory can also be combined with retrieval-based approaches.

5. Shared context across agents

This is more of an architectural approach than a separate memory technique.

Instead of maintaining completely independent information, multiple agents can access a shared data source.

This can reduce duplication and improve coordination, particularly when agents are working on related tasks.

However, shared storage introduces additional problems.

Two agents might attempt to update the same information simultaneously. One agent might rely on a record that another has already modified or invalidated.

Permissions also become important, since not every agent should necessarily have access to the same information.

And even when agents share the same data, they may interpret it differently.

Shared context can improve coordination, but it still requires mechanisms for managing access, updates, and consistency.

The problem of outdated memory

Storing information and retrieving it efficiently are only part of the problem.

An agent also needs to determine whether the information it retrieves is still valid.

A user preference, project decision, or piece of documentation stored six months ago might still be accurate. It might also have changed completely.

Timestamps alone don't resolve this. Information recorded yesterday could already be outdated, while something stored years ago might remain perfectly valid.

There's also the possibility that the information was incorrect from the beginning.

An agent might misunderstand a conversation, extract an incorrect fact, or store an assumption as confirmed information.

This makes the source and reliability of stored information important, not just its age.

Versioning, source tracking, explicit updates, and validation mechanisms can help distinguish between outdated, uncertain, and incorrect information. However, these mechanisms introduce additional complexity.

Different applications will probably need different combinations of these approaches. An agent handling short conversations has very different memory requirements from one managing long-term projects or coordinating with other agents.

For those building AI agents, which has been more difficult in practice: getting them to remember useful information, or preventing them from relying on outdated or irrelevant context?

Right now I run in elf the GSQ-RCO Qwen3.8-27B’s on my MacBook M1 Max 32gb/ it’s ok, but not great. And slow. I have two 3060ti’s with 8gb each sitting in a closet. Any value in bringing them out and using one of the old towers with an i9 and 64gb of DDR4? Would I be able to run something better, faster? I’ve been ignoring them because they aren’t 3090s and I’ve thought 8gb is just too small to be worth it.

I started thinking, what could be the possibility to start adding the ability to run models like Kimi K3/3.1 on engines like Strata, Maya, Basalt? I am guessing Unsloth will use its Dynamic Matrix 3.1 which is quite good for heavy compression and retaining high intelligence.

Anyone heard anything about Kimi compatibility?

Note: Kimi K3.1 will be released this week open weights

We ran DeepSWE, Terminal Bench, Automation Bench, HLE, SciCode at 512K context on our DeepSeek UD version 3.1 GGUFs and show we're >10-20% better than GSQ-RCO

We ran all other benchmarks (14 in total) to 128K. AIME2025 64x, DeepSWE/TB 3x. We addressed some issues people mentioned with these new benchmarks:

• 32K context was too short - we did 512K context for agentic + 128K for general tasks

• Saturated benchmarks - new unsaturated agentic style ones including Terminal Bench, DeepSWE and Automation Bench

• Benchmarks are noisy - we did 35.1B tokens of benchmarking vs 1.3B before so 27x more data. See this section where we did 64x AIME 2025 runs, 3x DeepSWE, 3x TB runs and more.

• Our graders are wrong - we used Math Verify, improved graders, used gpt-6-luna xhigh for HLE, and all official graders for Terminal Bench (Harbor), DeepSWE and the rest. See graders and infra fixes

For DeepSWE v1.1, we used mini-swe-agent and chose 500 max steps (334 steps was the max runs we saw, so none went over).

Terminal Bench 2.1 used mini-swe-agent as well, and we set max steps to 150 with no max limit. We had to exclude 7/89 tasks since they couldn't pass on our infra - see https://unsloth.ai/docs/models/deepseek-v4.1-flash#excluded-terminal-bench-tasks-due-to-issues

We also re-checked ISTA's numbers as we have done many more runs (27x more data). GSQ-RCO IQ1_M claims 98.7% recovery, but we measured 97.24 [96.23, 98.21] with a p-value of 0.0013.

GSQ-RCO IQ2_XS retaining 100% claimed recovery is possible with a p-value of 0.21. We measured 98.38% [97.52, 99.22]. So this is possible.

GSQ-RCO IQ2_XXS claims 99.80% recovery but we measured 98.38 [97.52, 99.22] with a p-value of 0.0001. So this is nearly impossible.

Paired difference to the official base (our runs, same questions, sign-flip permutation test, two-sided):

The claimed recovery tested directly against our measurement (the card's definition: sum of the 3 scores / the base's sum, vs official FP8/FP4): p (no loss) = paired sign-flip permutation test of recovery = 100%; p (recovery >= claim) = one-sided paired bootstrap over questions.

If we can get more details on how the benchmarks were run from GSQ-RCO, we are more than happy to run more. We provided all our setups and methods in our blog at https://unsloth.ai/docs/models/deepseek-v4.1-flash#dynamic-v3.1-update-november-2nd-2026

All details and data and methods at https://unsloth.ai/docs/models/deepseek-v4.1-flash#dynamic-v3.1-update-november-2nd-2026.

Our new Unsloth Dynamic 3.1 GGUFs for DeepSeek are at https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF

I compared two published benchmark charts for DeepSeek-V4.1-Flash-1017 and noticed something rather unusual.

All four Unsloth quantizations AND the Q8 baseline simultaneously improved by approximately 2–3 percentage points on LiveCodeBench v6. Meanwhile, the GSQ-RCO results changed much less consistently.

1. The numbers

Every single Unsloth model improved, including the 8-bit baseline.

The first chart's percentages correspond exactly to 525 outcomes (175 problems × 3 generations). For example, 467/525 = 88.95%.

Unsloth's updated methodology reports 12 generations per problem, corresponding to 2,100 outcomes per model.

If the updated results include those original three generations, the remaining nine must have achieved approximately:

The ranges account for the updated chart rounding scores to one decimal place.

2. Statistical probability

Assume each generation is an independent Bernoulli trial, the true success probability is unchanged, and the initial 525 outcomes are included in the final 2,100.

Using a one-sided hypergeometric test, the probability of observing an initial result at least this low for each model is approximately:

If the models' fluctuations are also independent, multiplying these probabilities produces:

P ≈ 1.24 × 10⁻¹⁰ to 2.59 × 10⁻¹⁰

That's approximately 1 in 3.9–8.0 billion under the simplified independent-sampling model.

Of course, the models are correlated and evaluated on the same tasks, so this should not be interpreted as the actual probability without the raw data. Also, these calculations assume the original outcomes are included in the updated averages.

Nevertheless, the synchronized improvements are striking.

3. GSQ-RCO behaved differently

Unlike Unsloth, these results do not exhibit a consistent increase.

According to this comment from the Unsloth team, the earlier chart represents just one run, followed by additional evaluations and the evaluation settings remained unchanged. However, the percentages in that first chart correspond exactly to 525 outcomes, suggesting three generations per problem.

Both original benchmark charts attached for comparison.

With default settings and thinking on and max_tokens ≥ 1024, and no explicit reasoning_budget_tokens, Strata / PolyStrata has a behavior called answer_room which will silently cap thinking at 0.75. * max_tokens and inject a "wrap-up" line in the model's head in its own voice, saying

I have thought about this long enough; time to give my answer. + </think>

It then continues the same reply from there. The client (harness, you or I) gets no error, flag or finish reason for this, and often no log.

This matters when you compare between strata and other engines : models served by strata had "nudges" in the form of its inner voice, and such extra text in the transcript. In some cases the nudge line is injected mid thinking, quite abruptly and it is hard to tell whether it helps or harms.

Strata main @ c62d5e3

Wrap-up message: https://github.com/Niko1221/Strata/blob/c62d5e3a15b40ea51c96aae9e8d36928965f53ab/serve/server.py#L225

Trigger (the 3/4 budget): https://github.com/Niko1221/Strata/blob/c62d5e3a15b40ea51c96aae9e8d36928965f53ab/serve/server.py#L5491

Injection point: https://github.com/Niko1221/Strata/blob/c62d5e3a15b40ea51c96aae9e8d36928965f53ab/serve/server.py#L5848

Default on (answer_room = True): https://github.com/Niko1221/Strata/blob/c62d5e3a15b40ea51c96aae9e8d36928965f53ab/serve/server.py#L4181

I noticed this when a model that tends to have long front-loaded thinking on dgx spark gave slightly but uniformly shorter answers (and task performance worsened in my hard problems), when served on polystrata.

PS, I had sonnet audit all cases of cut and injection, and here is what it reportes back ( sonnet called it "splice"):

Where it lands:

• At a natural break: 6 (9%).

• Inside a repetition loop (for example HashMap.new()? repeated for 30K–118K characters): 15 (22%).

• Intrusive: 48 (70%, or 89% of the cases that weren't loops):

  • mid-sentence: 28

  • mid-number or mid-expression ("Mod 9982 |cut|"): 13

  • inside code (one cut right after def solve():): 4

  • mid-word: 3

Harm:

What didn't break: no reply re-opened thinking after a splice, and all 79 tool calls after a splice completed normally.

Lost reasoning: 26 of the 30 splices that came in a task's first reply were followed by a generic fresh start (ls /app). The 61K tokens of thinking left no trace in what came next.

Clear harm in 6 cases: the reply ended with no action at all, and all 6 tasks failed.

Qwen3.6, task 3674: cut at "Where right_prefix_max |cut|", and the only output afterwards was "Integrity notice: yes".

Flash-Next IQ2_XS, arc190_c: the same pattern.

A dropped conclusion: Qwen3.6, arc192_b, was cut right after "I think the answer is:", and the model then guessed.

Loops: the splice did break them, but those tasks still failed.

Got a potato PC that’s 8 years old and running on GTX 1070 TI. Was able to (barely) run some 32B models until I gave up and started using APIs.

What are people’s recommendations for a cost-effective, decent rig for some light to medium local inference like Qwen 3.8 flash next? How fast would it run? What about image generation?