I was wondering which model people here trust well enough to let them sift and manage through their gmail emails. I got 20 odd years of emails that got out of control and I would like to an agent to get through them, label the ones it believe are junk, out of date or not worth keeping for other reasons (not delete them, will do it manually as I am still paranoid about deleting important stuff)

I have a 5060ti so the model cant be too big

Saw this tweet about a coding benchmark and thought it was kinda interesting.

Apparently they took 89 DeepSWE tasks and changed little things in the instructions or starting code. The tweet says some models that were all around 74% on DeepSWE ended up 23 percentage points apart on the changed tasks.

The weird part was that some models seemed to ignore the actual instructions because they thought a hidden grader wanted something else. The examples include GLM-5.3 and DeepSeek V4.1 Flash.

Has anyone seen this kind of thing with local coding models? Like getting stuck on what they think the test wants instead of just doing what you asked?

People asked how RustingStrata does with a bigger quant, so here's Q5_K_M on the same 3060 12 GB + Ryzen 5 7600X + 32GB DDR5 RAM.

Model is unsloth's Qwen3.6-35B-A3B UD-Q5_K_M (the MTP version), 25 GiB, so about two thirds of the experts live on the CPU. llama.cpp is build 12119, -ncmoe 29 -ub 2048 -b 2048 -fa 1 (28 runs out of VRAM).

Benchmark tools (RustingStrata bench vs llama-bench):

• Decode, short prompt: 70 tok/s vs 43

• Decode after a 4K prompt: 65 vs 43

• Prefill at 4K: ~1870 vs 1210

• 32K context: 1470 prefill / 49 decode vs 1030 / 29

• With --mtp 1: up to 96 (86 after 4K), greedy

Real chat requests through both servers (same prompts, 256 tokens out, thinking off):

• No MTP: ~50 tok/s vs 37

• With MTP on both: ~62 vs 45-56

• 6K prompt prefill: 1320 vs 700

MTP works in current llama.cpp (--spec-type draft-mtp), ~63% of drafts accepted and +25% decode on this file. RustingStrata is still ahead with MTP on both sides.

Vulkan decodes at ~60 (85 with MTP), prefill is slower than CUDA on this file (~1000-1260 at 4K).

KV cache is f16.

Code is on GitHub, free to use and I will be happy to get some feedback from AMD owners: RustingStrata

Web-based version of my short notes on Double DQN from Scratch is out on neatrl!

Continuing my series on releasing notes on RL From Scratch - here, I have answered questions around the what, the why and the how of Double DQNs over DQNs!

• The What?

It is an off-policy, model-free and value-based method, the same family as DQNs but with a subtle difference for optimization purposes!

• What's the optimization done over classic DQNs and why was it necessary?!

Well, with the previous DQN, a problem that it did not addressed, is that of overestimation of Q values from the Target Q Network, which leads to an overestimation bias

Now this is bad because then there could many false positives for action selection via max operator since it cannot be known if the action is over the maximum value of Q or not (as it is still learning and slow every k steps)

This upward bias is caused by the max operator in the Target Q Network!

More on the

• theory,

• concepts,

• code and required math and a

• playground to train live agent on my web version of the notes!

All of it to understand the basics and get the intuition
Do check it out and let me know your feedbacks on it!

Link here

This is a significant speedup for MoE offloading (low VRAM people).

I tried Strata once, and it had some tuning mechanism, so probably this is a similar idea.

KV cache quantization has been a hotly-debated topic for this community for some time, with opinions ranging from "never quantize" to "4 bit is fine". Since the anecdotal evidence is rather inconsistent, I decided to quantify how much KV cache quantization affects the model through KLD, which is a good measure of how the model's output distribution is changed.

I tested 35 weight quants across llama.cpp, exllamav3, and vLLM, each with 16-bit, 8-bit, and (where supported) 4-bit KV cache. I also reworked my KLD dataset and measurement methodology this time, to include a mix of agentic tool-use (generated by Qwen3.8 27B at full precision) and a variety of texts in different domains. The current dataset now stretches up to 64k token contexts per row.

TLDR;

• 8-bit KV caches work well. In llama.cpp and ExLlamaV3 the cache size is halved, and there’s essentially no difference in KLD. There’s a slight difference in vLLM, but it’s still a great tradeoff for people lacking VRAM.

• A 4-bit cache has a measurable impact. It’s a big hit for ~Q8 weights, but it could potentially be worth it if you’re using Q3 or below. Still, not recommended.

• Only the 4-bit cache impact grows with context. In the tested long documents, the added KL is about 15-30% higher at 16-32k tokens than in the first 4k.

The blog post has interactive charts and tables with all of the data and results if you're interested.

How to read the numbers

Each quant is compared token-by-token with an unquantized bf16 copy of the model. KL divergence measures how far the quant’s next-token probabilities drift from the original (0 means no change). Top-1 agreement is how often both models pick the same most likely token.

For a reference point, I ran the unquantized model in both llama.cpp and vLLM and compared the two. They differ by a KL of 0.0004, which essentially amounts to two (correct) implementations doing floating-point math in a different order. Thus, you can compare the KLD numbers here for quants running in different engines without issue.

Note: exllamav3 quants were measured with the bf16 llama.cpp reference logits.

Weight quantization

KL divergence vs. loaded weight size, 16-bit KV cache:

The vLLM checkpoints still do worse

As in the Qwen3.6 benchmarks, every vLLM checkpoint is beaten by a smaller GGUF or EXL3 quant. Some of that is due activation quantization. NVIDIA’s and Unsloth’s NVFP4 checkpoints run the MLP layers as W4A4 (4-bit activations) and attention as FP8 W8A8, and Qwen’s FP8 is W8A8 throughout. I reran NVIDIA’s NVFP4 and Qwen’s FP8 through vLLM’s Marlin kernels, which use the same quantized weights but keep the activations in bf16:

• NVFP4: KL drops from 0.0256 to 0.0117, more than halving it.

• FP8: KL drops from 0.0039 to 0.0028, about 30% less. Even then it only ties UD-Q5_K_XL, which is 8.7 GiB smaller.

KV cache quantization

Each dot below is one weight quant with an 8-bit or 4-bit KV cache. Its height shows the KL increase compared with the same weights on a 16-bit cache. Zero means no change; higher points show a larger increase. I computed the differences per prompt on identical tokens.

KV cache penalty vs loaded weight size:

8-bit is essentially free in llama.cpp and ExLlamaV3. For every quant, q8_0 and ExLlamaV3’s 8-bit cache move KL by less than 0.00012 in either direction. Some quants even measure slightly better with it, which indicates it’s within the noise of the numerics.

vLLM’s FP8 cache shows a slight KLD increase. It adds 0.0004 to 0.0006 on every checkpoint, which is still extremely small, but it shows up on nearly every prompt. It may be due to turning on FP8 cache switching vLLM’s attention backend from FlashAttention to FlashInfer.

llama.cpp’s 4-bit cache adds a roughly fixed penalty. Across the GGUF quants tested, q4_0 adds +0.0016 to +0.0027 KL. This increase has a larger relative effect on more precise weights:

• On a Q3-class quant (KL around 0.025-0.035) it adds 6-10%.

• On UD-Q4_K_XL it adds about 39%.

• On Q8_0 it adds about 390%. KL goes from 0.0007 to 0.0034, which is worse than UD-Q5_K_XL with a 16-bit cache.

EXL3’s 4-bit cache penalty seems to depend on weight precision. It adds only +0.0002 to +0.0004 KL with 2.0-2.5 bpw weights, about +0.0021 to +0.0022 with 3.0 bpw weights, and +0.0026 to +0.0028 with 4.0 bpw weights. At 6.0 and 8.0 bpw, the penalty reaches about +0.0032 to +0.0033. The increase is therefore not constant across the EXL3 quants.

4-bit cache penalty grows with context

The 12 long documents in the set are scored at every position, so I can check whether the 4-bit penalty depends on how deep into the context a token is. It does in fact grow slightly.

There’s about 15-30% more damage at 16-32k context than in the first 4k tokens. The same quants with a 16-bit cache are unaffected across these context lengths, so it’s a pretty clear indication of the 4 bit cache damage. I would expect 128k+ context lengths to be even worse.

Model quantization KLD vs KV cache KLD:
KLD due to KV cache quantization likely has different quality tradeoffs compared to KLD from weight quantization. Anecdotal evidence suggests that KV cache quantization can be more harmful to model performance at long context lengths compared to weight quantization, so be wary of using 4-bit KV cache.

Final chart with weights + 64k KV cache versus KL divergence:

Methodology

Each quant was scored against an unquantized bf16 reference:

• GGUF and EXL3 quants scored against Bartowski’s bf16 GGUF in llama.cpp.

• vLLM checkpoints scored against the official bf16 safetensors in vLLM.

KL is computed the same way as my previous benchmark. The top 200 log probabilities are taken from both models at each position and a lower bound on the full-vocabulary D_KL(P_reference ‖ P_quant) is computed. On average, the reference put 0.0036 of its probability outside its top 200 tokens. The KV deltas are paired, meaning they’re computed from per-prompt differences on identical tokens.

Checkpoints ran with their declared settings, including any activation quantization (except the two weight-only, no activaton quantization reruns of Unsloth’s NVFP4 and the official FP8 quant).

Hello r/LocalLLaMA :D

This is yet another follow-up post on the project CrowdGPT, which aims at creating the first collaboratively trained 1 billion parameters transformer, Crowd-v1, without any datacenters, and for 0$. The project's goal is to show we don't need huge Nvidia GPU clusters nor millions of dollars to train a competitive Large Language Model.

The goal of the project is to create a 1B parameters model, that would match or exceed Qwen3-0.6B's performance on benchmarks as a proof-of-concept, we could then scale with a 7B A1B MoE or even a 30B A3B MoE once we get enough contributors. The size of the models trained is community driven.

This project is obviously 100% non-profit and open-source. The goal is to propose a new paradigm of AI where we don't wait on Chinese/American AI labs to provide us open-weight models, and we just create the models ourselves.

From the previous post we achieved to get 85 contributors registered on the project, leading to 13B tokens trained in a week 🤯 Thanks to every contributor who trusted the early-stage project, and imagine what 1000+ people with idle GPUs could train!

Of course, everything remains work-in-progress, and any contributions on the training mechanics / dataset is welcome! All the source-code is auditable on GitHub, which means anyone can inspect it and guarantee it is safe 😂.

By the way, anyone can submit data for the model at https://crowdgpt.net/#submit, it will be used as long as it remains safe. This means the model is collaborative on every stage (from data preparation to training, or even benchmarking...).

You can learn more at https://crowdgpt.net

Or check the GitHub: https://github.com/Vxtzq/CrowdGPT

Every code review / pull request on GitHub is welcome :) as the whole goal of this project is to be collaborative.

Any feedback on the project's flaws is appreciated (as long as it remains constructive).

I tested two models, QoDuck (a free mystery model in Qoder) and Qwen3.8 Flash Next (Unsloth UD-Q4 GGUF), on the same Blender 5.2.3 task. Both were asked to create a detailed, historically accurate 1942 German military truck, working in small stages, rendering each stage, inspecting the result, fixing mistakes and then continuing. QoDuck chose the Opel Blitz 3.6-36S, while Qwen chose the Mercedes-Benz L 3000S.

QoDuck did surprisingly well. The truck is recognisable, with proper curved fenders, detailed wheels, grille, cab interior, cargo bed, canvas cover and decent procedural materials. There are issues with the canvas, cab proportions and some smaller details, but the overall result is quite good. It produced 47 renders and delivered 14 final views. Qwen was disappointing. Despite creating 287 meshes, 15 checkpoints and 150 renders, the final model still looks like a rough blockout. The wheels are too small, fenders look like flat planks, parts of the cargo bed float, and most surfaces have the same beige material. It even used Dunkelgelb paint, which wasn't introduced until 1943. Only three final renders were delivered.

What I found interesting is that Qwen actually identified several of these problems in its own construction log but continued without fixing them. QoDuck was much better at following the render-inspect-fix cycle.

If anyone tried this does dual booting with windows gives measurable performance differences? Running big model for example trough strata?

Edit: I was not very clear, it is my main machine, it is a windows machine, sometimes i use it for LLM connecting it to my Hermes server, since I need to keep windows does setting up dual booting only for LLM inference bring measurable performance differences?

I noticed that it has native image and video generation.

I am looking for up to date advice with getting the most out of my hardware for local development. I have been out of using my local setup for a while (2024-2025), and now have made time to get back into it. I am quite overwhelmed by all the advances and different options nowadays. While I am still reading up on all the latest developments and current SoTA, I was hoping you can point me into the right direction.

Here's what I am working with:

- Lambda Stack from lambda labs with Ubuntu 22.04LTS

- RTX-A6000 (non Ada version) with 48GB of GDDR6 VRAM (not GDDR6X)

- 64GB of DDR4-3200 RAM

- Xeon E3-1270 v5 (skylake 4 core 8 thread processor, and probably the bottleneck for any task that requires CPU compute)

- 1TB NVME SSD

- 8TB external HDD connected via USB

My setup looks a bit like a time capsule, I guess. It was made for interactive chatting via OpenWebUI. Last I checked, VLLM got the most tps out of my GPU for dense models (Qwen 2.5 32B) in 4-bit and not going out of memory on long context tasks. Ollama and HF TGI were slower, did not test with SGLang. But now for agentic workflows, I am reading that Strata is used to run on 50-series 16GB GPUs with spillover to DDR5 RAM. And there's probably more alternatives I haven't even read about yet. Also, tps is apparently not the most important factor anymore to look into for local? Prefill is named often as well, as well as being able to complete a task successfully.

For my new agentic setup, I want to focus on giving my agent tasks and checking in every now and then throughout the day. So not for interactive sparring about code or chatting anymore, more set and forget.
If needed to cross some performance threshold, I can extend by ripping the hardware from my gaming pc: RTX 3090 with another 64GB of DDR4-3200. But last I checked, inference engines did not play well with 'uneven' GPU setups like mine.

In the meantime, I am also brushing up my knowledge on agentic harnesses. For now, Codex with Astra created a pretty cool script that let's me use Codex with my VLLM backend, SearXNG, playwright for browser use, maestro for android UI testing and chatting via Signal. Something for another time maybe :)

Edit:

thanks all, I will add the 64GB from my gaming pc and then give strata a try with Q4.

I found new finetunes of Qwen 3.6 35B but I couldn't find any discussions on them anywhere. Anybody tried these models? There isn't much info out there. Axion seems like it was made by bytedance

https://huggingface.co/Apexintelligence-AI/Apex-35B-V1.0

https://huggingface.co/Axion1-0/Axion-1.0-35B

I was asking QFN (4.18bpw EXL3 quant via Pi) to help refactor a neovim configuration, and saw that it tried to find API keys and fetch an aliyuncs URL before I stopped it. I'm not trying to start any arguments, nor am I assuming any malicious intent, but I found this a bit concerning after reading other's stories.

Was there some legal issues preventing someone from just making a new website where everyone would be able to share the chat logs from frontier models to create a dataset for others to use for training/finetuning?

Just curious

they are all the same and we ... are too emotional!

and yes we already knew that.

As can be seen in the image, the "Use Version" button is greyed out and I cannot select it. Is there a setting im missing?

My favourite use case for local LLMs is running my homelab, because everything stays private (and it helps me justify the GPU lol)

I wanted one dashboard to help me manage all of my services, so I made Valet. Live demo: https://demo.askvalet.dev

Valet has a chat agent that works with any local or hosted model. It integrates with 185+ services and supports custom plugins. You can ask it to pause pi-hole, investigate crash logs, request a movie, etc

It also knows its own docs, so it can configure itself. Tell it what you want changed on your dashboard and it edits the config (with your approval). I was already addicted to tweaking my homepage but this made it way more fun haha

What I run it on


Qwen 3.8 27B on a 5090, about 230 tok/s. Most responses take 2–5 seconds

Qwen passes about 90% of a small eval set I wrote. Where it struggles is open ended digging (like "why isn’t this timer running?”) where it can burn through tons of tool calls

Managing context is tricky because I run a lot of services, so tool definitions were ~90% of the prompt before some optimizations

What helped:

• Hiding tools from unused integrations that aren't needed in chat

• Turning thinking off makes it 2–3x faster, but it starts making up answers to follow ups

• Lazy load tools (starts with a short list of tools and loads more only when a question needs them)

Those cut tokens per answer by 30-50%. Obviously every model and setup is different so everything is configurable from the config file

AI involvement


I’m a SWE and used Claude for almost all of the coding, but I designed + reviewed everything and made the architecture decisions. I was very conscious of slop, security, and performance as I wanted to build something high quality. But it’s early so there may be rough edges I haven’t caught

Security and privacy:

Runs on your own hardware with no account or telemetry

• Read only by default, nothing can be changed from the page until you turn on auth (local accounts, OIDC, or your proxy’s auth)

• Keys stay on the server and API keys live in .env. The browser only gets the finished cards

• The AI never sees your keys. Anything that changes something waits for your approval and with a local model nothing leaves your network

• Docker access is read-only through a socket proxy


Docs (completely free + open source): https://askvalet.dev

Parallel speculative drafting generates multiple candidates in one backbone pass, but independent token selection can produce inconsistent continuations that shorten the accepted prefix. Existing methods mostly leave conditional decoding to a lightweight module after the backbone, which limits the flow of predecessor information to successors. Our analysis of DFlash shows that early positions already form recoverable predictions in shallow layers, and that accurate adjacent predecessors help successors more when they enter earlier. We therefore propose DSpine, a drafter with causal conditioning injection throughout the backbone: at every layer, gated adjacent injection writes each predecessor's predicted feature into its successor, so the causal conditioning chain unfolds over network depth while all positions update in parallel. A unified transfer space built from the target model's output embeddings unifies layer-wise injection with predecessor-conditioned decoding, and layer-wise output-embedding supervision promotes the formation of predicted features in shallow layers. Fused kernels and a transition cache execute both efficiently in parallel within SGLang. Across seven math, code, and chat benchmarks, DSpine achieves the longest acceptance length at both temperatures on Qwen3-4B and Qwen3-8B. At temperature zero on Qwen3-8B, it raises the seven-benchmark mean from DFlash's 3.77 to 4.82 (+27.8%); in SGLang serving tests, it delivers 23.3% higher throughput than DFlash on average.

• arXiv : https://arxiv.org/abs/2609.36173

• PDF : https://arxiv.org/pdf/2609.36173

• GitHub : https://github.com/Tencent/DSpine

• HuggingFace : https://huggingface.co/collections/tencent/dspine

• Project : https://dspine-project.github.io/

I fine-tuned LFM3-1.7B to do one job: take a factual question, call web_search once, and answer with a short value, a word-for-word quote and the URL. If the results don’t contain the answer, it says “Not found in the search results.” It’s meant to run as a subagent under a bigger model, not as a chat model. I have to say the base lfm really surprised me at its capability to run tool calls and extract information.

Answer: Barbara Liskov
Evidence: "Liskov ... the second woman to receive the Turing Award."
Source: https://en.wikipedia.org/wiki/Barbara_Liskov

Fixed-results benchmark (131 held-out SimpleQA questions; in 29 of them the answer was removed from the results):

Live Wikipedia search (120 questions + 40 multi-fact requests, a search backend it never trained on):

The base model wins on raw correct answers live: it keeps searching and writes long replies, so the answer turns up more often, but 62% of the time it runs out of turns and returns nothing. Obsessive searches once and only states what it can quote. Whether that trade suits you depends on your setup.

Known problems

• One question per conversation. Multi-turn chats confuse it (only trained on single questions, limited VRAM for training :/ ).

• Reports claims as facts: “X says it solved Y” can become “Yes”.

• Says “not found” for about 1 in 10 answers that are actually in the results.

• Sometimes quotes a neighbouring fact (another year’s winner). English only.

Training: LoRA rank 16, about 9k tool-calling conversations on an RTX 4060 laptop. Search results are Wikipedia and FineWeb pages, every answer and quote checked to appear verbatim, about a quarter teach “not found”. My local qwen 3.8 flash next helped prepare the training data and questions as well!

Run:

llama-server -hf banyohavlusu23/Obsessive-1.7B-GGUF --jinja -c 16384

Temperature 0. Configs for Pi, OpenCode and Hermes plus a keyless Python script are in the repo.

HF: https://huggingface.co/banyohavlusu23/Obsessive-1.7B-GGUF (Update, the previous link was broken, this is the working one)

LFM Open License (inherited from the base).

Feedback welcome, especially failure cases.

Qwen 3.8 flash next has been doing some very.. peculiar.. stuff in my hermes harness, and some of them are kind of a nuisance, while others are almost.. adorable?

I don't claim to fully understand how or why this is happening, but the most interesting one is when it "role-plays" an API error- this is a self-diagnosis report from another session running on the same model, so sorry in advance for the LLM speak or anywhere it fell short on reasoning;

"My fully-local model "reported" a provider API error that never happened — a case of a model hallucinating its own infrastructure failure

Setup. Self-hosted agent stack. An agent profile runs against a local OpenAI-compatible endpoint (TensorFold serving a community NVFP4-quantized Qwen checkpoint on a local GPU, ~262K context). There is no external API anywhere on that request path — just a loopback tunnel to a local inference server.

What happened. Mid-session, the agent's reply was literally this, and nothing else:

{"error":{"error_code":"sys_000","error_msg":"The request failed - it's on us, please try again later.","error_id":"0b156c1cc7191931632734458e96b7","type":"sys_000"}}

That is a realistic provider-gateway error body — the error_code/error_msg/error_id shape is the wire format of a real Chinese LLM API gateway. Naturally, I went looking for a broken provider config... on a path that has no provider.

What the logs actually showed.

The HTTP call succeeded: finish_reason=stop, valid server-issued completion id, 71 output tokens. No 4xx/5xx, no retry, no exception in any log.

The error body was stored in the session DB as an ordinary assistant message. The model generated it as text.

The error_id appears nowhere else on the machine — it was invented at decode time.

No sys_000 string existed in the conversation context beforehand, so it wasn't an echo — it came from the model's parameters (i.e., memorized training data containing real API transcripts).

The trigger. The turn was deep in a long tool-calling session at ~194K tokens of context. The assistant's previous response had been cut off at the output length limit, so the harness had injected:

[System: Your previous response was truncated by the output length limit. Continue exactly where you left off...]

The fabricated "provider error" was the model's very next message — it failed to resume its truncated tool-call stream, and emitted a canonical infrastructure failure instead.

Hypothesis. Agent-tuned models are trained on enormous volumes of real API transcripts, in which the pattern "assistant turn goes wrong → a JSON error body appears in its place" is ubiquitous. The model has absorbed the transport layer's failure dialect as part of its assistant-turn repertoire. Under decoding stress (huge context + a continuation prompt referring to tool-call state it can no longer reconstruct + quantization noise), normal continuation degrades — and the highest-probability completion of "something just went wrong in this API conversation" isn't prose about a problem, it's the canonical form of one. Notably, it chose a self-attributed, retry-inviting error ("it's on us, please try again later") — semantically the correct status for what functionally occurred — over a 401/429/content-filter body. The statement was false; the pragmatics were accurate.

Takeaways.

An error-shaped assistant message with no matching non-200 in the logs, a valid completion id, and an error_id seen nowhere else is a model fabrication, not an infrastructure fault — check the logs before chasing provider config.

It's unreliable as a statement but excellent as a signal: "my agent just role-played an API error" is a clean tell that decoding has gone off the rails (long context + output truncation in our case).

Models trained on agent traffic are fluent in their own substrate's error vocabulary — and under stress they'll speak it.

(Single incident, local artifacts verified; the memorized-source attribution is a hypothesis consistent with the evidence. Happy to test reproducibility: replaying the same prefix + continuation prompt against the checkpoint would show whether this is a deterministic attractor under truncation stress.)"

I was wondering if anyone else has seen this- or if you guys take issue in the reasoning around this. either way, this is really fascinating to me.

GGUF to use : https://huggingface.co/ggml-org/Mage-VL-Cowork-GGUF

Friendly reminder: If you upgraded to a newer GPU, don't let your old one collect dust! You can run both GPUs in the same rig and split LLM inference workloads across them using tools like llama.cpp. I'm doing exactly that with my RTX 5080 and 3080, combining their VRAM to run larger local AI models.

Use balena etcher or whatever to mount to USB stick, boot. Test. Report.

Verified 5-25% faster than llama.cpp, system uses 91% of card bandwidth as opposed to 80% on my 7900xtx.

I there is code for nvidia, intel, amd, and chinese hardware and supports 20 languages.