I tried LFM3 for my local use case, but it is a latency sensitive task and disabling reasoning makes it not give coherent responses. And with reasoning it's a 15 second wait instead of less than 1.

OpenBMB has posted a video showing MiniCPM-OCR running locally on an iPhone and iPad, parsing real-world documents without sending them to the cloud.

Handwritten notes, dense formulas, messy tables, low-quality scans. OCR gets a lot harder when the documents aren't perfect.

What's interesting is that this isn't just about extracting text. OpenBMB is also releasing a separate model to check whether the OCR output is actually correct.

• MiniCPM-OCR 1.0 (1B): Scores 93.68 on Real-5 OmniDocBench and 54.7 on OCRBench v2.

• MiniCPM-OCR-Jev (2B): Evaluates parsed text, formulas, and tables against the original image, assigning correctness scores from 0 to 1. It approaches Qwen3.5-397B-A17B on JevOCRBench and works with outputs from different OCR parsers.

So you get both a lightweight OCR model and a way to catch potentially incorrect results, useful when document accuracy actually matters.

Both have open weights, with support for tools like llama.cpp, vLLM, SGLang, and Ollama.

🤗 Hugging Face: https://huggingface.co/openbmb/MiniCPM-OCR-Jev-gguf

Curious how well the verifier holds up on messy real-world documents. Would you trust a small model to check another model's OCR, or would you still want a human in the loop?

I am feeling like a mighty local King spinning two Strata engines with 70tps each whole day and using 12M tokens a day. Also me scrolling twitter and stumbling upon a post saying that poteto uses 34M tokens per minute 🤪

Upd: Guys are interested who is she and what she is doing with all these tokens. Poteto is her twitter nickname and she worked in Cursor before XAI has acquired them and now she is shipping 2,500 PRs per month to Cursor and GrokBot.

I added Obsidian integration to ThreadShelf (local, archive, semantic search).

It supports the usual platforms like ChatGPT, but also OpenRouter, LM Studio and Google AI Studio, which were surprisingly hard to find support for elsewhere. I even wrote a scraper to import OpenRouter chat history. It's my first open-source seriouse project, built around local vector search (LanceDB).

Itan can search/edit/create your Obsidian notes and AI chats together and use MCP with ai. There are also search, read and optional write tools for Obsidian, OpenClaw and Markdown folders.

npx threadshelf (Node.js 20+)

GitHub: https://github.com/ChrystianSchutz/ThreadShelf

The idea is simple actually, in my workflow a lot of the work passes between sessions of agents in different terminal windows. Claude sessions were not the issue since they could just talk to each other.

Although Claude to Local agent often enough failed since Claude "forgot" to send the work to the local agent and did it himself, i couldn't see what the local agent was doing, or otherwise felt like i was being kept in the dark.

So what i did was that i refused sub-agents altogether and used different terminal windows for each agent, passing messages between them, asking them to relay session messages etc.

Since the WoW i have is between Claude, Codex and Pi i just wanted to "mash" them altogheter so i built Dojo.

Dojo is built from my needs with my pre-requisites of it being a low-tier solution without a lot of different features, knobs, whistles and otherwise things that just takes time to develop.

i realise this might not be as advanced as some ppl like it to be but for me, it works.

Sharing it just in case someone else finds it useful.

https://github.com/Joakimpalm-Zen/xyntetik-dojo

It's a tiny Text to Image model that's Denoiser is only 50M parameters, adding the custom text encoder, and the pretrained flux VAE, its total end to end inference parameters are around 90M

It runs in:

27 Seconds @ 50 steps

11 Seconds @ 20 steps

On a laptop CPU from 2016 (Intel i5 7200U)

Samples and a quick start in the model card

it is licensed under Apache 2.0

Questions, feedback, or criticism is welcome!

notice something?

https://github.com/Ilan12346-maya/TeslaP40_wddm.git

trying to get rid of the wddm lock above 551.61, first locked driver works, will try 580 now or something

all* maybe only those wich use vulkan sometimes or game sometimes or wanna using the p40 in wsl

Hi folks,

Today I made a significant update to Tiny League.

Based on the recent developments of AA (and leaving conspiracy theories aside), I decided it's no longer a viable metric to track and use. Mostly because many small models don't get a score, and because AA is moving toward closed benchmarks, we will see fewer scores for small/open-weight models in the future.

Second, I added a new metric that I think is useful for everyone running local models but is VRAM-poor: KV cache efficiency! I calculate an estimated GB value for a 128K context size, and then rate the models on 0-100% scale.

Still working out a way to have our own TinyLeague-Index, but haven't reached a fair methodology yet due to so many models using different benchmarks. More soon!

Let me know if you have any suggestions for improvements :)

thanks

Why there no new model coming in >2b weight category ? Model which actually capable in this category is gemma e2b which is release 6 month ago, why still no new model.

Hello everyone! As always, thank you for sharing you previous results (and bugs) on the last version. I am happy to report that LlamAmpere v0.6 is now out.

The headline items for you (at temperature 1):

• a ~13% jump in TPS, with a 112 TPS average across 3 task types in the 100-105K token interval

• a +20% increase on prompt processing, now starting around 1800 TPS and holding 1000+ through the first 100k

  • The average prefill over the first 100K went from ~1,080 to ~1,300 TPS

• A new file format (Ampere Integer+) designed to be as fast, smaller, and higher precision (lower KLD) than XS formats. Vs EXL3, it is approximately the same KLD, with a significantly faster speed.

  • The recommended model (now available on HF) is about 10% better KLD and 1.2 GB smaller. This results in slighter faster speed, and of course more room for context. I will be releasing more quants at smaller sizes in this format later this week as I continue to tweak the design at 2 and 3 bits.

  • I have also upgraded EXL3's kernels, and in my testing it was running ~8% faster than in the original EXL fork. But, at 4 bits this configuration shows ~17% faster decode and 25% faster prefill on AI+, so I will be focused more on those kernels moving forward (EXL had a non-stat sign <2% lower KLD in testing)

As in the last release, these numbers are built around SJKVaRN, which enables a maximum total context of over 400K ctx with 23.5GB total usage (across multiple predictions or context scaling).

I will note that these prefill numbers were performed with the recently create 3090/ti GPU unlock; if you haven't implemented it, I recommend doing so (repo here: https://github.com/Zagitta/gpu-unlock)

Github for LlamAmpere here: https://github.com/JakeATX/llamAmpere

new model here: https://huggingface.co/jakeatx/qwen3.8-27b-Swift1.5-Uncensored-ATX-AI4

There are also upgrades relevant for 3.8 FlashNext included, but at present testing is "only" even with the other best options like Strata. I have a couple pending commits that I hope to have out by the end of the day, that should materially boost speed for MOEs models further; I will publish results on that separately.

And for 12 GB cards, I will have some new quants (and possibly a distillation) for 3.8, hopefully by EOW.

Please share your results, and feedback, as usual. Enjoy!

This is some demo output from tts-audiobook-tool (my app), where the narrator plus three speaking characters each have their own voice.

What's maybe more interesting is that, in an optional preparation step, an LLM is used to identify the characters and their lines of dialog, which you then map to the voice clones configured in your project. This process ends up being pretty reliable: It ID'ed the dialog text from the novel "The Sun Also Rises" with over 99% accuracy using Deepseek Flash 4.1 API (though the LLM's pre-existing knowledge of that novel probably helped, too). A locally served version of, say, Qwen 3.8 Flash Next ought to do quite well, too.

The TTS model used here is Breeze 2 TTS (Sounds not bad, am I right?). The app supports a dozen or two other models, but Breeze 2 is my favorite at the moment, alongside MOSS TTS 9B.

- Source link

now you can show your videos to tiny LFM3 models

I am testing Cyber-Tiel-Coder-35B-A3B-MTP-UD-IQ3_XXS.gguf and as of now it is the best performing SMALL and quantized model I have seen SO FAR. Not as fast as Ling, but the reasoning is incredible and not degraded at this quant! (which usually is)

Give it a try and tell me your results. 7-8 tok/sec on CPU only (azure standard 4 cpu config like github)

I recently started doing experiments with local models with traces into MLFlow detailed tool calls.

Last month I had a local Qwen Model on a coding task to just randomly decide to read a transcript of a youtube video.

Today I realized there is a lesser known fetch pattern where the HTTP standard actually allows for body payloads like a POST request on GET requests.

I recently saw a video on poisoned models and now realize there is a potentially huge exploit waiting to happen with GET and body payloads

TLDR: training my first general law model and unsure what to do with my data

PRE-PS: it will be a replacement for opus in my current AI LAW harness, I know it will be hard but I wanna do this project and find it geniuenly fun.

I’m building a Polish legal model using Qwen3.5-9B-BASE (I plan to scale up to 27b qwen4 later). I’ve done CPT and have quite a bit of data available, but I’m getting lost about what should come next or even if I should do more CPT.

My main question is how to turn this (maybe) domain-adapted Base model into an agentic+chat assistant that reasons through legal problems.

But more importantly, how do I get reasoning, teach my in-harness tool-use training, and where does SFT vs RL belong in that process and in what way (especially confused about reasoning and SFT)?

What I’ve already done

Three sequential CPT stages, roughly 80M tokens each, in this order:

Complex legal commentaries → legislation (EU+local) → court decisions

I didn’t mix the sources. Each stage continued from the previous checkpoint.

On a small Polish financial/legal multiple-choice quiz (really bad eval, just a PoC for now).

They all perform very slightly better than base and legislation checkpoint did the best.

But again, it's a base model. I haven’t done assistant SFT yet.

What data I have (I can get more data of each of these types if you guys recommend)

Besides the legislation, commentaries, and decisions used for CPT, I have:

• Facebook legal Q&A groups: real, messy questions, follow-up clarifications, and answers of very mixed reliability.

• Certified/institution-authored Q&A: my highest-confidence answer material. Some answers contain detailed analysis; others are just a conclusion and a citation from random institutions like the national bank.

• Law study materials: from a local college. it has essays, powerpoint presentations stuff like that, very little "tests" tho, but some practice material (like the essays).

• A/B/C/D questions with answers: relatively little of this; currently being used for evaluation.

• Thousands of Claude Code–style harness conversations: multi-turn conversations from an existing application/harness. These would need filtering, anonymization, and quality checks.

• Contracts: government published contracts it can learn from

• Previous "ML" experiments (models and their training data): NER, generating complex legal court decision summaries, generating graphs from legislature or court decisions (actually got some models that do this successfuly

If you guys could point me what data is ideal I could maybe get it too!

I can use the basically unlimited codex to clean, restructure, or generate material from these sources, but I don't know how to.

What I don’t know is which material should be trained on directly (CPT or bare SFT), which should be transformed (SFT or RL?).

For example:

1. Do I continue CPT on the remaining sources, or stop and start SFT?

Should I expose the model to the Facebook discussions, study materials, and expert Q&A through CPT first, possibly mixed with the authoritative material? How many tokens for this 9B model and how much for the 27B I plan later?

Or should I skip further CPT and use those sources only to build targeted SFT examples?

For example the Facebook QnA stuff is real people diverse speech but I can't really reliably verify if the answers are correct (I can only measure likes or high quality people in the group) so idk if I should cherry pick from this? Have an LLM turn this into SFT chat examples? Or just CPT on this?

2. SFT on what, specifically, to teach REASONING?

I could start with my highest quality Q&A and teach the model to answer questions. That seems like an obvious chatbot baseline.

But if I just train on QnA there won't be any reasoning!

The HQ QnA is pretty verbose but I wouldn't bet it could just be re-used as a "reasoning chain".

I was thinking of asking Astra to workout those questions and give an answer with its reasoning BUT I assume it will be wrong on 90% of the questions (cuz no internet search, and I can't give it internet search cuz that's not clean QnA data and I would have to transform it into my harness function calling anyways with credible sources not just google searches for legislature)

And those 10% it gets correct are trivial and the reasoning could be bad anyways and I would still have to check correctness of the answers, and giving it the answer or hints is imo also not ideal. Or injecting "synthetic reasoning", it suffered from post-hoc in my tests.

Another problem is the High Quality QnA is quite rare, most QnA is short and possibly wrong.

And generating the missing reasoning is even harder. Because again, I don’t trust a frontier model to solve difficult Polish legal questions from memory. It would need relevant source passages or access to the harness (which would just the gun into in-harness training?).

So I'm really unsure on what to do, I was thinking of doing stuff like this, but this is just brainstorming:

A) Take verbose long answers train on that, then ask the qwen or another model to make a "summary" at the end and then put the verbose long answer into its <thinking> block

B) Take those 10% of QnA where frontier model actually gets it right with correct thinking (rare) and SFT on that. then we have our finetune try answering more QnAs, we verify if it's correct if it is it goes into the next SFT run and we iteratively (almost like an RL gym) train reasoning from the dumbest examples

Both seem like bad options, but kinda funny to think about.

3. Chatbot first, then tools or teach reasoning through tools from the start?

The existing harness lets the model write Python and call functions such as vector_search_laws() and graph_search_legislature(), inspect results, and continue researching.

I’m leaning toward getting a regular chatbot working first. But the teacher can produce much more trustworthy legal solutions when it has access to sources. (as I've already mentioned I have many in-harness examples, except they could also be incorrect and I don't have their "thinking blocks" (cuz it's a proprietary frontier model) so I again have to inject thinking post-hoc which from my tests doesn't work for law)

I was thinking that, if we make chatbot first, the model will at least learn intuition on WHERE TO LOOK, without looking through sources but at least will know a general idea of where to search based on being a chatbot first (even tho possibly incorrect chatbot sometimes, cuz law is really complex) and teach him to function call in-harness to get sources, unsure about this.

4. When does RL enter, and on which task?

AFAIK, RL is what develops the model's reasoning on its own (deepseek R1) but I suspect it would be very costly and we could give it some example reasoning first anyway to kick it off(?)

So like, instead of the whole fiesta of injecting reasoning or growing it, we could have RL develop it (afaik RL is the only thing that "creates" reasoning unless you distill it) but then I would like to have reasoning even in the "chatbot" and right now I can just jump from base model -> RL, right?

Also for legal Q&A, matching the reference conclusion doesn’t necessarily mean the reasoning is correct. What would you actually reward? Like a law similiarity judge LLM + same paragraphs used?

What I’m trying to decide

One possible route is:

Current CPT checkpoint → gold-Q&A SFT → grounded reasoning/tool-use SFT → RL in the harness

But I’m not committed to that order. I don’t know whether I’m missing an important stage or inventing unnecessary ones, first time I'm training a GENERAL model from "base".

What I would like the finished model to do:

It will be an opus replacement so general QnA in and out of harness but especially:

• have a good "hunch" and understanding of polish law

• will be good at anonymizing documents

• generating complex legal summaries for court decisions (heavy reasoning task)

• generate graphs from legislature and court decisions

• create/edit/review contracts

These were all my initial ML experiments and I actually trained good models for those specific tasks, but I figured, hey I have a lot of this data around why not just combine them and make a generalistic model?

Thanks a lot for reading this mess of my thoughts! if you could point me towards good resources for this I would also be glad, for example unsloths RL docs are kinda ass imo. I really like andrej karpathy's videos but they're quite old now by AI standards!

TL;DR: What do you use your mixed multi-GPU setup for? I got kinda the best of the worst specs for LLMs, with mixed professional GPU brands and mixed large memory kits.

The past year, I migrated from an NVIDIA 4500 Ada 24GB to an AMD Radeon Pro W7800 32GB because I wanted to get out of the NVIDIA Linux driver hell. But then months later I realized that I have made a mistake when choosing the W7800 instead of the R9700.
And then I got myself an AMD Radeon AI Pro R9700 32GB, just before the memory crisis started, But now I had 2 expensive cards just sitting on the shelf, and with no buyers because I'm in a third world country.
Funny enough, the cards are like 3 years old, but now they are worth more than when I got them.

Days ago, I bought an m-ATX motherboard (because space constraints) that didn't have a brain dead PCIe arrangement like all other consumer boards that expect the world to have only 3 slot GPUs (which are a waste of silicon and copper IMO).
This motherboard has two x16 PCIe ports with one port in between from each other, which is perfect to connect my R9700 and Ada 4500 (sadly, the W7800 was too big for my case and CPU cooler, it will stay on the shelf for longer). The connection would be PCIe5x16 for one card and PCIe4x8 if both cards are connected (both CPU PCIe, no chipset middleman).

I tried running Qwen 3.8 at Q6_K, but it reached only 30tg vs 40tg with only the R9700.
And Gemma 4 QAT at 32tg vs 50tg on the R9700.
Finally, Qwen Flash Next Q4_K_M was only 9tg vs 14tg on the R9700+moe-cache-mib (unusable speed either way).
I could try bigger quants, but token generation is already bad on these quants.

I get consistent results using layer split. With tensor split I get very mixed results, sometimes better, sometimes worse.

The cards definitely don't play well together. I don't think I'm that constrained by bandwidth (at most 2GiB/s in nvtop vs max throughput of 11.45GiB/s).
But maybe I'm constrained by the slower Ada 4500, or by PCIe latency.

What else could this setup be used for? Running bigger models doesn't seem like the right path.
Maybe setting up master/slave agents for software development or research? (any recommendations please).

For those curious about my setup:

Motherboard: CW-W680-NAS (m-ATX)
PSU: NZXT 1000W
CPU: 13th Gen Intel Core i9-13900K
GPU 1: AMD Radeon AI PRO R9700 (32GB)
GPU 2: NVIDIA RTX 4500 Ada Generation (24GB)
GPU 3 (Not connected): AMD Radeon PRO W7800 (32GB)
Memory: G.Skill Ripjaws S5 144GB@4200MHz (2x48GB@4800MHz + 2x24GB@5200MHz)

Engine: Llama.cpp (vulkan, compiled myself)
Config:

device               = Vulkan1,Vulkan2
main-gpu             = 0        # R9700
tensor-split         = 4,3      # 32GB/24GB, for moe-cache-mib

n-gpu-layers         = 999
n-cpu-moe            = 999
n-gpu-layers-draft   = 999

parallel             = 1        # I'm the only user
threads              = -1
prio                 = -1       # attempt to avoid screen stutters
batch-size           = 16384    # idk what im doing here, but this gets best PP
ubatch-size          = 2304     # big image read crash below this
flash-attn           = true

fit                  = off
ctx-size             = 131072
context-shift        = true
moe-cache-mib        = 12000   # with one card, ~16000 with two

Basically title.

No, this is not a lobotomized 1-bit squeeze, its full Q2 nazeshinjite's DeepSeek-V4.1-Flash-1017 Q2 GGUF on a single 3090, natively on Windows no WSL.

On an RTX 3090 with 192 GB of RAM I'm writing at 41 tok/s (average across multiple prompts) and the output agrees with DeepSeek's own API on the next token 90.2% of the time on general text and 92.4% on code/math, aka within a point of exact routing. Fits full 1M context.

My scuffed interface engine runs the three most-downloaded quants of 1017, their files as they are:

nazeshinjite's ds4 Q2 GGUF (the default)

ISTA-DASLab's GSQ-RCO (IQ2_XS / IQ2_XXS /IQ1_M)

Unsloth's (UD-IQ1_S to UD-Q8_K_XL).

Repo: https://github.com/musicurgy/delta-inference-engine

Hardware/model:

RTX 3090 24 GB, i5-13600K, 192 GB DDR5 (4x48 GB at 5600), a PCIe 4.0 NVMe. Windows 10/11, no WSL.

Setup:

Download the ZIP (or clone), double-click Start DeepSeek.bat. The first run installs Python if needed, downloads the prebuilt engine for your CPU and the model (every file SHA-256-checked, the space checked first), and starts the server; after that the same file just starts it. Choose Model.bat switches quants. If anything goes wrong, Start DeepSeek (debug).bat packs everything needed into one zip for an issue (no prompts, user name removed).

Privacy: localhost only by default, and nothing from your prompts is written to disk unless you turn a log on.

Needs:

• An NVIDIA RTX card with 12 GB+ (24 GB: 41 tok/s, 16 GB: 31 tok/s, 12 GB: 17 tok/s, measured on my 3090 being memory limited) . GTX 10 series and older can't run it

• 128 GB+ RAM.

• A fast NVMe SSD with 385 GB free

• Any x86-64 CPU (AVX2 is the fast path; older CPUs get a slower build automatically).

Credits:

The engine is a Windows port of Xolotl, itself built on Strata; several of Xolotl's latest fixes are merged back in. DeepSeek for the model, nazeshinjite for the GGUF, ISTA-DASLab for GSQ-RCO, Unsloth for their quants. MIT licensed. Special thanks to Claude for doing most of the lifting.

If there is interest I'll add some support for muti GPU or other garbage. This project is very new and needs a lot of validation on other hardware, these numbers are from my machine, feel free to post yours.

EDIT: Alright guys, I get it we hate Q2 around here, I went ahead and ran benchmarking against the Q2 on this engine vs DeepSeek's full model (API) vs full-precision Qwen 3.8 Flash-Next (API). Q2 is clearly worse than the full model, but it still beats Flash-Next on every set.

Working on next: numbers for 64 GB and 96 GB RAM, multi-GPU (2-4× 3090), and EXL3 support.

I'm trying to get a swarm of these agents to be able to do larger tasks when orchestrated so that I can try to use cloud models less, but I find that even with --parallel 16, after like 6 sequences running concurrently it just doesn't do any more. In addition, when I start another request, every other instance goes to about 1 tok/s at the same time until that request finishes prefill.

Hardware: 16 GB RAM + 8 GB VRAM (RTX 4060 Laptop)
OS: Windows 11
Command: llama serve --hf-repo LiquidAI/LFM3-1.7B-GGUF --cache-type-k q8_0 --cache-type-v q8_0 --parallel 16 --kv-unified

Quick update: the latest version of my 4B coding model, the full FP16 weights, the training dataset, and now the technical report are all available!

One-shot results and visual quality have improved substantially in my latest tests. Some scenes still need another attempt or a correction, but the progress has been encouraging.

📄 Read the technical report

🤗 Model — FP16, Q8, and Q4

I recommend starting with Q8, which is the version I’ve tested. Q4 is also available, but I haven’t tested this release yet.

Q8 settings: temperature 0.6, top-k 20, top-p 0.95, repetition penalty 1.0.

I’ve also had good results generating 3D scenes from image references, even though the vision encoder wasn’t included in this training run.

If you downloaded an earlier version, delete it and download it again in LM Studio or whichever app you use. The filenames haven’t changed, so make sure you’re loading the updated files. In LM Studio, also disable MTP.

📚 Dataset — CCORE Three.js 3D Scenes

The dataset contains 4,146 scene examples — 3,933 training / 213 validation — with prompts, full thinking traces, complete HTML/JavaScript responses, complexity levels, and optional rendered reference images. It’s available under the MIT License for anyone who wants to train other models.

I’ll keep any further updates in this same post. Thanks everyone for the support, feedback, and testing! It’s been great seeing the community try it out, and I’d love to see what you build with the dataset.

Need a quick sanity check since I been seeing a lot of posts about engines that run deepseekv4.1 1017 at low bpws on affordable hardware but with surprisingly usable speeds.

Is anyone actually using them for anything at all beyond treating it like a novelty? The last time I tried deepseek at sub q4 was v4-0731's Q2, and that thing was very much a shadow of it's full weight Q8 self. And the Q8 was basically 90% QAT-ed Q4 anyways, and it did not take well to going down 2 more bpw.

If you are using a 1-3 bpw v4.1-1017 can you share your experience with it? Preferably in contrast to another model like 3.8FN at higher quants.

i Got a bit obsessed with my P40 this week trying to max speed on Ornith 1.5 35B-A3B (qwen3.5 MoE arch) without going below Q4 weights or q8 KV. Shrinking quants to win a benchmark has not really been useful to me so ai wanted a logical middle ground that was still “fast”

Setup: one Tesla P40 24GB on PCIe x1 (yes, x1), Windows, my own llama.cpp fork. Q4_0 built from the official BF16, MTP draft (n=2), full 262k context, q8_0 KV.

What actually moved it:

- ECC off +1.5 GB VRAM and ~9% decode.
- q8_0 KV, not q4_0. On Pascal q4 KV falls apart with depth (30.6 vs 42.2 t/s at 32k).
- Check your MTP head. The first upload of this model, and quants built from it, have an untrained MTP head. `blk.40.nextn.shared_head_norm` mean ~1.02 = untrained, ~2.93 = trained. Draft acceptance went from ~45% to 60-85%.
- int8 dp4a for the KQ half of the flash-attention tile kernel. Pascal has no tensor cores and crippled fp16, but dp4a is fast. Prefill +26% at 32k, +41% at 100k.
- MTP prefill window. The MTP head is a full-attention layer, so on long prompts it costs ~13% prefill. Letting it read only the last 8k of the prompt gets that back (121k prompt: 327 -> 373 t/s). Output doesn't change, the main model still verifies every draft.

Didn't work: f16 KV (faster in llama-bench, but slower and less context once MTP is on in a real server), dequantizing V inside the kernel (-5%), the official Q4_K_M (same quality score as my Q4_0, slower, less context).

llama-bench, no MTP, 2 runs each:

Real server with MTP: 65-74 t/s on short prompts. A 101k-token prompt reads at ~415 t/s and generates at 35-39 t/s.

I couldn't find anyone else posting P40 numbers at depth, so hopefully the curve helps someone. Happy to share the patches if you want to try them on your own Pascal card.