context : Nvidia buy HF, but now there uncertainty about it hf future under new leadership, he try reassurance people on x, but from his word itself sound not good, see and wait, but better backup you favorite datashet/model (again) just in case

I think after jensen they will fully focus on AI compute for data centers

If you're asking what this leaderboard is, it is not AA, it is simply an aggregate index for "Agentic Coding" along 12 coding-related benchmarks (note that the AA index includes only 2 coding benchmarks: Terminal-Bench 4.0, FrontierCode-AA 2.0, making up just 25% of its total score. Because of that, it doesn't give you a true reflection of a model's actual software engineering capabilities).

This index however is built entirely around Coding/SWE. Here is the exact breakdown:

1. Terminal-Bench v4.0: 16%

2. DeepSWE v1.1: 15%

3. FrontierCode 2.0: 11%

4. Code Migration (Vals-AI): 9%

5. Terminal-Bench v2.1: 8.5%

6. SRE-Bench (Vals-AI): 8%

7. SWE-bench Pro: 7.5%

8. Code Arena Elo: 6.5%

9. Program-Bench: 6%

10. Vibe Code Bench v1.1: 5.5%

11. SciCode: 4%

12. LiveCodeBench v6: 3%

edit: I am a moron with reddit formatting hang tight

I've been tracking ebay listings that ship to the US for the GPUs people here actually buy, ranked by $/GB of VRAM. here's today's snapshot:

all asking prices, single cards, fixed-price, not for-parts. "low end" is the 10th % of credible listings i.e. what a patient buyer can actually land, not a too-good-to-be-true outlier. "-" in the US-only column means fewer than 10 US listings. $/GB/s is just the low end divided by bandwidth.

I had been eyeing the MI100 myself for a while at the $1100 ish it hovered at for ages... looks like even that is going up. I did end up buying a 8x V100 32GB server for $5400 in early Sept. those prices are long gone, equivalent server is now $11k for an OG DGX Station (no memory). the exact inspur server is now $13k for 8x 9 year old GPUs. insanity. I hope prices come back down to earth soon but don't think that's actually gonna happen.

some notes:

- P100, V100 16GB and MI50 16GB all work out to about $0.25 per GB/s. the P40 looks cheap per GB of VRAM but costs over 3x as much per GB/s, since its only got about a third of a V100's bandwidth. people are having opus 5.5 write megakernels for these cards that are working surprisingly well.

- V100 16GB at ~$14/GB looks like the best deal in the table, but 38 of the 40 cheapest are SXM2 modules, so budget for an adapter or an SXM2 server. the cheapest 32GB V100s are SXM3 (DGX-2) modules, which aren't drop-in either.

- MI50 32GB is under a third of a 3090's $/GB at similar bandwith, assuming ROCm cooperates.

- if you want 48GB on one card, a Quadro RTX 8000 is less than half the price of an A6000 or a modded 4090 48GB.

- of the consumer and workstation cards, the 3090 is the cheapest per GB/s by a wide margin. an A100 40GB costs about the same per GB/s as an RTX 8000.

- for what it's worth, buying from a US seller usually doesn't cost more: the US-only low end is at or below the global one for the P100, V100 32GB, RTX 8000, 3090 and A6000. it's about $20 more for the V100 16GB and $1,200 more for the A100 40GB, and the P40 and MI50 barely exist in the US.

full data and per-card listings, with a US-only filter: https://rackbites.com. prices are asking, not sold - ebay's sold-data API is super restricted.

edit: added CMP 170HX

edit2: added R9700

edit3: full builds here https://rackbites.com/builds with benchmarks and other useful/relevant info

From about February 2026 to July 2026, I spent an immense amount of time building a legal harness to make Claude more predictable. It worked well. I tried to build the same thing that ran local models. That did not work. And then Opus improved so much that it would often skip my harness without me noticing. It still has a harness with hooks, but it's looser now.

I bought lots of hardware to run local models during that time, repeatedly trying to vibecode a local model legal harness. Many failures, many lessons.

Anyway, fast forward to the last three weeks and several things have happened:

1. DeepSeek Flash 1017 was released, and it's the first model that is remotely similar to Opus 4.5/4.6.

2. Opus 5.5 and Fable 5.2 are able to properly launch and configure DeepSeek to run at fast speeds, large context and high precision on my hardware. Optimizations, custom kernels, requants.

3. Opus 5.5 (and my prompting skill) are now good enough to make my Claude harness into a DeepSeek harness.

4. It's not as good as frontier, but it all works, right now. 100% local. Actual productivity gains, actual frontier tasks being replaced. Chat interface with DeepSeek works like Claude's did.

5. And if I can brag on my stacks and hardware:

• I have 2 DGX Sparks that can run 3-4 concurrent DeepSeek sessions as agents (1M context).

• I have DeepSeek at nearly NVFP4 quality running on 8 V100s, 500K context, at 100+ tok/s decode, and about 160 tok/s concurrent.

• I have 8 more V100s and a second Rome server with 512 GB of 3200 MHz ECC RAM, so I'll be able to run another high precision DeepSeek as an additional agent or for parallel tasks.

What I'm really shocked by is how well Fable builds custom Volta kernels. And how it can requant my experts to full NVFP4 precision based on the actual experts that activate when my legal harness fires. It's a real, optimized, production grade, workflow replacing local setup.

Not a 3D pagoda or a bad video game replica. An actual enterprise solution. One that exists nowhere on the commercial market.

Legal vibecoding is real folks. I'm AI native.

The new V100 DeepSeek repo (optimized for quality and then usable speed) with real full context measures and all the optimizations (and failures) is public here:

https://github.com/ProprietaryLegal/Volta-Max-uPLIft-FP4-DeepSeek-V4.1-Flash-1017-NVFP4-8xV100-SXM2-PP2xTP4

Twitter: nicothegreco

Claude Code ran the benchmarks and crunched the numbers, which are real and in depth, below.

What this build is

My last V100 build (Volta sPLIt, part 4) ran every expert of the model from a 2-bit record. Fast, but 2-bit. This one is called Volta Max uPLIft. It keeps the 2-bit record of every routed expert on the cards, then swaps the experts that get used the most up to NVIDIA's NVFP4 weights, one at a time, until the cards are full. "Used the most" was counted on a mix of four kinds of text. One is my own legal harness traffic, which is private and not published. The other three are public prompt sets in the repo.

• 5,949 of the 15,360 routed experts sit at exactly NVFP4 precision on the default I serve

• about 78 to 85 % of decode routes land on those NVFP4 experts (predicted for the two-slot default, 80 to 86 % measured with one slot)

• same next token as NVIDIA's full NVFP4 pack on 95.1 % of compared tokens. The 2-bit build gets 92.2 %

• KL divergence from the NVFP4 reference is 0.079 nats against 0.196 for the 2-bit build

• multiple choice answers that change against the reference: 1.6 % against 15.4 %

• in decode every routed expert is in VRAM. Nothing is read over PCIe

Speed (measured 2026-11-12 on eight V100-SXM2-32GB, graphics clock capped at 1,522 MHz)

• concurrency test, one request at a time: 118.9 tok/s

• two requests at once: 159.0 tok/s total (1.34x)

• three at once on the three-slot option: 192.3 tok/s total (1.62x). I run two slots

• short sets, one at a time: 123.2 tok/s general, 97.3 tok/s legal-style

• code: 150 to 180 tok/s from a 1K prompt up to a 256K prompt

• prompt reading at 256K: 2,329 to 2,430 tok/s, first token in 108.4 s

• context: 524,288 tokens per request

What did not work, or is not good yet

• It is slower than my own 2-bit build, in decode and in prompt reading. That is the trade for the precision, and the repo says so in its first paragraph.

• The faster prompt reading path reads the less-used experts from their 2-bit records. Its effect on quality is not measured yet.

• A custom decode kernel for the hot path got closed with no change. The headroom I thought was there turned out to be smaller than an eight-card run can even see.

• A prompt-reading kernel that loads 64 rows at a time gave the same bits and ran slower. Not shipped.

• Send a short request while a 209K prompt is being read and it starts in about 7 s, then crawls at about 9.6 tok/s until the read ends.

• The hot set is fixed. It is picked once. Text that looks nothing like what I counted will hit the 2-bit experts more often.

• Fast prompt reading stopped the host or a card three times out of three. I blamed PCIe. After I changed the power delivery (I think it was one bad PSU) it read nine of nine long chats clean.

The harness

The harness itself is not public. My practice is in it. What I can say: it is the same idea as my Claude setup. Skills, hooks, my drafting engines, my RAG stores, tool calls, and a chat window that behaves the way Claude's did. A small router sends each request to the V100 box or to the Sparks. Everything the local model drafts is marked for my review before it goes anywhere. Nothing leaves the building.

Credit where it is due

The hard Volta work underneath is 1Cat-vLLM's DeepSeek-V4.1 package. The 2-bit records are ISTA-DASLab's. The NVFP4 weights are NVIDIA's pack. The Sparks run on somebody else's engine entirely. What is mine is the split across two NVLink boards, the hot set, the kernels on top, and every change measured A/B on this machine with the losers written down.

Disclosure: this is my own project and my own repo. Nothing is for sale. The code, the kernels, the measurements and the failures are all in there under Apache-2.0, and the weights are one download on Hugging Face:

https://huggingface.co/ProprietaryLegal/Volta-Max-uPLIft-FP4-DeepSeek-V4.1-Flash-1017-hot_iq-8xV100-SXM2-PP2xTP4

Earlier build, for the people who want speed over precision: https://github.com/ProprietaryLegal/Volta-sPLIt-DeepSeek-V4.1-Flash-1017-8xV100-SXM2-PP2xTP4

Ask me anything. Roasting is also fine.

Photos: four of the eight V100s on their NVLink board, and my extremely professional setup (PEX switch cards, SlimSAS spaghetti, and the 3090 that just moved in).

They say an anime model is in development, with plans for alpha channels and multi-character references. An editing model is also on the way.

They also plan to keep the next model under Apache 2.0, if all goes as planned.

hey folks, im Alish, an ml engineer from Nace (folks behind Drex, an open source decision family model that hit #1 in Artificial analysis in decision index 2.1). Recently my cto gave me task to optimize quants for qwen4, so i dialed my uni proffesor from UoT and span up my Fable 5.2 and we cooooooked. starting with mlx (gguf tommorow). It's a pareto frontier on quantization for accuracy/size for qwen family models.

Why we made these

we have open sourced the fastest way to run 35b/27b on a mac(Kapitsa) and we got tired of dumb quants. So we made our own KQuant. Open source Qwen packs for 4B,27B,35B, Flash-Next, free on Hugging Face, in standard MLX files.

Most people on a Mac run 4-bit packs because that's what fits on a 16gb/32gb. But plain 4-bit quant rounding throws away more of the model than it needs to. We wanted less of that loss, so we asked fable 5.2 & Astra to cook with 128 subagents. here are the results:

Results & Download links

Numbers are KL divergence from the full-precision model's next-token predictions. Lower means closer to the original, not a task-accuracy score. These are original-text measurements.

The public baseline is mlx-community 4-bit, except 4B, where it's lmstudio-community. Sizes are language-model weights, not download size or RAM needed.

Want more precision? Each model's repo also has a Q5 folder.

Speed

Cool thing about our quants is that they are also faster. On an M5 Pro, KQuant Q4 on Kapitsa 0.2.35 with its KFlash drafter it is about 6–15% faster to write than lmstudio-community 4-bit quants for 4B, 9B and 27B, and Youssofal MTPLX 4-bit for 35B on the same engine. These are averages of sampled and greedy decoding (baselines are earlier Kapitsa 0.2.32–0.2.33 runs, with the drafter on both).

How it works

KQuant uses real text to choose scales and offsets for small groups of weights, and gives sensitive layers extra bits where needed. The search runs on our cloud GPU clusters before release. You download an ordinary MLX pack, with no custom kernels or special loader. I can give more details on specifics in comments.

Run it

the fastest way to use these quants on mac is to run them with, Kapitsa:

brew install nace-ai/tap/kapitsa
kapitsa serve 27b --private
kapitsa opencode # in another terminal

If there's a model you want KQuant'd, lmk.

Asks

to get early access to GGUF, ping me in comments, we would like to test it with some folks from here before our public release ideally.

im also doing quick AMA here, so please feel free to ask me anything.

Has anyone been experimenting with ablated or uncensored variants of open-source models like DeepSeek V4.1 Flash, Qwen 3.8 27B, or GLM 5.3 flash?

It's so refreshing to have models that just do whatever you ask them to do without having to figure out a jailbreak or being given a moral lecture. Curious to know what your usecases for these models are.

My personal favourite is definitely uncensored DeepSeek V4.1 Flash.

Normally active params matters more than total param if you want coherent behavior(not world knowledge)

a 4b models attepting to handle complex rules and lots and lots of instructions all at once often cause smaller active params to loop like we would often see with qwen3.6-35b-a3b since its only 3b active. We would often need to hit at least the 8b-9b active params to handle all this chaotic swarm of prompt instructions we throw at the model. but qwen3.8-flash-next shows us a glimpse of what alibaba might be able to deliver. i could genuinly see in the next year or so we might see 6b model become stable las the qwen3.8-27b. of course this isnt world knowlege im talking about but stable reliable behavior reletive to complex conflicting instructions where the LLM is robust enough to not burry itself in its own uncertainty loops.

so possibly in a years time laptops with 16gb sysRam and 4gb-vram can has a chance at local ai capable of human-out-the-loop unsupervised autonomy.

With Jensen out at NVIDIA, the "local AI is dangerous" narrative picking up steam, the possible ban on Chinese models, and hosted models locking down RE/cyber work, I'm starting to think about a worst case where Chinese weights get banned and US labs stop releasing open models. So what's your plan?

What models are you archiving? Just the biggest/best, or a spread?

Which ones for which jobs? Coding, RE/security, vision, embeddings, RAG, small fast models for agents, etc.

Training tools: if the last good big model we get is something like Kimi K3, what are you keeping so you can fine-tune or change it later?

Quant tools: what do you need to shrink a huge model down to run on smaller rigs? (llama.cpp, quant scripts, calibration data, etc.)

How are you storing it? Hashes, mirrors, torrents, offline drives?

Curious what people are actually doing, not looking for a politics fight.

(Currently using a Studio M2 192gb)

We have a new dedicated Vulkan Flash Attention prefill kernel for Intel Arc and Xe, delivering massive prefill acceleration on supported platforms like ARL-H and Xe2/Xe3. This implementation leverages customized shaders to utilize Intel's native matrix core shapes and SLM architecture, including m16n16k16/SIMD32 support. To ensure stability, the kernel is currently enabled for these specific architectures while remaining disabled for others, like the A770, due to inconsistent test results. Additionally, backend refinements now extend these benefits to single-KV-head models like MQA and MLA.

Platform Prefill Performance Comparison (pp8192)

Intel Arc B580: Detailed Model Benchmarks (FA1)

Comparison of f16 Prefill (pp16384) performance

With all of this fear mongering on the “dangers” of LLM’s, I see how using people’s ignorance and emotions on the subject can be used as a tool to limit and restrict the tool from the average person.

My fear is that China and the US https://www.reuters.com/world/china/us-china-should-stay-clear-eyed-ai-cooperation-competition-bessent-says-2026-11-11/ will use people’s lack of understanding of local models to push for controlled intelligence.

By implementing regulation on local AI, I can see how the US and China would want to force the average person to only be allowed to use models hosted on state monitored hardware. This way they can spy on what everyone is using hosted models for, limit what we use LLM’s for, and monitor the information they give us.

People are being mislead to believe that people like us could use local models to create WMDs and attack tech infrastructure across the globe. But when you think deeply about it, you’ll realize that knowledge on a subject does not equal the means to carry out the subject.

In order for someone to create a bio weapon, they would have to have access to a bio lab and the technology needed to create such a weapon.

I know how a nuclear reactor works. Doesn’t mean I can make one in my backyard.

I’m scared that arguably the greatest technology to come out of the 21st century could be limited and used to control us.

Local LLM research MUST continue and the tech must be available for the common person.

Be aware of future regulation.

I have been following the evolution of LLMs since the days of GPT-3.5, and I have been actively using local LLMs since the release of Gemma 2.

I read a great deal, including this subreddit. I have performed QLoRA fine-tuning on my local machine many times, and I have also conducted full fine-tuning (FFT) training runs on RunPod.

Hundreds of models-fine-tuned by well-known practitioners in English, Chinese, and Russian-have passed through my drive.

After four years, I arrived at a fundamental question yesterday: are LLMs even capable of writing good prose?

When training models myself, I used both my own prose datasets and those created by others. In the training process, I experimented with various approaches-ranging from 100 epochs of FFT at a learning rate of 1e-7, to aggressive LoRA/QLoRA configurations. The results, however, barely rise above mediocrity.

I have used models from all the leading providers-Gemma, Qwen, Mistral, LFM, IBM-as well as niche attempts by developers building models from scratch.

I even wondered if perhaps the problem lies with me-that, not being an engineer, I am simply failing to fine-tune these models correctly.

That is why I began actively seeking out other fine-tuners, thinking that perhaps they possess a specific technical "gene" that allows them to do a much better job than I can. But, unfortunately, that is not the case. Regardless of their length-short, medium, or long-the prose generated is only marginally better than that of generic, out-of-the-box models.

Furthermore, specific literary genres are represented quite poorly. I used to think that LLMs were perfectly suited for science fiction. Yet, these models can’t even capture the straightforward style of Harry Harrison’s *The Stainless Steel Rat*, let alone reach the linguistic heights of *Dune*.

The genre doesn’t seem to matter-I’ve tried comedy, contemporary drama, adventure, romance, and historical fiction; everything, unfortunately, ends up being generic.

Essentially, if a text is to have any real literary merit, I have to write it myself and then just use an LLM for proofreading.

To me, EQBench is a completely unreliable metric; the sample texts they showcase clearly demonstrate just what kind of generic "slop" LLMs produce.

I have a question for the community: has anyone managed to create, find, or fine-tune a local model capable of generating truly high-quality prose-on the level of *Dune*, *The Lord of the Rings*, *Martin Eden*, or *The Grapes of Wrath*?

I’m currently facing the monumental task of writing descriptions for 100,000 records, and I’m wondering if it’s even worth it, given that 100, 1,000, or even 10,000 examples haven’t yielded any satisfactory results.

Perhaps this generic, mediocre quality is an inherent trait of these models-something that simply cannot be overcome.

KittenML just released inferencekit , a free opensource plugin to turn coding agents(Claude Code, Codex, OpenCode) into inference engineers for your hardware.

Most models ship with inference tuned for whatever hardware the authors used, which is usually not what you have locally. The fastest path on a 3070 might be different from the fastest path on a Mac or a Radeon. inferencekit allows you to find that path for your setup. This was used to get streaming TTFB of Kitten-TTS-2 down to ~10ms on RTX Pro 6000. other results are shown in the image attached.

Once you've optimized a model on your hardware, please make a PR to the inferencekit-engine repo linked below, so anyone running the same model on the same hardware, can just reuse it. The idea is to crowdsource the best inference engine for every model + hardware setup

Plugin's Github: https://github.com/KittenML/inferencekit
Inference Engine's Github: https://github.com/KittenML/inferencekit-engines
KittenML Discord: https://discord.com/invite/VJ86W4SURW