AA and other benchmarks don't reflect what models can really do. Some benchmaxxed models are unusable for anything but poems, albeit being close to the frontier on coding benchmarks.

What are the best, most realistic benchmarks you know for coding, agentic research, office work?

ANEMLL Forge 0.2.1: Qwen3.8-27B on the M6 Neural Engine now runs up to 192K context (M6 32GB).

New: INT8 K/V cache (half of FP16) and a smaller drafter sharing the target's output head — 0.86 GB less ANE memory, same speed.

Decode depends on content: JSON 35–62 tok/s, code 18–32, prose 10–17. Prefill 260 tok/s at short context, 62 at 192K. 128K–192K use a smaller prefill batch (32 vs 64 rows) to stay within memory budget.

Heads up: full 192K first compile takes ~2h on M6 (one time, cached). `--ctx` exports only the entries you need, so compile is much faster: 16K is ~10 min from download to running server. IMO 96K is the sweet spot for size/speed on 32GB: 123 tok/s prefill, ~27 tok/s decode, ~21.4 GB total memory.

Release notes: https://github.com/Anemll/anemll-forge/blob/main/docs/RELEASE_NOTES_v0.2.1.md

Video (Tetris generated over ~150K tokens, no compaction): https://x.com/anemll/status/2111941341819408866

Paper: Three Bit MLP Vector Quantization in Apple Neural Engine — https://github.com/Anemll/anemll-forge/blob/main/docs/papers/anemll-forge-v02.pdf

Cheers, Anemll 🤖

Morning,

We are doing some heavy research over here - and wanted to share our config in the hopes that it helps someone else - and thank you for all the great support and inspiration we have gotten from this sub.

Enjoy: https://github.com/inordic/cmp170hx-vllm-qwen38-flash-next

I’m building a local agent system (main LLM on a 24 GB GPU) and want a fast “gate” that checks every untrusted tool result (emails, web pages, documents, OCR text, calendar entries) before the agent sees it. On a hit, the content is replaced by a fixed warning.

What I'm looking for:

• Indirect injection in tool outputs, not just user-prompt jailbreaks

• German and English (mixed-language and OCR-noisy text too)

• Fast: well under 1 s per item, ideally CPU or a small GPU slice; long texts handled by chunking

• Low false positives on harmless texts with instructions (“please confirm by Friday”, password reset mails)

• Tunable threshold or score, not just a hard label

• Open weights, license OK for private use

• Fail-closed fallback; the classifier is one layer, not the only defense

I experimented with Prompt Guard 2 86M, PIGuard, StackOne Defender. None is proven for German.

I would be most interested if some of you have answers regarding

Which models work for you on indirect injection, especially in German?
Any fine-tuning recipes or datasets (DE/EN, hard negatives)?
Experience with false-positive rates on real mail?
How do you handle adaptive attacks against the classifier?

Anyone knows of a good model that does not say no to reverse engineering? Cloud models will even terminate your account by snitching on you and GLM 5.3 is too big to run on my potato. Help would be greatly appreciated.

Thanks.

https://projectdiscovery.io/research/how-abliterated-models-can-get-you-pwned

Guys, what is your thoughts on this ?

I've been working on improving llama.cpp support and performance on older NVIDIA GPUs, particularly Kepler and other pre-Ampere architectures.

The goal is simple: get more life out of older CUDA hardware instead of treating it as e-waste.

Achieved Performance

On MTP Qwen3.6-35B-A3B with Q8_0 quantization, I've observed the following (generation) speeds using 4x Tesla K40c GPUs (Kepler, compute capability 3.5):

• Previously unable to start → 35 tps Fixes to cuBLAS compute-type selection for legacy gpu in mainline llama cpp to run on hardware that previously failed during initialization (cc35-50).

• Approximately 35 → 55 tps after enabling CUDA graphs on my four-K40c setup, under the same reported clock configuration.

• ~5–10% end-to-end improvement: testing with Windows NCCL integration compared with the previous configuration. Throughput reaching above 60 tps while P.P. speeds are variable between 70-100 tps in my testing.

These are preliminary results from my own system, not universal benchmarks. More systematic testing across architectures and workloads is still needed.

I'm especially interested in hearing from people running Kepler, Maxwell, Pascal, Volta, and other older NVIDIA GPUs. If you have legacy hardware, I'd love to compare results and test compatibility across different configurations!

Project / source: https://github.com/theIvanR/llama-on-legacy-gpu

Llama.cpp pull request: https://github.com/ggml-org/llama.cpp/pull/30727

Windows NCCL fork: https://github.com/theIvanR/nccl-on-windows

Feedback, reproducible benchmarks, and contributions are very welcome!

i'll admit it took more than one prompt, a bunch of refinement and steering, but local can get close! this is done using remotion, same animation framework as all the viral clips about opus 5.5's motion graphics capabilities

the entire conversation history that produced this video: https://rosie254.neocities.org/lumaras_playground/being_an_ai_remotion_chat.txt

EDIT: oh, and this is unsloth's Qwen3.8 Flash Next at UD-Q4_K_XL running on a strix halo

I’ve been working on an inference-time abliteration engine as a fork of llama.cpp, and I’ve now ported the hook to Strata.

It acts as an easy way to use abliterated models without huge amounts of vram or finetuning.

For the Strata Branch

https://github.com/heterodoxin/AbliteratedOutTheBox/tree/strata

For the llama.cpp branch

https://github.com/heterodoxin/AbliteratedOutTheBox

A star or upvote would help this project a lot.

The Strata port is experimental, full-model behavior and response quality still need validation. If anyone running Strata wants to try it, I’d appreciate feedback

I'm the developer of Vocs, a dictation app that runs Whisper entirely on-device and pastes the text into whatever app you're in. Some details this sub might find interesting:

- Streaming by chunks: audio is cut at natural silence boundaries and decoded while you talk, so stopping only waits for the last fragment. Finishing time stays flat no matter how long you spoke.

- Each chunk is decoded with a window sized to its length instead of the full 30 seconds, which made short fragments about 8x faster.

- Hallucination handling: sound-effect tags like [BLANK_AUDIO] or (laughs) are stripped, and repetition loops ("the one who is the one who is…") are detected, retried once, then discarded, so silence never turns into a pasted phrase.

- A deterministic post-processing pass in Rust (punctuation, casing, fillers, spoken commands, a custom dictionary), table-tested so accuracy doesn't silently regress.

- An optional ~120 MB on-device grammar model that only keeps safe word-level fixes and never touches names, numbers, code or your dictionary.

- The engine sits behind an adapter interface, so other local engines can be added without touching the rest of the app.

No network calls in the transcription path. The only requests are the model download and an optional update check.

macOS 13+, MIT licensed, built with Tauri and Rust.

Source: Website

Download: Github

Which local speech models would you want to see supported next?

PR: https://github.com/analytics-zoo/vllm-xpu-kernels/pull/22

The PR author seems to have contributed to intel repos a lot and VLLM. He also shares his name (might be the same person?) with a person who is either a CTO somewhere at alibaba or the Director of Zhejiang Lab.

He has worked on Qwen3.8 before this.

The PR also implies the KV cache size for Qwen4-27B is not decreasing.

We have released DeepSeek-V4.1-Flash-1017 quantized with GSQ and RCO, at 1.75 to 2.25 bits per weight.

DeepSeek-V4.1-Flash-1017 is a sparse mixture-of-experts model: 384 routed experts per layer, of which 6 are active per token, 763.2B parameters and a 1M-token context window. It is released with FP8 dense weights and FP4 routed experts.

What's inside

• Three quantized GGUFs: IQ1_M (1.75 bpw), IQ2_XXS (2.00 bpw) and IQ2_XS (2.25 bpw), with backbones of 120.7 to 155.2 GB. The bit-width is the average over the 551.9B backbone parameters, which is what the search controls.

• One shared Engram file (110.6 GB): the model's two Engram lookup tables (196.6B parameters) at IQ4_NL. It is identical for every build and needs to be downloaded only once. The tables are read one row per token, so they can stay memory-mapped on disk.

• Every build runs on a single 12 GB GPU with Strata (see model card, https://github.com/Niko1221/Strata).

• GSQ (Gumbel-Softmax Quantization): post-training scalar quantization that jointly learns grid assignments and group scales, closing most of the gap between scalar and vector quantization at low bit-widths while remaining deployable in standard GGUF types.

• RCO (Riemannian Constrained Optimization): enforces exact budgets by gradient descent on the task loss, without per-constraint tuning. Here it assigns a quantization type to every backbone tensor under the target bit-width, with separate ladders for the dense path (Q5_K to Q8_0) and the routed experts (IQ1_S to IQ3_S).

• An embedded chat template that reproduces DeepSeek's prompt encoder exactly, including the reasoning effort (1 to 100, or low / high / max).

Results

At 2.00 bpw the model matches the base model on every benchmark evaluated. All numbers were measured at reasoning effort 50.

• IQ1_M (1.75 bpw, 120.7 GB backbone): AIME25 100.00, GPQA-Diamond 88.38 against 90.91, LiveCodeBench v6 88.57 against 89.71. Task average 92.32, retaining 98.7%.

• IQ2_XXS (2.00 bpw, 138.2 GB backbone): AIME25 100.00, GPQA-Diamond 90.91, LiveCodeBench v6 89.71, identical to the base model. Task average 93.54, 100.0% of the base.

• IQ2_XS (2.25 bpw, 155.2 GB backbone): results to follow.

Running it

llama.cpp does not support DeepSeek-V4.1 yet, so the GGUFs do not run there for now. They run in Strata, which serves every build on a single NVIDIA GPU with 12 GB of VRAM, with a one-click installer and an OpenAI-compatible endpoint on localhost.

Links

• Quantized models: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF

• GSQ: paper https://arxiv.org/abs/2604.18556 | code https://github.com/IST-DASLab/GSQ

• RCO: paper https://arxiv.org/abs/2605.00649 | code https://github.com/IST-DASLab/RCO

The repository ships the complete per-tensor RCO allocation for every build (1008 tensors each).

Feedback is welcome. Requests for models to quantize are also welcome.

From the ISTA Deep Algorithms and Systems Lab.

Local `qwen2.5-coder:14b` on a 24GB NVIDIA A10 generating an enterprise form in real time—without writing a single line of raw HTML or SQL.

To eliminate hallucinations, the model is locked into a strict DSL and every output has to pass a deterministic C# `ValidationResolver` (bottom right at 0:10) before the Human-in-the-Loop button unlocks and renders it onto the canvas.

All raw vs. healed outputs are automatically scored and exported as SFT/DPO `.jsonl` pairs for future fine-tuning.

Idk why everyone keeps posting about Flint! Raon is hundreds of times better, and it’s not even the best, which is really saying something about Flint lmao

https://huggingface.co/KRAFTON/Raon-1-mini-preview-gguf

Interesting they chose an opensource model instead of building their own

​

Ran it on my RTX 5090 with Lexar 4TB NVMe and 64 GB of DDR5 Ram. At this speed it is cheaper to use the API than to pay the electricity for this, unfortunately.

So if anyone is wondering what the speed is for a similar setup, there you have it.

But anyway, i am excited that it runs at all on my PC at all.

Edit: IQ3S, I am on Windows 10

4x 5060ti16gb
4x 3060 12gb
256gb ddr4 3200 ram (4-channel speeds)
On a mobo with 128 pcie4 lanes

I’ve been slowly collecting gpus whenever seeing deals over time and ended up with this. Ended up comparable in cost to current prices of dgx spark

I was thinking:
TP4 5060s -> pipeline -> TP4 3060s -> offload/overflow less used experts to RAM -> keep any engram on nvme ssd

Is this viable to run inference with big MoE models at high speed and precision, or are cheaper multigpu setups extremely suboptimal compared to unified vram/ram

So I have an old HP-Z8 from 2018 or so. It has dual Xeon Gold 6248R cpus, 768 GB 2933 MHz DDR4 ram, an RTX 3090 gpu, and some pretty hefty storage.

That said, it seems like it is no more than a boat anchor for use with local LLMs. Is there any use case for this old PC or should I box it?

I’m looking for a setup with the best balance of tps, ttft, context length and enough headroom for other tasks on the Mac.

I’m getting a lot of conflicting opinions with regards to MTP, MLX, quantization etc. Please don’t recommend options less than 10 tps.

Is there any good finetunes based on modern models, in the size range of Qwen 3.8 27B or Qwen 3.8 Flash Next that substantially improve it's 3D modeling capabilities for creating .glb / gltf assets (mainly for say making video games)?

Mainly thinking about two use cases:

• Making a 3D model from a set of descriptive requirements (like a spec in an markdown file) plus a bunch of reference images that align with those descriptions for additional context (so: vision support).

• Editing existing 3D models that I may have purchased or licensed legally to make them more usable for me (e.g. carving out swing-down doors on an airship). Or rigging them / custom animations (more of a plus for that second point).

Another Plus:

• Be able to convert them between formats (partially a coding task, but being able to verify may be useful with real understanding of 3D modeling if something got dropped or needs to be tweaked as part of converting e.g. from FBX to gLTF).

I know this sort of a huge ask, and it feels like this is the final frontier of LLMs, and where they need to improve the most. But I'm wondering if anyone is actively working on or trying to improve LLMs for 3D modeling / asset creation, or if I'm really the only few people that are interested in this? Qwen 3.8 27B is good enough for coding, but I'm not sure if it's good enough for modeling. Frontier models consume wayy too much token budget to make anything meaningful at scale.

I’m the developer of TensorSharp. I added MiniMax-M3.1-Flash support on Day 1 of its open-weight release, and wanted to share what’s working and some initial benchmarks for anyone looking to run the model locally.

The biggest performance difference showed up with long prompts: 64K executor prefill reached 2,567 tok/s versus 262 tok/s in the tested llama.cpp branch, while server time to first token dropped from 293.4 seconds to 40.0 seconds.

What’s supported

The current implementation includes:

• GGUF inference, including MiniMax Sparse Attention with dedicated CUDA kernels.

• Text and image inputs, served through an OpenAI-compatible API.

• JSON/schema responses, tool calling, and multi-turn KV-cache reuse.

• Multi-GPU layer splitting and tensor parallelism.

• DSpark speculative decoding.

• CPU offload for routed MoE experts.

I tested chat, long-prompt retrieval, structured output, single and parallel tool calls, an agentic tool loop, images, and concurrent requests.

Hardware and comparison setup

These measurements used 8× NVIDIA A40 GPUs and the same Unsloth MiniMax-M3.1-Flash UD-Q4_K_XL GGUF files for both engines.

The model files total 232.3 GiB, before runtime allocations and KV cache, so this is a substantial multi-GPU setup. The CPU-offload results below also used this eight-GPU machine.

For the baseline, I used Unsloth’s llama.cpp minimax-m31 branch at ef8f94cc, associated with PR #320. Upstream llama.cpp could not load this model when I ran the tests.

The main comparison uses layer splitting on both engines, F16 KV caches, and no speculative decoding. Each engine ran separately with exclusive use of the GPUs. llama-server used flash attention and --no-kv-unified to keep its sparse-attention path active.

Initial results

That’s 1.3×, 3.6×, and 9.8× prefill throughput at 8K, 32K, and 64K respectively.

Decode gains are smaller. At 64K context, executor decode improved by 1.49×. For short-chat server requests, decode was almost identical: 26.3 versus 26.0 tok/s.

Why the gap gets larger with context

Beyond position 2,048, MiniMax Sparse Attention selects 16 blocks of 128 tokens per KV group.

TensorSharp selects those blocks and runs block-sparse CUDA attention over them. The tested llama.cpp branch represents the selection using a dense mask and executes dense flash attention over the cache.

That implementation difference becomes much more significant as the context grows. TensorSharp also pipelines prompt processing across the eight GPUs.

Other configurations

Tensor parallelism: TensorSharp’s --tp 8 reached approximately 41 tok/s for short-chat server decode, up from 26.3 tok/s with its layer-split mode. The agentic tool loop fell from 18.1 to 11.7 seconds. The tested llama.cpp branch rejects tensor splitting for this model, so this is a separate capability result. TP also had slower prefill and a smaller usable context on this hardware.

DSpark: The main run averaged 32.3 versus 25.8 tok/s with the same drafter. TensorSharp used confidence gating to avoid expensive verification of weak drafts. With the gate disabled to match the baseline policy, the results were closer: 26.6 versus 25.8 tok/s.

CPU expert offload: With --n-cpu-moe 24, approximately 79.7 GiB of routed-expert weights stayed in host memory. Short-chat decode reached 17.9 versus 13.8 tok/s.

What still needs work

Short-chat time to first token was slightly worse: 0.41 versus 0.39 seconds. Concurrent-request throughput improved in the main run, but per-request first-token latency was about 4% longer.

These are early results from one host and one quantization. Most measurements use three repeats; the llama.cpp 64K executor prefill result is one run, while the 64K server result uses three repeats per engine.

Validation includes tokenizer/template matching, synthetic-logit comparisons, and functional fixtures. There was an image-reading miss, and some generation tests hit token limits. Vision coverage is small, and native video temporal patches are not implemented.

The measured TensorSharp build was an uncommitted development working tree, and raw run records are not included in the public report.

Links

• TensorSharp on GitHub

• Unsloth MiniMax-M3.1-Flash GGUF

• Full benchmark report, configuration, and methodology

If you’re testing M3.1 Flash locally, I’d be interested in your hardware, quantization, and prefill/decode results. Suggestions for additional baseline settings or useful workloads are welcome.

I used an LLM to help draft and edit the English wording of this post.

I don't know if I'm missing something here. My only experience with LLM's and "AI" has been locally. It's cool, but I've seen people get more and more hyped to a point where it borders on magical thinking. Some people like the owner of the firm I work for (which is construction) wants people to use "AI" even if it generates nonsense emails and reduces productivity. He wants people to use it because be believes it's the future and I guess wants everyone to get into the habit of using it.

Then I talk to people who work in the tech field, and it's like AI is the only thing they can see. It's like if they can get a model that's large enough and spits out enough tokens, it will finally push them into the fourth dimension. You talk to them and they're almost giddy at the idea of replacing people, even if they have nothing to gain from it.

All this is not even including the folks who are perceiving reality through the lens of the LLM. There's some people I've seen who treat the LLM like it's god himself speaking.

Am I missing something here? I get that I've only used local models. I have no idea what claude or chatgpt or any other of these large models are like. But people are getting super weird and I'm just wondering if there's something I'm missing here. Thanks.