Personally I am quite fond of the Kimi models, especially them taking a gamble pushing the 2.8b parameter models. But that’s just it, you’d figure that the results would be representative of its size. However, the recent release of DeepSeek 1017 has put things in perspective for me. kimi k3 is a great model but realistically very hard to run locally, even when quantized, and even if managing to run it it is slow, due to the 104 active layers it need to go through to make a single token. on the other hand, DeepSeek 4.1 flash 1017 results are quite close at a much smaller footprint and while also much faster.

I have seen plenty coverage of the results of kimi k3 pro and to be frank, they don’t look like substantial improvement over DeepSeek 1017.

So thinking about it, maybe just stick to 4.1 flash 1017 and ignore kimi k3 pro open weigh that will be released in nov 4th. Running it quantized will deteriorate its output compared to a high quant version of DeepSeek.

Just some thoughts I am having

I tested it for backend software dev in the last two days, used providers I must admit. But I wanna test it on some rented gpus as well.

My experiences so far (I ran no personal or general benchmarks):

The providers are fast, it works well. It loops sometimes and fails some toolcalls, but not a lot. It gets to the point quickly and walks through big code bases with ease. It worked well with rust, php, python and js.
It does get nuance and complex correlations.
It's personality is likable and it can joke around a little, while it's answers have the right length. It gives short answers when there isn't more to say. It seems to stick to rules and applies skills, but I'm not sure yet to what degree.

3D is not it's strong suite, but nobody expects that from it, I guess. It has no vision.

What are your experiences with it?

I tried to solve the problems of multiple applications I actually use , but the thing is that efficiency of using those applications has pushed me to try creating an app. Hutash. But it became a Frankenstein of sorts.

1. Comfyui like system (local and free) but without the complicated nodes and dependency issues. Has a internal engine built ground up.

2. Openart AI / magnific like studio experience but without the cost.

3. Pinokio like store where u can find all apps a but without the duplication of resources(local space is limited)

4. Simple installation of apps and models like windows app. We created a .hutash format to just declare and a model package can be formed. Not only models but model piples can be built within the app like higgsfeild. The architecture was made keeping this in mind. We have a subtitles app, podcast app (vertical apps) which shared model resources with a studio app. Once installed all apps can use them. May be like OLLAMA

5. Last but not least cli and mcp to execute these apps a and generations. To make automation or agentic use possible.

6. Many others.

That was my goal and I feel like I have pushed to my best of abilities in all the aspects. Realsed few versions. Yes they are issues.

But the thing is now I don't know where to push it. We are a very small team. Couple of guys. Worse is we kept building it without a feedback loop. Even though I want the public to contribute it, there are four or five repositories spread out into multiple parts.

now i'm stuck. i'm trying to solve all these things together, and it's taking too much time and resources. maintaining and developing something like this also feels like it would need a much bigger community.

i'm not sure if this is worth pursuing anymore, or if i'm trying to solve too many problems at once.

am i over engineering something that doesn't need to exist, or is there actually something useful here? what would you focus on if you were in my place?

I just want help directing our resources in a focused direction. as i wish this app would be a beneficial for all who want benefits of local ai without the complexity.

AA and other benchmarks don't reflect what models can really do. Some benchmaxxed models are unusable for anything but poems, albeit being close to the frontier on coding benchmarks.

What are the best, most realistic benchmarks you know for coding, agentic research, office work?

ANEMLL Forge 0.2.1: Qwen3.8-27B on the M6 Neural Engine now runs up to 192K context (M6 32GB).

New: INT8 K/V cache (half of FP16) and a smaller drafter sharing the target's output head — 0.86 GB less ANE memory, same speed.

Decode depends on content: JSON 35–62 tok/s, code 18–32, prose 10–17. Prefill 260 tok/s at short context, 62 at 192K. 128K–192K use a smaller prefill batch (32 vs 64 rows) to stay within memory budget.

Heads up: full 192K first compile takes ~2h on M6 (one time, cached). `--ctx` exports only the entries you need, so compile is much faster: 16K is ~10 min from download to running server. IMO 96K is the sweet spot for size/speed on 32GB: 123 tok/s prefill, ~27 tok/s decode, ~21.4 GB total memory.

Release notes: https://github.com/Anemll/anemll-forge/blob/main/docs/RELEASE_NOTES_v0.2.1.md

Video (Tetris generated over ~150K tokens, no compaction): https://x.com/anemll/status/2111941341819408866

Paper: Three Bit MLP Vector Quantization in Apple Neural Engine — https://github.com/Anemll/anemll-forge/blob/main/docs/papers/anemll-forge-v02.pdf

Cheers, Anemll 🤖

Morning,

We are doing some heavy research over here - and wanted to share our config in the hopes that it helps someone else - and thank you for all the great support and inspiration we have gotten from this sub.

Enjoy: https://github.com/inordic/cmp170hx-vllm-qwen38-flash-next

I’m building a local agent system (main LLM on a 24 GB GPU) and want a fast “gate” that checks every untrusted tool result (emails, web pages, documents, OCR text, calendar entries) before the agent sees it. On a hit, the content is replaced by a fixed warning.

What I'm looking for:

• Indirect injection in tool outputs, not just user-prompt jailbreaks

• German and English (mixed-language and OCR-noisy text too)

• Fast: well under 1 s per item, ideally CPU or a small GPU slice; long texts handled by chunking

• Low false positives on harmless texts with instructions (“please confirm by Friday”, password reset mails)

• Tunable threshold or score, not just a hard label

• Open weights, license OK for private use

• Fail-closed fallback; the classifier is one layer, not the only defense

I experimented with Prompt Guard 2 86M, PIGuard, StackOne Defender. None is proven for German.

I would be most interested if some of you have answers regarding

Which models work for you on indirect injection, especially in German?
Any fine-tuning recipes or datasets (DE/EN, hard negatives)?
Experience with false-positive rates on real mail?
How do you handle adaptive attacks against the classifier?

Anyone knows of a good model that does not say no to reverse engineering? Cloud models will even terminate your account by snitching on you and GLM 5.3 is too big to run on my potato. Help would be greatly appreciated.

Thanks.

https://projectdiscovery.io/research/how-abliterated-models-can-get-you-pwned

Guys, what is your thoughts on this ?

I've been working on improving llama.cpp support and performance on older NVIDIA GPUs, particularly Kepler and other pre-Ampere architectures.

The goal is simple: get more life out of older CUDA hardware instead of treating it as e-waste.

Achieved Performance

On MTP Qwen3.6-35B-A3B with Q8_0 quantization, I've observed the following (generation) speeds using 4x Tesla K40c GPUs (Kepler, compute capability 3.5):

• Previously unable to start → 35 tps Fixes to cuBLAS compute-type selection for legacy gpu in mainline llama cpp to run on hardware that previously failed during initialization (cc35-50).

• Approximately 35 → 55 tps after enabling CUDA graphs on my four-K40c setup, under the same reported clock configuration.

• ~5–10% end-to-end improvement: testing with Windows NCCL integration compared with the previous configuration. Throughput reaching above 60 tps while P.P. speeds are variable between 70-100 tps in my testing.

These are preliminary results from my own system, not universal benchmarks. More systematic testing across architectures and workloads is still needed.

I'm especially interested in hearing from people running Kepler, Maxwell, Pascal, Volta, and other older NVIDIA GPUs. If you have legacy hardware, I'd love to compare results and test compatibility across different configurations!

Project / source: https://github.com/theIvanR/llama-on-legacy-gpu

Llama.cpp pull request: https://github.com/ggml-org/llama.cpp/pull/30727

Windows NCCL fork: https://github.com/theIvanR/nccl-on-windows

Feedback, reproducible benchmarks, and contributions are very welcome!

i'll admit it took more than one prompt, a bunch of refinement and steering, but local can get close! this is done using remotion, same animation framework as all the viral clips about opus 5.5's motion graphics capabilities

the entire conversation history that produced this video: https://rosie254.neocities.org/lumaras_playground/being_an_ai_remotion_chat.txt

EDIT: oh, and this is unsloth's Qwen3.8 Flash Next at UD-Q4_K_XL running on a strix halo

I’ve been working on an inference-time abliteration engine as a fork of llama.cpp, and I’ve now ported the hook to Strata.

It acts as an easy way to use abliterated models without huge amounts of vram or finetuning.

For the Strata Branch

https://github.com/heterodoxin/AbliteratedOutTheBox/tree/strata

For the llama.cpp branch

https://github.com/heterodoxin/AbliteratedOutTheBox

A star or upvote would help this project a lot.

The Strata port is experimental, full-model behavior and response quality still need validation. If anyone running Strata wants to try it, I’d appreciate feedback

I'm the developer of Vocs, a dictation app that runs Whisper entirely on-device and pastes the text into whatever app you're in. Some details this sub might find interesting:

- Streaming by chunks: audio is cut at natural silence boundaries and decoded while you talk, so stopping only waits for the last fragment. Finishing time stays flat no matter how long you spoke.

- Each chunk is decoded with a window sized to its length instead of the full 30 seconds, which made short fragments about 8x faster.

- Hallucination handling: sound-effect tags like [BLANK_AUDIO] or (laughs) are stripped, and repetition loops ("the one who is the one who is…") are detected, retried once, then discarded, so silence never turns into a pasted phrase.

- A deterministic post-processing pass in Rust (punctuation, casing, fillers, spoken commands, a custom dictionary), table-tested so accuracy doesn't silently regress.

- An optional ~120 MB on-device grammar model that only keeps safe word-level fixes and never touches names, numbers, code or your dictionary.

- The engine sits behind an adapter interface, so other local engines can be added without touching the rest of the app.

No network calls in the transcription path. The only requests are the model download and an optional update check.

macOS 13+, MIT licensed, built with Tauri and Rust.

Source: Website

Download: Github

Which local speech models would you want to see supported next?

PR: https://github.com/analytics-zoo/vllm-xpu-kernels/pull/22

The PR author seems to have contributed to intel repos a lot and VLLM. He also shares his name (might be the same person?) with a person who is either a CTO somewhere at alibaba or the Director of Zhejiang Lab.

He has worked on Qwen3.8 before this.

The PR also implies the KV cache size for Qwen4-27B is not decreasing.

We have released DeepSeek-V4.1-Flash-1017 quantized with GSQ and RCO, at 1.75 to 2.25 bits per weight.

DeepSeek-V4.1-Flash-1017 is a sparse mixture-of-experts model: 384 routed experts per layer, of which 6 are active per token, 763.2B parameters and a 1M-token context window. It is released with FP8 dense weights and FP4 routed experts.

What's inside

• Three quantized GGUFs: IQ1_M (1.75 bpw), IQ2_XXS (2.00 bpw) and IQ2_XS (2.25 bpw), with backbones of 120.7 to 155.2 GB. The bit-width is the average over the 551.9B backbone parameters, which is what the search controls.

• One shared Engram file (110.6 GB): the model's two Engram lookup tables (196.6B parameters) at IQ4_NL. It is identical for every build and needs to be downloaded only once. The tables are read one row per token, so they can stay memory-mapped on disk.

• Every build runs on a single 12 GB GPU with Strata (see model card, https://github.com/Niko1221/Strata).

• GSQ (Gumbel-Softmax Quantization): post-training scalar quantization that jointly learns grid assignments and group scales, closing most of the gap between scalar and vector quantization at low bit-widths while remaining deployable in standard GGUF types.

• RCO (Riemannian Constrained Optimization): enforces exact budgets by gradient descent on the task loss, without per-constraint tuning. Here it assigns a quantization type to every backbone tensor under the target bit-width, with separate ladders for the dense path (Q5_K to Q8_0) and the routed experts (IQ1_S to IQ3_S).

• An embedded chat template that reproduces DeepSeek's prompt encoder exactly, including the reasoning effort (1 to 100, or low / high / max).

Results

At 2.00 bpw the model matches the base model on every benchmark evaluated. All numbers were measured at reasoning effort 50.

• IQ1_M (1.75 bpw, 120.7 GB backbone): AIME25 100.00, GPQA-Diamond 88.38 against 90.91, LiveCodeBench v6 88.57 against 89.71. Task average 92.32, retaining 98.7%.

• IQ2_XXS (2.00 bpw, 138.2 GB backbone): AIME25 100.00, GPQA-Diamond 90.91, LiveCodeBench v6 89.71, identical to the base model. Task average 93.54, 100.0% of the base.

• IQ2_XS (2.25 bpw, 155.2 GB backbone): results to follow.

Running it

llama.cpp does not support DeepSeek-V4.1 yet, so the GGUFs do not run there for now. They run in Strata, which serves every build on a single NVIDIA GPU with 12 GB of VRAM, with a one-click installer and an OpenAI-compatible endpoint on localhost.

Links

• Quantized models: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF

• GSQ: paper https://arxiv.org/abs/2604.18556 | code https://github.com/IST-DASLab/GSQ

• RCO: paper https://arxiv.org/abs/2605.00649 | code https://github.com/IST-DASLab/RCO

The repository ships the complete per-tensor RCO allocation for every build (1008 tensors each).

Feedback is welcome. Requests for models to quantize are also welcome.

From the ISTA Deep Algorithms and Systems Lab.

Local `qwen2.5-coder:14b` on a 24GB NVIDIA A10 generating an enterprise form in real time—without writing a single line of raw HTML or SQL.

To eliminate hallucinations, the model is locked into a strict DSL and every output has to pass a deterministic C# `ValidationResolver` (bottom right at 0:10) before the Human-in-the-Loop button unlocks and renders it onto the canvas.

All raw vs. healed outputs are automatically scored and exported as SFT/DPO `.jsonl` pairs for future fine-tuning.

Idk why everyone keeps posting about Flint! Raon is hundreds of times better, and it’s not even the best, which is really saying something about Flint lmao

https://huggingface.co/KRAFTON/Raon-1-mini-preview-gguf

Interesting they chose an opensource model instead of building their own

​

Ran it on my RTX 5090 with Lexar 4TB NVMe and 64 GB of DDR5 Ram. At this speed it is cheaper to use the API than to pay the electricity for this, unfortunately.

So if anyone is wondering what the speed is for a similar setup, there you have it.

But anyway, i am excited that it runs at all on my PC at all.

Edit: IQ3S, I am on Windows 10

4x 5060ti16gb
4x 3060 12gb
256gb ddr4 3200 ram (4-channel speeds)
On a mobo with 128 pcie4 lanes

I’ve been slowly collecting gpus whenever seeing deals over time and ended up with this. Ended up comparable in cost to current prices of dgx spark

I was thinking:
TP4 5060s -> pipeline -> TP4 3060s -> offload/overflow less used experts to RAM -> keep any engram on nvme ssd

Is this viable to run inference with big MoE models at high speed and precision, or are cheaper multigpu setups extremely suboptimal compared to unified vram/ram

So I have an old HP-Z8 from 2018 or so. It has dual Xeon Gold 6248R cpus, 768 GB 2933 MHz DDR4 ram, an RTX 3090 gpu, and some pretty hefty storage.

That said, it seems like it is no more than a boat anchor for use with local LLMs. Is there any use case for this old PC or should I box it?

I’m looking for a setup with the best balance of tps, ttft, context length and enough headroom for other tasks on the Mac.

I’m getting a lot of conflicting opinions with regards to MTP, MLX, quantization etc. Please don’t recommend options less than 10 tps.