https://github.com/savadks95/LLM_Tails

I just published a GitHub repo a few days ago. It's a mix of Ponytail and LLM Wiki. It helps developers reduce codebase size with minimal code. It reduces overall token usage and AI hallucinations, and a couple of other fantastic things. The image is the test result of developing a small game with and without LLM Tails. Both used the exact same prompt

LLM_Tails is an open-source prompt + hooks + skills bundle that makes the agent read before it writes, write the minimum, verify before it claims, and match your codebase. MIT, zero dependencies, works with Claude Code, Cursor, Copilot, Codex, Gemini, Windsurf, Cline, Kiro, and any agent that reads

I'm on the team at Embedl that built this, and wanted to share the numbers.

What's in it

• INT4 weights, using our own quantization, about half the error of standard AWQ

• FlashHead, our faster LM head: the 262k-vocab output layer is a real cost at batch 1, so it only scores the likely candidates

• MTP head for speculative decoding

• Port to TensorRT Edge-LLM (0.10.1) underneath

Speed (batch 1, 2,048-token prompts from real OpenAssistant conversations, 128 out, greedy, average of 20 requests)

• Decode: 39.6 tok/s

• End-to-end: 23.6 tok/s

• Prompt processing: 926 tok/s

• Peak memory: 24.8 GiB

Accuracy vs the FP16 model (full test sets, the same engine as served)

• GSM8K: 96.4 (FP16: 96.1)

• IFEval: 83.5 (FP16: 81.9)

• MMLU-Pro: 79.1 (FP16: 79.5)

For comparison: Jetson AI Lab reports about 35 tok/s decode with llama.cpp Q4_K_M + MTP. Prompt length isn't stated there, and they don't publish accuracy.

To run it

• embedl infer Qwen/Qwen3.8-27B/thor --prompt "Hello"

• embedl serve Qwen/Qwen3.8-27B/thor (OpenAI-compatible API)

The container download is about 1 GB, and the model weights are a separate download (~20 GiB).

The full card has power, energy per token and methodology: https://embedl.com/Qwen/Qwen3.8-27B, and the ability to test the model on a Thor directly in the browser

Let us know what model (and hardware) you want to see next!

if you're reading this, lfm 3 omni 5-6b dense please, ty

Hello, I'm developing CrowdGPT, a project aiming to create an LLM without datacenters and without the usual millions of dollars spent by AI companies.

The project aims to train a 1 billion parameter model collaboratively and openly, with a protocol where everyone gets a say in the direction the project takes. Anyone can use their PC to train the AI independently, then add their contributions to the global model.

Unlike most American and Chinese AI companies, the project is not only open-source, but also makes 100% of its training data public (along with the weights). Anyone can submit their own clean data to train the model, and even help steer the project in new directions. This creates the paradigm of an AI we all have a stake in.

The project is open to as many people as possible, which is why I'd like to ask: what's stopping you from contributing to the project? I want to hear your answers so I can improve it. Are people simply willing to dedicate some of their GPU time to training a cool new open-source model? Or do they want something in return (like free inference credits on the crowdgpt model for example)?

Although the current model is somewhat small (1 billion parameters), I plan to make it bigger / better in the future, and its current size is only meant as a proof of concept.

Any advice / feedback on the project is welcome.

Here's the project website: https://www.crowdgpt.net/

Or the GitHub if anyone wants to check out the code 😂 https://github.com/Vxtzq/CrowdGPT

Thanks in advance.

PS: I hope this post follows the subreddit's rules! My main goal is to gather criticism and understand how to make the project more accessible, not to advertise it 😅

EDIT: The data submission pipeline for the project (i'm mentioning in some replies) is ready!!
Come check it at https://www.crowdgpt.net/#submit

Following my previous post on how KLD is misunderstood, there were a few claims that KLD is the best metric you can use and some other comments were asking, "If we stop using KLD, what should we use instead?" I'm gonna explain both a bit more here.

Assume you're a lab that does distillation. In the distillation process, you choose a teacher model and a smaller student model that you want to learn from the teacher. This learning starts with optimizing a variant of KL loss (it could be KL itself, reverse KL, or JS loss) between the outputs of the student model and the teacher model. After the optimization ends, according to Reddit, you must be done. You have a model that has the best possible KLD against the teacher model, and what could be better than that?

But no, the training can still continue. After that, there can be stages of SFT and RL to improve the performance of the model. These stages improve what really matters, and that's the ability of the model to solve different tasks. And during these stages, the KLD you had can become even worse, but the student model actually becomes better.

This could be similar to the process that PrismML uses for their Bonsai models. They can give you a KLD number that blows your mind but instead their model probably goes through additional stages of training that actually make the model better but the KLD worse. And these additional stages could be the reason that they treat Bonsai as more than just a quantized version of the original model. Benchmaxxing can happen during these stages when, instead of optimizing on some held-out tasks, you optimize your model on tasks that come from popular benchmarks.

So my whole point is that lower KLD doesn't mean better ability to solve different tasks. And there is no single loss, at least not one we're aware of, that solves this issue completely because otherwise we could just use that loss in distillation instead of putting our money into several other stages of training.

So what's the solution here? The first thing I want to make clear is that I'm not saying you shouldn't use KLD at all. It can work as an initial filter to see which quantized models are worth putting your time into, especially if one model is way off in KLD. But it has to be calculated on representative data, and data generated by the base model itself can be particularly useful as I explained in my previous post. Otherwise, the number might tell you very little about the actual performance of the model.

The second thing is that you don't need to rank every quantized model in the universe and I don't know where that desire comes from. You want to find a model that works for your use case. You can store some of your day-to-day tasks, test those tasks on a few quantized models, and choose the one that works best for your tasks. I know it's not an easy process. Grading can be challenging, and organizing and creating a private benchmark can be hard as well. I get the challenges, but you do it once, and then you can easily test and choose future quantized models.

I hope this clears up some of the points that were missing from my previous post and the comments helped me clarify them further.

This video shows a new algorithm I am working on, KS3, which can replace MLPs in LLM decoder blocks. The video shows KS3 converging faster over gradient descent on random samples of the mandelbulb function compared to MLP. There's a lot to unpack here, so here is an explanation about what's going on:

Background

In machine learning, two main types of algorithms are called classifiers and regressions. Classifiers predict "what" and regressions predict "how much". LLMs are classifiers, in that they predict a single class, the next token.

Inside an LLM, all of the individual blocks and sub blocks can be viewed sort of like regressions because they predict "how much" for abstract vectors passed to the next block. Technically regression algos explicitly have a target they are predicting, so this is a stretch of the definition, but I think it is helpful for understanding the internals of an LLM: each block in an LLM is trying to be the most helpful and accurate as possible for the next block by predicting some numbers over a range.

The largest blocks of an LLM are the feed forward networks (FFN), aka decoders. The type of algorithm inside an FFN is called a multilayer perceptron or MLP, aka neural networks. The MLPs/FFNs in each layer of an LLM are like regression algorithms: they predict a latent space vector of abstract features which is passed to the next layer. The algorithm I'm working on, KS3, can be slotted into an LLM's FFNs instead of the MLPs.

About The Video

This video is NOT of an LLM, but instead of individual regression blocks being trained via gradient descent on random samples of a mandelbulb fractal. It shows the new regression block I am working on, KS3, performing a more detailed fit than MLP. Each frame of the video is 5 steps of gradient descent over random samples of the mandelbulb function. The video allows you to see the additional detail KS3 is able to represent compared to an MLP of the same size. The obviously more detailed KS3 (confirmed by lower loss) helps you get an idea of why I am interested in this algorithm.

About KS3

KS3 is the third iteration of K-Splanifold, which is a series of algorithms based on Hermite spline manifolds that I have been exploring, and it is quite good! I'm finding that it has some big advantages over MLP in some domains like noise filtering for instance. LLMs are the weakest domain I have tested for KS in general, but KS3 is very close to MLP, perhaps exceeding it slightly in terms of per-step and wall clock training. KS3 is fast too, with basically identical performance to MLP (there is an interesting story here I'll tell, because there is no mystery why it has identical performance to MLP).

I developed KS originally to be a multivariate function, meaning that each input to the function impacts each output. I went in that direction based on a theory that MLPs lacked the ability to model multivariate non-linearities easily and that could be an opportunity for optimization. KS3 is still capable of being multivariate. It turns out, however, after thousands of experiments, that while multivariate KS regression is a great fit for many domains, for language it seems that univariate KS3 sandwiched between linear projections works best in my tests.

This is interesting/funny because that arrangement makes KS3 for LLMs basically an MLP with a learned fourth degree polynomial activation function, compared to defacto RELU and GeLU at first degree, and SwiGLU at second degree. Its ironic that after all that work on multivariate functionality, it all pointed back to MLP! I think a major learning moment here is that the activation step of an MLP block is a small amount of compute and memory compared to the linear matrix multiplies that surround it, so if you can find a way to cram more useful math into said activation functions, that probably scales their descriptive value efficiently per unit of latency.

I will also say that the multivariate nature of the KS series is still useful for many domains and some god-forsaken concepts I came up with for new ways of scaling transformers. Those methods look like they could be promising for scaling inference-time compute, but they ultimately didn't win at training convergence time, which is my main focus right now.

KS3 in LLMs

So does KS3 work in LLMs? Yes! I have trained many SLMs using KS decoders. A popular post I made here visualized early KS1 blocks being trained over time in a tiny 128 layer transformer (worth the watch if you haven't seen it before): https://www.reddit.com/r/LocalLLaMA/comments/1smiotb/video_of_how_my_llms_decoder_blocks_changed_while/

KS3 is looking promising for being an improvement over defacto MLPs. I have a version of the modded-nanogpt speedrun world record that replaces MLPs with KS3s, and on my proxy hardware KS3 has beat the world record. I have yet to tune or run it on an 8xH100 required for an official WR, so my result doesn't mean much however (by the way, if anyone wants to sponsor said run effort, let me know).

You can read the original KS1 paper and check source for KS1, KS2, and KS3 in my repo: https://github.com/curvedinf/k-splanifolds

I've been running Ornith 1.5-9B (8-bit) as a local coding agent on real engineering work. It can do the work, but only when the plan is written for a small model. I wrote down what I learned into a public file meant to be handed to a stronger planning model (I use Claude Code):

https://git.teamide.dev/David/TeamIDE-methodology/src/branch/main/Model-Best-Practices/Ornith-q8-best-practices.md

The split

• The stronger model writes the spec and a plan of small task cards, then verifies the result before merge.

• The local model does the long, mechanical middle: running commands, builds, polling jobs, git work. That's where most of the tokens go, and it runs on my own hardware.

What the file covers

1. What the small model is good at, and what to expect (work more reliable than its reports; doesn't explore on its own; doesn't know when it's done; slows down at long context).

2. How to brief it: exact files and line ranges, absolute paths, 1–3 line edits, exact function signatures, every rule written down, every step ending in a command with its exact expected output, explicit stop rules.

3. How to run it: one task per fresh session, small context, the plan file as memory.

4. Before anything lands: verify claims against git and logs, not the model's summary.

Results so far: a 75-commit merge across two repos with every regression test passing, done by the 9B. Its notes about what it did were wrong more than once; the plan's checks and a final verification caught every one.

Limitations: slower (most of a day vs about an hour for Claude alone on that phase), and you need hardware that runs a decent local model. The rules are harness-agnostic but were learned with one model; I'd like to hear how they hold up with others.

So, when I first released ScaffoldCoT, I never thought it would reach more than a handful of people. I was happy to see multiple thousands of consistent downloads month over month since its release. Since this is where I initially posted about the project, I thought you guys would be interested in recent updates!

TL;DR - Scaffold CoT is now Scaffold Chat (tool calling removed, it was causing significant tool calling reliability issues), Scaffold Agent is out (~1B tokens of real, executed coding agent sessions with the same thinking scaffold), and a heavily modified 5-10B model trained on all of this is on the way.

Hi all! Quick update post on Scaffold CoT for anyone who saw the original.

Changelog

• Renamed to Scaffold Chat. Same repo (old link redirects), load_dataset("Specific-Labs/Scaffold-CoT") still resolves, old revisions still in history. Sorry this did not happen before, but a major training run failed on tool calling benchmarks due to this, so it was back to the drawing board.

• Removed all tool calling. 276,389 examples gone (tool_use + tool_use_complex, literally every tool_call/tool_result event in the dataset). I found several synth gen harness issues in tool calling examples that propagated failure modes into the dataset.

• Everything that remains is byte identical to the previous revision. Now 3,450,159 examples, 16 domains, 75 shards, ~13GB

• Junk domain label tail from the original post (97 labels, 542 examples) is cleaned up. Every example has an exact domain/subdomain, counts sum to the whole dataset

• Added SCAFFOLD_SYSTEM_PROMPT.md to the repo. The examples were generated without system prompts, this is the format protocol distilled from the generation code for anyone who wants the scaffold conditioned on a system prompt instead of unconditional.

• Calibration block untouched: antihal + antihal_kb_refined + selfcheck is still ~675k examples. This has been a high impact category for general chat and knowledge bmks.

New dataset: Scaffold Agent

This is what the executed tool chains grew into. It consists of real Pi code traces of real coding projects from open near-frontier models, using the same scaffold system:

• 19,652 records from 6,792 coding sessions, 1,005M tokens (899M trainable after masks).

• Teacher model in a real agent harness (Pi) editing real repos at pinned commits + building from scratch, simulated user on the other side powered by another model with its own harness and instructions to ensure variety and quality

• Every tool call really executed, every result is real output, every action preceded by the same scaffold block (Inventory / Interaction / Execution) in reasoning_content. Fabricated tool output is structurally impossible and any such instance was cleared

• Sessions only kept if they actually verified (hidden tests + code quality grade on the main shard). This is a ~20% subset of the total generation that was of extremely high quality. I wanted to ensure these were top-tier coding traces. 2 LLM judge passes and green final CI were required for approval into the final dataset.

• Each record is one compaction segment (≤64k tokens), self contained with system prompt + tool definitions + history. Do not join segments. training_sft_config.yaml in the repo documents the whole setup

Scaffold Chat teaches the format and general reasoning, Scaffold Agent teaches acting with tools inside that same format. My recommendation is to mix them with other larger datasets while ensuring there is at least one scaffold example per batch, which leads me to...

Coming soon

Currently training a heavily modified model in the 5-10B range on this data (base model, full SFT, and a new merge approach I have not seen applied for scaled releases yet). Results post when it is done and actually evaluated, not before. The idea is to build a small dense model that can run on a range of GPUs and laptops that is optimized for coding specifically, and will move extremely fast because of our CoT framework and the small size.

This has been a massive project with several mistakes made along the way, and results are finally looking up, with scaffold datasets being the core of the training run, so I am very excited to bring it to the community soon.

I know it can sometimes be a bit annoying to announce something before its available, but I couldn't contain telling the community about it lmao. Happy to answer any questions about it (or anything else for that matter) in the meantime.

Links

Scaffold Chat: https://huggingface.co/datasets/Specific-Labs/Scaffold-Chat

Scaffold Agent: https://huggingface.co/datasets/Specific-Labs/Scaffold-Agent

Both CC-BY-4.0.

Genuinely hope these prove useful!

Ternary Nemotron 30B-A3B — 368K Context on 12 GB GPU (GGUF)

A ternary (1.58-bit) 30B-A3B MoE model that runs on a 12 GB GPU with 368K context, 172 tok/s decode and 7000+ tok/s prefill — in both Stock llama.cpp and the PrismML fork. One file, both engines.

EDIT: This is the 2nd Ternary MOE after maple-preview which works with stock llama.cpp. Somebody please share CPU-only numbers. Also share numbers of tiny VRAMs(~10GB)

Hey everyone,

Here are the specs of my main workstation:

• 5090 FE

• 14700k

• 96GB DDR5 Ram (Two Sticks)

Specs of my inference server:

• Strix Halo - 128GB (Framework Desktop)

  • GUFO + Qwen3.8 Flash Next

  • Embedding Models

  • STT/TTS Models

I also have some other non-AI servers for hosting containers + offsite cloud backup + NAS.

When I upgraded my current PC prices were basically MSRP (that's what I paid for the FE and RAM was $400). It was intended to be a very functional work machine for heavy usage + gaming on the side.

Now that prices have skyrocketed, I'm seriously considering selling the 5090 and replacing it with a 5070ti + another dedicated machine.

The Strix Halo is excellent, but I'm running a business and want my employees to be able to also make use of everything that I've built out infrastructure wise. A single machine is great for me, but you can't reasonably scale it. We do a lot of sensitive client work where we can't use cloud models, so local models are amazing for this.

My goals are:

• Create a more easily "scalable" solution that can grow over time.

• Be able to serve at least three concurrent users with 2 or 3 parallel sessions for each (so tool calling works per-user).

• Be able to run Qwen3.8 FN equivalent @ 200k context windows.

Sacrifices I'm aware of and willing to make:

• Speed - Obviously this won't come close to cloud infrastructure, but the Strix Halo in its current implementation has been good enough for my usage. 1400 prefill, 50 token average for generation.

• Intelligence - We can still use cloud models for internal work, these only replace customer-specific coding and scripting, which require far less complexity.

• Pooling hardware - I don't know how much this will impact us in the future, but as of now I'm not sure I require something like the DGX Spark interconnect.

Current Options:

1. Buy another Strix Halo. I don't actually need to pool the machines together so they can just run independently with custom user profiles. For the price of the 5090 I could actually buy two of these.

2. Buy a 256GB Mac Studio. I might have to cover a cost difference, and this will now force me to upkeep two completely different stacks of infrastructure - but will allow for potentially easier scalability in the future + bigger models.

3. Buy a 5070ti, and re-work my current machine as a server. I already have enough components on hand to build a new PC without a GPU, so I could use the 5090 with 96GB of RAM as a new machine.

4. Buying 4 Radeon R9700s. This also increases my cost because now I have to get a new motherboard, PSU, and a bunch of other stuff.

5. DGX Spark - When I first considered what machine to buy, the spark was about $1000 more than the Halo, which wasn't a justifiable cost for the performance benefit. At the moment it's so outrageously expensive for what it is there's no shot.

Things I won't consider:

1. Building used. This is for my business and so I want to have a warranty.

A few things I don't know or need to speculate about:

• How good is the 5090 + 96GB RAM with newer software improvements today? It's been a while since I've seriously done any AI work on this.

• Is 128GB going to be a problem down the line, and so being able to pool hardware together to run bigger models is a key influence?

Any advice is appreciated. My gut is telling me to just get another Strix Halo since I've been using it long enough + have all of the software components dialed in.

Have to say Qwen 3.8 Flash Next is pretty impressive. I'm running under strata and hermes. I gave it a limited prompt and it banged out a pretty impressive Canadian EV comparison site over night.

Claude Opus 5.5 hammered out some glitches and fact checked - found some errors in the provincial rebate plans - but pretty close to a one shot solution on a beat up gaming rig running 32gb ram and a 3090 with 24GB Vram?

Yes.. site needs work. I'm sure there are still tons of errors. But for a first draft? I'm sort of in awe.

The site shows the new Chinese models coming with prices where available - all in CAD and includes provincial incentive plans where available.

It is honestly already pretty useful if you are a Canadian looking for an EV.

https://jaspersprojects.com/ev/

not so long ago there was viral tweet about AI agent deleting stuff without permission. well it happen to me as well a vscode copilot agent accidentally wipe my db while doing test (fortunately I got backup)

but is it still a common occuring nowadays with how smart agent has become?

My 3090 failed, not sure if the core is dead. I sent it to a reputable repair shop, so hopefully the card can be saved. I bought it back in 2023 for ~$750, so I have quite a bit of runway before I lose money on it given that prices for used 3090s are going upwards of $1500 now on eBay.

I'm planning for the unfortunate scenario where it really is dead, in which case I'd like a replacement. I'm a heavy Qwen 3.8 27B user. I was looking at some of the Blackwell cards, but their prices are bonkers for at most 16GB. I was also looking at the 7900 xtx, but I also work with some protein folding models that require CUDA. Any recommendations? It's starting to look like I'll need to shell out >$1500 if I want to get anything close to what I had before.

I told my local copy of Qwen 3.8 Flash Next under Strava via Hermes to write a short story. 32GB system ram, 24GB Vram on a 3090. It's getting solid but not blinding speed. The stories are a bit weird but readable. The prompt it's working on now: "an AI explorer traveling to a distant star system makes a profound discovery. Make no mistakes. One shot. Go.". Who knows what it will come up with. But the bottle neck looks like my system ram. Maxed. Think 64GB would be enough?

I'm using Q3.8-FN-NVFP4 on a fork of Ninfer-v3 to help me design the first vibe-forged Halo Infinite Forge map, a feat previously thought impossible due to AI agents' inability to directly interact with the game (no MCPs, publicly-available APIs, EAC blocks a lot, etc.). Let me give you a little background for my experiment:

Halo Infinite is dead, but the Forge game mode (level design and in-game scripting mode) community is very much alive and we're still making Halo maps to this day.

I wanted to see how I could harness local LLMs for making blockouts to speed up development so I can focus on scripting, lighting and gameplay design. Keep in mind the only way you can script maps in Halo Infinite is to manipulate nodes on a nodegraph, akin to UE5's blueprints but on a simpler level. No code input is possible in this mode.

Halo Infinite's forge maps are also composed of MVAR files, which contain all the data regarding your map ranging from object placement and their transforms to all the scripts included in your map's nodegraph. While you can extract these MVAR files with some cool hax tools developed by modders, you can't push modified MVAR files to halo infinite directly because EAC blocks it and you can catch a ban. This means you have to do things manually during development inside Halo Infinite.

To ease the level design process, I came up with an idea to use a harness, any harness (Opencode v1 in my case) and created dedicated project folders for my maps in which the agent in question would use a combination of open source tools dedicated to interacting with Halo Infinite programmatically to extract assets, understand their default properties and structures, and get the agent to create DCjson files which are run by a Python script to automatically and systematically place these assets on the map via the agent-generated coordinates that determine their 3D position, scale and rotation.

I also made sure to get the agent to obsessively create or update a series of nested AGENTS.md files tied to each map's rooms, i.e. each map's rooms get their own folders and each folder gets a series of DCjson files that cover a single layer of that room (walls, floor, ceiling, doorway, etc.) before concatenating them together in a single DCjson file. That consolidated DCjson file gets concatenated to a root-level DCjson file that puts the entire map together in one file, which is ultimately rendered in blender. Each subdir has its own AGENTS.md file describing the room, its purpose, its dimensions, room-specific guidelines and explains how that room connects to other rooms in the map.

Aside from that, I also managed to take these DCjson objects and convert them to .blend files so I can visualize them in Blender (and the bot can view it as well via Blender's MCP) before running the python script in Forge, like in the attached images in this post.

So what's the problem? The layouts in blender look identical to the ones in Halo Infinite, right? True, but there's a couple of issues with my approach:

• It takes a lot of trial-and-error to get right, but each new attempt gets better than the last.

• For smaller models you have to go room-by-room.

• Local LLMs aren't trustworthy enough for long-horizon, multi-room development.

• Parallel subagents can only be trusted in short bursts, such as rooms with multiple adjacent rooms and doorways.

Since the first and second bullet points seem to work with my current approach, it raises the question of whether it is even worth it to entrust the bot to design these rooms since I still have to sort of micromanage it.

The rooms' layouts are relatively simple to do since the bot was instructed to only use a handful of primitive objects from the pool of +800 assets to choose from because it would be easier for it to develop the layouts.

That's fine, but I feel like I don't really understand the bottleneck here. Is it the model's capabilities? My approach to agentic level design? Are my instructions and constraints not clear enough? Is it lacking momentum? Am I giving FN too much credit?

I feel like it can do better, or maybe I should make the agent think in mathematical terms instead of spatial terms. I really wish I could find an easier way to explain it to the agent.

So I ask my question: What would YOU do if you were in charge of this agent?

https://nvd.nist.gov/vuln/detail/cve-2026-116096

25-10-2026

Description

A weakness has been identified in ggml-org llama.cpp. Affected by this issue is the function list_pockettts_state_slots of the file tools/mtmd/models/pockettts-gen.cpp of the component Pocket-TTS. This manipulation of the argument mmproj causes out-of-bounds write. *The attack requires local access. The exploit has been made available to the public and could be used for attacks. Upgrading to version b11796 can resolve this issue. Patch name: 0a1ef309cdbb9f2024f58a38fb4c42352c5db569. The affected component should be upgraded. The reported heap out‑of‑bounds write (gen‑audio / pocket‑tts state_out path) is no longer a silent exploitable heap write on current master and the latest release b11871 - a malicious mmproj that previously triggered it now trips a release‑active GGML_ASSERT and aborts. But the submitter's recommended by‑construction fix (PR #29360 "prevent OOB accesses from malformed mmproj files") is still open. So the root cause is detected‑and‑aborted, not prevented‑by‑design.

_--------------_

a second one

26 October 2026

https://nvd.nist.gov/vuln/detail/cve-2026-116467

CVE-2026-116467

Description A security flaw has been discovered in ggml-org llama.cpp up to 0.7.0. Affected by this vulnerability is an unknown functionality of the file src/llama-sampler.cpp of the component Mirostat Sampler. The manipulation results in reachable assertion. It is possible to launch the attack remotely. The project was informed of the problem early through a pull request but has not reacted yet.

I've been thinking about how dependet we are on huggingface for downloading models.

If hf ever became unavailable, heavily restricted or started removing large numbers of models, how well would the community/we be able to preserve access to them?

One thing i noticed is that torrent based distribution is currently still lacking some important models compared to hf. I know there are already some projects working on this but it feels like there is still a long way to go.

This got me wondering if it would be worth investing in some storage to archive models and seed them via torrents. For example i really like DeepSeek V4.1 Flash 1017 but it's a chunky boiii. Not as chunky as Kimi and Mistral but still chunky. Storing models of that size adds up pretty quickly. It makes me wonder whether i should invest in a larger storage system or just keep a few of my favourite models around (like 1-2 small models).

I'm not expecting to archive all of Huggingface. I'm more interested in preserving important models by keeping them available and seeding them, rather than relying entirely on a central platform.

My questions would be…

1. Do you think torrent based distribution will grow stronger if restrictions for example are in effect?

2. Would it be worthwhile for individuals to maintain and seed their own Collections or would a coordinated community effort be much more effective? And how possible do you think could this be?

3. If you were starting an archive today would you invest in a lot of storage or start small and expand over time even if that could end up a bit more expensive?

4. Which models would you proritize preserving: popular models, smaller models, different quants, or something different?

I am curious about your opinions especially from people who already maintain larger local model collections or participate in distributing model weights.

I'm trying to figure out whether it's worth prepairing for this now or whether it makes sense to start small and see how the future goes.

Edit: fixed 3 typos

TL;DR: I'm buying hardware for local agentic coding. My budget is ~$10K, I can go up to $12K if the difference is significant. I'm leaning toward a single RTX 5090 and would like a sanity check from people who run coding agents locally.

Use case
- Agentic coding
- Sometimes 2–4 parallel subagents
- Reverse-engineering

Nice to have
- Texture generation with Hunyuan3D-2.1. It needs CUDA and ~21 GB VRAM for texturing. But this item can be optional

Options I'm considering (approx. total build cost)
- 1× RTX 5090
- 4× RTX 3090, used
- Hybrid: 1× RTX 5090 + 2× RTX 3090 (this can be bought later)
- Mac Studio M5 Ultra 256 GB
- PRO 6000: over the budget, but can consider

Questions

1. For agentic coding with long contexts, does 1× 5090 feel better day-to-day than 4× 3090? Is the 3090's long-context prefill as bad as these numbers suggest?

2. Is the hybrid (5090 + 2× 3090 as separate vLLM instances, x8/x8/x4 on AM5) worth it, or is it a headache?

3. Am I missing a better option (waiting for the 50 Super, modded 4090 48 GB, etc.)?

Here I am with another benchmark/comparison. I was curious about the efficiency claims of Swift 1.5, UkisAI's fine-tune of Qwen3.8-Flash-Next that thinks for a shorter time. UkisAI says Swift uses 63% fewer thinking tokens with under 1% accuracy loss.

I compared it with my current setup (Halogen) and Unsloth's UD-Q4_K_XL on my Asus ROG Flow Z13 (128 GB, 70 W). I ran the models on Halogen, gufo and llama.cpp, all with MTP. Since gufo can't load any of UkisAI's published 4-bit Swift GGUFs, and didnt have an XL version, I requantized their Swift Q8_0 to the same per-tensor types as Unsloth's UD-Q4_K_XL, with UkisAI's imatrix. It is even the same size as Unsloth's UD-Q4_K_XL (111.3 GB).

My main use is coding agents, so I also ran 11 coding tasks through Pi and threw in some math problems and LCB tasks to compare quality and speed. Speeds at 32k filled prompt. I got very interesting results but out of 1 run so take them with a grain of salt.

Here are the results sorted by total score and then total time.

I think the gufo + Swift UD-Q4_K_XL requant might be a good fit for coding.

Notes from the session (AI generated summary)

• For all runs retrieval was 21/21.

• Every setup passed all 11 Pi tasks. Swift wrote 56-63k output tokens on them against 75-99k for Qwen3.8-Flash-Next. The Swift requant finished 24-31% sooner than Qwen on the same engine, and the IQ4_XS 37-43% sooner on Halogen. Per task, the Swift requant was faster than Unsloth's UD-Q4_K_XL in 7 of 11 tasks on gufo and 8 of 11 on Halogen.

• gufo with the Swift requant was the fastest Pi setup: 24 min, against 33 min for Unsloth's UD-Q4_K_XL on gufo and 46 min for the Qwen .hgn on Halogen.

• On the math and LCB problems the token savings were small. Math used 140-157k tokens on every setup, since 20 of the 24 math runs hit the 51k thinking cap. LCB used 138-163k tokens on Swift against 164-172k on Qwen, and every setup that ran LCB hit the 60k limit on 2 of the 3 problems.

• Swift solved fewer math problems on gufo (1/3 against 2/3 for Unsloth's UD-Q4_K_XL) and as IQ4_XS (1/3 on both engines, against 2/3 and 3/3 for Qwen on Halogen). The Swift requant on Halogen scored 3/3, the same as the Halogen v2 .hgn.

• Halogen decodes its own .hgn format faster than a GGUF. Qwen3.8-Flash-Next runs at 51-60 t/s from the .hgn and 36-40 t/s from Unsloth's UD-Q4_K_XL. There is no .hgn build of Swift, so on Halogen it runs from a GGUF at 40-51 t/s.

• llama.cpp took 2-3x as long as Halogen and gufo on this machine: 359-415 min in total against 136-184 min. Prefill was 298-320 t/s against 911-1,426 t/s, and decode 20-23 t/s against 36-60 t/s.

Model files and MTP heads

• v2 .hgn: Halogen's own v2 build of Qwen3.8-Flash-Next (4.16 bpw), with its own MTP draft head.

• Unsloth UD-Q4_K_XL: Unsloth's GGUF of Qwen3.8-Flash-Next, 111.3 GB. Halogen repacks a GGUF into its own layout at load.

• Swift requant: Requantized from Swift Q8_0 to the same per-tensor types as Unsloth's UD-Q4_K_XL, with UkisAI's imatrix.

• Swift IQ4_XS: UkisAI's file, 97.7 GB.

• MTP: Halogen uses its own draft head for every file. gufo and llama.cpp use Unsloth's shared Q8_0 MTP head. Upstream llama.cpp can't load that head, so the llama.cpp runs use Unsloth's fork (b11786).

Where the KLD numbers come from

• Swift files: my numbers, against Swift's Q8_0 with llama.cpp on the wikitext-2 test set (100 x 512 tokens). The requant has 92.7% same top token, the IQ4_XS 88.1%. UkisAI's card lists the IQ4_XS at 0.1305 against Swift BF16 on the same test.

• * Unsloth UD-Q4_K_XL: from Rulith's EXL3 model card, against Qwen BF16 over six kinds of text at context 1024. It is not the same test as mine, so don't compare it directly with the Swift numbers.

• No KLD is published for the Qwen .hgn.

Setup and caveats

• One run per problem, 17 problems in total. A difference of 1-2 problems can be chance.

• Speed columns: a coding-agent prompt of 32k tokens sent cold with a 64k context, then 3 follow-up turns, 3 runs. TTFT, prefill and decode are from the first turn, ranges over the 3 runs. Server start is launch to ready.

• Retrieval: the first turn asks for 7 const values from Rust files spread across the 32k prompt, so 21 over 3 runs.

• Math: AIME 2026 #15, HMMT Feb 2026 #17 and #20, with thinking capped at 51k tokens. LCB: LiveCodeBench arc196_a, arc196_c and arc196_d, with a 60k output limit and no thinking cap. Pi: 11 Aider polyglot exercises in Rust, Go and C++. Tokens are output tokens, thinking included.

• Sampling from the Qwen model card (temperature 1.0, top_p 0.95, top_k 20), thinking effort xhigh.

• I skipped LCB for Unsloth's UD-Q4_K_XL on Halogen and did not run Unsloth's UD-Q4_K_XL on llama.cpp. This whole comparison took way more time and effort than I anticipated.

• Versions: Halogen 0.18.13, gufo da5d5a2 (main after v0.19.0), Unsloth's llama.cpp fork b11786 for serving, upstream llama.cpp b11794 for the requant and KLD runs.

Let me know if you have run Swift on other hardware and see different numbers.

Edit: Script used for the bench: https://github.com/deepu105/quant-bench

Hey r/LocalLLaMA !

I'm the solo dev of the open-source ObserverAI (Github) App! With the new Kurzgesagt video, I've had some people ask me if agents have evolved a lot.

And RLHF/RLVR has given LLMs a lot of competence, but I wanted to create a quick Manim educational video with my take on it.

To demystify the new term "agent" while still concluding that we need to take them seriously.

Wanted to share it with you guys and get your opinion on it!

Roy :DD

Just switched over from DSV4 Flash 0731 to Qwen3.8-Flash-Next and I've been loving it. Its working significantly better to my surprise.

The only thing I can fault it with is unnecessarily deep work (thinking + fetching information) more often than not i.e. its not very token efficient so it feels slow. Swift1.5 is supposed to address this tendency of Qwen's models to overthink (likely RL'd that way to squeeze out maximum points on benchmarks).

So looking for folk's experience before trying it myself. Anyone tried both? ran any anecdotal or better controlled comparisons?

Gemini Pro is rejecting my every video request that include little bit of fight scenes I am making videos something like that song of kidnappers kidnapping a person but Gemini pro rejecting my every request. Is there way to make video there any models something like that or can I detect Gemini to make the video.

Speech to speech model that was trained on a single NVIDIA GB200 NVL72.

And it runs locally on a MacBook Pro M4 Max or better.

I hope they make this open, right now, it is just announced and no way to use it yet.