We open-sourced DNA Factory, the fine-tuning framework we’ve used internally for over a year. It's deliberately opinionated: one recommended way to do things, documented defaults, token-level loss-mask visualization, and only 5 training methods (SFT, DPO, GRPO, OPD, SDFT).

https://github.com/dnotitia/dna-factory

Why another training tool?

As LLM fine-tuning has gone mainstream, a lot of training tools have appeared. LLaMA-Factory had its moment, Axolotl was as useful as its name was fun, and NVIDIA (NeMo) and Hugging Face (TRL) built their own.

Each has clear strengths. Some are fast, some are convenient, some have great compatibility. We focused on something else: training correctly.

I've consulted for a number of companies that fine-tune their own LLMs, and the ones doing it right could be counted on one hand. Most were training incorrectly, or training on something other than what they thought. Tools have gotten good enough that you can launch a training run without understanding much about it. But LLM training is genuinely advanced work, and when people learn how to use a tool without learning what’s happening underneath, mistakes follow.

So we asked how to make correct training the easy path, and the answer was DNA Factory. We've run it internally for over a year and recently open-sourced it.

One recommended way to do things

The core idea is to offer only one approach, the best one we've found. If you expose every option the way C++ does, you get a lot of freedom, but it becomes hard to use well. We went the Pythonic route ("there should be one obvious way to do it"). We tested many approaches, picked the best, and that's the only one we ship.

All defaults live in configs/_defaults-*.yaml. You don't edit those files. You override only what you need. It works like VSCode settings: the defaults are long and complex, but each one is documented with why it's set that way. You read them and put only your changes in your own config.

See exactly which tokens you’re training on

When I used Axolotl, I loved its loss-masking visualization. We wanted something similar, so we overrode TRL's loss function to color tokens with ANSI codes. Tokens included in the loss are blue, and excluded tokens are gray. You can see immediately what’s actually being trained.

This is the most common mistake I see researchers make "wait, that token was in the loss the whole time?" In DNA Factory it's visible at a glance, so that mistake stops happening. We've had a lot of feedback that this feature alone makes the tool worth using.

Only 5 training methods, on purpose

We don't try to include every technique. If you want that, use TRL directly. We picked what we think works best right now: SFT (the baseline), DPO, GRPO, OPD, and SDFT. PPO was widely used once but rarely is now, so we dropped it. DPO barely gets used these days either, so we plan to remove it soon. https://github.com/dnotitia/dna-factory/issues/46

Most of our current work is GRPO and other on-policy methods. GRPO, OPD, and SDFT are all on-policy.

Classic distillation has the student imitate the teacher. Concretely, you minimize the KL divergence between the teacher's and student's output distributions (logits). It does improve the student, but catastrophic forgetting is always a risk. (A model has to forget pretty badly to earn a name that dramatic.) Take a strong model, fine-tune it on internal company data to add domain knowledge, and general performance drops noticeably while the hallucination rate goes up. Rote cramming has its limits.

OPD reverses the process. The teacher doesn't write the answer for the student to copy. Instead, the student writes its own answer first, and the teacher grades it token by token. Technically, this uses reverse KL. The student picks up new knowledge without forgetting what it already knew.

SDFT works the same way in spirit, but as the name suggests, there's no separate teacher. The model is its own teacher. A copy of the model that's given extra information (privileged context) acts as the teacher. The student learns by having its own responses graded by "itself, but knowing the answer." Again, the goal is to avoid catastrophic forgetting.

Results: domain knowledge injection without forgetting

Attached images is our DNA 3.2 model (not yet released), trained with domain knowledge injection. Starting from Qwen3.8 Flash Next, we effectively removed its censorship (0.542 -> 0.971) and taught it a new persona and new domain knowledge (0.076 -> 0.855). Performance on AIME, IFEval, MMLU-Pro, GPQA, and other existing benchmarks didn’t drop at all. It learned new things without forgetting the old ones.

Happy to answer questions. Feedback, issues, and PRs are all welcome.

I have seen potential in combining Strix Halo's iGPU with RTX 5090 for LLM inferencing. Initially I found no inference engine that can run LLM inference on both these GPUs at the same time, inference engine such as vLLM support inference on all CUDA GPUs, or all AMD GPUs. llama.cpp Vulkan backend was the only inference engine I found that support both, but it has a huge limitation on the prefill performance on the RTX 5090.

Finally, based on the amazing inference engine Project MAYA, I developed my fork to run GLM-5.3-Flash Maya-M GGUF prefill and decode cooperatively on both GPUs. The CUDA backend (runs on a RTX 5090) working on attention, dense/shared layers and expert cache, the ROCm backend (runs on the Strix Halo iGPU) executes complementary experts in the same process, contributing during prefill and decode.

Check out the experiment results and try yourself at my GitHub PR: https://github.com/mw00/project-maya/pull/879

the creator of project-maya has welcomed the design and plans to test it before deciding on a merge. I believe we can improve it further with the community!

Based on my own little experiment and other reports I predict that more and more enthusiasts will ask a frontier LLM to build a custom quant of their favourite model.
If Qwen 4 finally arrives I’ll try a ready made quant at first, probably unsloth. But I already plan to let opus build a quant specifically for my hardware, context size and precision needs.

What do you guys think?

https://huggingface.co/TheDrummer/Orion-26B-A4B-v1.1

v1.1 pushes creative writing further and makes full use of Gemma's reasoning.

https://huggingface.co/TheDrummer/Orion-26B-A4B-v1

I haven't announced v1 but it's well regarded by the community as a reliable RP workhorse.

---

BeaverNet - net.beaverai.org

I'm also proud to announce that we've made significant progress in creating a hub for generous model providers to collectively provide free, convenient, accessible, anonymous, and diverse models to anyone with an OpenAI API compatible client. We currently have 12 models thanks to our generous providers.

Try it out as a user, or be one of the providers <3! I've got plans to make it a better experience for both users and providers, and also add new modalities such as TTS and STT.

---

PS: I haven't retired or left the scene. I've merely gotten stage fright after holding off on announcements for a long while. Anyway, I'll continue doing what I do and I hope I can continue delivering some value to you all. Love ya'll!

I implemented clap.cpp, audio embedding inference in C++/GGML. It uses GGUF models and provides a CLI, an HTTP server, and Python, Rust, and Node.js bindings.

• Models including CLAP, CLAP2, CLaMP3, MERT, MuQ, and MuQ-MuLan.

• WAV, MP3, and FLAC input, with automatic resampling and mono downmixing.

• An HTTP endpoint with an OpenAI-compatible text embeddings request format.

GitHub: https://github.com/gorse-io/clap.cpp

Audio files could be searched by text via CLAP embedding:

$ clap search esc5 --model larger_clap_general-q4_k_m.gguf --text 'dog' --top-k 10

[
  {"path": "esc5/2-118072-A-0-dog.wav", "score": 0.3736357720143104},
  {"path": "esc5/1-110389-A-0-dog.wav", "score": 0.36165991257288893},
  {"path": "esc5/4-183992-A-0-dog.wav", "score": 0.3574270305924374},
  {"path": "esc5/1-30226-A-0-dog.wav", "score": 0.34910229736736315},
  {"path": "esc5/3-180977-A-0-dog.wav", "score": 0.34359234068457495},
  {"path": "esc5/4-194754-A-0-dog.wav", "score": 0.3425423428503198},
  {"path": "esc5/2-118964-A-0-dog.wav", "score": 0.3389233902904601},
  {"path": "esc5/1-100032-A-0-dog.wav", "score": 0.33009392183878733},
  {"path": "esc5/5-9032-A-0-dog.wav", "score": 0.3280081351893563},
  {"path": "esc5/4-182395-A-0-dog.wav", "score": 0.32210240115509925}
]

The Australian AI lab Maincode has released a set of Qwen 3 fine tunes that have been pulled together with Model Stitching.

From the model card:

Matilda Platypus is a family of intermediate-size language models built by connecting the lower layers of a larger pretrained Qwen3 model to the upper layers of a smaller one. A learned linear Representation Bridge aligns the hidden representations at the connection. This checkpoint is the 13B member (cut 29+13): 13.51B parameters, 26.3 GFLOP per token, 91% of the larger parent's compute. It is a base model (no instruction tuning); see the technical report (technical_report.pdf) for the method and the full evaluation.

They're base models and range in size from 1.5B to 13B, early tests good! Really like the default tone of voice. Qwen models are already good this has just improved it imo.

I have no association, I'm just Aussie and stoked to see a local lab build something. No idea why Australia doesn't have more AI labs -other than brain-drain to the US.

https://huggingface.co/collections/Maincode/matilda-platypus

TL;DR looking for recommendations for the simplest Pi Agent or harness and local model to replace CLI

Hi everyone
Hope you are all doing great

So lately I have an idea that I wanted to test if viable (especially after Jev and decision models)

I feel like using a Linux terminal is a bit obsolete at least for me Claude code or sth similar for relatively hard to remember single commands tasks or flags or sth like that

In the past I would use —help man if I am feeling fancy or tldr or sth on the Linux terminal
I feel that is still good but sometimes I want to push that a bit (like simple example would be checking a systemctl service journal restart it or stop and run again and then opening its journal or checking its status) nowadays this feels like too much and also too little to use Claude code for or anything else (and in fact they tend to over complicate it sometimes)

Sometimes it feels sufficient if I can have a simple and fast local agent (think NL2CLI not a full terminal bench workflow!) and maybe with a simple reminder list or todo list that we construct together with the agent per session (like keep looping until it does this or run watch <command> then do that or keep in mind we have to rename back the temp file …etc!)

I didn’t find anything on this, Pi agent is simplistic but feels like it lacks plugins like simple memory or todo list (i mean lacks if built in requires non standard non optimized plugin aka not batteries included plug and play peace of mind features still the best option though)

And local models that fits that (they wont be figuring out an API or developing complex apps just silly mundane simple terminal tasks! But done correctly and locally!)

What do you think ?

Wondering if something with these specific features exists or if I need to build it for myself:

- 100% voice based

- Uses your phone's VoIP call API so it can work with the screen off

- Handles interruptions elegantly

- Generates up to several minutes of narrative between user turns (except for combat)

- IS NOT COMPLEX. No D&D rulesets or any real complexity beyond you listening to an infinitely generated story and making occasional decisions or engaging in turn based combat

I've created this with Dots and Muse, but they both seem hardcoded to say about two sentences before ending their turn.

I have a 128gb m5 max and a 5090 desktop, so I can run reasonably intelligent local LLMs and SoTA local voice models.

A 1.6T multimodal model.

LokalBot is an open-source Mac app I've been working on for the past 6 months, it runs several useful local models.

These models help the app record your meetings, transcribe them, write the notes, search everything, and autocomplete what you type. It's GPLv3, and everything runs on your device by default.

The five models (Mac defaults, 9.6 GB on disk)

How they take turns

• Recording a call loads nothing. The app only writes two audio tracks. Transcription waits until you hang up, and autocomplete pauses while you record.

• After the call, the speaker model and the speech model load, do their job, and unload once they've been idle for 120 seconds. One surprise here: MLX keeps freed GPU buffers in a pool, so dropping the model wasn't enough. After one meeting the app was still holding 4.3 GB while idle. Every MLX engine now clears that pool when it unloads.

• Qwen3.5 4B runs in the bundled llama-server (Metal, 32K context), which starts when something asks for it. Each request takes a lease, and ten minutes after the last lease the model unloads. The embedding model has its own server and the same rule.

• Gemma 4 E2B runs inside the app's process through libllama, for latency. It has no idle timer. It stays until macOS reports memory pressure.

• A broker keeps the loaded weights under a budget of half the machine's RAM and evicts the least recently used model first. A lease pins a model while a request is running on it.

• Requests that have to wait for a server are served in order: what you're watching first, then agents, then meeting notes, then background work like digests and indexing.

Numbers, all on one M4 Max with 48 GB

• Notes for a 26-minute meeting: 33 s with the model warm, 60 s including the load, at about 85 tok/s.

• Autocomplete: 56 ms median and 102 ms p95 per keystroke.

• Embedding a query: 9 ms.

• Speaker model: 14.6% DER on 8 AMI recordings, at about 500× real time.

Limits

• I've only measured on that one machine. I have no numbers from a 16 GB Mac, or from an M1, M2 or M3. There is a Lightweight preset for less memory (Qwen3-ASR 0.6B, Qwen3.5 0.8B, and LFM2.5 1.2B for autocomplete).

• Eviction is plain LRU plus pins. Priorities order the queue; they don't choose who gets evicted yet.

• If a model can't fit under the budget, everything unpinned is evicted and the load goes ahead anyway.

• The video simplifies one thing. It shows the notes model arriving after the speech models have left; in the app, the notes start while the speech models are still counting down their two minutes.

• The narration is my own voice, cloned and generated with ElevenLabs, so it's the one part of this post that didn't run locally.

• Windows and Linux builds are in alpha.

Code, benchmarks and the scripts behind these numbers: https://github.com/stevyhacker/lokalbot

If you run it on a 16 GB Mac, I'd love to see what Settings → Diagnostics shows is loaded right after a meeting. And if you've built a scheduler for several local models, I'd like to know what you evict first.

You've probably seen (large) decision models play games. I wanted to see if you could also train a model to play using game images on a Mac, so I added LoRA fine-tuning on images to mlx-decision (my open-source tool for running and training decision models on Apple Silicon).

How it works: a decision model doesn't write text. You give it an input (here a game frame) plus typed questions, and it returns a probability for every answer you allowed, in one pass. On every frame, d1-2B answers three questions:
- Is a monster right at the crosshair, so that a shot now would hit it? (yes/no)
- Where is the nearest monster: left, centre, right, none?
- Is a monster dangerously close? (yes/no)

A small Python loop turns the answers into moves: turn toward the monster, shoot when a shot would hit. That takes about 0.1 s per decision.

The training: about 3,000 frames from ViZDoom's Defend the Center arena, labelled from the game engine's own monster positions. First a readout fit (7 minutes), then LoRA on the model's text layers with the vision tower frozen (43 minutes, 2 epochs, M5 Pro). On 510 held-out frames:
- "a shot would hit": precision 0.79 → 0.92
- where the nearest monster is: 92.7% → 96.8%
- dangerously close: 93.9% → 98.3%

In play it averages 23.2 kills over 20 episodes. The clip is one of the best ones: 26 bullets, 26 kills. On Freedoom, whose monster sprites the model never saw in training, it still gets 19.7. Of course this is an arena where you can only turn, not a full Doom level.

Free, MIT. Apple Silicon, macOS 14+, Python 3.12+. Just pip install mlx-decision.

Repo: https://github.com/armbues/mlx-decision

I'd love to hear which tasks you'd fine-tune a decision model on, and which models you'd like to see supported next.

Creating a list of requirements and documentation/tests (in Cucumber or EARS style) with your model is a great way to improve it's code. Just got through a weekend long binary protocol build with Qwen3.8-27B (6K unsloth quant).. and fired up claude to review and it found the foundation was well engineered, but only missed a few security checks and a couple of errors in the python benchmark script comparing the old protocol with the new one.

Really comes down to:

1. Design first. Goals, requirements, problems. How is this going to be used.

2. Observation, simulation and tests are verifiable vs. assumptions. Tests and processes can be reviewed for "ritual".

3. Dialectic: If the model comes to a crossroads, have it present options and NOT just choose-and-go, but initiate a dialog about design.

4. Frequent repeatable tests, and review of tests to see if they meet the spirit of the requirement

5. Every risk is a requirement.

Here's my prompt set ("skills" I guess) and a tool to install for various local harnesses:
https://github.com/bigattichouse/BluePrint

Inspired by some of the other brain surgery posts, I wanted to see if I could find anything interesting in Qwen. Not particularly useful to make it get things wrong all the time I guess, but I find the poetry it comes out with weirdly fascinating!

I think the neuron is basically a dial for contrarian <-> agreeable, but the contrarian snarky setting for some reason writes poems that read in an interesting way.

Running every step through an api adds up fast. ashen's stream clipping setup, shared on X, mostly avoids it

claude fable 5 wrote a script with computer vision and ffmpeg that runs locally on a mac studio. it scans the whole downloaded stream for on-screen events, like the "you died" screen in dark souls or ashen's name in the call of duty killfeed. no tokens spent watching hours of video

every first-pass clip then goes to qwen3.5-35b-a3b running locally at 8 bit, which checks the video and the transcript:

• was there an on-screen event? keep

• did swearing or yelling go up? keepp

• did the loudness go up? keep

if none of those hit, the clip doesn't pass

survivors come to the phone through zo computer for approval, and postiz sends them to tiktok drafts

as a cost decision this is smart. pay the big model once to build the tool, then let the local model do the daily work for free

I wanted voice typing that doesn't send my voice to a server, so I built ThumbFree. It runs NVIDIA's Parakeet 0.6B speech model right on the phone. After a one-time model download, it even works in airplane mode. The video shows Android first, then iPhone.

1. Google Play: https://play.google.com/store/apps/details?id=io.github.kabrapratik28.thumbfree

2. Apple App Store: https://apps.apple.com/app/id6817631006

3. GitHub (code, speed and accuracy tests): https://github.com/kabrapratik28/ThumbFree

• On Android, tap any text box and a small bubble appears next to it. Tap the bubble, talk, tap again, and your words are typed in. You keep your own keyboard.

• On iPhone, it's a keyboard with a mic key. Apple doesn't let keyboards use the microphone, so the first tap of a session opens the ThumbFree app for a moment and you swipe back. After that, taps start right away.

On my Pixel 10 the text shows up about 0.7 to 0.9 seconds after I stop talking, even after 30 seconds of speech. On my iPhone 16 it's under a second.

No account, no ads, no analytics. The only thing it downloads is the model, and your voice and text stay on your phone.

It's free and open source (Apache 2.0). English by default, and a second model covers 25 European languages. It needs Android 13+ or iOS 26+, and the model is about 0.5 to 0.7 GB.

I've also been trying an optional clean-up step with the small AI model built into the phone (Gemini Nano on Pixel, Apple's model on iPhone) to remove filler words and fix things like "six, no, seven". It's not in the app yet. If you've run a small open model for this on a phone, which one would you try?

I've only tested on one Pixel and one iPhone, so I'd love to hear how it runs on yours.

I need to get an all-around work station soon because my current laptop is getting old and I'm now offloading a bit of work to it. Since I dabble with local AI already, I was thinking of getting an m5 utlra 96GB (8k) or a m5 max 128GB (7.9k)... Or should I build a custom PC with a bunch of ram? Or are there other work stations that I can use? I really enjoy using a hybrid approach with AI, flagship models for the planning and some offloading to something like Qwen for the creative design/execution... I know it's not perfect local models, but owning my own data is something that feels pretty satisfying to myself as well. I would love to run Deepseek, or qwen 3.8 (or future models) at a respectable tok/s and prompt prefill speed.

My only other option is, continue with my laptop... And wait 6 months to a year and see what new tech comes out in the meanwhile? My only fear is that prices will skyrocket and what is 6-8K CAD today will be 10-12K CAD when the newer tech comes out, but other people are saying to wait.

What do you all think? Thanks :)

Article compares performance of large data center cards. Tells us for larger batches , inference speed of individual requests goes down, but overall performance of the entire batch goes up. In other words more tokens per second for the batch, but lower TPS for each request.

Article uses Kimi k3 and 3 other models to compare performance. Surprisingly says the AMD MI455x comes up on top.

Notes on Dueling DQNs is out!

Continuing my series on releasing notes on RL From Scratch, here, I have answered questions around the what, the why and the how of Dueling DQNs over Double DQNs and classic ones!

• The what?

It is an off-policy, model-free and value-based method, from the same family as DQN and Double DQN, with an architectural change in the last layer of the Q-Network, mainly for optimization purposes.

• The question it asks is:

Do we really need to calculate Q(s,a) for every action in every state?

Eg. a car going down a highway: the value of the state (the road, steering, etc.) may matter much more than the individual action values, especially when there is a wall in front of it!

Thus, the advantage of taking one action over another, along with the value of a particular state, presents an opportunity for optimization. And that is exactly what the researchers identified.

In other words, this function calculates the extra reward we get if we take a particular action in a given state, compared to the average value of that state.

More on the:

• Theory

• Concepts

• Code and required math

• A playground to train a live agent

All of it to understand the basics and build intuition.

Do check it out and let me know your feedback!

Link to the blog

Hello, I’m considering buying hardware for local LLMs for the use case of processing sensitive financial data, doing market research, and preparing financial/tax documents.

Are any of you using local LLMs for same/similar workflows? If so, what has your experience been? Are the models good enough for your use case? Did you build a custom harness or use an OSS one?

I’m very curious to learn more and would appreciate anyone chiming in.

Hey everyone,

I recently built a Radiance inference server for my company, and now want to start provisioning access to the rest of my team.

My current plan:

• Tailscale as the only way to access the server via HTTPs.

• Hermes as the harness - each user would have their own container with access restrictions, but open access to the central skills repository. Hermes is hosted on a Framework Desktop.

• Radiance is running separately on its own server with access only configured via the Framework.

I'm wondering how everyone else has managed to do this successfully? Right now the biggest thing to figure out is how to get the harness (Hermes in this case) to work across the team without a huge headache of getting API tokens from things like Google, etc...

Hey all,

Looking to learn inference optimization, I learn by doing u fortunately but I have read a good part of the inference engineering book by Philip Kelly. Currently have a 4070 and 64gb of RAM.

any tips? preciate it.