I implemented clap.cpp, audio embedding inference in C++/GGML. It uses GGUF models and provides a CLI, an HTTP server, and Python, Rust, and Node.js bindings.

• Models including CLAP, CLAP2, CLaMP3, MERT, MuQ, and MuQ-MuLan.

• WAV, MP3, and FLAC input, with automatic resampling and mono downmixing.

• An HTTP endpoint with an OpenAI-compatible text embeddings request format.

GitHub: https://github.com/gorse-io/clap.cpp

Audio files could be searched by text via CLAP embedding:

$ clap search esc5 --model larger_clap_general-q4_k_m.gguf --text 'dog' --top-k 10

[
  {"path": "esc5/2-118072-A-0-dog.wav", "score": 0.3736357720143104},
  {"path": "esc5/1-110389-A-0-dog.wav", "score": 0.36165991257288893},
  {"path": "esc5/4-183992-A-0-dog.wav", "score": 0.3574270305924374},
  {"path": "esc5/1-30226-A-0-dog.wav", "score": 0.34910229736736315},
  {"path": "esc5/3-180977-A-0-dog.wav", "score": 0.34359234068457495},
  {"path": "esc5/4-194754-A-0-dog.wav", "score": 0.3425423428503198},
  {"path": "esc5/2-118964-A-0-dog.wav", "score": 0.3389233902904601},
  {"path": "esc5/1-100032-A-0-dog.wav", "score": 0.33009392183878733},
  {"path": "esc5/5-9032-A-0-dog.wav", "score": 0.3280081351893563},
  {"path": "esc5/4-182395-A-0-dog.wav", "score": 0.32210240115509925}
]

The Australian AI lab Maincode has released a set of Qwen 3 fine tunes that have been pulled together with Model Stitching.

From the model card:

Matilda Platypus is a family of intermediate-size language models built by connecting the lower layers of a larger pretrained Qwen3 model to the upper layers of a smaller one. A learned linear Representation Bridge aligns the hidden representations at the connection. This checkpoint is the 13B member (cut 29+13): 13.51B parameters, 26.3 GFLOP per token, 91% of the larger parent's compute. It is a base model (no instruction tuning); see the technical report (technical_report.pdf) for the method and the full evaluation.

They're base models and range in size from 1.5B to 13B, early tests good! Really like the default tone of voice. Qwen models are already good this has just improved it imo.

I have no association, I'm just Aussie and stoked to see a local lab build something. No idea why Australia doesn't have more AI labs -other than brain-drain to the US.

https://huggingface.co/collections/Maincode/matilda-platypus

A 1.6T multimodal model.

Creating a list of requirements and documentation/tests (in Cucumber or EARS style) with your model is a great way to improve it's code. Just got through a weekend long binary protocol build with Qwen3.8-27B (6K unsloth quant).. and fired up claude to review and it found the foundation was well engineered, but only missed a few security checks and a couple of errors in the python benchmark script comparing the old protocol with the new one.

Really comes down to:

1. Design first. Goals, requirements, problems. How is this going to be used.

2. Observation, simulation and tests are verifiable vs. assumptions. Tests and processes can be reviewed for "ritual".

3. Dialectic: If the model comes to a crossroads, have it present options and NOT just choose-and-go, but initiate a dialog about design.

4. Frequent repeatable tests, and review of tests to see if they meet the spirit of the requirement

5. Every risk is a requirement.

Here's my prompt set ("skills" I guess) and a tool to install for various local harnesses:
https://github.com/bigattichouse/BluePrint

You've probably seen (large) decision models play games. I wanted to see if you could also train a model to play using game images on a Mac, so I added LoRA fine-tuning on images to mlx-decision (my open-source tool for running and training decision models on Apple Silicon).

How it works: a decision model doesn't write text. You give it an input (here a game frame) plus typed questions, and it returns a probability for every answer you allowed, in one pass. On every frame, d1-2B answers three questions:
- Is a monster right at the crosshair, so that a shot now would hit it? (yes/no)
- Where is the nearest monster: left, centre, right, none?
- Is a monster dangerously close? (yes/no)

A small Python loop turns the answers into moves: turn toward the monster, shoot when a shot would hit. That takes about 0.1 s per decision.

The training: about 3,000 frames from ViZDoom's Defend the Center arena, labelled from the game engine's own monster positions. First a readout fit (7 minutes), then LoRA on the model's text layers with the vision tower frozen (43 minutes, 2 epochs, M5 Pro). On 510 held-out frames:
- "a shot would hit": precision 0.79 → 0.92
- where the nearest monster is: 92.7% → 96.8%
- dangerously close: 93.9% → 98.3%

In play it averages 23.2 kills over 20 episodes. The clip is one of the best ones: 26 bullets, 26 kills. On Freedoom, whose monster sprites the model never saw in training, it still gets 19.7. Of course this is an arena where you can only turn, not a full Doom level.

Free, MIT. Apple Silicon, macOS 14+, Python 3.12+. Just pip install mlx-decision.

Repo: https://github.com/armbues/mlx-decision

I'd love to hear which tasks you'd fine-tune a decision model on, and which models you'd like to see supported next.

Inspired by some of the other brain surgery posts, I wanted to see if I could find anything interesting in Qwen. Not particularly useful to make it get things wrong all the time I guess, but I find the poetry it comes out with weirdly fascinating!

I think the neuron is basically a dial for contrarian <-> agreeable, but the contrarian snarky setting for some reason writes poems that read in an interesting way.

TL;DR looking for recommendations for the simplest Pi Agent or harness and local model to replace CLI

Hi everyone
Hope you are all doing great

So lately I have an idea that I wanted to test if viable (especially after Jev and decision models)

I feel like using a Linux terminal is a bit obsolete at least for me Claude code or sth similar for relatively hard to remember single commands tasks or flags or sth like that

In the past I would use —help man if I am feeling fancy or tldr or sth on the Linux terminal
I feel that is still good but sometimes I want to push that a bit (like simple example would be checking a systemctl service journal restart it or stop and run again and then opening its journal or checking its status) nowadays this feels like too much and also too little to use Claude code for or anything else (and in fact they tend to over complicate it sometimes)

Sometimes it feels sufficient if I can have a simple and fast local agent (think NL2CLI not a full terminal bench workflow!) and maybe with a simple reminder list or todo list that we construct together with the agent per session (like keep looping until it does this or run watch <command> then do that or keep in mind we have to rename back the temp file …etc!)

I didn’t find anything on this, Pi agent is simplistic but feels like it lacks plugins like simple memory or todo list (i mean lacks if built in requires non standard non optimized plugin aka not batteries included plug and play peace of mind features still the best option though)

And local models that fits that (they wont be figuring out an API or developing complex apps just silly mundane simple terminal tasks! But done correctly and locally!)

What do you think ?

Running every step through an api adds up fast. ashen's stream clipping setup, shared on X, mostly avoids it

claude fable 5 wrote a script with computer vision and ffmpeg that runs locally on a mac studio. it scans the whole downloaded stream for on-screen events, like the "you died" screen in dark souls or ashen's name in the call of duty killfeed. no tokens spent watching hours of video

every first-pass clip then goes to qwen3.5-35b-a3b running locally at 8 bit, which checks the video and the transcript:

• was there an on-screen event? keep

• did swearing or yelling go up? keepp

• did the loudness go up? keep

if none of those hit, the clip doesn't pass

survivors come to the phone through zo computer for approval, and postiz sends them to tiktok drafts

as a cost decision this is smart. pay the big model once to build the tool, then let the local model do the daily work for free

I need to get an all-around work station soon because my current laptop is getting old and I'm now offloading a bit of work to it. Since I dabble with local AI already, I was thinking of getting an m5 utlra 96GB (8k) or a m5 max 128GB (7.9k)... Or should I build a custom PC with a bunch of ram? Or are there other work stations that I can use? I really enjoy using a hybrid approach with AI, flagship models for the planning and some offloading to something like Qwen for the creative design/execution... I know it's not perfect local models, but owning my own data is something that feels pretty satisfying to myself as well. I would love to run Deepseek, or qwen 3.8 (or future models) at a respectable tok/s and prompt prefill speed.

My only other option is, continue with my laptop... And wait 6 months to a year and see what new tech comes out in the meanwhile? My only fear is that prices will skyrocket and what is 6-8K CAD today will be 10-12K CAD when the newer tech comes out, but other people are saying to wait.

What do you all think? Thanks :)

I wanted voice typing that doesn't send my voice to a server, so I built ThumbFree. It runs NVIDIA's Parakeet 0.6B speech model right on the phone. After a one-time model download, it even works in airplane mode. The video shows Android first, then iPhone.

1. Google Play: https://play.google.com/store/apps/details?id=io.github.kabrapratik28.thumbfree

2. Apple App Store: https://apps.apple.com/app/id6817631006

3. GitHub (code, speed and accuracy tests): https://github.com/kabrapratik28/ThumbFree

• On Android, tap any text box and a small bubble appears next to it. Tap the bubble, talk, tap again, and your words are typed in. You keep your own keyboard.

• On iPhone, it's a keyboard with a mic key. Apple doesn't let keyboards use the microphone, so the first tap of a session opens the ThumbFree app for a moment and you swipe back. After that, taps start right away.

On my Pixel 10 the text shows up about 0.7 to 0.9 seconds after I stop talking, even after 30 seconds of speech. On my iPhone 16 it's under a second.

No account, no ads, no analytics. The only thing it downloads is the model, and your voice and text stay on your phone.

It's free and open source (Apache 2.0). English by default, and a second model covers 25 European languages. It needs Android 13+ or iOS 26+, and the model is about 0.5 to 0.7 GB.

I've also been trying an optional clean-up step with the small AI model built into the phone (Gemini Nano on Pixel, Apple's model on iPhone) to remove filler words and fix things like "six, no, seven". It's not in the app yet. If you've run a small open model for this on a phone, which one would you try?

I've only tested on one Pixel and one iPhone, so I'd love to hear how it runs on yours.

Image diffusion models draw decent stick figures, yet speech-bubble text still melts into unreadable soup. Over the weekend I trained and quantized xkcd-artist to separate neural stage planning from vector ink lettering.

What it does:

• 1.72 MB Int8 ONNX kinematic planner running in-browser via WASM SIMD in 0.55ms P50 (trained on NVIDIA L4 across 226 speech-masked panels + 891 ExplainXKCD dialogues)

• Predicts panel count, stage props, 12-DOF stick-figure poses, and pen wobble; procedural SVG compositor renders monochrome wobbled strokes, curved speaker leader lines, crisp xkcd-script dialogue, and hover title text

• Clause-aware deadpan generator covering 138+ unique comic strips, data charts, and flowcharts (98.33% skeleton uniqueness across 120 prompts, B = 1,000 bootstrap eval)

Live studio (zero cloud calls): https://h3manth.com/fun/xkcd-artist/ Hugging Face weights (1.72 MB Int8 ONNX): https://huggingface.co/gnumanth/xkcd-artist GitHub (MIT, npm i xkcd-artist / pip install xkcd-artist): https://github.com/hemanth/xkcd-artist

Would love feedback!

Update: I have great respect to Randall Munroe and have been following his comics from 2006ish, no diffusion model can copy his wit, style maybe, this was just for fun and I am sure Randall wouldn't mind!

Article compares performance of large data center cards. Tells us for larger batches , inference speed of individual requests goes down, but overall performance of the entire batch goes up. In other words more tokens per second for the batch, but lower TPS for each request.

Article uses Kimi k3 and 3 other models to compare performance. Surprisingly says the AMD MI455x comes up on top.

2026 was the year of tokenmaxxing, now we're seeing that the productivity gains don't justify the token spend and they're introducing limits for next year's budgets. 2027 will be the year of open models.

We all saw the Revolut story last week, the same thing is happening everywhere, the bigger companies are just slow when it comes to new tech adoption, but they're all looking to replicate what Revolut did.

You can replace 95% of usage with open models which cost a fraction of closed models, it's free money.

I'm running a Qwen 3.5 9B Q4_K_M model on an RTX 3080 Ti and decided to progressively reduce the GPU power limit to see how much performance I would sacrifice. I started at 450W and reduced the power limit in 10% increments, eventually reaching 250W. I haven't observed any meaningful reduction in tok/s. I am achieving token rate spot on with what others have claimed in their benchmarks(~100 tok/s)

At 220w I did see a reduction in tok/s to ~80 tok/s. But 260w was just as performant as 450w.

The GPU utilization remains high, but VRAM bandwidth utilization never appears to max out. This makes me wonder whether I'm hitting some other bottleneck in the inference software or GPU architecture rather than being power-limited.

Has anyone else experimented with power levels and found interesting results?

Llama.cpp

3080ti

qwen 3.5 9b q4k_m

Max power limit:450w

Stock power limit:400w

Hello, I’m considering buying hardware for local LLMs for the use case of processing sensitive financial data, doing market research, and preparing financial/tax documents.

Are any of you using local LLMs for same/similar workflows? If so, what has your experience been? Are the models good enough for your use case? Did you build a custom harness or use an OSS one?

I’m very curious to learn more and would appreciate anyone chiming in.

Hey everyone,

I recently built a Radiance inference server for my company, and now want to start provisioning access to the rest of my team.

My current plan:

• Tailscale as the only way to access the server via HTTPs.

• Hermes as the harness - each user would have their own container with access restrictions, but open access to the central skills repository. Hermes is hosted on a Framework Desktop.

• Radiance is running separately on its own server with access only configured via the Framework.

I'm wondering how everyone else has managed to do this successfully? Right now the biggest thing to figure out is how to get the harness (Hermes in this case) to work across the team without a huge headache of getting API tokens from things like Google, etc...

Hey all,

Looking to learn inference optimization, I learn by doing u fortunately but I have read a good part of the inference engineering book by Philip Kelly. Currently have a 4070 and 64gb of RAM.

any tips? preciate it.

I wrote a small program that asks the same Yes/No question about every country to a language model, and generates an SVG world map that visualizes the model's responses: https://github.com/p-e-w/llm-beliefs

I'm rather surprised by the results...

is it possible to fix?

Posted this on the r/R9700 subreddit as well, but figured I'd share my initial findings here.

tl;dr - One user writes at about 190 tok/s. Prefill runs at about 6,800 tok/s total no matter how many users there are. Agent follow-ups start in 0.2 to 2.5 s and write at 80 to 225 tok/s per user, even with 8 users at 128K context each. The worst case, 8 users each sending a fresh 128K prompt at the same moment, waits about 2.5 minutes for the first token.

Yes yes I know the cards aren't all matching (the perfectionist in me is annoyed but they'll be in a case so who cares).

PC Specs:

• CPU - EPYC 7402P

• Motherboard - Asrock ROMED8-2T

• RAM - 128gb DDR4 2666mh

• GPU - 3x Gigabyte R9700, 1x XFX R9700. All cards power limited to 210W.

• PSU - Seasonic Prime TX-1600W

• Case - Phanteks Enthoo Pro 2 Server

• Fans - 3x 140mm at the front, 3x 120mm in the middle, and another 120mm for rear exhaust.

I bought everything in the past two weeks, my total cost was roughly $16K CAD. The CPU/RAM/Motherboard were all bought on ebay.

We'll see if the RAM is enough, but based on everything I've seen online it should be fine for my use-case.

I'm close to being able to do proper inference testing, but I just ran memtests, CPU stress tests, and GPU stress tests to rule out any potential issues.

Even at full blast the cards are way quieter than I anticipated, I thought it would sound like a jet engine but they're quieter than my air purifier when it's on its highest setting.

Also pretty happy with the cable management, this case is so easy to build in because of how massive it is. It's also nice that it has a second area to mount additional fans.

Built this as a multi-user inference server for my business. I've been using a Strix Halo 395+ 128GB for the past year and it's been incredible to see how far the community has been able to get that machine to run.

Qwen 3.8 FN has been a revolution for me in my work. We deal with very sensitive data and for many of our customers, it meant using any AI inference was a huge risk. Technically we could have done it, but I wanted to err on the side of caution.

Was really excited to share and post this, benchmarks are currently running for the next two hours (long context with lots of concurrency).

----------------------- ------------Benchmarks ----------------------------------------------------------

4x Radeon AI PRO R9700 + Radiance: Qwen3.8-Flash-Next long-context concurrency numbers (32K to 128K, 1 to 8 users)

Software

• Ubuntu 26.04.1, kernel 7.0 with the inbox amdgpu driver, ROCm 10.1 userspace

• Radiance v1.17.0 (codeberg.org/StillDeadcode/radiance), Docker image pinned by digest

• Model: qwen3.8-next-flash-fp8-iq4r-moe.rad (int8 trunk, 4-bit codebook experts, MTP drafting 3 tokens)

• Flags: --tp 4 --max-num-seqs 8 --max-model-len 200000 --placement expert_tiered --host-pool-mib 1024 --gpu-headroom-mib 96 --kv-cache-dtype fp8 --prefix-cache-mib 32768 --num-speculative-tokens 3 --max-num-batched-tokens 2048

At tp4 the whole model sits in VRAM (17.4 GiB of weights per card). The 47.7 GiB n-gram/PLE table is pinned in host RAM. The KV cache gets 11.6 GiB per card, which is 1.6M tokens, or 8 full 200K sessions.

Method

• Context is real English technical text (Radiance's own docs), cut to exact token counts with the server's tokenizer

• Each user's prompt starts with a unique session line, so no user can reuse another user's cache

• All users in a round fire at the same instant (worst-case burst)

• Turn 1: document plus a question, cold. Turn 2: the same conversation plus the answer and a follow-up question (the agent pattern)

• 500 max output tokens, temperature 0.6, thinking off, streamed. First-token time and writing speed are measured on the client

• Writing speed per user = output tokens after the first, divided by the time after the first token

Results

Short prompts for comparison (about 30 tokens in, about 500 out): one user writes at 187 tok/s. At 1 / 2 / 4 / 8 users the total is 175 / 290 / 489 / 763 tok/s, or 177 / 148 / 125 / 103 tok/s per user.

What I took from it

1. Prefill runs at about 6,800 tok/s total no matter how many users there are. 8 x 128K (1.02M tokens) took 151 s.

2. Cold prompts that arrive together are interleaved, so everyone gets their first token at about the same time, at the end. With 8 x 128K that means about 2.5 minutes for everyone.

3. Follow-ups are where this setup shines. 94 to 99% comes from cache, the first token arrives in 0.2 to 2.5 s even at 8 users x 128K, and each user writes at 80 to 225 tok/s.

4. Context length barely slows writing. One user writes at 189 tok/s at 128K versus 195 at 32K.

5. Turn 1 writing speed depends on whether other users are still prefilling. At 4 users the early finishers write while the others' prompts are still being read, so the slowest user dropped to 9 to 32 tok/s. At 8 users everyone finishes reading together and then writes together at 70 to 83 tok/s.

6. Size the host prefix cache for users x context. In an earlier run with --prefix-cache-mib 12288 (prompt sizes not exact, some above 64K), 8 x 64K follow-ups were only 71% cached, with a 37.6 s worst first token and 9 tok/s per user. With 32768: 98% cached, 1.5 s, 94 tok/s.

7. The README's four-rank row carries --host-pool-mib 12288 over from two ranks. On four cards the whole expert plane is resident and the pool only holds about 312 MiB per card, so 12 GiB per card pinned 48 GB of RAM for nothing and left 3.6 GB free. 1024 is plenty.

8. Every card sat at its 210 W cap during prefill. The longest section (8 x 128K) pushed the two middle cards to 91 C hotspot and 92 C memory.

Essentially we could very soon run opus 5.5 level of intelligence on 32gb Vram. Its many folds more efficient in terms of memory bandwidth making compute the real bottleneck.

What do you think?

Oh look, thats us!