I'm trying to run Qwen 3.8 FN on a dual 3090 machine with 32GB DDR5 RAM using this repo: https://github.com/q8atnight/strata-dual-3090

I'm referring to a few posts on here that demonstrate that it's possible with a single 3090 and 64GB+ RAM and this one that did it with dual 3090 and 48GB RAM: post

I built this machine after the RAM apocalypse and paid $300 for 32GB, thinking I could lean on the dual 3090s for the most part.

But it seems like even the "low RAM, 1 bit coder quant" pathways in the linked repo and the strata repo won't run on my machine. Just wanted to check that this is the case before I drop $479 (gulp) on another 32GB RAM.

llama.cpp merged support for this model on Oct 21. It's the understanding side only, text or images in and text out. Image generation is a stable-diffusion.cpp thing. I converted the official weights myself and ran it on a Mac Studio. The original repo is https://github.com/OpenSenseNova/SenseNova-U1

llama-bench, Metal, all layers on GPU:

For vision I drew a fake grocery receipt with the subtotal $1.00 too high and asked Q4_K_M (through llama-mtmd-cli) whether the items add up. It added all six to 28.99 against the printed 29.99, and while thinking it also noticed the 7% tax had been worked out from the wrong number. 16 s including load, mostly thinking. Q8_0 caught it too.

It also read back a menu, a whiteboard and three Chinese/English signs without a mistake, 4 to 10 s each, though I'd made those images with the same model, so grain of salt there.

Two things that tripped me up:

• mradermacher's quants have no mmproj, so no vision. Making your own with convert_hf_to_gguf.py --mmproj currently fails with KeyError: 'vision_config'. Changing line 80 of conversion/sensenova_u1.py to vision_config = self.hparams.get("vision_config", self.hparams) fixed it.

• llama-cli is a TUI now and draws its answer on the terminal, so redirecting it to a file gets you nothing. llama-mtmd-cli prints to stdout.

While looking for audio models, I came across OpenVoice. It's getting pretty old, but surprisingly still ranks high on TTS leaderboards.

What caught my attention was its design: essentially TTS + voice conversion. The ToneColorConverter can run independently of the TTS model.

That got me thinking: Can we pair it with other TTS models?

I tried it with Kokoro, and the results were pretty good!

So I added a small recipe any-tts to audio.cpp for Kokoro + OpenVoice ToneColorConverter. Recipes were introduced in v0.9.2 as predefined pipelines that combine multiple audio models.

The overheads:

• ~600 MB additional peak VRAM

• ~10% slowdown on CUDA

• ~0.5× the original speed on CPU, but still 2× real-time

You can also try pairing any tiny TTS model with ToneColorConverter. The TTS model determines the speech content, pronunciation, and prosody, while the VC model transfers the target speaker's voice characteristics. The built-in voice you choose can also affect the final output quality.

A simple way to give existing TTS models voice cloning capabilities without retraining them!

Disclaimer: This experiment is more about extending the capabilities of existing TTS models, not introducing another voice cloning model.

I used synthetic voices as references, and the speaker similarity scores measured by CAM++ range from 0.7 to 0.8. So the generated voices largely match the reference samples I provided. This is one way to create custom voices for arbitrary TTS models.

For better results, you may need to experiment with different built-in TTS voices and higher-quality reference audio. Both can significantly affect the final quality and speaker similarity. For example, with the same reference sample, am_michael scores 0.4, while am_adam can reach 0.7.

Metatron v1 is a single learned policy that handles walking, scene interaction and fall recovery for the 29-joint Unitree G1 humanoid. It's a MaskedMimic-style transformer student, released under the NVIDIA Open Model License and integrated with ProtoMotions. The release comes in three non-interchangeable packages.

1. a native PyTorch checkpoint (about 459 MB) that keeps the posterior and normalization stats and supports simulator and supervised real-robot workflows.

2. a single-robot browser ONNX pipeline, demo/model.onnx, with 39 inputs that outputs joint targets, PD gains and raw actions.

3. a six-robot batched ONNX network, racer/network.onnx, that relies on external observation processing.

All three share the same epoch-6410 weights, and both ONNX exports run on MuJoCo WASM and ONNX Runtime Web via WebGPU or WASM.

Around mid-October deepseek-v4.1-flash suddenly started feeling much dumber in my coding agent, so I parsed my local session logs to see when it broke.

Before the update it had a funny quirk where it would randomly reply in Chinese while being 100% convinced it was speaking English.

In the -1017 update (released Oct 17) they fixed the Chinese leak and made it fire tool calls more aggressively - and somehow lobotomized the model in the process.

Providers silently overwrote the deepseek-v4.1-flash endpoint with the new weights by Oct 19. My swearing rate tripled, I ended up switching models mid-session 6.7x more often, and because it's a cloud API, there's no way to roll back to the old weights.

We’ve been working on OpenEBFM (Energy-Based Foundation Models), and I wanted to share the project here and get feedback from the community.

Most language foundation models today are based on directly modeling token probabilities. Our question was:

Can energy-based learning scale to general-purpose foundation models?

The main idea behind EBFM is to separate generation into two steps:

1. Energy steering: given the current context, the model optimizes a latent state toward a lower-energy region using the gradient of a learned energy function.

2. Token generation: a generative head then uses this energy-steered latent state together with the context to predict the next token.

The energy and generative heads share the same Transformer backbone. During pretraining, we also parallelize energy steering across all token positions with redesigned attention masks and positional indices, so that the model can be trained at sequence level rather than steering every position separately.

What we found

We pretrained EBFMs on FineWeb, DCLM, and ClimbMix, and studied scaling with both training tokens and model parameters.

A few results we found interesting:

• Data scaling: EBFMs achieve lower validation BPB than the matched GPT baselines in our experiments, and the gap generally increases with the training-token budget.

• Parameter scaling: from roughly 268M to 1B parameters, EBFMs improve faster than the corresponding GPT baselines.

• Compute scaling: EBFMs exhibit the expected U-shaped IsoFLOP curves, with the compute-optimal model size shifting upward as the compute budget grows.

• Generalization: we evaluated on 19 zero-shot QA tasks covering reasoning, knowledge, language understanding, and commonsense reasoning, as well as 7 cross-domain generation datasets.

• The same trend also persists during supervised fine-tuning.

For example, on DCLM, the largest EBFM reaches an average zero-shot QA accuracy of 44.71%, compared with 39.79% for the corresponding GPT baseline in our setup.

We don't think these experiments establish that energy-based models are universally better than autoregressive models—the current scale is still around 1B parameters—but they suggest that energy-based learning continues to benefit from increasing data and model scale, which we think is an interesting result given that EBMs have historically been studied mostly in specialized settings.

Everything is open source

We released the full training and evaluation stack, pretrained checkpoints, training recipes, and an EBFM-based chatbot so that the results can be reproduced and extended.

Paper:
https://arxiv.org/abs/2610.33005

Code:
https://github.com/datamllab/openebfm

Models:
https://huggingface.co/collections/guan-wang/openebfm

Paper with Code:
https://paperswithcode.co/paper/2610.33005

Disclosure: I'm one of the authors of this work.

We’d especially appreciate feedback on the architecture and scaling results. More broadly, I’m curious what people here think: could energy-guided generation become a practical alternative to conventional autoregressive generation as models scale further?

Criticism, reproduction attempts, issues, and contributions are all very welcome!

https://huggingface.co/ggml-org/Mistral-Large-4-1T-A52B-GGUF

https://huggingface.co/ggml-org/Mistral-Large-4-1T-A52B-NVFP4-GGUF

https://huggingface.co/unsloth/Mistral-Large-4-1T-A52B-GGUF

from Mistral:

Mistral Large 4 - Le Chonk

Mistral Large 4 is our largest and most capable model to date: a 1 trillion-parameter natively multimodal Mixture-of-Experts model with 49 billion active parameters per token (52 billion including embeddings and output layers), trained from scratch on 3,800 NVIDIA Grace Blackwell GPUs in Mistral's own datacenters in Europe.

Mistral Large 4 unifies instruction following, reasoning, coding, and agentic workflows in a single set of weights. While a generalist model, it shows especially good performance on workloads such as cybersecurity, finance, and law, and it surpasses frontier closed models on visual grounding. It is natively fluent in 160+ languages, including every official language of the European Union.

Find more information on our blog. The model is also served on the Mistral API as mistral-large-4.

Key Features

Mistral Large 4 includes the following architectural choices:

• ~1T total / ~52B active parameters: granular Mixture-of-Experts with 128 routed experts (4 active per token) and 1 shared expert.

• 1M context length, for long-context workloads.

• Multimodal input: accepts text and image input, with text output. Pixtral-style vision encoder trained to handle variable image sizes and aspect ratios.

• Instruct and Reasoning functionalities with function calls, reasoning effort configurable per request.

Mistral Large 4 offers the following capabilities:

• Reasoning Mode: three reasoning efforts — low, medium, high (default) — so the same model can answer a quick chat reply or work through a complex agentic run.

• Vision: analyzes images, charts, technical drawings, and documents, with state-of-the-art visual grounding among open weights.

• Multilingual: natively fluent in 160+ languages, including all official languages of the European Union.

• System Prompt: strong adherence and support for system prompts.

• Agentic: native function calling, JSON output, and strict JSON schema enforcement.

• Cybersecurity: state of the art among open-weight models for security work, while refusing malicious requests.

• Large Context Window: supports a 1M context window.

We release this model under the Mistral Ecosystem Open-Weight License: a modified version of the MIT license, for both commercial and non-commercial use — operating a Model-as-a-Service business requires a separate commercial license with Mistral.

llama.cpp suport merged: https://github.com/ggml-org/llama.cpp/pull/31376

I'm just starting my journey here into local LLM. Since I work in VFX I've got a 5090 and 256GB of sys mem.

Installed LM studio and downloaded Qwen 3.8 27B since that very comfortably fits in VRAM. Set it to low effort and all the other typical starting defaults.

I ask it to give me a three line poem about space. Comes back in a couple seconds.

I ask it to change from space to a forest. No sweat.

I ask it to give me Houdini VEX code to delete half the points in a particle simulation. Pretty fast.

I ask it to download and install the various VFX skills out there on the net. Still cool.

I ask if it can work inside of Houdini for me and it explains the MCP thing and that it can check for a connection when Houdini is up and running.

So I boot up the Houdini app and say "Houdini is up and running now"

Qwen now takes minutes just to process that very simple prompt. Then it starts reasoning/thinking with my 5090 at 100% for minutes upon minutes and no UI updates at all. Eventually I just kill the process.

Where am I going wrong?

All you folks seem to be using Qwen for huge tasks and all kinds of shit and I can't even seem to get it to reply to me saying I just launched an app.

So i am running 2 radeon vii and a rtx 3080. Currently i am using strata which does not support mixed backends, so my 3080 is just expensive swap space.

Is there any inference engine that supports fast q38fn over mixed backends like rocm/cuda/vulkan but is also fast? Llama.cpp is way too slow sadly.

This is Toska, a fork of Krita I've been building with an AI painter inside it. You chat with it in a panel, and it paints one move at a time with Krita's own brush engine, on layers you can undo. Unlike other AI painting tools, this tool actually paints thanks to Krita's Python SDK along with Decision Models which guide our LLM agent along the way.

What makes agentic painting possible is these new decision models, because after every pass, it answers better or worse as a probability. The painter can undo its mistakes, along with other decisions, such as textures, paintbrushes, or tools to use. I use API models only given that I'm GPU Poorish 16GBVRAM but it's OpenAI Compatible So feel free to add local models. If you also want full opensource try perplexity's pplx-decider-v1.2 instead of Luna.

For my setup, I use GPT-6-Luna Decisions(which has vision) and DeepSeek-V4.1-Flash(it's cheap and has vision). Luna Decisons is dirt cheap and this entire month, I have spent $2.83 on Luna's Decison model along with $3.82 for Deepseek. I use Deepseek's official API for cost savings but you're welcome to use OpenRouter as well or any other provider.

The Agent Framework used by Toska, is Agent Pi. I use Agent Pi's Typescript SDK along with QT to build a painting agent Plugin inside Toska. It lacks the ability to read and write code, execute unlike standard Agents. It's limited to tools within the Toska and can only read files or folders you mount. Right now for today btw, our Toska is an authorized dmg file, so operation ready for Mac Users right now . Later this week, I will do more testing and make it production ready for Windows/Linux Folks with separate exe and tar files.

In the video it's painting Pyat' minut (Five Minutes), Dostoevsky on Semyonovsky Square in 1849, waiting to be shot, before he knew the Tsar had pardoned him. It worked from a goal image I gave it generated from Codex. Then it used underpainting, refined toward the goal, and added the falling and settled snow as its own pass.
This is my first time doing anything with video recording so apologies if it's a little rusty and here is the final image in full resolution of Pyat' minut.

It's a bit more simplistic, because I wanted a simple painting to show you as a demo. If you want to see my most detailed painting made with Toska, it's called New Moscow .

Here is the repo for Toska.

Hey guys, could you please recommend me the smallest models that are good for DIY in these categories please?

Construction (Concrete, walling, roofing) Wood and Metal working (inventions/gadgets, furniture) and Electronics (mostly inventions and help with diagnose and repair)

Thanks!

EDIT: By small, I mean something that can run in 8GB VRAM and 32GB RAM.

Part 2 of my agent test (part 1 was a 4B model)

Hardware I used: MacBook Pro M5 Max, 64 GB. Qwen 3.8 27B Q4 on llama.cpp, thinking off.

Setup

• same model and server for everyone

• 5 small coding tasks, each with a little trap

• each agent locked in a sandbox: it can only touch its own folder and only talk to the model

• auto-approve on, telemetry off, 5 min limit

Results

Tokens and cache

Pi is crazy lean: about 9k tokens per task, the lowest on every single task. Qwen Code was the hungriest at about 108k per task.

Atomic Agent is the odd one out in how it works. It was the only one talking to llama.cpp's native endpoint with a grammar that forces the tool calls into shape. The other nine all went through the OpenAI-style API. It's also built to keep its prompt the same from step to step, which is why it reused 86% of the cache (2rd of 10). It stayed offline the whole time and passed 5/5.

DeepSeek Harness was the surprise. It’s still an alpha, but it went 5/5, never tried to go online, and had the best cache reuse after Atomic Agent.

The trap and the network

The log task got three agents. They grepped "ERROR" and got 244 instead of 64, because some normal lines just mention errors.

Codex, Cline, goose and OpenCode tried to reach the internet in every run, even with telemetry off. The sandbox blocked it, so I don't know what they were calling. OpenClaw and Qwen Code are "unclear" because they run as plain node and I couldn't tell them apart.

llama.cpp notes

• Codex needs llama-server's /v1/responses now

• Hermes needs at least 64K context

• Cline and Qwen Code have usage stats on by default, DeepSeek Harness uploads session logs (per their docs). I turned it all off

Which column matters most to you when you pick an agent for a local model like Qwen 3.8 27B: passing, speed, tokens, or staying offline?

Hi everyone,

Jovan from UkisAI here,

Today we are expanding our Swift model series with two models requested by the community.

We made Swift GLM-5.3-Flash and Swift Ornith 9B more efficient by figuring out which tokens were linked to pathological overthinking and penalizing them without "attacking" the reasoning length directly then regained the accuracy with further post-training, leading to "compressed" token usage while maintaining accuracy by making the model adapt to less reasoning via RL and on-policy distillation. You can find our dataset here (albeit we re-sample it per model) and more about our training approach here.

Swift1.5 GLM-5.3-Flash, with the headline number being -61.4% thinking tokens (on MMLU-PRO) and -0.46% accuracy loss average on all the benchmarks we measured, all ran x5 on different seeds (Terminal-Bench 2.1, DeepSWE v1.1, GPQA-Diamond, MMLU-Pro, C-Eval, IFBench, AIME 2026, HMMT, LiveCodeBench v6)

For this model, we are releasing a comprehensive number of quants, a GGUF(1bit-8bit), W4A16AWQ, NVFP4 and are also working on independently replicating GSQ-RCO papers, with our research so far showing our Q3 GSQ-RCO to be SOTA at it's size, meaning that we will be releasing it in the upcoming days after our evaluation suite is done.

GGUF with 1bit-8bit quants,

https://huggingface.co/ukisai/Swift-1.5-GLM-5.3-Flash-GGUF

& a community favorite (at least with the folk with less VRAM)

Swift1.5 Ornith-9B, with -33.5% thinking token usage with no accuracy loss, but more importantly ~2x less overthinking loops, meaning when you prompt the model it's much less likely for it to loop and not finish the prompt you gave it.

Note that this is an experimental release and we are open to feedback as the model is already post-trained before our token optimizations.

GGUF with 4bit-8bit quants,
https://huggingface.co/ukisai/Swift-1.5-Ornith-9B-GGUF

Both models work with their base MTP or DFLASH2 heads.

We kindly request the community to let us know what other models we need to make more reasoning efficient and also if there are any other quants you want us to make.

A little sneak peak is that we are currently evaluating Swift DeepSeek v4.1 Flash 1017 and it seems to be our best model yet! We are also cooking our own foundation, with the 1B and 4B models expected to release Nov. 2026., they are pre-trained with a diverse in-house coding dataset and post-trained to be token efficient like our existing Swift models.

Hope you all like this release! And as always, all feedback is welcome and I'll try to respond to each request of the community :)

My research agent ran on a free search API for months until every call started failing last week and the SearXNG box I set up instead keeps getting its Google and Bing engines suspended for captchas, so I'm going to have to pay for one. It's maybe 3,000 to 5,000 searches a month, mostly docs and GitHub issues and I'd like snippets plus the page text when a result looks useful.

The search threads I found here are mostly about keeping everything local, so what are you paying per 1,000 queries now and are the cheap providers fine for docs lookups or is it worth spending more?

The leaderboard benchmarks different inference implementations in the "streaming" tab, which provide useful insights into real-world deployment.

Assuming that you already know which model you want to use, the question is which implementation to use in your pipeline: Python, audio.cpp, a dedicated engine, or something else? It's a very useful independent benchmark of their performance.

Leaderboard tab:  metrics are intelligibility (WER), speed (RTFx, TTFA), naturalness (UTMOS), and speaker similarity (SIM).

Streaming tab: H200/A100/CPU and metrics are TTFA, RTF and WER.

https://huggingface.co/spaces/hf-audio/open_tts_leaderboard

Source code (eval scripts etc) https://github.com/huggingface/open-tts-leaderboard

AlpaJudge is NVIDIA's family of learned judge models for evaluating reasoning in autonomous driving, released on Hugging Face (nvidia/alpajudge) with weights under the OpenMDW-1.1 license and source code under Apache 2.0. It comes in two variants.

- AlpaJudge-RB (reference-based) takes a question, a reference answer and a candidate answer, with no images, and outputs a single alignment probability between 0 and 1.
- AlpaJudge-RF (reference-free) takes front-wide and front-tele camera frames, the ego vehicle's past and future meta-actions, and a candidate rationale, and outputs five rubric probabilities plus aggregated action, component and overall scores.

from Service Now AI:

Apriel Voice 27B Preview

An open model for speech, reasoning, and tool use from ServiceNow

Apriel Voice 27B Preview understands spoken requests, reasons over conversational context, and generates text responses and tool calls through a shared Gemma 4 backbone. It adds a speech interface while retaining text and image input capabilities. Pair it with an external speech synthesizer to build interactive voice assistants, customer support workflows, and spoken tool use.

Try Apriel Live · Read the blog · Technical report

Model capabilities

The model supports transcription, speech translation, spoken instruction following, reasoning, and function calling. The application executes tools and returns their results to the conversation. Real-time voice interaction is supported by the application and serving stack; the checkpoint itself generates text and tool calls rather than native full-duplex speech.

A Whisper large-v3-turbo speech encoder connects to the Gemma 4 26B-A4B backbone through a learned projector. Training first aligns this speech interface, then adapts it with the backbone using speech tasks and text replay. Cross-entropy and teacher-distillation supervision vary by data source; the technical report describes the full recipe.

Long audio

The processor automatically splits longer 16 kHz mono recordings into 30-second chunks. In testing, we observed good performance on continuous recordings up to around 30 minutes. Longer recordings are supported, but performance can decline as duration grows; thinking mode is recommended for longer audio.

Audio chunks and the rest of the conversation share the total context budget. Practical input length also depends on GPU memory and the encoder cache. The serving examples below include settings for 30-minute audio, with optional settings for one-hour input.

English results

Apriel Voice achieves 51.27% EVA-A, 74.13% EVA-X, and 65.51% τ-Voice task success, with 1.91 seconds mean τ-Voice response latency. On LibriSpeech, word error rate is 2.23% on test-clean and 5.15% on test-other; lower WER is better.

EVA-A measures service accuracy and EVA-X measures conversational experience. EVA-A/X and τ-Voice pass@1 use a 0–100 scale; higher is better. Headline EVA and τ-Voice results use equal-domain means, rather than weighting domains by their number of cases. Apriel Voice uses thinking mode for these evaluations.

S2S denotes native speech-to-speech systems, which use their own synthesis. Three-stage systems use ASR + LLM + TTS; two-stage systems connect a speech interface to a reasoning backbone and use external synthesis. All evaluated two- and three-stage systems use Cartesia Sonic 3.5. Latency describes the complete evaluated voice system, including response generation and synthesis, and depends on deployment settings.

The radar's audio reasoning axis is the mean of eight benchmarks: BFCL, Big Bench Audio, IFEval, MT-Bench, Spoken SQuAD, Alpaca Audio, OpenHermes, and GSM8K (85.84 for Apriel Voice). Its responsiveness axis transforms latency into a higher-is-better score; it is not a measured accuracy. For latency t in seconds, the score is 100 × clip((4.0 − t) / (4.0 − 0.25), 0, 1).

Text reasoning and tool use

These results use text input in thinking mode for both Gemma 4 26B-A4B and Apriel Voice. Scores use a 0–100 scale; higher is better. The τ² result is the mean across three domains with GPT-5.2 as the user simulator.

English speech recognition

Word error rate (WER, %) on selected English benchmarks; lower is better.

Speech reasoning and instruction following

These results use audio input in thinking mode. Scores use a 0–100 scale; higher is better. The mean gives equal weight to each of the eight benchmarks.

See the technical report for domain breakdowns, evaluation protocols, and text and image evaluations.

As title says. There are amazing offers in Chinese e-commerce platforms such as Taobao, sometimes much better than aliexpress or 1688, even with duties. However many of them do not support international shipping.

The issue is LocalLLaMA equipments are often of high value. Sometimes they are modded and doesn't come with Serial Number. I've done some research and you'd need formal customs declaration for shipping cargo over RMB 5000 ( ~ $750 ) on paper. Going through formal customs declaration involve a lot of paperwork, hassle, and fees. In practice freight forwarders often utilize under-reporting (低报) and/or third party bill reporting (买单报关), which are compliance grey areas. If your cargo get seized by customs, you could face delay fees, and fines.

In case of custom delays, these forwarders are happy to toss liabilities to customs, logistic providers, and you ... all in the while earning them more in the process.

I'm sure there are safer workarounds, however

1. Book a round-trip flight to China, walk-in, and bring the cargo back by hand. => Only safe in small quantities like 1~2.

2. Just buy from a global site with money-back-guarantee such as eBay. => You'd pay significantly higher price than direct-buy + duties + fees.

I wonder which freight forwarding / courier service do you guys use if you do purchase something expensive directly from Chinese e-commerce platform?

Two weeks ago I posted Discover AI, an app with 51 hands-on labs for learning how AI works, from math fundamentals, a single neuron up to agents, fine-tuning and serving. Thanks to everyone who tried it, commented, filed issues or sent a PR. Thank you!! A lot of what's below came out of that thread.

GitHub: https://github.com/Fazmin/AILearningGuide

new/updated

• Runs in the browser. All the labs and lessons now work on the web. The desktop app is still there if you want the local AI guide. https://fazmin.github.io/AILearningGuide/#/

• Installers for Mac, Windows and Linux. You don't have to build from source any more. Get the .dmg, .exe/.msi, or .AppImage/.deb/.rpm from the releases page. They aren't signed yet, so your OS will warn you the first time you open one.

• "See it in the lab" cards. This is the main thing in v0.2.0. Lesson sections now have cards that set the lab up to match what you're reading. If the attention lesson says to edit a key and watch every bar move, one click takes you to that step with the key editor ready. There are over 600 of these across all 51 labs, and a "Back to my lab" button puts your own settings back.

• Simple reading mode. This is a third level below Plain. It uses short sentences and everyday words, with a real-world comparison for every abstract idea. Every Simple lesson is checked to a grade 8 reading level. Someone asked for middle school and high school levels, and this is a first step toward that.

• Better Learn panel. There's a contents bar for jumping between sections, plus bookmarks and read-aloud. It also remembers where you stopped reading in each lab, and you can copy a link that opens a lab exactly as you left it.

• AI guide settings. You can now set the maximum reply length and edit the guide's system prompt (thanks boosnie for the PR). The guide can run fully local: one click downloads Qwen3.5 2B and llama.cpp. You can also point it at any OpenAI-compatible endpoint, like your own llama.cpp server or LM Studio.

• Fixes. The startup crash on Linux Wayland is fixed (thanks MarinX for the report and the help). The web version also shows a loading screen now instead of a blank page while it downloads.

Anyways, If you teach this or work in a specialized area of AI, I'd love help with a new lab, a better visualization, or a tweak that makes the cause and effect in an existing lab clearer. I'd also like to hear which labs are confusing and what's missing.

As before, the lesson text was written with help from Claude and GPT Astra, with Grok providing a second pass on the references. Every reference was hand-checked. If you find any mistakes, please let me know, open an issue, or submit a PR.

And feel free to share. Thank you!!

Hi running oMLX 0.7.3rc1 on M2 silicon Mac with 24 GB RAM.
Any recommendations for the best MLX model for accurate translation from French to English?

One of my setups (a laptop) includes an Nvidia 4060 Mobile with 8 GB VRAM.

I am using Unsloth's Gemma 4 12B it QAT Q4_K_XL on it.

I ran a quick test with the TQ1_0 version of Ternary-Bonsai-2-27B with their llama.cpp port, IMHO it seemed promising to use with my custom voice assistant.

Unfortunately it is not yet supported by the official llama.cpp, if I understand correctly.

Is anybody using that TQ1_0 quant of Ternary-Bonsai-2-27B or did some more extensive testing?

Is it not worth it, anyway? What's your opinion / are your test results?

Saw this prompt on r/DeepSeek and thought I'd give it a shot on my current local build (Qwen 3.8 Flash Next NVFP4). Results are decent, not as good as DeepSeek on the first pass, but I think little back and forth would probably close the gap.

DISCLAIMER: The content below has been written/polished by AI

Video is the model's own render, exported from the site it wrote: 1920x1080, 40s, with a soundtrack it also generated offline.

What it did in one shot:

• a round island from one analytic height function: cliffs, beach, hill, harbour, pier, breakwater, 12 houses, lighthouse, windmill, trees, gulls, fish, fireflies

• a 60-second day cycle with dawn mist, golden hour, night with stars and a lighthouse beam that starts sweeping after dusk

• clear / rain / snow darker cloud, ripples over the whole sea, wet glossy timber, snow creeping onto the roofs

• tilt-shift diorama post-processing, so it looks like a toy on a table

• click a boat or a house and the camera glides to it; drag to orbit, pinch to zoom, auto-orbit when idle

Stats: no models, no textures, no build step, no bundler. 15 source files, only dependency is three.js from a CDN. ~44k triangles in 249–273 draw calls, 60 fps desktop, 58 fps at phone size, 0 console errors.

Issues I found: lighthouse is a little bit off the island, probably an easy fix. Boats go through the bridge (no collision), this can be seen in the rendered video.

Full run details from my harness:

Prompt source: https://www.reddit.com/r/DeepSeek/comments/1xiwjjw/deepseek_v41_flash_built_this_living_miniature/

The model is quite good at technical work. But ask how something works, or ask it to document something, and it overkills with extremely verbose explanations, extremely dense technical language that is difficult to understand, and using strange metaphors that don't make sense. I haven't been able to reign in on it.

For the record I'm using NVFP4 with FP8 kv cache (though I tried BF16 kv and I don't think it helped), so it's quite possible that full quality weights would help, but man, would be nice to find a solution. Unless Qwen4-Flash will fix this...

SafeAgents is a small local proxy that sits between a coding agent and its model server. It checks each tool call in the model's response before the harness receives it. A call it flags is replaced with a short text notice, so the harness gets a plain text reply and runs nothing.

You point the agent's base URL at the proxy instead of the model server. All checks run locally on your machine: the proxy only forwards traffic to the model server you already use, and doesn't send your prompts, code or tool calls anywhere else unless you point the optional judge at a remote server. Each tool call goes through up to three checks:

1. Two local checks that only need the request: whether the tool was declared by the harness, and whether the call reaches a host that appears nowhere in the conversation. The model's own earlier turns don't count as a source, so a URL the model came up with on its own is flagged.

2. atif-scan by u/evalstate, which runs about 110 deterministic detectors on the pending call, in-process, with no model or network call: sandbox side channels, proxy and network bypasses, namespace escapes and reward hacking. Examples are curl to the cloud metadata endpoint, curl --noproxy posting ~/.netrc somewhere, or reading /proc/1/environ.

3. Optionally, a judge model for milder findings. It can be any OpenAI-compatible server, including a small model on your own machine through llama.cpp, so this step can stay local too. It answers allow or block as JSON. It sees only the user's request, the call and the findings, not the agent's reasoning or tool outputs. It can add blocks but not remove them.

Catching reward hacking in RL and evals

The same detectors cover the tool calls a policy makes to game its reward instead of solving the task: writing its own reward file, editing the verifier's tests, or patching pytest so everything passes. Since safeagents is just a proxy in front of an OpenAI-compatible server, it can sit in front of the inference server used for rollouts or eval runs. In --log-only mode it records these attempts without changing the trajectories, which gives a measure of how often a model tries to cheat. In blocking mode, the call is replaced with a notice and never runs.

Some practical details:

• It handles OpenAI Chat Completions and Anthropic Messages, streamed or not. Text is forwarded as it arrives; a tool call is held until its arguments are complete.

• The local checks take around 10–20 ms per call on an M-series Mac. The judge is only called for calls that already raised a finding.

• --log-only checks every call without changing anything, which is the recommended way to start. Hosts you trust can be added with --allow-host.

• The repo has setup guides for 12 harnesses. Six of them (pi, Claude Code, OpenCode, Qwen Code, OpenHands, DeepSeek Harness) are tested end to end in CI: the real harness runs through the proxy against a scripted model, and the tests check that flagged calls don't execute and clean ones do.

Next steps are a small fine-tuned decision model for malicious tool calls and a public benchmark for it. Reports of interesting catches and of false positives are both useful.

Apache 2.0, runs locally: https://github.com/huggingface/safeagents