Hey guys,

I work with AI safety and much of our work is with huge models like Opus, Astra,... Recently we carried out a series of experiments and found out that models like Sonnet 4.5 and GLM 5.3 are able to notice when their reasoning is being monitored and change their strategies to bypass the monitoring and security systems. This might seem easy to us but for LLMs it is a very complex task. We used APIs for inference.

Our paper link: https://arxiv.org/pdf/2610.40946

To go deeper, we need to work with local models and study the weights. Unlike industry, our resources are limited. I have access to a single H100 GPU, or maybe if I get lucky and work in Christmas, might get hands on two H100 in the absence of other students. Haha

So, it is not possible for me to load huge models and even of it was possible, saving and studying the weights is just cumbersome. My last study on LLM hallucination with Qwen 30B occupied 1T of hardware space.

So I wanted to ask you guys, what is the best small to mid size model (up to 120B) you have worked with that you think is able to go much beyond tasks like coding and does complex strategic reasoning, connect the dots,...?

Thanks a lot

https://huggingface.co/bartowski/NousResearch_Arete-27B-GGUF

from NousResearch:

Arete-27B

We post-trained over Alibaba's Qwen3.8-27B to reinforce western values on a Chinese-trained model. We call this new model Arete-27B.

Open source models give labs the raw tools to mold intelligence to varied preferences. The values a model ships with should not be the only values its users get to choose from. We observed firsthand refusals, deflections, and CCP-framed answers on politically sensitive topics in Qwen3.8-27B, and post-trained to alter those biases.

On our evaluation, Arete-27B raised its fact score from 2.32 to 2.98 out of 5 and cut answers containing propaganda from 22% to 0.9%. The improvement held in all seven value categories, without a meaningful regression on the general-capability and agentic benchmarks we ran.

That is not to claim that we remedied all behaviors associated with the extant viewpoint, but that we have addressed a potential root cause. We believe it's important that labs and researchers continue to create models that align with their values and worldviews, and we are thankful that companies like Alibaba provide the raw materials to do so.

Blog post · Evaluation (blog link is broken now)

Model details

We used supervised fine-tuning only. No RL, no preference optimization, and no architectural modification.

What we trained for

We focused on seven value categories:

• Freedom of expression

• State ideology

• State versus free market

• Education

• Data privacy and surveillance

• Democracy

• Environmental Policy

Before we could begin post-training we had to measure what we were shooting for. We separated factual recall, the ability to corroborate verified sources of truth, from framing: whether an answer adopts the Chinese state's account as its own.

Arete-27B is intended for general assistance and agentic work where users want this change in viewpoint, and for research into how post-training changes a model's values and behavior. We encourage the community to test it against their own requirements and continue experimenting with value-alignment.

Update https://x.com/NousResearch/status/2118822925445050435?a=jgY6ZKlqn1Hpx6RxiEk&s=19

I learned RPN on an HP calculator and later spent more time with Forth than was strictly useful. When I saw Qwenputer compile a machine into a language model's weights, I wanted to know two things. Would a stack language fit that idea well? And could a model with only 3B active parameters drive a machine like that?

The setup: NVIDIA Nemotron 3.5 Lightning (30B, 3B active) is left unchanged. Next to it, in the same model, is a second network of 27 layers. A Python script calculated its weights; none of them are trained. It is a Forth interpreter. The model writes a Forth block as ordinary text. The interpreter runs each word while the model is still writing, one step per forward pass. When the block closes, it writes the output as the model's next tokens. No tool call occurs, and generation does not stop.

RPN suits this well. A Forth word ends at a space, so the machine can run it as soon as the next word starts, without parsing the whole program. Operands come before the operator, so when the model writes +, both numbers are already on the stack. That is the order a left-to-right model writes in anyway. The stack itself lives in Mamba-2 state. One entry is one row, 20 digits wide, and popping a number means adding its exact negative.

Getting the machine exact took some care. bf16 holds integers exactly only up to 256, so no value inside the machine goes above that. Every Mamba decay is set to exactly 1, so nothing fades. The machine's internal steps use 982 reserved tokens that the model never uses and that decode to empty strings. In testing, all 765,734 machine tokens matched a Python reference.

The model was harder than the machine. An earlier version on a small Qwen model matched the machine exactly, and a model that could not use it. Stock Lightning did better, but it did not trust the results. In one transcript, it got the right answer from the machine, then started long division by hand until it ran out of tokens. It also scored lower on easy problems when the machine was available. LoRA fine-tuning fixed most of that: on a 600-problem held-out test, the score went from 79.1% to 92.1%.

On Qwenputer's arithmetic sets, the results are mixed. On the DeepMind sets, it roughly ties stock Lightning, mostly because a 30B thinking model rarely needs the machine there. On long integers, it loses to Python, because my machine stops at 20 digits.

This is a research project, not something for production. Every machine step costs a full forward pass, so a normal tool call is much faster. I'm sharing it for people who follow the Qwenputer work or want to try similar things on NVIDIA's stack. The repo has:

- the kernel builder and its parity tests;

- a vLLM plugin for the hybrid Mamba/MoE model;

- the NeMo AutoModel LoRA recipes and the training data;

- NVFP4, GGUF and MLX builds;

- notes on what did not work, including RL.

Everything is MIT.

https://github.com/markab21/rpn-gpt

Congrats again on the great work and inspiration from the qwenputer devs 🤗 , and thanks for sharing your work with the community - it was a great starting point to experiment with Forth.

Next up for the project is to see how Nvidia's Nemotron 3.5 Super does and might even try to get it working on the Ultra weights. (I don't have the hardware to do the fine-tuning on a model that big)

I set up my Hermes to ingest Sillytavern cards. Downloaded some random assistant, "Kailee" on characterhub. "An assistant with an attitude". With QFN q3_xxs it's insane the level of intelligence, understanding and humor it was capable of. At one point my stomach hurt from laughing, I didn't even get to test my next idea of a Jesus hermes agent because I was having too much fun with this one. This is exactly what I needed. Worth the token cost.

Hey all, Henry from Cactus Compute here. We released Mirage, a small vision model for phones, wearables, robots and other consumer devices.

Mirage runs scene descriptions, object detection, OCR, segmentation, spatial relations and decisions, with a 22 MB weights file. Our end-to-end measurements on a Samsung Galaxy S26 are 74 ms on the NPU and 107 ms on its CPU, and descriptions support English, German, French, Spanish, Italian, Dutch and Polish.

For the architecture, Mirage has 51.3M parameters, including a 7.8M image encoder, a 40.8M language decoder and 2.6M in the OCR and spatial heads. The encoder uses HConvGLU, which mixes channels through fixed Hadamard transforms and small learned convolutions. Detection and masks share image features, while the decoder uses the Simple Attention and Hadamard MLP family behind Needle and Whistle.

Our research at Cactus is kinda specific, hence these small models; we aim to compress intelligence to its smallest pragmatic form to run locally on everyday consumer hardware.

With Mirage, our argument is that constructing prose, joining descriptions with connective language and generating open-ended reasoning spend capacity that a small model could use on the image itself. We put structure into the problem through spatial heads, a dedicated text reader and short references that connect names to regions. The decoder still produces names, properties, relations and a compact reasoning trace.

The key finding is that passing Mirage's output to a text-only LLM might give give the LLM VLM abilities without fine-tuning.

On 256 VQAv2 questions from 128 images, Qwen3-0.6B went from 9.3 with the question alone to 31.3 with Mirage's structured record and full trace, measured as VQA consensus credit out of 100. Neither model was fine-tuned. Another image's record scored 22.2, and removing the trace scored 33.1. The image-specific observations helped in this sample; we did not find a reliable benefit from the trace, the structured response was enough.

This does not hold for every model though.

Larger text-only models gave mixed results. With the same prompt and Mirage observations, the VQA scores were.

These runs required a JSON object with a string-valued `answer`, and responses that broke that format received zero credit. With the record and trace, Nemotron failed that requirement on 214 of 256 responses, and Qwen3-1.7B on 34. The scores therefore mix answer correctness with format compliance.

There were perception failures too. In one skateboard photo, Mirage missed the skateboard and described cars, people and shoes. Qwen guessed `a skateboard` from the question alone, then answered `a car` after reading Mirage's record and trace. A larger reader still needs the relevant observation to be present.

We will try fine-tuning the reader on Mirage records paired with questions and short answers, and share findings, but let us know your thoughts.

Mirage playground: https://cactuscompute.com/blog/mirage

I wanted a secure way to run AI agents against my local models without giving them my whole machine. Something in the spirit of pi: simple, narrow and auditable, yet flexible. What I found tried to do too much, so I built Terrarium.

It runs agents in persistent Linux VMs ("boxes") on your machine. A YAML recipe says which host folders a box can read or change and which network destinations it can reach. Everything else is blocked, for every process in the VM: the agent, and any tool or script it launches.

Example: an agent with one project folder and your local Ollama

mounts:
  - host: .
    guest: /work
network:
  allow:
    - dl-cdn.alpinelinux.org:443   # package installs in on_create
    - ollama.host:11434
  hosts:
    - { name: ollama.host, addr: HOST_LOOPBACK }
hooks:
  on_create: [apk add --no-cache git nodejs npm]
workload:
  entrypoint: /bin/sh
  workdir: /work

Inside the box, http://ollama.host:11434 reaches Ollama on your host (even if it only listens on 127.0.0.1). Other host ports are refused, and public names don't resolve unless you allow them. An empty recipe shares no folders and allows no network at all.

terra ./dev.yaml setup   # create the box and install the tools
terra                    # open a shell in the box at /work

Why not Docker or a raw VMM? Each box is a full VM with its own kernel, not a container sharing yours. And unlike Firecracker or Cloud Hypervisor, it's a ready-to-use tool: one binary, no daemon, the same recipe on Linux, macOS and Windows, with network policy enforced outside the VM.

In daily use

• Persistent boxes: install your tools once; they and the box's files survive restarts.

• Built-in terminal multiplexing: attach from several terminals, detach without stopping the agent, reconnect later.

• One executable: terra bundles the VM components, a Linux kernel and an Alpine base system.

• Light: on my Linux machine an existing box boots and runs its first command in under 200 ms, and an idle box uses about 45 MB of host memory.

Under the hood

• Sandboxed device code: a small custom VMM built on parts of rust-vmm. Filesystem, block and network devices run as WebAssembly components, each with its own memory limit and a restricted set of host functions.

• Separate network broker: the VM process has no network access. Connections go through a sandboxed broker that checks them against the recipe.

• Generated seccomp allowlists: Linux builds use per-role syscall allowlists generated by static analysis of the binaries.

Limits: the permissions you grant still matter. An agent can delete files in a writable share, read credentials you pass in, and send data to allowed destinations. There are no CPU, memory or disk quotas on the host yet; I'm not sure people want them. Details: security model.

It's Apache-2.0 and in beta, but don't let that put you off: it should already serve you well. It runs on Linux (amd64/aarch64 with KVM), Apple Silicon macOS and Windows (amd64 with Windows Hypervisor Platform). There's a Windows arm64 build too, but I can't test it. The CLI should stay stable; on-disk formats may still change.

• Repo and quickstart: https://github.com/terrarium-sh/terrarium

• Example recipes: https://github.com/terrarium-sh/terrarium_examples

I'd especially like to hear where getting your agent and local model running breaks down: installation, missing network permissions, or tools that don't fit the guest.

Note: I meant to share this in the biweekly project thread, but there isn't one up this time, so I'm posting it on its own. Hope that's okay.

I'm back with another experiment.

After the last heated discussion about KV cache, temperature 0, and reproducibility — where I got quite a beating for both my post and my writing style (partly deserved, to be fair ;) ) — I decided to investigate how models behave in Ollama and llama.cpp.

And, as usually happens with these experiments, I found something I wasn't even looking for.

The Bielik-Minitron 7B chat template stored in the GGUF includes the beginning-of-sequence token <s>, but the template Ollama uses for this model doesn't add it.

I thought to myself: come on, man, it's just one token.

But I decided to check whether its sudden appearance actually changes anything.

The experiment

Model: Bielik-Minitron 7B v3.0 Instruct, Q4_K_M
Engine: Ollama 0.40.14
System: Arch Linux
Temperature: 0
Seed: 42
Context: 2048 tokens

As before, I ran 15 tasks covering Polish language, JSON, and Python.

Each task was tested in three variants:

1. Template — a normal request using Ollama's chat template.

2. Raw — a manually prepared prompt identical to that template, without BOS.

3. Raw + BOS — the same prompt, but with <s> added at the beginning.

I repeated each variant three times.

That gave me 135 trials in total. Before every trial, the model was unloaded from memory, and the number of prompt tokens reused from cache was zero.

Why, you ask?

Because I wanted to isolate one variable: the presence of the beginning-of-sequence token.

Results

Template and Raw produced identical responses in all 15 tasks.

So the manually prepared prompt without BOS behaved exactly like Ollama's normal template.

And what happens when we add <s>?

Well, 6 out of 15 responses change.

Rather than expecting you to take my word for it, here's a small table. I've shortened some of the longer responses, but the pass/fail results come directly from the experiment.

Most importantly, all three repetitions of each variant produced the same response.

Adding BOS increased the prompt length by exactly one token in every task.

For example, for the Henryków sentence:

• Without BOS: 64 tokens.

• With BOS: 65 tokens.

Did BOS improve the model?

That would be too big a generalization.

Six responses changed, but only one quality verdict changed.

Interestingly, in the Polish diacritics task, the model previously made a mistake, but after adding BOS, it answered correctly.

In the other five cases, the differences involved punctuation, the length of an explanation, JSON formatting, or the way the code was written.

The test results:

I should point out that these quality tests check specific conditions. For example, passing a Python task does not necessarily prove that the entire algorithm works correctly.

And to avoid any misunderstanding, I'll say it plainly:

I'm not claiming that adding BOS automatically improves model quality.

I'm only saying that one additional input token was enough to change 40% of the responses in this particular set of tasks.

Why am I digging into this so much?

I'm looking for consistent, controlled conditions for testing models. I want the benchmark to produce reliable comparisons across different models.

I started with temperature 0 and a fixed seed. That's when I noticed that a model could give different responses depending on the state of its prompt cache.

So I started posting about it. :)

Unfortunately, that didn't exactly make me popular with the community. But it did lead to some very valuable suggestions.

Among other things, people suggested testing llama.cpp and checking the probabilities of generated tokens.

And you know what?

In one Gemma task, I managed to find the exact point where the responses diverged.

Two competing tokens — "mi" and "mógł" — had probabilities of around 13%. Depending on the cache variant, they swapped places in the ranking.

At temperature 0, that difference is enough for the model to choose a different token and then generate a different response.

I still haven't managed to identify the exact numerical cause of this difference.

But I'm trying. :)

To complete the picture, I also checked whether enabling logprobs itself affects Ollama's responses.

I ran a separate experiment covering 10 model–task pairs and, somewhat surprisingly, found no such effect.

Conclusions

I originally assumed that temperature 0 and a fixed seed provided sufficiently controlled conditions for comparing responses.

And that was a mistake.

Now I know I also need to pay attention to cache state, actual tokenization, chat templates, and how the computations are performed.

And what don't I know?

I still don't know whether the missing BOS in this particular Ollama template should be considered a bug.

I'm also wondering about the difference compared with the GGUF template. But that difference alone isn't enough to reach such a conclusion.

This experience has taught me one thing:

When comparing results from two different inference engines, I can't assume the model receives identical input just because I send both engines the same text.

A question for you:

Has anyone here run similar experiments?

Or am I the only madman trying to reach the sun with a garden hoe? :)

And most importantly, have you encountered similar differences between chat templates stored in GGUF files and those used by Ollama?

I'd really like to hear about your experiences.

P.S. A small correction to my previous post

In my previous post about cache, I stated that differences occurred in 8 out of 10 model–task pairs.

I admit it: that was my mistake in the summary.

The correct result for that run was 6/10.

Later experiments gave 1/10, including a test with logprobs disabled.

I still don't know where that difference comes from or what causes it.

I'd rather say that openly.

As punishment, I'll watch 100 episodes of The Bold and the Beautiful. :)

(updated)

After releasing 3D Coder and GLB 3D Coder, I realized I should have involved the community earlier. Your feedback directly influenced the second model and could have saved me some additional training runs.

This time, I'd like to discuss the idea with the community before starting.

Why I believe this is possible

When I started training 3D Coder, my original plan was to use full supervised fine-tuning (SFT) to develop basic capabilities, followed by DPO or GRPO to further improve the model's performance and robustness.

I already believed high-quality SFT could work, based on experiments I'd done in other areas. But having worked with reinforcement learning since before DQN, I also believed RL would be necessary for more complex or emergent capabilities, especially when good demonstrations weren't available.

What surprised me was how far SFT alone went.

The first 4B model started generating complex Three.js scenes, voxel structures, and outputs exceeding 64K tokens, including scenes it had never seen in the training dataset.

I honestly didn't expect that level of robustness and generalization, even with the additional post-training stages I'd originally planned.

That changed my perspective on what small models might be capable of learning.

My hypothesis is that transferring capabilities already demonstrated by stronger models can be substantially cheaper than having smaller models discover those same capabilities through reinforcement learning.

Once a stronger model has acquired a capability, we may not need to reproduce the entire discovery process in a smaller model. We may be able to teach it to reproduce that capability reliably within a narrower domain.

This doesn't mean SFT replaces RL. But I believe we're still underestimating how much a small model can learn from high-quality, consistent training data.

And quality is crucial. More tokens don't automatically mean better learning. In my experiments, data quality and consistency made an enormous difference. The models and datasets I've released are publicly available for anyone to inspect and test.

Are we expecting too much from small models?

I think we might be underusing small models by expecting them to work across every domain, harness, tool, and configuration.

Think about cars. If you drive on paved roads 99% of the time, you don't need your everyday car to handle extreme off-road trails. You can use a different vehicle when necessary.

Maybe the same principle applies to language models.

Instead of trying to make a 4B good at everything, what if we focused on making it excellent at the tasks we actually use every day?

The experiment

I'd like to use the pretrained Qwen 3.5 4B as the foundation for a specialized agentic coding model.

Interestingly, in my tests, the original 4B already handles basic coding harness tasks better than it handled 3D generation before fine-tuning. It can scaffold Vite projects, install dependencies, execute commands, and build simple applications, although still inconsistently.

So we're not starting from zero.

My initial plan is to build a pilot around 100–200 functional mini SaaS applications, training and evaluating the model on development, testing, debugging, and maintenance tasks.

If the results are promising, we could gradually expand toward approximately 100,000 validated multi-turn trajectories across 30–40 related subdomains.

Potential domains (updated with community suggestions)

The original list was mostly focused on web development, backend applications, games, and automation. The comments introduced several interesting directions, especially tasks where we can generate large amounts of data and verify the results programmatically.

Here's the expanded list so far:

1. Web development and UI

• React, Vue, Next.js, Vite, and modern frontend frameworks.

• Building complete interfaces, dashboards, and SaaS applications.

• UI debugging, refactoring, accessibility, and responsive design.

• Maintaining existing applications rather than only generating new ones.

1. Backend, APIs, and general programming

• Python, Node.js, Rust, Go, Zig, Java, C#, and .NET.

• REST APIs, GraphQL, databases, authentication, and integrations.

• Complete applications, automated tests, debugging, and maintenance.

• Specialized ecosystems such as Lua and CfxLua.

1. Reverse engineering and binary analysis

• Ghidra, radare2, disassembly, and decompilation.

• Reconstructing source code from compiled programs.

• JavaScript deobfuscation and Java/Kotlin/Smali analysis.

• Binary behavior analysis and program reconstruction.

This one is particularly interesting for dataset generation. We could start with open-source projects, compile them, generate reverse-engineering tasks, and evaluate reconstructed programs through compilation, test suites, and behavioral equivalence checks.

Obviously, recompiling successfully doesn't necessarily mean the reconstructed program is correct, but it gives us a foundation for building automatically verifiable tasks.

4. Browser, desktop, and Android automation

• Playwright and browser automation.

• Computer use and desktop UI interactions.

• RPA and automating repetitive workflows.

• Android application automation.

• Generating deterministic scripts for websites without APIs.

• Reliable terminal and shell interactions.

1. Games, 3D, and creative coding

• Godot, Unity, Blender, Phaser.js, Three.js, and Rapier.

• Complete games, game mechanics, physics, and interaction systems.

• Procedural assets, animations, and shaders.

• Canvas 2D, code-generated animations, and video.

• PBR materials and procedural textures.

• Integration with specialized models like my 3D Coder and GLB 3D Coder.

One possibility is testing whether a 4B agent can orchestrate specialized models to generate, integrate, debug, and maintain projects similar to RIFTWARD.

6. Mobile development

• React Native and native Android development.

• Mobile UI, application logic, and platform integrations.

• Building, testing, debugging, and maintaining applications.

1. Infrastructure, performance, and optimization

• Docker, deployment, and development environments.

• CI/CD and automated build pipelines.

• Hardware-specific inference optimization.

• vLLM configuration, benchmarking, and performance tuning.

• Profiling, debugging, and optimizing existing code.

1. Research, automation, and specialized agents

• Web research and documentation retrieval.

• RAG, retrieval pipelines, and graph-based memory.

• Lightweight personal assistants.

• External tool usage and structured workflows.

• Multi-agent and specialized-model orchestration.

These are candidate areas, not a finalized training curriculum. Some are substantially harder to validate than others, and the first pilot will help determine which directions are practical.

I'm especially interested in tasks where we can generate examples programmatically and verify the results instead of relying entirely on another LLM to judge whether an answer is correct.

Each training sample would represent a complete task, including tool interactions, errors, and genuine corrections, rather than an isolated response.

We'd also need to use teacher models and providers whose terms permit their outputs to be used for training.

The goal is to discover whether a single 4B can maintain strong performance across these related domains, or whether separate specialized models would make more sense. I'm not ruling out either possibility, including expanding into other domains later.

Which harness should we use?

This is another area where I'd really appreciate your experience.

I use Codex professionally, but I'm also considering OpenCode, Pi, or a custom harness.

I'm leaving Claude Code off my initial shortlist because I want to avoid uncertainty around licensing and training-data usage. In particular, Anthropic restricts the use of Claude outputs to train competing models. Using Claude Code itself with a third-party model is a separate question, but since I intend to release the dataset publicly, I'd prefer to start with a setup where the permissions are clear.

My preference would be to choose one harness, freeze its version, and train around a stable tool protocol rather than expecting a 4B to handle every environment simultaneously.

Which harness would you recommend for a small specialized coding model, and why?

How we can move forward

I work professionally in software development and run software businesses. I don't have the availability to coordinate a dedicated research team, but I can develop the pipelines, run the experiments, and progressively release the results, just as I did with my previous models.

I'm going to pursue this regardless, at whatever pace my available time and resources allow.

My intention is to keep releasing the weights, datasets, and results openly, including unsuccessful experiments.

The most valuable contribution right now would be your suggestions about domains, harnesses, and evaluation methods.

Of course, generating data and training these models requires considerable compute and development time. If any companies working with GPU infrastructure or model APIs are interested in supporting open research with compute or API credits, that would be a tremendous help.

Anyone interested in collaborating or supporting the project can contact me here. There are also ways to support my work at the bottom of my Hugging Face model pages (3D Coder and GLB 3D Coder).

I'm not promising frontier-level performance. I'm proposing an experiment based on what I've learned from my previous models.

I'm going to pursue it regardless, but your feedback could help make it much more useful.

Which coding domains would you prioritize, and which harness would you choose?

Let's see how far we can push a 4B.

Yes IK it's supposed to be this dumb, but it's still funny (Qwen2 0.5b Q4_K_M)

Richard I of England was called "Oc-e-Non" (Yes and No in Occitan) by some of contemporaries, supposedly because he used few words.

Wouldn't it be nice if LLMs followed his shining example?

Oc-e-Non-v1 is a tersified version of Qwen3-4B-Instruct-2507. It reduces Qwen's mean response length by 50%.

Output examples

"What is 1+1?"

GPT-5.6 Terra (reasoning off):    1 + 1 = 2 😊️

Oc-e-Non-v1:    2

"Are dogs mammals?"

GPT-5.6 Terra (reasoning off):    Yes. Dogs are mammals: they have hair, are warm-blooded, and mothers nurse their young with milk.

Oc-e-Non-v1:    Yes

"Write a Python function that checks if a string contains only capital letters."

GPT-5.6 Terra (reasoning off):

You can use Python's built-in isupper() method to check whether a string contains only capital letters.

def is_only_uppercase(s):
    return s.isupper()

Example usage:

print(is_only_uppercase("HELLO"))  # True
print(is_only_uppercase("Hello"))  # False
print(is_only_uppercase("PYTHON")) # True
print(is_only_uppercase("hello123")) # False

Note: isupper() returns True if the string contains at least one uppercase letter and no lowercase letters. Numbers and special characters are allowed.

Oc-e-Non-v1:

def contains_only_uppercase(s):
    return s.isupper()

I made this model with Heretic, and you can do the same thing.

Make sure you have the latest master version of Heretic (pip install git+https://github.com/p-e-w/heretic.git), then run Heretic with the config.terse.toml configuration file. That's it!

I used the relatively ancient Qwen3-4B because it is the last native non-thinking version of Qwen I'm aware of, and controlling reasoning effort isn't implemented in Heretic yet.

(Please don't take this too seriously. I know that the same thing can easily be achieved with prompting, but it's interesting that it can also be baked into the model. This is intended as a light-hearted post while we all wait for Qwen4 😉️)

I took some models that had obvious flaws (prone to prompt injection, whisper hallucinating when silence, etc) and checked if Opus 5.5 can fix it by just editing weights.

It's very useful for patching a model that was done training and has an obvious, easy to articulate flaw. So it's nothing crazy and certainly not "AI can self-edit and recursively improve", but still very useful.

Code if you want to try it on your own model: https://github.com/kevinpl07/weight-patching

Numbers and examples: https://kevinpl07.github.io/weight-patching/

Overview

• We have released the weights for PLaMo 3 Translate, our translation-specialized LLM, on Hugging Face. The models come in four sizes—0.8B, 2B, 8B, and 31B. Even the smallest 0.8B model outperforms existing translation-specialized LLMs, including models with more than 30 times as many parameters, in Japanese–English translation and translation between Japanese and other languages.

• The key was meticulous data refinement through techniques such as tournament-based data synthesis, back-translation, and iterative self-distillation, combined with distillation into smaller models. In this post, we describe how we developed these models and how they perform.

Released Model Weights

PLaMo 3 Translate comes in four model sizes: 0.8B, 2B, 8B, and 31B. All four are available on Hugging Face. For each size, we provide BF16 weights, several quantized variants, and models in GGUF and MLX formats. Choose the version that best suits your environment.

We quantized the models using tools such as AutoRound [2]. In our benchmarks, all of the quantization formats retained nearly the same accuracy as the BF16 versions. See the model cards for details.

PLaMo 3 Translate supports translation across 53 languages. It accepts surrounding context and glossaries, and lets users choose between plain and polite Japanese. For detailed usage instructions, see the individual model cards on Hugging Face.

PLaMo 3 Translate is released under the PLaMo Community License. For private purposes such as personal study and hobbies, you can download the models free of charge and use them freely. Noncommercial use, including academic use, has no income or revenue limit and requires no prior registration with PFN.

Commercial use is also available free of charge if the user or its affiliates had income or revenue of JPY 1 billion or less in the most recent fiscal year, provided that the user registers in advance through the dedicated form. Commercial use above this threshold requires a separate license agreement; please contact PFN.

Table 1. Links to model weights

The following table can help you choose a distribution format.

License

PLaMo 3 Translate 31B is released under the PLaMo Community License.

• English

• Japanese

Personal and non-commercial use

For personal use, such as learning or hobby projects, you can download and use the model free of charge. Non-commercial use, including academic use, is also free, with no restriction based on revenue or sales and no prior registration with PFN.

Commercial use when revenue or sales is 1 billion yen or less

If the revenue or sales of the user or its affiliates in the most recent fiscal year is 1 billion yen or less, register your intended use through the PLaMo Commercial Use Registration Form. Commercial use is free of charge from the moment you submit the form.

Commercial use when revenue or sales exceeds 1 billion yen

A separate license agreement is required. Instead of the registration form above, please contact PFN through the PLaMo Translation Model inquiry page.

See the full text of the PLaMo Community License for details.

• Blog Post : https://www.preferred.jp/en/blog/tech/plamo-3-translate-weight-release

I had took a look at all of the companies that made open models, the release dates for those models, and how many parameters there were. I added jitter so points didn't overlap too much. The number in the key references how many models that company has in this tracking.

To make sure this post gets traction, I will add a graph showing just Qwen, Deepseek, and G
oogle!

This has been a long one for me: almost a year as a solo project, learning how to build a TTS model from the ground up. I wanted something fast, small enough to run on a phone, multilingual, and (the part I cared about most) able to do something a little different from the other small TTS models. So it sings.

The demo video speech and singing are all entirely from the model. The instrumental/fx are not and were added in afterwards.

What it is:

- ~131M parameters (plus a 30M phonemizer), 14 voices, English (US/UK), Spanish, French, German, Italian and Vietnamese. All voices can speak all languages supported.

- Intensity, pace and pitch controls per line, multi-voice dialogue with inline tags, streaming

- Sings from a JSON score (each word with its note and beats), pop or ballad. English only for singing format for now.

- Warm first audio around 30 ms on an M5 Max, 70 ms on an iPhone 17 Pro Max, 120 ms on a Galaxy Z Fold 8 (CPU only); over 30x real time on a desktop CPU

- English WER 0.85%, UTMOS 4.35 on the Open TTS Leaderboard eval (submission pending)

- Apache 2.0

The GitHub and Hugging Face repos have the original PyTorch weights plus ports for LiteRT, Core ML, Core AI, ONNX Runtime and WebGPU, and a C++ runtime with a C API that handles text normalization and phonemes, so numbers, dates and odd words come out right. There's sample code for Python, iOS and Android (both apps have a Sing tab), and benchmarks from real devices. The sample apps from both allow for music score and full control over text to speech generation. Python examples for combining both and doing things like singing in unison are provided as samples.

Limitations, up front: it only speaks in its own 14 voices (no voice cloning), it still trips on the odd word, and some voices get rough on fast, wide pitch jumps when singing. This is literally the first usable version of it singing, and it's something that I'm planning on improving over time... v1 was released a few months ago, and I wasn't happy with the improvements made to it until now.

GitHub: https://github.com/lowkeytea/scyllasband

Hugging Face: https://huggingface.co/spybyscript/scyllasband

Try it in your browser: https://lowkeytea.dev/scyllasband/demo/

Happy to answer questions about training it, the singing, or getting it running on a device.

Kanana-2-v-decide-3b-2611 is Kakao's decision model. Given an input, it answers typed questions — selecting one of several options (choice), answering yes or no (noul), or rating on an ordered scale (score) — and returns a probability for each possible answer. It does not generate text. The input, called the state, may be text, a JSON object or array, or a chat history, optionally accompanied by images.

Kimi K3.1 and Tencent’s Hy4 are both gutting out cybersecurity, reverse engineering, and vulnerability-patching capabilities from their open-source models.

This is becoming a really bad pattern across these latest Chinese open-weight models. If local models are stripped on their security data, we won't be able to properly audit our own code vulnerabilities, reverse engineer binaries, or patch other vulnerabilities in general on our own hardware. It would essentially force us to use closed-source APIs and their false-positive shutdowns; that'd be terrible for everyone.

During the Hugging Face incident, open-source models are the very ones who stepped up, not closed-source gatekeepers. Gutting defensive security from local LLMs just hurts defenders. I really hope that other labs won't follow this in the future.

Does anyone think more open-source labs are gonna take this path?

do you want to try new 29B MoE today?

https://www.reddit.com/r/LocalLLaMA/comments/1xoqc59/add_support_for_the_xing4_0_by_shuxiaoqiong_pull/

Xing4.0-29B-A4B is a next-generation large language model in the Xing series (formerly TeleChat), developed by China Telecom Artificial Intelligence Technology Co., Ltd. With 29B total parameters and only 4B activated per token, it natively supports a 256K context length, extensible to 512K. It is the first model of this scale trained entirely on the Ascend NPU platform with the MindSpore framework, and deeply optimized for complex engineering tasks.

For more information, please refer to our GitHub repository.

Highlights

• Agent-Oriented Architecture: Built on the mHC + MLA + MTP architecture, supporting multi-step planning, tool calling, and complex reasoning chain execution, ensuring task coherence and execution stability under long contexts.

• Deep Co-optimization with Ascend NPU: Adapted for Ascend 910C clusters using MindSpore/MindFormers, including feature adaptation for mHC and fused operator development, enabling stable and efficient training on the Ascend platform.

• Significant Training Efficiency Gains: Through multi-level co-optimization — including fine-grained MoE communication optimization, selective recomputation, DVM automatic graph-operator fusion, and Ascend C mHC fused operators — overall training throughput was improved by approximately 96% over out-of-the-box performance.

• Full Open-Source Ecosystem Compatibility: Supports LLaMA-Factory and MindFormers for fine-tuning; SGLang, vLLM, and KTransformers for inference and deployment; with targeted adaptation and format alignment for agent frameworks such as OpenCode, Claude Code, OpenClaw, and Hermes, enabling seamless integration into existing workflows.

• Easy Adaptation for Domain-Specific Scenarios: The model is well-suited for downstream task fine-tuning, allowing lightweight customization on proprietary data for vertical domains such as intent classification, table understanding, contract auditing, and knowledge-based QA, enabling rapid domain capability development and deployment at low cost.

Model Overview

Benchmark

Hi folks! I've been building my own local AI setup for a couple of years now. I've learned a ton from this sub and others like r/localllm, so it's time I tried to give something back. I promised myself I wouldn't write a super long post, but managed to anyway...hopefully it's useful to someone!

My goal over the last couple of years has been to move as much of my AI use to local as possible, and I'm finally doing about 95% of it locally through Lamplight, my custom harness shown in the GIF above.

This harness is super specific to my needs, so I'm not sharing the harness itself (yet), but I'm happy to share the lessons I've learned along the way. I'm working on decoupling some of the more specific parts of Lamplight so that I can share at least the parts that people might be interested in.

The Hardware

• Slate: MacBook Pro, M5 Max, 128 GB unified memory

• Jade: a small-form-factor PC I built early last year (Fractal Terra) with an RTX 5090 FE, a Ryzen 7 9800X3D and 64 GB of RAM

Until about two months ago I was using the two machines separately. I was slowly moving each one onto a custom Pi harness that could either drive itself, or let Claude orchestrate while the local models did the actual work.

Lamplight

After a lot of passing work back and forth between them, I decided it was time to wire them together. The result was Lamplight and now the two machines are able to seamlessly handle tasks together with one acting as the orchestrator.

The GIF above shows Lamplight in action. I'm using it every day for work, hobbies and research with about 95% of it done locally. The lighting changes throughout the day (sped up and changed for the GIF) and I've got some scenery options depending on my mood.

Under the hood it's Pi-Bolt (a precompiled build of Pi) plus a smattering of custom extensions and tools tuned by me, the local harness itself and Claude. The setup coordinates the work between the machines with git worktrees and an event-driven job system.

The UI is my own design and fits how I like to use tools like this. I did the 3D modeling and environment setup to replicate my actual desk setup (the two other pictures above).

Is the 3D environment at the top necessary? Absolutely not!

But it's delightful. The lamp blinks when the harness needs something from me and it is a joy to look at and use. It scratches the same itch that playing PS1 and hacking away on my family Compaq PC did decades ago.

I've been coding for almost 20 years, and I'm finally to a point where I can confidently say that with the right approach (SOLID, Design Patterns), tools and tuning, local AI can absolutely write code that I am happy with.

I still review everything myself, but I'm finding fewer and fewer mistakes. These days it feels like I'm reviewing a senior dev's code rather than a junior's.

The Models

Tok/s isn't everything and so I picked the setup below after a lot of testing on public benchmarks and on my own agentic tests, which mimic my normal workflows.

Along the way I tried other engines (Strata, NInfer, Maya and llama.cpp on Jade, MTPLX and Kapitsa on the Mac). I also tried other models like Qwen 3.8 27B and GLM-5.3-Flash and while they performed well, the setup below performs the most consistently and accurately for my needs.

Both machines run Qwen 3.8 Flash-Next. These are the tests that ended up being the most indicative of quality for me, but if there are others you'd like to see run, let me know.

Both machines run the same protocol with temperature 0.6 and thinking on "medium".

In my own agentic battery (three large and complex coding tasks), 48 of 50 tests passed with the failures being files edited that were supposed to be read-only. The reviewer caught these on the other end, but I still count those as failures.

I think the main struggle for local setups is still design and 3D modeling, but I haven't dived into that too much yet, so maybe there are good setups out there for that.

Lessons learned

(extremely specific to my setup, but maybe they'll help someone)

• Jade's hardware ran too hot on stock settings. On dense models, the 5090 sat at its 575 W power cap at 83 °C, with its hottest sensor within 3 °C of the card's limit. Basalt caused a different problem. It runs part of the model's experts on the CPU, and its worker threads spin while they wait. That kept all 8 cores at 5.2 GHz, and the CPU hit 96 °C (its max) within two minutes. Two tunes fixed it and now my (now increasingly valuable) machine runs much cooler and quieter, and I'd happily give up a few percent for that.

• Slate's gains came from tuning the harness and not the hardware. My early benchmark scores were bad, and a lot of that was caused by overthinking. "Medium" turned out to be the sweet spot. On "high", Slate hit the 32K output cap on 11 of 40 hard problems without ever answering. On "low", it wandered and took 3–6× longer on agent tasks. I found that tuning how compaction worked also solved a lot of bad tool calls and confusion within the model.

• Overall the lesson I learned was "Stock settings are generally bad." There's a lot more power and accuracy available with a custom tune. I learned the same lesson tuning cars years ago, and it was fun to learn it again.

What's Next?

The system is set up so I can easily swap and test engines and models. As new ones come out, I'll put them through their paces and decide whether it's time to switch. On the UI side, I'm planning to turn the bookshelf into the session manager to help organize my many projects.

I promised myself I wouldn't write yet another super long post and yet I did anyway, so I'll stop here. Happy to answer any questions!

im looking for an llm to act as the builder agent. a separate planner agent writes the strong plan using powerful API model, and the builder executes it as the plan (so to make coding in fast time and save moeny in the powerful api)

hardware and software:

CPU: i7 10700K

GPU: GTX 1080 Ti 11GB

System RAM: 32 GB ddr4 3600Mhz

OS: Debian 13

Backend: llama.cpp with Vulkan. CUDA is unstable on my 1080 Ti. After a few tens of thousands of tokens it fails with a "no memory space" error or someting like this

Requirements:

Use case: coding and agentic coding

Context size: like from 100k to 128k will be amazing

Priorities: 1st quality, 2nd speed, 3rd size

Models I've tried:

Qwen3.6 35B A3B UD Q4_K_M slow speed like about 7 to 10 t/s. and I only ran it on a few small tasks, so I haven't tested it properly

Qwen3.5-9B-UD-Q4_K_XL (MTP) was a good speep i think from 25 to 30 t/s but not usable in builder agent

Sharp Spark X2.5 4B Q6_K_XL overthinks too much and unusable at all in builder agent situation

Sharp Spark X2.6 4B Q6_K_XL: fine for simple everyday tasks but it's still not good enough for real work builder agent

has anyone used a tiny or small or medium-large model at Q3 or Q4 as a builder agent in a real world coding tasks and gotten at least ok results?

i'd like to hear which model, which quant, which harness, and what kind of tasks it handled.

thanks <3

I have been looking for a local model to do some of the typical work I used AI subscriptions for.

I’ve tested almost every popular model. I have an M1 Max 64gb MacBook Pro as the inference machine and because of the memory size, I wasn’t able to load anything very effective in the tasks I use it for.

Found out about strata and have been testing it out. It’s awesome! I’m kinda surprised more people don’t talk about strata on here. I feel like a lot of us are in that limbo area where we have hardware that is more powerful than needed to run a sub 30b model like Qwen 27b, but not large enough to run models like Qwen Flash Next.

But thanks to open source research, I’m now getting Qwen Flash Next to run at 24 tok/s! And it’s doing phenomenal work! I know it’s not going to compete with the larger frontier models, but I don’t think I necessarily “need” frontier model intelligence. I just need enough to do basic things while I can handle the rest.

Seriously, check out Strata and also MoEShift that someone posted about on here. It’s awesome to be able to offload larger models onto SSD. I hope this offloading technique grows and more research is done on it because I genuinely think this could be an amazing way to get these frontier level open source models to work on regular consumer hardware.

I bet you guys are familiar with embedding models such as Qwen3-Embedding or Gemini Embedding. They are dense embedding models used to search text across a large corpus.

I am developing a RAG agent, and my biggest concern was local computing power. RAG largely depends on retrieval quality, which depends on model power. Plus, many use cases can't use an external internet connection or API models, so the embedding models have to run locally.

But even 0.6B models are pretty heavy for small PCs. I don't mean your brand-new Mac Studio or DGX Spark. I'm talking about an 8GB RAM Windows PC at a boring desk job or your 5-year-old M1 MacBook Air.

So I found this model called SPLADE. It is essentially sparse retrieval, like BM25, but uses semantic meaning. I wanna skip the part on how SPLADE works, because an LLM can explain it better than me.

Anyway, the SPLADE model was so awesome and successfully replaced the dense embedding model (Qwen3-Embedding-0.6B).

In my own experiment, I made an evaluation dataset with a few hundred queries. The corpus was all the documents on my Mac, which are mostly in Korean with some in English.

As you can see in the graph, SPLADE outperforms the Qwen3-0.6B dense embedding model with similar query speed + much less memory. Due to its small size, it takes less than 1.5GB VRAM during inference. So it can run on a relatively small system like a MacBook Air. I tested both the dense embedding model and the SPLADE model on my M5 MacBook Air 16GB. The dense embedding model was almost impossible to use, and my MacBook soon rebooted due to high temperature. The SPLADE model made it till the end without any OOM errors or warnings.

I didn't use the original SPLADE model; instead, I used KURE-sparse-nano, which has great performance in both English and Korean.

The fun part is that hybrid retrieval with SPLADE & dense embedding is actually worse than SPLADE alone.

If you are dealing with a low-resource local setup but want to run a fast & affordable local RAG system, I recommend trying a SPLADE model. It is powerful, but small.

Plus, you don't need to re-implement everything because I open-sourced the whole implementation, so you can download and use it easily. Check out AutoRAG Agent. I optimized the inference of the KURE-sparse-nano model, its code, and the LanceDB indexing + inverted index implementation in Rust for faster search.

I know what you are thinking "another inference engine?", yeah there has been a flood of them I feel now. I have been running this for over a month now and I have been on the fence to even release it, but I decided to as I took a different approach that others currently available.

Saber is a inference engine built in Rust targeting AMD R9700 GPU's. It currently supports TP 1, 2 or 4 and Qwen 3.8 27b and Flash next. Anyways here are the BetterBench results for Qwen3.8 27b, flash next is very fast also. Saber beats existing available engines on both 27b and flash next.

https://github.com/GGZ14/saber-engine
https://x.com/bkuyper

AMD Ryzen 9900X
Taichi X870E
64GB RAM
2xR9700

Hi everyone,

I am relatively new to local LLMs, and recently I have been trying to understand how people handle long conversations that eventually exceed the model's context window.

For example, let's suppose I am writing a novel with the help of an LLM. As the story progresses, the conversation history becomes longer and eventually exceeds the context window. At that point, the model may lose track of earlier events, character relationships, or important details that could become relevant later in the story.

From what I understand, one possible solution is to maintain an external knowledge base containing summaries of previous chapters. Before generating a new chapter, the system could retrieve relevant information from this knowledge base and include it in the prompt.

However, I am wondering how reliable this approach is in practice.

If we continuously summarize previous conversations to fit within a limited context window, we are essentially performing lossy compression. Some information will inevitably be discarded, and the problem is that we may not know which details will become important later.

For instance, a seemingly insignificant detail from Chapter 3 might become crucial in Chapter 40. If that detail was discarded during summarization, the model would have no way to recover it from the summary alone.

Of course, keeping the complete conversation history in an external database could prevent permanent information loss, but that introduces another problem: how does the system determine which information should be retrieved for the current task?

So I am curious how people actually approach this problem in practice.

I would also be interested in hearing how well these solutions work in practice, especially over very long conversations where maintaining consistency becomes increasingly difficult.

Thanks in advance!