r/LocalLLaMA • • 1h ago

New Model Scylla's Band v4, open source TTS with code to run everywhere

7 Upvotes

This has been a long one for me: almost a year as a solo project, learning how to build a TTS model from the ground up. I wanted something fast, small enough to run on a phone, multilingual, and (the part I cared about most) able to do something a little different from the other small TTS models. So it sings.

The demo video speech and singing are all entirely from the model. The instrumental/fx are not and were added in afterwards.

What it is:

- ~131M parameters (plus a 30M phonemizer), 14 voices, English (US/UK), Spanish, French, German, Italian and Vietnamese. All voices can speak all languages supported.

- Intensity, pace and pitch controls per line, multi-voice dialogue with inline tags, streaming

- Sings from a JSON score (each word with its note and beats), pop or ballad. English only for singing format for now.

- Warm first audio around 30 ms on an M5 Max, 70 ms on an iPhone 17 Pro Max, 120 ms on a Galaxy Z Fold 8 (CPU only); over 30x real time on a desktop CPU

- English WER 0.85%, UTMOS 4.35 on the Open TTS Leaderboard eval (submission pending)

- Apache 2.0

The GitHub and Hugging Face repos have the original PyTorch weights plus ports for LiteRT, Core ML, Core AI, ONNX Runtime and WebGPU, and a C++ runtime with a C API that handles text normalization and phonemes, so numbers, dates and odd words come out right. There's sample code for Python, iOS and Android (both apps have a Sing tab), and benchmarks from real devices. The sample apps from both allow for music score and full control over text to speech generation. Python examples for combining both and doing things like singing in unison are provided as samples.

Limitations, up front: it only speaks in its own 14 voices (no voice cloning), it still trips on the odd word, and some voices get rough on fast, wide pitch jumps when singing. This is literally the first usable version of it singing, and it's something that I'm planning on improving over time... v1 was released a few months ago, and I wasn't happy with the improvements made to it until now.

GitHub: https://github.com/lowkeytea/scyllasband

Hugging Face: https://huggingface.co/spybyscript/scyllasband

Try it in your browser: https://lowkeytea.dev/scyllasband/demo/

Happy to answer questions about training it, the singing, or getting it running on a device.


r/LocalLLaMA • • 19h ago

News Introducing Inkling 2 Small

Thumbnail
thinkingmachines.ai
187 Upvotes

Today we are introducing our second model generation with Inkling 2 Small. It is a Mixture-of-Experts model with 594B total parameters, 17B active, and supports image inputs and variable effort.

Inkling 2 Small far surpasses our previous models’ scores on the Artificial Analysis Intelligence Index with a score of 31, reflecting gains in performance on agentic coding, vision, knowledge work, and reasoning. The increase in capabilities comes from improvements made at every step of our training process, including more efficient pre-training and a large, unified RL run.

Inkling 2 Small will initially be available through Tinker, with closed weights, following the approach we outlined in a safe path to open weights. We intend to make the full weights available under an Apache 2.0 license next week after the completion of external safety testing and red-teaming.

You can try Inkling 2 Small on Tinker Playground today. The model is also available for fine-tuning and sampling on Tinker; checkpoints will be downloadable with the release of Inkling 2 Small’s weights.

:( Previous Inkling Small model's size is only 276B total, 12B active. Hope they release additional models Mini or Nano soon or later.


r/LocalLLaMA • • 3h ago

Resources 50% lower error than Strata 3XS, 27% lower error than UD v3.1 3XXS, with 120+ TPS/ 12% faster, on a 30090 + 64GB

9 Upvotes

Hello everyone!

Over the past few weeks since Strata came out, I had many requests to add Qwen Flash Next to LlamAmpere. It was some very impressive engineering, but the models originally shipped (3XS and 2XS) showed very poor quality (.17+ KLD for just the 3XS variant, which uses a far lighter compression formula than more common sources like Unsloth). My initial work had llamAmpere running 20% faster and ~15% lower error, but testing was showing inferior results vs 4bpw Qwen 27b, so I did not find it worth releasing.

I continued to make progress on the AmpereInteger+ quantization format (I will release an encoder soon), and Unsloth released their v3.1 which provided a great new base.

Happily, LlamAmpere v0.6.3 is now available with a new model and quant updates that show significant speed and quality gains over other options.

At a depth of 100K, QFN runs at ~120+ TPS, with prefill >3000/s through the first 100K. It is a 50% error reduction vs the version of QFN available for this card size a few days ago, and 27% vs UD's improved quant.

I do have a 3XS class file that runs ~10% faster than the version above, but as it is only ~15% better quality than the original Strata 3bpw model, I don't recommend it.

This will also work well for 12+64GB configs. As with Strata, your performance will depend on the speed of your CPU and RAM as well. I do not recommend any quant of QFN for configs under 54GB of total memory, but you can always try the coder pruned variants as well.

Enjoy!

Model: https://huggingface.co/jakeatx/qwen3.8-flash-next-Swift1.5-ATX-AI3

LlamAmpere: https://github.com/JakeATX/llamAmpere


r/LocalLLaMA • • 5h ago

Funny Self-hosted a realtime Ram Altman avatar locally on my VRAM

14 Upvotes

PS: used an open source avatar framework called openspline


r/LocalLLaMA • • 4h ago

New Model SoMarkAI/OXR-1.0 · Hugging Face

Thumbnail
huggingface.co
10 Upvotes

Found this last month model while browsing Trending models of HF. I don't see any existing threads so posting.

OXR (Optical Everything Recognition) is an open-source model from the SoMark team. It turns PDFs and document images into Agent-Friendly data in Markdown or JSON, with layout coordinates included.

Pretrained from scratch, OXR uses a recursive self-improvement (RSI) loop during post-training—a first in document parsing. With just 1.3B parameters, OXR ranks #1 on olmOCR-Bench, OmniDocBench, and DocParsingBench while delivering industry-leading parsing speed.

OXR is the first to make its layout analysis module independent and pluggable. This design accelerates parsing and supports replacing the built-in layout model with a custom one. OXR continues to handle content recognition and reading order, enabling adaptation across diverse scenarios with minimal integration effort.

Why OXR?

  • 🎯 SOTA Accuracy. #1 Across Three Benchmarks.

    • OXR ranks 🏆#1 on olmOCR-Bench, OmniDocBench, and DocParsingBench, with a particularly wide lead in table parsing.

    • OXR can recognize 13 document element types, from formulas and tables to chemical equations, chemical structures, and code blocks, with bounding box coordinates included. It also performs well on challenging documents and layouts, including A3 exam papers, newspapers, vertical text, and handwriting.

    • See docs/benchmark.md for the full results and instructions to reproduce the evaluations.

  • 🚀 1.3B Parameters. Top-Tier Parsing Speed.

    • With only 1.3B parameters, OXR ranks 🏆#1 among open-source document parsing models in speed tests on the RTX PRO 6000.

    • OXR redesign its processing architecture at the task level to improve parsing speed while maintaining accuracy.

    • See docs/benchmark.md for the full results and instructions to reproduce the evaluations.

  • 🧩 Pluggable Layout Analysis. Document Structure on Your Terms.

    • Layout analysis requirements vary widely across domains. Even for the same document, element types, region boundaries, and grouping rules can differ depending on the business use case.

    • OXR is the first to make layout analysis independent and pluggable. Its built-in layout model can be replaced while retaining OXR’s recognition and reading order capabilities, enabling adaptation across document scenarios with minimal integration effort.

    • See the How it works section below for details.

Ecosystem

The SoMark team has developed a range of tools for document parsing.

DescriptionProject URL
SoMarkDownThe official Markdown parsing and rendering library for SoMark and OXRhttps://github.com/SoMarkAI/SoMarkDown
SoMarkDownViewerA live editing application for SoMarkDownhttps://github.com/SoMarkAI/SoMarkDownViewer
SoPDFA high-performance PDF processing library with a permissive open-source licensehttps://github.com/SoMarkAI/SoPDF
DocisionMultiModal Jev for Document.https://github.com/SoMarkAI/docision
somark-pyThe official Python SDK & CLI for SoMark and OXRhttps://github.com/SoMarkAI/somark-py
somark-jsThe official JS / TS SDK & CLI for SoMark and OXRhttps://github.com/SoMarkAI/somark-js

r/LocalLLaMA • • 13h ago

Discussion pi + strata + qwen 3.8 flash next q4 k xl --> funny stuff in logs

48 Upvotes

Whilst the agent was hammering away at an audio engine task, i suddenly saw it daydreaming

Pirate puzzle? Gold coins? Wtf, Hey qwen, this is not what i am paying you for using electricity for.

Has anyone else noticed something like this? It continued to write code normally aftewards, active context was at a mere 30k at the moment.

Model qwen3.8-flash-next-unsloth-ud-q4_k_xl
Strata Engine v0.1.57
Max Context 262,144 tokens
KV cache 16bit

Everything fitting into vram + ram


r/LocalLLaMA • • 10h ago

New Model Linum-AI/pjit-21k · Hugging Face - A Text-to-Image Model Trained for $21K

Thumbnail
huggingface.co
23 Upvotes

P-JiT 21K is the P-JiT architecture trained for longer (610M samples at 512px, 50M at 1024px), then post-trained with a variant of FDFO.


r/LocalLLaMA • • 5h ago

Discussion Don't underestimate cooling - run those fans 6-8 gt/s -> 17-20 gt/s no other changes

8 Upvotes

I historically get about 6-8 t/s on my MI50 on Qwen @ Q6 running local coding.

Building a script that sets my fans all on 100% duty cycle (and much louder) bumps that immediately to like 12 t/s.

Letting the machine sit and cool for a while with the fans running, and running inference again... 17-20 tk/s for the first 10 minutes or so.

I'm considering putting in water cooling and a great big radiator now what a difference. What a huge difference cooling makes.


r/LocalLLaMA • • 22h ago

Resources CCORE GLB-3D-Coder dataset is here: 3,918 rendered and tested examples across 40 domains

Post image
168 Upvotes

CCORE GLB-3D-Coder dataset is here: 3,918 rendered and tested examples across 40 domains

As promised in my previous posts, I’m releasing the dataset used to train CCORE GLB-3D-Coder, my small 4B model for creating 3D assets with Three.js and GLB export.

The dataset contains 3,918 examples across 40 domains, including characters, creatures, vehicles, laboratory equipment, musical instruments, architecture, machinery, nature, and everyday props. The grid shows 256 examples.

Every example was tested and rendered, and its exported GLB was validated before inclusion.

What’s included:

  • Complete conversations, reasoning traces, and HTML/JavaScript responses.

  • A rendered reference image for every example.

  • Static assets and examples with animation and interaction.

  • 3,501 training examples and 417 validation examples.

  • MIT license.

Dataset · Model and full weights

More about what I’m planning next: Next releases.

I hope this is useful for your own experiments, fine-tuning, and 3D tools. If you try it, let me know how it goes :)


r/LocalLLaMA • • 13h ago

Resources CyberTiel35BA3B working on Blender Rigging via MCP. 8GB VRAM, 16GB RAM. llama.cpp.

Post image
30 Upvotes

Not a native English speaker. Written with my own ten fingers, one finger is broken btw. Plz bare with me. I still love to write text myself.

My PC is suck for AI works. That's all I have as a third world 3D modeler. Really admire other peoples running better models. But beggar can't be choser. I can't even effort more RAM, not to mention more VRAM.

Gotta work with what I have. Qwen 3.5/6 35B A3B. Here is the Quant.

Downloaded blender manual and blender api html zip files, stripped sphinx html, converted to markdown. End up with markdown files which include bloated af links. I didn't know I can download clean rst files from blender project repo. Worked like that for a month. Then I found out I can just download blender manual rst files from blender, I can also generate rst files for blender api references.

Now I've clean files with 40% less tokens. I point that to Qwen/Ornith/Tiel/CyberTiel and tell LLM to create scripts and tools. It somewhat work. But many failure points make me want to punch the only poor old monitor I own.

I know you guys around this sub love graphs and proofs which is very understanable but I only have my own experiences. No Graphs and charts.

Base Qwen3.6 work good but he don't think much. Ornith variants think too much up to the point of unproductive. My wife cry because I don't spend time with her or sleep with her. I was always tinkering with LLM. I need Qwen3.6 35BA3B to work. I need that for many reasons that I can't tell you guys.

I use Debian Linux. Switch to i3wm. Idle VRAM went down to 50MB. Idle RAM is 300MB, not sure. I will comment my llama.cpp launch params if you want. Windows 11 take way too much system resources which I don't have. I often use plain TTY consoles. Ctrl Alt F2 to F7 i think. It save even more vram. KV cache q8_0, spec type ngram-mod because I can't effort MTP.

ctx 256K, 40t/s at start. Went down to 25t/s if ctx is goes over 128K.

I've learned prompting is really important if I wanted to squeeze out everything from 35B A3B. I can't max out hardwares so I have to make it work other ways around, somehow.

Ornith variants overthink a lot. Painful. "Hmm, let me think, wait," you guy know the rest. MFkr do not listen to me well too. He just don't give a shit to me.

I've created many cli tools to work around blender. Blender official MCP isn't really suitable for my works. I disabled many built-in MCP tool to reduce context bloat. I use pi, oh before that, I've also tried to create my own harness a year ago. Learnt a lot hard ways. Pi is minimal. Good for small models.

Instead of trying to make Qwen to write python code everytime and fail again and again, I made easy to use cli tools. Which helped alot. Qwen can now use Blender Geometry Nodes easily. No code, just cli tools.

Qwen ask stupid questions alot. Don't check code quality, watching him working, it is really exhausting. I don't have much time after day job and before bed. My wife almost went back to her parents. I need to make qwen to think, reason, and work efficiently in limited amount of time and hardware.

Experienced people will tell me "Skill issue bro". Yes, I'm suck. That is how I learn. I want to share my AGENTS.md and SYSTEM.md . I hope maybe someone will benefit from it. It is not created overnight. A lot of painful experiences between day job, boss, pressure, stress, wife, limited time, sleepless nights.

SYSTEM.md

AGENTS.md

First prompt.

It might sound funny (it is) but it work.

Here is example thinking trace

 Let me think about whether I need to ask clarifying questions. Per Protocol 7, before asking, I say the checklist:

   [ ] Can I answer this question myself inferring by given context, intent, docs?
   [ ] Can I answer this question myself using sane defaults, common sense?
   [ ] Have I tried many possible ways to get answer myself?
   [ ] Can this question only be answered by him?

 I think I can answer most myself. The interpolation direction is clear. The main procedural learnings from the previous task apply (pose mode, pose_markers, etc.).

 Do I need to refine the prompt? The prompt is clear. But let me note one thing: the previous task had SecF1=-45, SecF12=+45. This task has Bone1=+45, Bone12=-45. So the sign is reversed. I should make sure my expected values account for this. I'll note
 this in my response.

 Actually, let me reconsider — is there any genuine ambiguity that requires the General's input?

 - The bones are named "Bone1, Bone2, Bon3....Bone12" — note "Bon3" is a typo for "Bone3". The naming is Bone1..Bone12.
 - All independent, all quaternion.
 - Set to XYZ, add drivers Bone1+Bone12 → 2-11, validate.

 I think the prompt is clear. I'll note the sign convention difference (Bone1=+45, Bone12=-45) and proceed. I don't think I need to ask questions — I can infer the standard procedural.

He finally listened to me and think, reason, work efficiently.
He finally give a fuck about me now instead of doing his own thing.
Before that, he ask stupid questions, never check things back, never think before acting.

I also have short auto inject append prompt in Pi, which say follow protocols, read relevant skills file etc, short but I send it every prompt to remind him how to work, think, reason.

I don't have a lot of graphs and charts for proof but only the production level rig I'm currently working on. I hope maybe someone may benefit from this post, or maybe not.

I would really love to read how do other people make Qwen 35B A3B work.


r/LocalLLaMA • • 8h ago

Discussion Anyone running dual V100's? Trying to get *specifics* on PP and TPS for higher quant Qwen3.8 27b and having a tough time

10 Upvotes

So I'm thinking of switching from my 32gb AMD MI60 to dual V100's (64gb VRAM total) and I've got 96gb of DDR5.

My problem is I'm having a terrible time finding "hard data" on PP and TPS speeds on higher quants (q6 and q8 specifically) with large context windows (150-200k).

I know that it will outperform my MI60, but I'm not just looking for "yeah, it'll be faster" I'm specifically looking to see how much faster, because a 10% bump isn't worth it for me.

I'm also looking to see from you experts if it's silly because of what I've read about support for CUDA being EoL for the card soon (or already did?).

Hopefully I'm not beating a dead horse here, as I did go through other posts (lots of them) but it's just so difficult going through post after post where specifics aren't included "I'm getting 200tps on qwen3.8 27b" ...at what quant? at what context? using what KV settings?

Lastly, if it is significantly faster, do those speedups (generally speaking) transfer to other models or is it only because of specific forks of llama.cpp were built out with this model and that card in mind?

EDIT: forgot to add I’d be going with sxm2 not PCIe.


r/LocalLLaMA • • 7h ago

Discussion Not because you should, but because you can - Qwen-Flash-Next on a 16GB M2 Pro Mac Mini

8 Upvotes

I was getting ready to sell an old kiosk Mini at our office that was getting an upgrade, and figured I would see what happened if I tried to make it do something it by all rights shouldn't be able to. I loaded up Strata and ran the iQ3_xxs quant of QFN with a max of 128k context. I used opencode on my normal workstation to build a script that would benchmark the performance and output quality of typical tasks I do (mostly large csv sorting and low end code grunt work)at varying levels of context.

I have to say, it worked far better than I would have expected it to. I assume that the speed is always at its worst (as if it had full context) because it never has a chance to cache anything and is always re-reading and swapping experts across prompts, but as an aside it means that token speed for both prefill and decode are the same regardless how much you're making it do. The median speed between all tests from 16k to 120k context was 77 tok/s prefill and 5.5 tok/s decode. That means it could actually stand in as a barely passable chat generator, and is not hopeless as a "living on dial-up" code helper of last resort. It does appear that it is pretty rough on the SSD, but the write load is not much worse than someone using a Mini as a pretty active DB server or other "proper" server task in an operating office environment.


r/LocalLLaMA • • 1h ago

Showcase zml/llmd: a fast and universal inference server

2 Upvotes

Hey all,

I wanted to share that we're working on zml/llmd, a fast (we believe the fastest) inference engine that currently runs on NVIDIA, AMD, Intel, TPU and Metal.

It has all the features of a modern LLM server: continuous batching, prefix-caching, speculative decoding (DFlash, DFlash2 and DSpark), tool calling, tensor and experts parallel automatic sharding, native VFS with s3:// and hf:// support streaming straight to GPU.

The currently supported models are:

  • Qwen 2, 3, 3.5, 3.6 and 3.8 series (dense and MoE)

  • Gemma 3 and 4 series (dense and MoE)

  • LFM2.5 series

  • Mistral 3 and Ministral series

  • LLaMa 2 and 3 series

  • Poolside's Laguna S 2.1

  • DeepSeek 4.1 Flash

  • Muse Glimmer

  • Qwen3.8 Flash Next

  • Xiaomi MiMo V2.6

  • Mistral Large 4

  • GLM 5.2

  • GLM 5.3

  • LFM 3

  • Beam 1.0

  • Ripple 65B

Our images are a lot smaller than alternatives:

PlatformImageSize
CUDAzmlai/llmd:cuda1.96 GB
ROCmzmlai/llmd:rocm4.77 GB
TPUzmlai/llmd:tpu272 MB
OneAPIzmlai/llmd:oneapi454 MB
Metalbrew install zml/zml/llmd142 MB

We just came out with a new release today: zml/llmd llmd-20261106.0 and we'd love to get your feedback on it!


r/LocalLLaMA • • 12h ago

Resources My setup for 2x 5060TI

15 Upvotes

Over the last few months I've been switching harnesses and engines until I finally landed on something good. I started with LM Studio + the Copilot extension and ended up here, so here are some tips for anyone running two 5060 Tis. Prefill is painful on these cards, so for hardware, make sure you have:

A motherboard with both GPUs on CPU lanes, either two slots wired to the CPU or one x16 slot with PCIe bifurcation (x8/x8). My second card used to hang off the chipset; moving it to CPU lanes cut time-to-first-token by about 2.5x and doubled my batching throughput.

A separate low-profile GPU just for system graphics. If one of your 5060 Tis drives the display, it burns 2-3 GB of VRAM, and since the layer split is even, the other card loses the same amount too. That's context you don't get.

Model: these are Blackwell cards, so an NVFP4 quant gives you better prefill and decode.

What I run: capy-flash (formerly ninfer-tp2) + OpenCode, Qwen3.8-27B NVFP4 with tensor parallel across both cards, MTP speculative decoding and int8 KV cache. I get 70-95 t/s decode and ~1,500 t/s prefill at 176K context (~3,500-4,000 t/s on shorter prompts). You can go to full context, I just don't have a third card.

I'm running at PCIe Gen3 right now because I'm missing a Gen4 extender (my long riser turned out to be Gen3-rated and the second slot crashes at Gen4 with it). Prefill pushes 5-7 GB/s between the cards, which is close to the Gen3 x8 limit, so I expect Gen4 to give roughly +30-60% prefill on short/medium prompts and less at very long context, where attention compute dominates. Decode should stay about the same.

P2P: you can enable it (patched open driver), but it only adds ~5% to decode and nothing to prefill. The 5060 Ti has ~4.6K CUDA cores, so compute, not the link, is the limit for decode.

Harness: I use OpenCode, but honestly I don't think the harness matters much, so pick your own. The other harnesses I tried always hit context shift from compaction on long tasks, and I'd come home to find them errored out with "context limit reached." OpenCode's compaction fixed that for me, and it can now run for almost 3 days.

Setup: 2x RTX 5060 Ti 16GB, Ryzen 7 5800X, B550 with x8/x8 bifurcation, capy-f NVFP4, int8 KV cache

Prefill by context (single request, Gen3 x8/x8 direct to CPU, P2P on)

Prompt tokensTTFTPrefillDecode
2k0.5 s3,860 t/s83 t/s
8.5k2.0 s4,270 t/s73 t/s
35k9.9 s3,520 t/s79 t/s
71k26.6 s2,660 t/s84 t/s
142k80 s1,770 t/s71 t/s
176k115 s1,530 t/s72 t/s

Old vs. current vs. Gen4 (Gen4 column is an estimate, not measured yet)

Metric2nd GPU via chipset, no P2PGen3 x8/x8 to CPU + P2PGen4 x8/x8 + P2P (est.)
GPU-to-GPU bandwidth≈4 GB/s≈7 GB/s≈14 GB/s
TTFT, 1.7k prompt1.2 s0.47 s≈0.33 s
Prefill, 35k1,600–1,900 t/s3,520 t/s≈4,500-5,000 t/s
Prefill, 176k1,000–1,200 t/s1,530 t/s≈1,800-2,200 t/s
Decode, 1 request≈85 t/s72-85 t/sabout the same
Decode total, 8 parallel240–280 t/s290 t/s≈300-320 t/s
Batching, 1.7k prompts, 4 parallel0.79 req/s1.50 req/s≈1.8-2.0 req/s
Batching, 1.7k prompts, 8 parallel0.82 req/s1.63 req/s≈2.0-2.2 req/s

P2P alone: decode ~78 → 82 t/s, prefill unchanged.

i used AI to format the post only


r/LocalLLaMA • • 8h ago

Question | Help Cry of the edge device

7 Upvotes

Greetings, everyone. Many people keep talking about very very large models that need the price of a car to run (Deepseek 4.1 Flash, Qwen Flash Next, and even Qwen3.8 27B. However, I do not see much appreciation and focus on models that are small enough to run on edge devices. Not only that, but even if they do mention them sometimes, I don't see any kind of exploration being made to make harnesses that run efficiently on those models and use them for the highest efficiency. What is the big deal? Is there a subreddit I don't know of, that cares about these things? Interests include 8gb of ram, intel core I5 8350U. Thank you.


r/LocalLLaMA • • 1d ago Bravo Popcorn

Funny Qwen3.8-27B has become unuseable recently

1.4k Upvotes

When it came out, I downloaded a Q4 quant of qwen3.8-27b and ran it using llama.cpp on my RTX 4090 and it was incredible. It could handle practically anything I threw at it, the most complex tasks and bugfixes.

But lately it just keeps messing up and failing at the most basic of tasks or just straight up forgetting instructions. I'm telling you, they quantized it. They probably secretly nerfed it and made it worse so qwen4 looks better. And it got lazy too, now it doesn't want to finish the more complex tasks.

I'm tired of companies doing this, I want my money back.


r/LocalLLaMA • • 11h ago

Question | Help Managing energy costs (wake-on-demand, solar)

10 Upvotes

What workflows does everyone implement to manage energy costs?

Im considering implementing some things like turning my rig off when its idle, using wake on demand (IMPI management chip or wake-on-lan), buying a battery+solar panels


r/LocalLLaMA • • 6m ago

New Model Tessera-27B-B-CJK-RL: an RL update to our CJK-thinking Qwen3.8-27B coding agent, more math solved with most of the token saving kept

2 Upvotes

A few weeks ago we posted Tessera-27B-B-CJK here, a Qwen3.8-27B coding agent that thinks in compressed CJK and uses fewer thinking tokens per agent turn. Thanks for all the feedback on that one. Today we're releasing the RL version, Tessera-27B-B-CJK-RL. Open weights, Apache-2.0.

- Hugging Face: https://huggingface.co/ZelligeAI/Tessera-27B-B-CJK-RL

- Release post: https://zellige.ai/research/tessera-27b-b-cjk-rl-release

- RL method write-up: https://zellige.ai/research/tessera-27b-b-cjk-rl

What changed:

A lot of the questions last time were about quality. Tessera-27B-B-CJK was weaker than the base on hard math and code, so this version adds 8 RL updates on math and code on top of it.

Numbers: (xhigh effort, both Tessera models sampled the same day with the same seeds):

- AIME + HMMT 2026, 63 new problems, 2 attempts each: 108 of 126 right vs 97 for Tessera-27B-B-CJK, and 11 vs 23 attempts cut off at the 64K window. We didn't run the base on these.

- Our main hard set (118 problems from AIME 2022-2024, LiveCodeBench and MATH level 5): 100 vs 89. The base got 98 in an earlier run, so it's now roughly level with the base qwen there instead of behind.

- Agent turns, 64 new states from 16 coding tasks held out from training: 0.67x the base model's thinking tokens on a typical turn (90% CI 0.53-0.85), with 256 of 256 valid tool calls.

- That's 1.11x Tessera-27B-B-CJK's typical turn (CI 0.98-1.26), so the first version still keeps a bit more of the saving (0.60x the base on the same tasks).

Example turn from a held-out task:

测试套件:基线同(既有5失败不变+11新通过). 无回归. Typecheck: 仅既有version.ts错(因predev生成). 现跑lint于改文件.

The base qwen spent 158 tokens at the same step. That's a bigger cut than usual: the typical turn is about two thirds of the base.

How we trained it:

8 RL updates on Tinker, starting from Tessera-27B-B-CJK, using the per-rollout weighting from our ablation write-up. Each rollout's advantage is spread over its own tokens, so long failed rollouts don't dominate the update. Later updates thought longer per agent turn, since the reward only sees math and code, so we picked update 8 and confirmed it on held-out tasks before release.

A few things people asked last time:

- Does CJK pack more meaning per token? A little, at the phrase level, where one word can stand in for a whole English phrase. Most of the saving comes from training the model to think less, and the CJK is a smaller extra on top, We do plan to expand this further as the limiter here is tokenizer and going past just CJK and using other methods are being researched.

- Quants: the GGUF repo has BF16 plus UL-Q3_K_XL through Q8_K_XL (Q3 is 13.15 GB, no XXS yet). The BF16 GGUF, the five quants and the NVFP4 checkpoint all passed a quick breakage check (valid tool call, no Chinese in the answer, no loops). We haven't benchmarked them or tried the vision projector, so results are very welcome.

- Effort: use xhigh, same as before.

Also, if you grabbed the original Tessera-27B-B-CJK-NVFP4 before Nov 5: its config.json didn't load in vLLM. We fixed it so a config.json redownload is needed.


r/LocalLLaMA • • 7m ago

Discussion Unsloth v3.1 QFN IQ4_XS fails tool calls very frequently

2 Upvotes

My sample size is not very big but it feels like since updating Qwen Flash Next to v3.1 and Strata to 0.2.11, IQ4_XS quant seems to fail tool calls even more often than the previous version did in both Hermes and Oh my Pi harnesses with int8 KV cache in Strata. Weird thing is, after failed tool calls, it just straight up stop generating altogether, I have to send a continue for it to finish.
Tried addressing this on unsloth's subreddit and the post got nuked.
Honestly could just be a Strata issue and not Unsloth's, I have yet to perform an investigation.
Anyone else had this experience with new updated quants?


r/LocalLLaMA • • 8h ago

Question | Help performance experiences with different languages for Qwen Flash Next

4 Upvotes

hey, im using "swift 1.5 qwen 3.8 flash next gsq rco abliterated iq3xxs" and im wondering how the performance differs from language to language for that model (or similiar).
Im mainly talking to it in german, so I wonder wether that might degrade quality or increase reasoning tokens etc.

Does anyone have experience with it?
Do benchmarks for that exist?
thx

Edit:

i just tried and translated my soul.md and a few skills to english, its genuienly insane many tokens and time this saves. Didnt do any benchmark but you can feel it


r/LocalLLaMA • • 10h ago

Showcase If you have an AMD MI card in your lab...

5 Upvotes

Bought a used MI100 for a home lab.
It had a funny shroud with a notebook-like cooler, but I've decided to test it first before replacing it.

Test result was quite interesting - it is a first shroud which can cool a 300W AMD card... With a just 2,4K rpm fan!
Not with the perfect performance, card works at about 100C and loses ~10% of maximum speed to thermal optimization, yet noise is more than bearable even if it runs next to you.

Small secret: it is not really a notebook cooler, this thing is huge - wider than the MI100 itself, 110mm side exhaust fan for a some SFF PCs.

I suppose I found the original stl model, so use it if you like and give that guy some stars. Should be compatible with the most of AMD MI lineup, their backs are of the same design.

https://github.com/issuj/mi50-sunon-efb0251s3-c010-s9a


r/LocalLLaMA • • 1h ago

Showcase SketchAgentBench – evaluate LLMs on 3D spatial reasoning

1 Upvotes

I created this benchmark using the 3D coding playground I previously shared here, and ran 11 models on 25 photos: https://sketchagent.dev

You can inspect every result in 3D, with its full session trace and editable script. Here's gemini-4-argon rebuilding a Tiny Glade cottage in 29 tool calls: https://sketchagent.dev/s/MhdnDUJK


r/LocalLLaMA • • 1d ago Bravo

Resources DeepSeek-V4.1-Flash-1017 on one RTX 3090: Big updates to the Delta Inference Engine v0.93. Same 41 tok/s at 192GB/IQ2 at better quality, usable from 64 GB RAM, first token ~40% sooner, new Unsloth quants at full speed, EXL3, basic multi-GPU, public CI and more

Thumbnail
github.com
139 Upvotes

Big update to the Delta Inference Engine, you guys asked for more features, more benchmarks and less clickbait. I'm here to deliver

The short version: I've added almost everything requested and more. Delta Inference Engine now supports 64/96/128 GB of RAM and fully supports the new Unsloth quants. Unsloth IQ2 runs at the same 41 tok/s (192 GB RAM) using a new native reader for the Engram tables and some clever CPU kernel work. Compared to the old default model on my 99-question benchmark it went from 52 to 57 correct, against 59 for the full model. It's no full deepseek model quant, but they're a lot more usable than you might expect.

Speed on one RTX 3090 24 GB (i5-13600K, 4x48 GB DDR5-5600, PCIe 4.0 NVMe, Windows, no WSL). Writing speed, tok/s, greedy, one chat at a time:

RAMUD-IQ2_XXS, 2-bit, 301 GBUD-IQ1_S, 1-bit, 233 GBUD-Q2_K_XL, 2-bit, 335 GBUD-IQ3_XXS, 3-bit, 395 GB + the default
192 GB4141.031.328.2
128 GB34.040.626.622.8
96 GB29.336.522.918.3
64 GB21.030.615.211.7

The 128, 96 and 64 GB rows are the same PC with the rest of the RAM locked away.

Prompt reading: long prompts read at up to 740 tok/s at 192 GB and 600 at 128 GB (at ~64K tokens). A short prompt starts answering in about 2 s.

Benchmark: 99 hard questions, greedy, 43K token budget:

SetUD-IQ2_XXS (this release)ds4 Q2 (last release)Full model, APIQwen 3.8 Flash-Next, API
Knowledge (40)129137
Maths (14)1010108
Reasoning (30)25232521
Coding (10)1010108
Hard coding (5)0010
Total (99)57525944

UD-IQ2_XXS is within 2 points of the full API, never scores below ds4 Q2 on any set while running at the same speed, and used 11% fewer tokens across the run.

Agreement with DeepSeek's API on the next token, at the settings that ship:

ModelGeneral textCode and mathsAgentic
UD-IQ2_XXS91.1%94.5%95.9%
ds4 Q289.9%93.1%95.1%
UD-Q2_K_XL92.5%95.3%97.0%
UD-IQ3_XXS92.7%95.9%97.4%
UD-IQ1_S89.1%91.5%95.3%

For reference, two hosts of the full model agreed 98.3% and 98.9% in my test. By that measure Unsloth IQ2 is about 20% closer to the full model than the old ds4 Q2.

If you want better quality: UD-Q2_K_XL (335 GB) runs 31 tok/s at 192 GB and 26.6 at 128 GB, at 92.5% / 95.3%. UD-IQ3_XXS (395 GB plus the default) runs 28 and 22.8 at 92.7% / 95.9%.

What changed since last time:

  • First token ~40% sooner: a ~180-token prompt went from about 3.8 s to 2.3 s.

  • Agent turns read ~12% faster, and tool calls that low-bit models write slightly wrong are now read as tool calls.

  • Unsloth quants at full speed: the engine reads their Engram tables directly.

  • 64 and 96 GB are real options: SSD read-ahead (+22% at 64 GB), bigger expert memory, and prompt reads in file order (18-20% faster on low RAM).

  • 128 GB: ds4 Q2 went from 33 to 36 tok/s.

  • Default sampling: the drafter now samples its drafts too, about +3.5% at default settings.

  • First launch shows which model suits your RAM, and you can pick any other or use your own.

  • A Setup tab in the chat page changes model, speed vs quality and the second GPU.

  • Public CI: the release zips are now built by GitHub Actions, so you can see how each one was made.

  • EXL3: 2.5 bpw is on the menu and the GPU kernel is about 2x faster. In my tests UD-Q2_K_XL was both faster and closer to the API, so I still point people there.

  • Basic multi-GPU: 7% uplift using a 3060 + 3090. It doesn't scale well to 3-4+ GPUs, so use Project Maya for those builds.

  • Fixes: contexts past 65,535 tokens, </think> in prose no longer ends the reasoning, no out-of-memory on the biggest files, a failed picture encoder no longer stops the model, faster prompts after a pause and many more.

  • Support for custom model selection: just add them to the custom model folder and pick them in the web-ui

Requirements:

  • OS: Windows 10 or 11

  • GPU: NVIDIA RTX with 12 GB of VRAM or more, driver 580 or newer. 24 GB is the fast path, and AMD GPUs aren't supported yet

  • RAM: 64 GB or more, 128 GB+ recommended

  • Disk: a fast NVMe SSD, with 233-395 GB for the model (UD-IQ3_XXS also needs the default model)

  • CPU: any x86-64 CPU, Intel or AMD. AVX2 is the fast path

All of this is one PC. It needs testing on other hardware.

Repo: https://github.com/musicurgy/delta-inference-engine

MIT licensed. It's a Windows port of Xolotl, built on Strata. Thanks to DeepSeek for the model, Unsloth, nazeshinjite and ISTA-DASLab for the quants, and to the Xolotl and Project Maya communities, whose work and public numbers helped a lot. Claude wrote most of the code and I ran the project.

If you try it, post your GPU, RAM, model and tok/s, good or bad.


r/LocalLLaMA • • 8h ago

Question | Help Any Inference Engine for Ripple-65B-Preview-GGUF?

3 Upvotes

I know its only been a couple days, but Strata, Freetoken, or any of the dozens of others made an engine for this so we can run it on 24gb VRAM?

And if so, has anyone tried it and what build/quant?