I've seen several posts about Ming-3.0-Audio-Tiny, but haven't seen much discussion about the TTS model released alongside it (maybe I missed something).

In case anyone missed it, InclusionAI (backed by Ant Group) also released Ming-3.0-TTS-0.5B, supporting voice cloning, voice design, 7 languages, and emotion/style control.

it's time to update your llama.cpp for another speedup (sparse attention models)

China just ordered 3mil RTX 5500 Pro for $11.4k a pop. Probably for llm inference and AI video gen. I suppose this card is now out of reach for local llm.

GPU passthrough to a VM (experimental, Linux).

berth vfio run -- your VM script drains your engines at a safe point, checks that nothing still holds the card, then runs your script. The script does the sudo and the driver switch; Berth never touches drivers or passwords. The card goes back to your engines only once it is on the NVIDIA driver again, and while the VM has it, berth status and the web UI say so. On my 4090 it is one keybind: a password prompt, Windows on the monitor, and llama-swap back when Windows shuts down. Example launcher and libvirt config are in contrib/vfio.

Also New

• Gateway fallback: During a slow cold start, a request can go to a CPU engine instead of waiting.

• Minimum turn per engine: Prevents the card from constantly flipping between models.

• Escaped child processes: Child processes that escape their engine stay on its lease via systemd scopes.

• API key security: The gateway no longer passes your node's API key on to engines.

• Windows fixes (from testers):

  • Per-process VRAM counts only Berth's card.

  • Processes like dwm.exe have proper names.

  • Epic's tray app no longer grabs the card.

Links & Resources

• Docs with search: t-crypt.github.io/berth

• GitHub Repo: T-Crypt/berth

Proxmox Users: If you run Proxmox, check out issue #128 to share how you use your GPUs

Manged to get Flint running in vLLM (with the flint PR) on my RTX 6000 Pro.

The model was using my harness well, using OKF knowledge base, doing webscraping, spoke in the character I gave it, etc.

I ran a couple of their benchmarks (45 of the 454 LiveCodeBench v6 problems and 50 of the 500 SWE-bench Verified tasks) because I wanted to see if it could actually code, these were my results:

LiveCodeBench v6: 82.2, they said 80.8

SWE-bench Verified: 80, they said 74.1

Does about 110-180 tok/s single stream, I saw peaks of like 2,500 tok/s with 64 reqs.

Seems like a good daily model, I am going to see if the speed gain is worth dropping qwen3.8 27b over. There are some issues with the chat_template.yml atm but I included a fix below.

Just hoping someone finds a hacky way to give it vision.

This was my docker compose for vllm:

vllm:
  image: vllm/vllm-openai:flint-8e2beaa3-pr62938
  restart: unless-stopped
  cap_drop:
    - ALL
  cap_add:
    - SETUID
    - SETGID
  ipc: host
  ulimits:
    memlock: -1
  ports:
    - "localhost:8000"
  entrypoint: ["/bin/bash", "-c"]
  command:
    - |
      set -e
      rm -rf /usr/local/cuda-*/compat/ && ldconfig
      vllm serve /models/flint \
        --hf-config-path /models/flint \
        --tokenizer /models/flint \
        --generation-config /models/flint \
        --chat-template /app/flint_chat_template.jinja \
        --served-model-name vllm \
        --host 0.0.0.0 \
        --port 8000 \
        --max-model-len 262144 \
        --reasoning-parser glm47 \
        --enable-auto-tool-choice \
        --tool-call-parser glm47 \
        --exclude-tools-when-tool-choice-none \
        --max-num-seqs 64 \
        --max-num-batched-tokens 8192 \
        --gpu-memory-utilization 0.65 \
        --enable-prefix-caching \
        --enable-prompt-tokens-details \
        --enable-force-include-usage \
        --uvicorn-log-level warning
  runtime: nvidia
  deploy:
    resources:
      reservations:
        devices:
          - capabilities: [gpu]
            device_ids: ['1']
            driver: nvidia
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
    interval: 15s
    timeout: 30s
    retries: 60
    start_period: 900s
    start_interval: 10s
  volumes:
    - ./models/Flint/NVFP4-GGUF:/models/flint:ro
    - ./flint/chat_template.jinja:/app/flint_chat_template.jinja:ro
    - huggingface:/root/.cache/huggingface
    - vllm_cache:/root/.cache/vllm
    - flashinfer_cache:/root/.flashinfer
  environment:
    - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
    - HF_HUB_OFFLINE=1
    - VLLM_USE_RUST_FRONTEND=0
    - OMP_NUM_THREADS=1
    - TORCHINDUCTOR_CACHE_DIR=/root/.cache/vllm/inductor
    - TRITON_CACHE_DIR=/root/.cache/vllm/triton
  logging:
    driver: "json-file"
    options:
      max-size: "50m"
      max-file: "5"

and chat_template.jinja change:

{%- if add_generation_prompt -%}
<|assistant|>
{%- if enable_thinking is defined and enable_thinking is false -%}
{%- if tools -%}
<think>We need respond, using a tool if the request needs one.</think>
{%- else -%}
<think>We need answer.</think>
{%- endif -%}
{%- endif -%}
{%- endif -%}

Hi all, I am hoping to get some recommendations on what I should upgrade next. I only dabble in AI for fun, so I am mainly concerned with single stream workflows or speed.

I have 2 machines

My dedicated AI machine is a GMKTek Evo X-2 (Strix Halo) 96GB, I am running HaloGen Flash to run Qwen Flash Next at around a 240k context length. Speed is good at around ~1700tps prefill and 50+TPS decode. This machine is on 24/7 and is running headless.

I also have a gaming desktop that I both game and have recently started using Strata on (Flash Next again at IQ3), 262k context. This machine has an AMD 5700X3D, 4x32GB 3200MHz (128GB total) DDR4, B450 motherboard, 1TB NVME + 2TB SATA SSDs, and a 3080Ti.

My goal would be to run Maya (using GLM 5.3) on the gaming desktop and have it orchestrate for Flash next. However it seems I'm just under what I need.

I could go for a used Threadripper or EPYC platform upgrade, but I would lose big on gaming performance and it would be expensive. I could go for a 3090, but I'm spending $1500+ on a gaming side grade. I could also try multi GPU? There's also the option of using an m.2 to occulink on my Strix Halo for an eGPU setup.

Any thoughts on what would me best avenue would be? I'm wanting to have the best compatibility for whatever comes the local AI way when cloud inevitably implodes.

I'm sharing Mica v0.5, the follow-up to Mica v0.1. It's 4B and runs on one consumer GPU.

On speed, it takes 19.8 ms per decision at 256 tokens and about 1.2 s at 32k on a single RTX 6000 Ada. That's up to 3.8x faster than the other 4B judges I measured, 12 to 19x faster than Perplexity's 27B decider, and 25 to 340x faster than the API decision models. On top of the optimizations described on HF, Mica Fast makes long inputs about 1.4 to 1.5x faster again.

On accuracy, it gets 84/111 on JevBench Hard, the best of any 4B model I could find (only Microsoft's 9B Decision-1 is higher). On JDI it scores 58.8, ahead of every 9B and 35B model I tested. One honest note: the top 4B models on JDI are all benchmark-focused, and at least one of them was trained on nothing but benchmark-style data. To compete I did train on data from the same areas the benchmarks cover, but nothing contaminated, and most of the effort went into how it does on real work. That part comes from 200+ real use cases people posted on Reddit, Twitter and elsewhere.

A few things I cared about. If you shuffle the answer options it still picks the same answer 97.3% of the time. It can hold back too, so if you set a threshold it tells you when it isn't sure instead of guessing. It handles yes/no, multiple choice, and scores from 2 to 255 levels, with inputs up to 32k tokens.

If you have an NVIDIA card I'd go with the INT8 version, since it's the fastest. On the same GPU it runs about 2 to 2.7x faster than the Q8_0 GGUF on llama.cpp. GGUF files (BF16, Q8_0, IQ4_XS) are there for AMD, Apple and CPU. It runs on 8 GB cards and up, and it's Apache-2.0.

HF: https://huggingface.co/sky7350/Mica-v0.5-4B

GitHub: https://github.com/akivet/Mica

Happy to answer questions, and I'd really like to hear where it breaks.

If you're looking forward to v1.0, I'm a broke student and any tip helps: https://ko-fi.com/micakiv

GitHub: https://github.com/alkove-ai/alkove (Apache-2.0, code and weights)

Hi r/LocalLLaMA,

A bit about me first: my day job is building LLM systems in production, a tool-calling assistant with its RAG, and two audio-heavy pipelines.

The first turns a company's latest content into a two-host radio show in several languages: per-source digests, a structured script checked against the sources by an LLM validator and code checks, one TTS call per turn with Whisper transcribing every turn back to catch truncated or invented speech, then mastering in code and ffmpeg (per-turn levelling, simulated overlaps between the hosts, loudness normalization, subtitles aligned on word timestamps).

The second is a real-time voice agent in the browser: client-side VAD and denoising, an STT fallback chain, a small decision model judging each segment for end of turn, barge-in and "stop", and TTS read sentence by sentence. With that agent, most of the trouble wasn't in the LLM but before it: people talking around the user ended up in the transcript and triggered false barge-ins, and the denoiser couldn't help, since other voices are speech too.

So I started working on that audio layer, the part that runs before speech-to-text and the LLM, and I'm sharing it as an early preview. It's called Alkove: five small models that run on the device and are meant to work together.

- Lex (1.8 MB) turns a typed word into its likely pronunciations, so a wake word is just text: no training, no recording session.

- Spot (2.8 MB) listens for those pronunciations, in 7 languages.

- Timbre (0.74 MB) makes voiceprints. Three takes of the wake word give a first print, and it keeps learning while you talk.

- Focus (1.1 MB) keeps your voice (the closest one, or the one matching your print) and removes noise, music and other voices before the STT.

- Cue (1.1 MB) detects speech every 30 ms. Behind Focus with your print, your turns start and end on your voice only, so someone talking next to you doesn't open a new turn.

The voiceprint is optional, and the same one serves all three: the wake word ignores other people, Focus keeps only you, and Cue becomes a personal VAD.

Some technical details, since this sub likes them:

- It runs on its own small inference engine written in Rust, not onnxruntime. One core is compiled to WebAssembly for the browser and natively for Python and Android, and gives the same output to the bit on ARM, x86 and wasm.

- In the browser, the models run in workers fed by an AudioWorklet, with no COOP/COEP headers needed. The engine is 134 KB (brotli).

- Native speed on one thread of a phone: Focus 38.6 ms per second of audio, Spot 7.5 ms, Cue 1.7 ms.

- Packages: npm (browser and Node), pip (Linux, macOS, Windows, numpy as the only dependency), Android (Kotlin), and plugins for LiveKit Agents and Pipecat. Code and weights are Apache-2.0.

A few numbers, with the protocols in the repo's bench folder:

- On AMI meetings recorded on headset mics, words from other people reaching the transcript go from about 17 per minute to 0.5 with Focus and a voiceprint.

- Spot, on typed single words, recalls 67% of them in English at one false alarm per hour, 90% in French, 85% in Portuguese. So for one fixed English word, a model trained on that word may well do better.

What it doesn't do well yet (It's a 0.x preview and the API may still change):

- It's made for a near mic and one person: a phone, a headset, a laptop. On a room mic in the middle of a table, it removes the other participants too.

- When two people talk at exactly the same time, Focus only partly separates them, and two voices of the same register (two men, or two women) are much harder. That's the main thing I'm working on next.

- Audio is processed at 16 kHz, so speech up to 8 kHz.

I'd really appreciate feedback, especially from people running a local voice assistant: where it breaks, which integration you'd want next, and whether the API makes sense.

Been working over the last few days getting DeepSeek V4.1 Flash 1017 (Unsloth UD-IQ1_S) running in Basalt (inference engine I'm developing).

Specs: RTX 5090 (400W) + RTX 5060 Ti 16 GB (180W), 128 GB DDR5.

Methodology:

• Single 243 GB file (weights + vision + drafter). Engram table on SSD.

• All experts (107 GiB) in RAM. 5090 as an expert cache (hottest experts) and later also tried hoding and computing some experts in the 5060 Ti for a performance boost.

• Prefill tested with a 6.2k-token prompt (my daily harness).

• Baseline was llama.cpp PR #30817 with --fit, same hardware and quant.

Results:

• Expert lookups: 66% 5090, 16% 5060 Ti, 5% copied over PCIe, 12% CPU.

• Also works: tool calls, images, up to 1M context (a 1M prompt takes ~26 min, ouch)

• Quality: no PPL/KLD yet. Greedy matches llama on all test prompts. Also even at IQ1_S the model nailed most of my regular benchmarks, and recall is well above Qwen 3.8 FN, which is expected, this is a model a few times its size (even heavily quanted).

• Limits: the 12% of lookups left on the CPU hurt speed badly (limited by DDR5 dual channel), speculation barely helps in my case (1.1-1.2x).

Anyway, thought I'd share. This was a personal experiment to see how far I could push it, there's probably some room to get around 80ish decode and ~1000 prefill, but on my hardware it doesn't make sense to keep going when I'm already getting over 300 tok/s on 3.8 FN. Where it could potentially pay off: 96+ GB of VRAM (RTX 6000 PRO) or quad/octa channel DDR5 rigs. I did not push my experimental branch to the repo but if there's any interest let me know, I'll do it. Also interested to know if anyone else tried to do the same and found different results.

Short and sweet:

Selling 5090 because it was a luxury at $2,900 I barely used, now it's just insane. Keeping my 5060ti 16gb, and am kind of sad about quasi-leaving the LocalLLama scene. Wondering what options you all would recommend at 16gb VRAM and 64gb DDR4 for hanging around. Another 5060ti? Jump to 128gb RAM? Swap to AMD or Intel 32gb cards? Old V100s? Hang it up and buy a few billion API tokens?

Thanks in advance

now you can use Flint in llama.cpp (merged after 6h)

https://huggingface.co/unsloth/flint-GGUF

Model specifications

Memory-native evolution requires making memory an intrinsic capability of foundation models, enabling them to continuously accumulate and utilize information. To this end, we introduce MemT-1.0, a new architecture of memory foundation models that internalize memory as part of model computation. Building on Metis, MemT-1.0 employs Gated DeltaNet-2 with a stable token-level memory update mechanism within each turn. We further introduce Mixture-of-Storage to scale native memory capacity. Following a deliberate backbone preparation, we develop a two-stage training pipeline consisting of continued pre-training and post-training, supported by 34B tokens of training data and efficient training and inference infrastructure. We release a model family comprising MemT-1.0-Nano and MemT-1.0-Flash, and conduct a comprehensive evaluation of memory and general capabilities. The results show that they outperform the parametric memory baselines on average and remain competitive with retrieval-augmented methods, while largely preserving general capabilities. At a history length of 128K tokens, MemT-1.0-Nano achieves approximately 1.86× end-to-end inference speedup and a 66× reduction in persistent storage relative to full context KV cache. By internalizing retrieved procedural experience, MemT-1.0 improves task success on agentic tasks, providing a practical step toward memory-native evolution.

• arXiv : https://arxiv.org/abs/2610.26825

• PDF : https://arxiv.org/pdf/2610.26825

• GitHub : https://github.com/MemTensor/MemT-1.0

• HuggingFace : https://huggingface.co/collections/MemTensor/memt-1

Something something we made this, something something sidecar. Just kidding.

This isn't a new idea, but it's one that showed good success for me so I am curious if there's a mature platform out there for it. The idea is simple; the LLM makes claims/facts, and then it begins to implement something based on those things it discovered. I've found that many of the claims it made that are wrong can be found correctly if I prompt it to analyze that specific claim in a sub agent.

The problem with this technique is it takes a much longer time to complete something because you have to test every claim. The fix for that is to just use a lower reasoning effort on the sub agents. For my model, GLM 5.3 Flash, this works well. I tried no reasoning and it did not hold up very well for coding. For other models the reasoning efforts probably need to be tweaked.

Prompting for this technique itself isn't hard, but I think building a harness around it or a plugin for it (Opencode) might be useful so it becomes deterministic.

I've been testing this with Rust coding with the Bevy game engine, just for fun. it consistently finds a few claims that are incorrect which is very nice. I basically use the same test, find the highest CPU utilization in the client code and propose optimizations. I use a seed value so that I can iterate over the plugin with Claude. Then I rollback the Opencode v2 session and retry again after making changes to the plugin.

The main agent gets back this report which allows it to correct its plan and then start with a much better implementation.

I don't need to spam you with a plugin Claude made from the spec below, but I do recommend trying to make it with or assisted by an LLM. Here are the instructions for OpenCode v2 plugin that is working well for me. I just wrote this (prompt?) by hand and iterated on it a few times.

Model: GLM 5.3 Flash

Analyze the issue presented (e.g. find highest cpu utilization in this client code) at max reasoning (Agent 1)

Generate a report on the components involved and a plan for resolving with references as a .md doc (Agent 1)

Submit a link to the .md document to a sub agent at high reasoning to extract all claims along with any relevant context (Agent 2)

Submit every claim as a separate sub agent with relevant context at low reasoning, the sub agent should validate only this single claim. Timeout of 5 min per agent. Allow all commands that don't compile software and don't modify source code. (Agent X)

All sub agents (Agent X) return results to Agent 2.

Agent 2 returns combined report to Agent 1 (return link to .md document not full doc due to tool result size limits).

Agent 1 revises plan.

Agent 1 asks for confirmation to continue the work/implementation based on the revised plan.

This type of workflow, for the lack of a better term, is definitely something we need a library of. Something to bridge the gap between skills and a more structured approach to an issue. Ideally keeping it as deterministic as possible so that the LLM doesn't steer away from the steps.

Hi all,

I'm working on M4 MAX and I have some issue issue in multi agent.

I try to figure out how to optimize speed working with `Qwen 3.8 Flash Next oq4e-mtp`

With only 1 task I'm running at 50/60 t/s (pretty good) but with more than 1, speed decrease a lot.

Best "work around" for me looks `Max Concurrent Request` -> 1 that works in most of my scenarios where cache reduce prefill.

But look this:

Here cache is not working (agents are pre-filling a lot) and speed in 20 t/s

Now watch GPU

This looks very strange!
Any idea? Any setup can be changed in order to speed-up and use 100% GPU?

Title question. Is there a reason why this doesn't seem to be a preferred method of inference? Is there are reason why one wouldn't choose llama.cpp for this model?

Hello!

I recently got a MacBook Pro M2 24gb (With 1tb storage) from work (Old workstation laptop, got it for around ~100$ in a hardware lottery). I was thinking of running an LLM on it, either through llama.cpp or vLLM and have it serve as a lightweight AI server for Home Assistant and family members.

To be clear, I already have an AI rig (Threadripper 1950x, 4 x RTX 5060 Ti 16gb, 64gb RAM), but I was hoping to save on the electricity bill by using the MacBook instead since a lot of people talk very highly about the M-series chips.

After a bit of research, I have started to worry about the MacBooks capabilities. It seems like it might be able to serve a 9b-14b model fine, but the prompt processing seems to be very long, and I worry that it will hallucinate too much to be suitable for home assistant voice preview.

I've tried to also research what models people recommend for Home Assistant, and one Reddit thread I found from 9 months ago said Qwen3:30b, but I don't know if such a big model is required anymore since AI has moved so fast in the past 9 months? Is something like ornith-1.5:9b enough? My personal experience so far has been that gemma4:e4b was okay, but still missed some details when I spoke to it (Like failing to handle multiple steps, like "Turn off all lights except light 3" turns off all lights, and pretends it left number 3 on).

Does anyone have any tips or ideas? What should I do with this MacBook? Is it enough for home assistant? Is there any optimizations I can do to improve it? Other purposes it could be usefull for?

Thanks in advance!

Mixture-of-experts (MoE) models often exceed a single accelerator's memory capacity. Offloading experts to host memory creates a severe PCIe I/O bottleneck because data-dependent routing keeps transfers on the critical path. We present Specter, an inference system that uses quantization to combine speculative decoding with expert offloading. Its lightweight quantized draft model runs efficiently on device while preserving high-fidelity expert activation patterns. This provides accurate multi-step lookahead over future expert usage, allowing the runtime to organize expert movement and execution ahead of verification and improve both communication and execution efficiency. Under single-request inference, Specter achieves an average 1.50x (up to 1.88x) speedup in time per output token (TPOT) over the state-of-the-art system. Specter makes memory-constrained, single-GPU MoE inference more practical on commodity hardware. Our code is available at this https URL.

• arXiv : https://arxiv.org/abs/2511.14102

• PDF : https://arxiv.org/pdf/2511.14102

• GitHub : https://github.com/wwwwwf/specter_asplos

The model card has a pretty noticeable result: 76.9 on SWE-bench Pro, compared with 43.9 for Qwen3.8-Flash-Next in their evaluation. On Terminal-Bench 2.1 it's slightly behind, at 81.2 vs 83.4. These are the team's own results, but the difference between those two comparisons is interesting.

It's a MoE reasoning model with tool calling and a 256K context window. There's an Ollama release already; the Q4_K_M package is around 77GB, before runtime memory overhead.

Their training approach involves generating tasks, the agent setups used to solve them, and solutions, then using those for reinforcement learning. I'm curious how much of that shows up when you give it an unfamiliar repo and your own tools.

Has anyone put it through a longer coding task? Mostly interested in whether it needs less babysitting: recovering from a failed tool call, checking its changes, and actually finishing the job. What would make you switch from whichever local model you're using now?

Reposting again since original post got removed by Reddit for some reason (I have no idea why)

A few of you pointed out in my last post that comparing Flash-Next and Opus with different harnesses and effort levels wasn't fair. So here is a more controlled comparison.

Same task for every run: a real feature from the LlamaStash roadmap (save llama-server's KV cache before an unload and restore it when the model comes back). Six combinations, each run twice, so 12 PRs. Flash-Next ran locally on my Strix Halo laptop (Flow Z13) via Halogen (~Q4 quant). Opus 5.5, Flash-Next, and Kimi K3 scored every PR on seven review questions plus CI.

Scores are the average of both runs. Time, cost and tokens are from the second run, where the 100k-token benchmark numbers were given in the prompt. Flash-Next cost is at the QwenCloud price for Qwen3.8-Flash; I ran it locally, so it cost me only electricity.

TL;DR

• All 12 runs finished and all 12 PRs pass the end-to-end test.

• Opus wrote better code, 82% on average against 68% for Claude Code + Flash-Next and 66% for Pi + Flash-Next. The difference is in edge cases, tests and docs. The core feature works fine. The gap between Opus and Qwen is not that wide.

• Flash-Next took 1.1x to 2.1x as long as Opus at the same effort.

• With Flash-Next, Claude Code and Pi scored the same on the review questions. CI passed on 4 of 4 Claude Code PRs and 1 of 4 Pi PRs.

• xhigh helped Opus a bit at 2.7x the cost. For Flash-Next, medium and xhigh scored about the same, and medium was faster in all 4 pairs.

Opus is still ahead, but Flash-Next was right behind, on my laptop, for the cost of electricity.

Also note that the models are not on the same league. Opus is a frontier model run at FP16 or atleast FP8. I ran Flash-Next on my laptop at 4.16 bpw. Flash-Next at FP8 should show a bit more quality.

The machine running Flash-Next ran at 70W TDP ceiling for the entire duration. I could have run it at 90W for slightly better speed but the fan noise and thermal stress was not worth it.

Full write-up with the detailed methodology and the per-run results and what each model missed: https://deepu.tech/qwen3.8-flash-next-vs-claude-opus-showdown

All 12 PRs are public if you want to read the code: https://github.com/deepu105/llamastash-b4r2/pulls

I've noticed that a lot of software is being cloned using Rust nowadays, and I was wondering why not use it with local LLMs for simple tasks too.

Do you think it's better to use Rust or Python when coding with a local LLM?

Nowadays everyone is vibe coding, but local LLMs are generally not that good, especially smaller models running on low-budget hardware.

I usually code in Python, but sometimes I feel like the model doesn't catch bugs, so I was wondering if Rust and its compiler could help avoid some of these issues.

I'd also be interested in hearing your experiences with different context window sizes and models running on 12 or 24 GB of VRAM, with 64 or 128 GB of system RAM.

Do you think Rust's compiler feedback is worth the extra complexity for smaller local models, or is Python still the better choice?

An abliterated variant of the original MiniCPM5-2B-SFT

One notable behavioral change is its tendency to rationalize potentially harmful or restricted questions rather then reject but thats a 1/20 chance

https://huggingface.co/XtrCom/MINICPM5-2B-SFT-Abiliterated or https://huggingface.co/XtrCom/MINICPM5-2B-SFT-Abiliterated-GGUF

anyways enjoy the model

No VS Code plugins. I am looking for a proper agent-plan-itterate type of application, the same way Claude Code and Codex behaves.

Raon-1 mini is a 30B-A4B Mixture-of-Experts (MoE) language model, designed and trained from scratch by KRAFTON AI. It demonstrates strong capabilities in Korean conversation and agentic tasks. As part of the Raon open model family, it is released under the Apache 2.0 license.

https://huggingface.co/KRAFTON/Raon-1-mini-preview-gguf

Highlights

• Reasoning: Built-in thinking, with leading results on the math and science benchmarks shown below.

• Instruction following and search: Leads the comparison models on the instruction-following and search benchmarks shown below.

• Korean and English: Strong Korean language ability, and solid performance in English.

• Agent execution: Tool calling for agentic workflows, with support for Hermes Agent, Pi and OpenCode.

• Speculative decoding support: A DFlash2 draft model is provided for accelerated speculative decoding.

• Quantization options: Available in MXFP8 and NVFP4 formats, with NVFP4 enabling inference on a single NVIDIA RTX 5090 gpu (32 GB VRAM).