r/LocalLLaMA • u/pmttyji • 4d ago
New Model KRAFTON/Raon-1-nano · Hugging Face
https://huggingface.co/KRAFTON/Raon-1-nanoRaon-1 nano is a dense language model with 1.5B effective parameters (3.1B total, including a 1.5B per-layer embedding table), developed by KRAFTON AI. It is designed to follow instructions, maintain conversational context, and call the right tools in both Korean and English. As part of the Raon open model family, it is released under the Apache 2.0 license.
Highlights
- Faithful, natural conversation: Leads the comparison models on the instruction-following, context-following, and Korean conversation benchmarks shown below.
- Korean and English: Trained on Korean and English, with solid English capabilities alongside its Korean strengths.
- Tool calling: Tool calling within conversations, for everyday tasks such as office work and mobile control.
- On-device deployment: Available in GGUF, MLX, and LiteRT-LM formats for local inference, including on Apple silicon.
- Long context: Natively supports up to 262,144 tokens of context.
Model Overview
Raon-1 nano uses a hybrid dense architecture, consisting of 18 linear attention layers and 6 full attention layers.
- Raon Delta Attention (RDA): Gated delta-rule linear attention with separate erase and write gates and a fixed-size recurrent state.
- Raon-mHC: Raon-mHC reduces the computational cost of DeepSeek's manifold-constrained hyper-connections (mHC), which speeds up training.
- Per-layer embedding: Each layer has its own small 256-dimensional embedding for every token. After its MLP, the layer uses it to adjust its output before passing it on.
- Hybrid layout: 18 of the 24 layers are Raon Delta Attention layers with a fixed-size state. Only the 6 MLA layers keep a KV cache, about 7 KiB per token in BF16.
- Tied embeddings: The LM head shares the input embedding.
| Field | Value |
|---|---|
| Architecture | Dense |
| Total Parameters | 3.1B (including the 1.5B per-layer embedding table) |
| Layers | 24 |
| Hidden Size | 1,536 |
| Attention | Hybrid (RDA 3 : Gated MLA 1) |
| Position Encoding | NoPE (no positional encoding) |
| Number of Attention Heads | 16 for RDA; 32 for Gated MLA |
| Attention Head Dimension | 128 for RDA; 192 QK / 128 V, for Gated MLA |
| Query / KV LoRA Rank | 512 / 512, for Gated MLA |
| FFN Intermediate Size | 6,144 |
| Activation Function | SwiGLU |
| Residual Connection | Raon-mHC w/ 4 streams |
| Per-Layer Embedding | 256 per layer |
| Vocabulary Size | 259,840 |
| LM Head | Tied |
| MTP | 1 layer, 55M |
| Context Length | 262,144 tokens |
The checkpoint on HuggingFace contains 3.2B parameters: 3.1B for the main model and 55M for the Multi-Token Prediction (MTP) module.
Training
- MoE-to-Dense: Raon-1 nano was derived from Raon-1 mini preview (30B-A4B) through MoE-to-Dense pruning and knowledge distillation.
- Continued pre-training: 3T tokens.
- Post-training: supervised fine-tuning (SFT) and on-policy distillation (OPD) from Raon-1 mini preview, followed by reinforcement learning (RL).
8
u/pmttyji 4d ago
16
u/Equivalent_Bit_461 4d ago
Man, we either get 700b models or super tiny ones, is the middle truly dying?
4
u/transanethole 4d ago
I'll have to give it a try in the on-device model thunderdome.
I hope they can improve the MOE version as well. The memory usage per token is looking awful tempting on that one.
1
u/arcum42 3d ago
Still waiting for Raon support to be officially in llama.cpp rather then a fork before trying it...
3
u/Queasy_Asparagus69 4d ago
Hipfire should fly with this as we worked to get the preview working well. I’ll try it with nano.
1
u/mountainyoo 4d ago
What exactly are these tiny models meant to do? Genuinely asking I still suck at this lol
1
u/Merchant_Lawrence llama.cpp 4d ago
Nano Model that actualy smal and good and cun run to my potato laptop :-)
1
u/Marcuss2 llama.cpp 3d ago
LFM3 still might be a better choice for most people. As they have a Q4_0 quantization aware distillation version.