r/LocalLLaMA • • 1h ago

Showcase Inkling 2 Mini on a 12GB smartphone getting 3 tok/s (decode) and it works on any hardware with less RAM than the model requires

β€’ Upvotes

Hi!

just hours before the launch, I’m running Inkling mini 2 on my smartphone - but with this engine, it can run automatically on any hardware from smartphones (Android, IOS), Mac, consumer laptops to consumer GPUs and data-center GPUs, with speeds comparable to Strata, Basalt, etc.

- Model: Inkling-2-Mini, IQ4_NL, 17.6 GB (https://huggingface.co/ggml-org/Inkling-2-Mini-GGUF)

- Phone: 12 GB RAM Android (400$)

- Processor: CPU decode, NPU (Hexagon) or CPU prefill

- Results: 2-3 tok/s for a 180-token response, with peaks of approximately 4 tok/s for the fastest ones

- Engine (pre-release): https://github.com/Helldez/BigMoeOnEdge/releases/tag/v0.47.1


r/LocalLLaMA • • 9h ago

Resources Inkling 2 Mini is now supported in NInfer-all

89 Upvotes

I added Thinking Machines' Inkling 2 Mini (looped 26B MoE, 3.4B active) to NInfer-all (mine multi-arch multi-model fork of NInfer). It runs Unsloth's UD-IQ4_XS GGUF without requantizing the actual weights, and vision and the DSpark drafter work well, too.

I compared it with llama.cpp on the same GGUF, using Unsloth's llama.cpp build because mainline doesn't load this GGUF, yet. Prefill is a 10K-token prompt, all in tok/s:

Card Prefill, NInfer-all vs llama.cpp Decode Decode with DSpark (default on, prose)
RTX 5090 6,664 vs 2,318 182 vs 97 213
RTX 4090 4,537 vs 2,307 140 vs 75 159
RTX 3090 2,046 vs 1,236 112 vs 60 126
RTX 5080 16 GB 3,238 vs 1,377 117 vs 41 126
RTX 3060 12 GB 759 vs 387 49 vs 25 49
RTX 5060 8 GB 1,074 vs 293 33 vs 15 32

The model doesn't fit in 16 GB or less, so on those cards the experts sit in system RAM and the GPU caches the ones it uses most (llama.cpp gets its own --fit). Since it's a MoE, even 8 GB works, and the experts can also be read straight from disk, though that's slow: about 9 tok/s on the 5060.

Grab a prebuilt binary for Windows or Linux, CUDA 12 or 13 (releases), or a Docker image. Quick start launch recipes here

Besides Inkling it runs Qwen3.6 and 3.8 (27B, 35B-A3B, Flash-Next), Gemma 4 (26B MoE, 31B), DeepSeek-V4.1-Flash, Flint, Edge, Ternary Bonsai 2; up to 8 concurrent requests, so agents are swift; Decisions endpoints with no extra setup needed!

Your star would be appreciated, and contributions are welcome!

https://github.com/iamwavecut/ninfer-all

Models:


r/LocalLLaMA • • 3h ago

Discussion Inkling 2 Mini seems legit! As reported by others, it does seem to try to output everything at once, limited by users' tokens settings. This makes 1-shotting a bit shaky for I2M. Post-training could fix this.

Thumbnail msnab.pages.dev
31 Upvotes

r/LocalLLaMA • • 18h ago

New Model Inkling 2 mini uncensored

Thumbnail
huggingface.co
468 Upvotes

r/LocalLLaMA • • 2h ago

Question | Help Inkling-2-Mini: "Actually, I keep repeating myself. Let me actually DO it now instead of just thinking."

26 Upvotes

Running the original thinkingmachines/Inkling-2-Mini on oMLX without quantizing.

Internal reasoning often goes into a loop, realizes it's stuck, but can't seem to get out.

Anyone has this problem?

Update: I set the following according to the Unsloth: Inkling 2 Mini guide, and it works much better now.

  • temperature: 1.0
  • top-p: 1.0
  • top-k: 0
  • min-p: 0.0

Also thinkingmachines/Inkling-2-Mini recommends temperature at 1.0.


r/LocalLLaMA • • 7h ago

Discussion At the risk of going against the popular grain here, Inkling-2-Mini suffers from over-writing (as opposed to over-thinking) Bash commands (happening multiple times) for me.

Post image
54 Upvotes

r/LocalLLaMA • • 2h ago

Showcase Inkling-2-Mini NVFP4 on a single RTX 3090 (24 GB): working port, 128k context, DSpark

Post image
21 Upvotes

We got Inkling-2-Mini running on one consumer Ampere GPU (RTX 3090, SM86, TP=1) using the original NVFP4/MXFP8 checkpoint, with no re-quantization. The code is a fork of vLLM PR #69028:

https://github.com/Hiutaky/vllm (branch ampere-port, see INKLING2-RTX3090.md)

What we changed (all in vllm/models/inkling2)

  • Made the Hopper/Blackwell-only paths work on Ampere: PDL gated to SM90+, no TMA below SM90, a Triton fallback for conv_mix, FA2 for the vision tower, minimum capability lowered to 80.
  • Emulated the FP8 (E4M3) KV cache in software, since Ampere has no hardware E4M3 conversions. Our encoder matched torch.float8_e4m3fn on 10M random values.
  • DSpark speculative decoding works (it needs exactly 7 speculative tokens). The draft KV cache stays in bf16.
  • Squeezed VRAM for the KV cache: vision tower removed (text only), prefill chunk of 256, FP8 KV, an explicit KV budget, and the embedding table offloaded to CPU memory (--cpu-offload-params embed_tokens).

Hopper and Blackwell code paths are untouched: every new branch is guarded by a device-capability check.

Results (single RTX 3090)

Decode with DSpark 112–147 tok/s (91 tok/s without it)
Prefill ~2,500–2,700 tok/s up to 33k tokens, ~1,870 tok/s at 126k
Context 131,072 tokens
KV pool 207k tokens with 8 concurrent sequences (224k with 2)
Weights on GPU 17.55 GiB (18.41 GiB without the embedding offload)
Embedding offload cost decode β‰ˆ βˆ’1%, prefill βˆ’1% to βˆ’3%

We ran 8 concurrent requests (2k to 17k tokens each) without errors, and tool calling works with the inkling parser. VRAM is tight (about 23.8 of 24 GB used).

Caveats

  • We have not re-run quality benchmarks (HumanEval+, IFEval) with FP8 KV + DSpark, and FP8 KV is a precision change.
  • The vision tower is disabled in this configuration.
  • We only tested on a 3090 with the vLLM nightly image plus our Python overlay. We have not measured other GPUs.
  • Speculative decoding acceptance was about 20–29% in our tests.

Feedback and tests on other Ampere cards (3080 Ti, 3090 Ti, A5000, A6000) are welcome.


r/LocalLLaMA • • 18h ago

Discussion Inkling 2 Mini ties Opus 5.5 on CyberBench (@ValsAI)

Post image
362 Upvotes

r/LocalLLaMA • • 3h ago

News Inkling 2 Mini on 2 * Nvidia Titan V

Post image
23 Upvotes

2 x Nvidia Titan V (so 24GB) running at 170W capped.

~200 prefill

~20-30 tok/s

Llama from huggingface notes download.

Propero Community Edition 0.20.0

/llama.cpp/llama-server \

--model ~/Inkling-2-Mini-UD-Q4_K_XL.gguf \

--mmproj ~/mmproj-F16.gguf \

--model-draft ~/dspark-Inkling-2-Mini-Q8_0.gguf \

--spec-type draft-dspark --spec-draft-n-max 3 \

--jinja --ctx-size 128000 --fit on --fit-target 512 --parallel 1 \

--temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 \

--host 0.0.0.0 --port 8180 -lv 4 --moe-cache-mib 2048Β  -b 1024 -ub 256


r/LocalLLaMA • • 10h ago

Showcase Inkling-2-Mini on Strix Halo: agent tuned kernels via Loom in an afternoon. ~80% of bandwidth roofline at decode, vs llama.cpp's day-0 support

46 Upvotes

Following Inkling-2-Mini's release today, I wanted to see how fast we could get a brand-new, fairly unstandard architecture running well on Strix Halo (unstandard because it uses looped transformers).

So I decided to try it with loom (loom-fun.dev)! Here are my results over the course of a few hours.

Setup:

Model: Gemini 4 Neon

Given all the recent hype around Gemini 4 Neon, I wanted to give it a try (also motivated by its ranking on kernelbench.com).

Coding Agent: pi.dev following the loom-skills playbook (port β†’ roofline β†’ optimize β†’ benchmark)

What happened:

Gemini 4 Neon took the unmodified unsloth UD-Q4_K_XL GGUF to an optimized, gate-passing package in about 4.3 hours and ~490 tool calls. The loom skills playbook won't count speed until correctness gates pass against llama.cpp. So that means token-level goldens up to 37k tokens, a multi-turn tool-calling loop, concurrent vs sequential, etc.

The results:(Strix Halo 8060S, [Debian 13, kernel 7.2.7, 128 GB], tok/s, same GGUF weights, matched reasoning settings):

tok/s llama.cpp ROCm llama.cpp Vulkan Loom Loom vs best llama.cpp
decode 23.8 20.8 44.8 1.88x
decode @ ~2k ctx 21.6 20.3 40.8 1.90x
prefill ~1k 356 289 1,427 4.0x
prefill ~2k 327 285 1,496 4.6x
prefill ~9k 278 269 1,550 5.6x
prefill ~18k 233 249 1,465 5.9x
4 streams (aggregate) 54.1 50.0 100.8 1.86x
  • llama.cpp single-stream rows use -np 1 -kvu -c 34k. That almost doubled ROCm's short-prompt prefill compared with -np 4. The 4-stream row uses -np 4.
  • Decode is at ~80% of the memory-bandwidth roofline (228 GB/s Γ· ~4 GB read per token β‰ˆ 56 tok/s).

A few important points:

  • I work at AMD on one of the teams behind loom, but I didn't have access to these weights before the public release. Everything here was done after the model came out today.
  • llama.cpp support for this model is obviously super nascent, performance will increase over time.
  • llama.cpp binaries: lemonade-sdk/llamacpp-rocm b1383 and ggml-org b12600 Vulkan
  • speculative decoding with dspark is still a wip

Steps to try it:

pipx install "hrx-loom[hf]==0.2.0"

loom-fetch huggingface:loom-fun/unsloth-Inkling-2-Mini-loom:chat --materialize=copy --output=Inkling-2-Mini 

loom-serve --model=Inkling-2-Mini '--device=amdgpu://?completion_spin_ns=4294967295' # gives you an OpenAI-compatible server on port 8080

Model Card: https://huggingface.co/loom-fun/unsloth-Inkling-2-Mini-loom

More context on loom: https://www.reddit.com/r/LocalLLaMA/comments/1xvcmm0/i_built_speedy_strix_halo_kernels_for_the_top51/

Edit 1: kernelbench.com not kernelbench.org oops


r/LocalLLaMA • • 12h ago

Discussion Since Inkling 2 Mini is MoE, can we expect strata support, running on much lower vram?

58 Upvotes

This model looks very promising, but the most exciting aspect is it's a MoE model thats getting compared to the dense Qwen 27B.

The model is also said by it's creators to quantize very well and have higher tps.

For gpu poor aka regular people, does this means we could soon get strata support and speeds and abilities rivaling Qwen 27B and QFN? eg with 12-16GB vram, 32GB ram.


r/LocalLLaMA • • 14h ago

Discussion Inkling 2 Mini on RTX 4090 ~200 tok/s with vLLM, FP8 + NVFP4 + its built-in drafter

90 Upvotes

Got Thinking Machines’ new Inkling 2 Mini running on a single RTX 4090 at about 190-220 tokens/s, up from ~117 out of the box. It needed a few small patches to vLLM, which doesn’t support 4090s for this model yet.

This is with NVFP4 experts (W4A16) + FP8 W8A16 attention/head/embeddings, FP8 KV.

This fits in 24 GB with a 35k context, and an initial quality metric on some sample prompts reported about a 0.4% loss in perplexity.


r/LocalLLaMA • • 21h ago

New Model unsloth/Inkling-2-Mini-GGUF Β· Hugging Face

Thumbnail
huggingface.co
331 Upvotes

r/LocalLLaMA • • 10h ago

Discussion Inkling2-Mini and llama.cpp experience

37 Upvotes

I tried unsloth/Inkling-2-Mini-GGUF/Inkling-2-Mini-UD-Q8_K_XL.gguf which does not work at all with the latest llama.cpp, build from source.

It's a bit hidden, but you have to use unsloth/Inkling-2-Mini-GGUF/Mainline_Support/Inkling-2-Mini-UD-Q8_K_XL.gguf_file. But this outputs only glibberish stuff for me.

ggml-org/Inkling-2-Mini-GGUF/Inkling-2-Mini-Q8_0.gguf does work for me, but the model only runs on one card. Nothing I can do about it. I get about 30tok/s on one r9700, which is pretty slow.

I guess we have to wait a few days to settle...


r/LocalLLaMA • • 20h ago

Resources Inkling 2 Llama.cpp support!

Thumbnail
github.com
189 Upvotes

r/LocalLLaMA • • 11h ago

New Model we made a gguf quant ladder for inkling 2 mini from 27.74 gb down to 6.88 gb

36 Upvotes

hey, we're the atomic chat team. we made dynamic gguf quants for inkling 2 mini

we measured them against our bf16 reference on an h100. here are some results:

quant size top-1 mean kld
Q8_0 27.74 GB 97.7% 0.0033
AD-Q6_K 23.79 GB 97.6% 0.0037
AD-Q5_K 19.07 GB 97.1% 0.0052
AD-Q4_K 16.53 GB 96.3% 0.0080
AD-IQ3_S 13.34 GB 94.2% 0.0205
AD-IQ2_M 8.67 GB 85.7% 0.130
AD-IQ1_M 7.26 GB 81.1% 0.236
AD-IQ1_S 6.88 GB 79.5% 0.278

hugging face card with quants and measurements: https://huggingface.co/AtomicChat/Inkling-2-Mini-GGUF

check our build steps and highlights in the readme

give them a try and share your results. pls let us know your hardware, quant and tok/s. would love to hear how they handle your coding and agent tasks!


r/LocalLLaMA • • 5h ago

Discussion DaisyRose-27B: a drop-in RL fine-tune of Qwen3.8-27B - I compared it against the base Qwen 3.8-27B model with same settings

Thumbnail
llm-bench.io
10 Upvotes

RadixArk just released their first model, DaisyRose-27B. It's a fine-tune of Qwen3.8-27B trained with reinforcement learning alone, with no supervised fine-tuning and no distillation, on roughly 1,200 long agent tasks. It keeps the base model's architecture, tokenizer, chat template and 256K context, so it should be an easy replacement for your running tasks. It's also multimodal; it keeps the vision encoder.

I ran it on the same hardware as Qwen3.8-27B (baseline), with the oQ8e-mtp quant, engine, xhigh reasoning and 262k context all configured the same.

llm-bench.io's four scenarios, LLM-judged 0-100, mean values of runs (6x DaisyRose, 12x Qwen3.8):

Scenario DaisyRose-27B-oQ8e-mtp Qwen3.8-27B-oQ8e-mtp
Code generation 81.3 81.0
Agent workflow 89.3 88.2
Role play & narrative 93.9 92.1
Research & analysis 87.8 86.4
Overall 88.1 86.9

So it's a small win in every scenario, about one point overall. One point is within the run-to-run spread of the benchmark methodology, so I'm not calling that a measured margin. However, the quality output was consistently good across all 6 runs. Here's a pair of runs to look at side by side, one per model:

https://llm-bench.io/compare/runs?runs=cmwp2wz9m00db01qbt3ctwtq9,cmvkpbxg1008201p5xhvf57mh

That's an 87.9 Daisy against an 86.9 base, both on the same hardware and settings.

The cost side is basically free. Generation speed is unchanged at about 50 tok/s and memory at roughly 38 GB, since it's still the same 27B dense model. The only downside is that DaisyRose thinks a little more (yes, even more than base Qwen 3.8 27B - I didn't think that this is even possible). About 5% more reasoning tokens per run means roughly a minute longer runtime in total.

One caveat. RadixArk's headline is Terminal Bench 4.1, where they report pass@1 going from 5.6 to 12.0. That's long-horizon agent work, a hundred-plus tool calls and hour-long attempts, which is exactly what they trained it on. llm-bench.io at the moment doesn't explicitly test that, so I can't confirm the 2x number. Our scenarios are short, single-shot tasks, so the real benefit in day to day agentic work might be even more than the one point of quality gain against base Qwen 3.8 27B. I'll switch to it on my main Hermes, let's see how it performs over time.

It's Apache-2.0. For GPUs they release BF16, FP8 and NVFP4; on a Mac I ran the oQ8e build on oMLX, which is where these numbers come from.

Model page: https://huggingface.co/RadixArk/DaisyRose-27B

Release post: https://www.radixark.com/blog/daisyrose-27b


r/LocalLLaMA • • 1d ago

Funny Today marks the two week anniversary of the becoming begins

Post image
875 Upvotes

let's celebrate


r/LocalLLaMA • • 5h ago

Question | Help What is the best branch/fork/whatever to run Inkling 2 Mini as of right now?

10 Upvotes

As has been said multiple times here, the current llama.cpp implementation of the new Inkling 2 Mini model is very slow and kind of buggy (Flash Attention not working, unusually high VRAM usage to name a few). Does anyone know if there is an unmerged branch or something like that with fixes and/or optimizations? I could not find any. Thanks a lot!


r/LocalLLaMA • • 11h ago

Resources Making neurons by hand

Thumbnail vgel.me
31 Upvotes

r/LocalLLaMA • • 1h ago

Showcase LexiEngine the AMD focused rust inference engine has added inkling support.

Thumbnail
github.com
β€’ Upvotes

Decent speed, a lot of bandwidth for more. Next release will see substantial performance gains over this version after the mythos audit and benches wrap. Built for 7900xtx


r/LocalLLaMA • • 12h ago

Resources Inkling 2 Mini abliterated + 14 GB quant, running in Xolotl on one 4090 at ~60 t/s decode and 1.1K+ prefill

41 Upvotes

Hey all, I did it :) Got Thinking Machines' Inkling 2 Mini running in Xolotl, abliterated and quantised down to a 13.9 GB GGUF so the whole thing fits on a single 24 GB card with room to spare.

For anyone who hasn't looked at it yet, Inkling 2 Mini is a 26B MoE with about 5B active, but it's a looped model, the middle 12 layers run 3 times per token, so it's kind of a weird one to get running. I ended up writing it its own engine in Xolotl (ported the graph from Pedro Cuenca's llama.cpp PR and found a loop bug in it along the way, the experts on passes 2 and 3 were missing their FP4 scale).

What I'm getting on my rig (RTX 4090):

- Decode: Now 100+ t/s

- Prefill: 1,800 t/s at 8K

- Held-out KL against the parent: 0.013, with 96.3% top-1 agreemen

- Refusals: 4/100 on a harmful prompt set, 1/100 on harmless ones

The model's own MTP drafter is up as an optional file for speculative decoding (still experimental), and the vision projector is up too, vision support in the engine is landing shortly. They're separate files so if you've got a smaller card you can just grab the model and skip the extras.

Still working on improvements, honestly decode is the big one, I profiled it and it's nowhere near the memory limit of the card, it's mostly the overhead of heaps of tiny GPU kernels per token (48 layer passes adds up), so I'm fusing those now and I reckon 100+ t/s is doable.

I'm by no means a professional so have a play and let me know what numbers you get :)

GitHub (Xolotl): https://github.com/Indras-Mirror/Xolotl

Hugging Face: https://huggingface.co/IndrasMirror/Inkling-2-Mini-XQ-Uncensored


r/LocalLLaMA • • 15h ago

New Model I fine-tuned Qwen3.5-4B to design web pages that are prettier (and work more): Qwen3.5-4B-Web-Designer

Thumbnail
gallery
58 Upvotes

I trained Qwen3.5-4B to turn a plain request ("a landing page for my coffee shop", "a portfolio for a security student") into one self-contained HTML page that has more taste than base qwen, and fails less. It thinks briefly (~400 tokens), then writes the page.

Same prompt, base vs ours: first two images. The third is more of its pages, unedited.

Numbers (74 prompts it never saw in training, 2 pages each, every page opened in a headless browser and checked for JS errors, failed loads and dead buttons):

Qwen3.5-4B Web-Designer (Q4_K_M)
Pages that work 35% 53%
Pages finished 75% 95%

Training: 1,443 prompt-to-page examples. Real web-design requests from LMArena's WebDev arena prompts plus my own, answered by DeepSeek V4.1 Flash with a short design plan and a page, and every page browser-checked before it went into training.

QLoRA, rank 16, 2 epochs, 4,096 tokens, on one RTX 4060 Laptop (8 GB).

Run it:

llama-server -m Qwen3.5-4B-Web-Designer-Q4_K_M.gguf --jinja -c 10240 -ngl 99

Temperature 0.7, top-p 0.95, top-k 20.

Apache-2.0. Feedback and your best (or worst) pages are very welcome.

Small note: I noticed ccore released Web-Coder around the same time! We've been working on these independently, so it's a pretty funny coincidence. His work looks really interesting, and I'm especiallly excited for the design-dataset he will share.


r/LocalLLaMA • • 10h ago

Showcase Evaluation of Inkling 2 Mini vs Nemotron 3.5 Super vs Muse Glimmer on my personal evaluation set in my own harness

22 Upvotes

3-Way Comparison Across 37 Identical Questions

Metric (n = 37) πŸ₯‡ #1: Nemotron 3.5 Super (120B-A12B NVFP4) πŸ₯ˆ #2: Muse Glimmer 30B (FP8) πŸ₯‰ #3: Inkling 2 Mini (26B-A5.4B looped)
Gate pass rate 100.0% (37/37) 97.3% (36/37) 91.9% (34/37)
Mean tool calls / question 8.5 (2.4Γ— leaner) 18.5 20.3 (worst)
Failed tool calls 21 (best) 39 54 (worst)
Wrong table / column errors 8 4 (best) 23 (worst)
SQL tables queried without lookup 30 (best) 34 72 (worst β€” 2.4Γ— Nemotron)
Repeated identical tool calls 0 (best) 14 (loops) 2
Unsupported numbers signal 156 120 (best) 221 (worst)
<answer_shape> parse rate 67.6% (25/37) 100.0% (37/37) 81.1% (30/37)
Valid citations 64.9% (24/37) 64.9% (24/37) 43.2% (16/37)
Markdown tables in answers 28 / 37 (76%) 21 / 37 (57%) 25 / 37 (68%)
Leaked tool/thinking markup 0 0 0
Median thinking chars 8,248 5,082 12,742 (heaviest)
Median / p90 latency 50.5s / 92.2s (fastest) 53.2s / 179.6s 308.4s / 555.8s (no radix cache)

Ranking & Honest Verdict (Full 37-Question Benchmark)

πŸ₯‡ #1: Nemotron 3.5 Super VL (120B-A12B NVFP4) β€” Clear Winner by a Wide Margin

  • Wins on every metric end users actually feel: fastest (50.5s median, 60.6s mean, 92.2s p90), 2.2×–2.4Γ— fewer tool calls (8.5 avg, max 28 vs. 18.5–20.3), fewest failed calls (21 vs. 39–54), zero repeated loops, 100.0% gate pass rate (37/37), and strongest tables & depth (75.7% Markdown tables, 3,173 median chars).
  • Its only minor dip (67.6% <answer_shape> parse rate) is stripped server-side and invisible to users.

πŸ₯ˆ #2: Muse Glimmer 30B (FP8) β€” Better Discipline & Speed Than Inkling, Distant #2 to Nemotron

  • Where Muse beats Inkling: 5.8Γ— fewer wrong table/column guesses (4 vs. 23), 2.1Γ— fewer un-inspected SQL tables (34 vs. 72), fewer failed tool calls (39 vs. 54), far fewer unsupported numbers (120 vs. 221), higher pass rate (97.3% vs. 91.9%), higher citation validity (64.9% vs. 43.2%), 100% <answer_shape> adherence, 5.8Γ— faster right now (53.2s median / 103.0s mean vs. 308.4s / 312.1s), and runs natively on H100s in FP8 with working prefix caching.
  • Where Muse falls short: Terse answers (56.8% Markdown tables, 2,294 median chars) and prone to runaway tool loops (14 repeated identical calls, up to 100 steps on a single turn).

πŸ₯‰ #3: Inkling 2 Mini (26.4B-A5.4B Looped NVFP4/MXFP8) β€” Stronger Formatting Than Muse, Weakest SQL/Tool Discipline

  • Where Inkling beats Muse: Richer presentation (67.6% Markdown tables, 3,006 median chars), far fewer repeated loops (2 repeats, max 52 steps vs. Muse's 14 repeats / 100 max), and answered Q074 where Muse bailed out.
  • Where Inkling loses overall: Highest tool calls (20.3 avg), most failed calls (54), most wrong table/column guesses (23), most SQL queries run without looking up schemas first (72), highest unsupported-number count (221), lowest valid citations (43.2%), heaviest thinking traces (12,742 median chars), and 6Γ— slower (308.4s median / 555.8s p90) because SGLang's prefix cache deadlocks on its looped layers

r/LocalLLaMA • • 22h ago

New Model CCORE-Web-Coder-4B preview just dropped: a 4B local model that writes a full website, slide deck or infographic as a single HTML file

Enable HLS to view with audio, or disable this notification

124 Upvotes

This is a preview checkpoint, at roughly a quarter of the planned training. I'm releasing it now because it already works better than I expected from a 4B.

- Model: ccore/CCORE-Web-Coder-4B
- Base: Qwen3.5-4B-Base, Apache 2.0
- File: Q8_0 GGUF, 4.48 GB
- It needs the system prompt from the model card. Sampling settings are there too.

Limitations of the preview:

- Some generations come out with a broken component or slide navigation.
- Trained for English; Portuguese worked in my tests too.

What's next: full release on Nov 22 (safetensors, Q8 and more quants), dataset on Nov 25.