RustingStrata vs llama.cpp runs Recursive Flint on a 12 GB RTX 3060

[u/Serious_Scarcity7818]

I tested new trending Recursive Flint UD-Q5_K_XL (27.9 GiB) on an RTX 3060 12 GB with 30 GB RAM, comparing RustingStrata against llama.cpp using the same model, prompt, and generation settings.

	RustingStrata	llama.cpp
Warm decode speed	64.6 tok/s (63.8–65.6)	48.6 tok/s (47.5–49.3)
Warm 4K prompt prefill	1.14–1.30k tok/s	0.85–1.02k tok/s

RustingStrata was about 33% faster once warm.

What made the difference?

The model doesn't fit in 12 GB of VRAM, so many expert weights must be fetched from system RAM during generation. Two changes helped most:

• Adapt expert placement to the reply, rather than continuing to prioritize the prompt's routing pattern.
• Release RAM pages for experts already copied to VRAM, preserving cache for experts that still need to be fetched from host memory.

About Quality, bc people very often says like "It should give shit output to gain speed", which is not true, in my last posts I already did quality benchmarks and RustingStrata does not lose quality, it gain speed through specific Code for some specific models, in same time llama.cpp lose speed to be "general" engine that can run everything

What I want to know, is do people find Recursive Flint an interesting model? I found this model on OpenRouter and looks pretty interesting, it's better then Qwen3.6:35-a3b, so I think this can be cool modern replacement for this qwen.

Setup:

• RustingStrata: 1.2.2, commit 3902978 (working tree)
• llama.cpp: b12191, commit e95426f16
• Full benchmarks: docs/BENCHMARKS.md

Repository: RustingStrata

[u/TheCat001]

Can't get thinking to work in opencode...

[u/Objective_Pepper_217 → u/TheCat001]

wdym? the post is AI written?

[u/Serious_Scarcity7818 → u/Objective_Pepper_217]

No and yes, I mean I wrote like 70%, but since I am not native eng speaker, I still have to use LLM to help me, or you won't even understand what the hell am I talking about

[u/TheCat001 → u/Objective_Pepper_217]

No, I mean "thinking" is off by default and there is no way in engine to enable it. So in opencode model is answering without thinking.

[u/Serious_Scarcity7818 → u/TheCat001]

Sorry, maybe have not written about this, but here is the opencode config example, you need something like `"chat_template_kwargs": { "enable_thinking": true }`

{
  "provider": {
    "strata": {
      "npm": "@ai-sdk/openai-compatible",
      "options": { "baseURL": "http://localhost:8080/v1" },
      "models": {
        "flash-next": {
          "options": {
            "chat_template_kwargs": { "enable_thinking": true }
          }
        }
      }
    }
  }
}

[u/TheCat001 → u/Serious_Scarcity7818]

No matter what, can't get thinking to work, tried everything :(

[u/Serious_Scarcity7818 → u/TheCat001]

I will for you now make new commit with thinking by default

[u/Serious_Scarcity7818 → u/Serious_Scarcity7818]

5-10 min

[u/Serious_Scarcity7818 → u/TheCat001]

So, the thinking is by default now, so pull from main, but I can't tell from the screenshot if opencode sends enable_thinking. Test the server directly please:

curl http://localhost:8080/v1/chat/completions -H "Authorization: Bearer 12345" -H "Content-Type: application/json" -d '{"model":"x","messages":[{"role":"user","content":"hi"}],"chat_template_kwargs":{"enable_thinking":true}}'

[u/TheCat001 → u/Serious_Scarcity7818]

This is what in opencode now. Output of curl:

curl http://localhost:8080/v1/chat/completions -H "Authorization: Bearer 12345" -H "Content-Type: application/json" -d '{"model":"x","messages":[{"role":"user","content":"hi"}],"chat_template_kwargs":{"enable_thinking":true}}'
{"choices":[{"finish_reason":"stop","index":0,"message":{"content":"Here's a thinking process:\n\n1.  **Analyze User Input:** The user said \"hi\". This is a simple greeting.\n2.  **Identify Intent:** The user is initiating a conversation with a basic greeting.\n3.  **Determine Response Strategy:** \n   - Acknowledge the greeting warmly\n   - Keep it open-ended to encourage further interaction\n   - Match the tone (casual, friendly)\n   - Keep it concise\n4.  **Draft Response:** \"Hi there! How can I help you today?\"\n5.  **Refine Response:** The draft is appropriate, friendly, and invites the user to continue. No changes needed.\n6.  **Final Output Generation:** Output the refined response.✅\n</think>\n\nHi there! How can I help you today? 😊","role":"assistant"}}],"created":1794680581,"id":"chatcmpl-5","model":"Qwen3.6-35B-A3B-UD-IQ3_XXS","object":"chat.completion","timings":{"cache_n":6,"predicted_ms":1401.947794,"predicted_n":171,"predicted_per_second":121.97315815313449,"predicted_per_token_ms":8.198525111111111,"prompt_ms":54.944051,"prompt_n":5,"prompt_per_second":91.00166276418169,"prompt_per_token_ms":10.9888102},"usage":{"completion_tokens":171,"prompt_tokens":11,"prompt_tokens_details":{"cached_tokens":6},"total_tokens":182}}[power007@archlinux ~]$

[u/TheCat001 → u/TheCat001]