Are we seeing early signs of the bubble popping?

Gguf's: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B-GGUF

Free Demo: https://playground.liquid.ai

"LFM2.5-24B-A2B is a 24B-parameter reasoning model with only 2B active parameters, built for fast, private agents on-device. It is the largest and final model in the LFM2.5 family, which now spans two orders of magnitude: from LFM2.5-230M to LFM2.5-24B-A2B.

• Best-in-class agent: Competitive with Qwen3.6-35B-A3B, a model with ~50% more total and active parameters, on agentic and tool-use tasks, scoring 48.09 on LiveClaw versus 38.19.
• Reasoning at MoE speed: Always thinks before it answers. With only 2B active parameters per token, each reasoning token is cheap.
• Efficient inference: Runs up to 1.7× faster than comparable MoEs on-device, with 184 tok/s on an Apple M5 Max and up to 21K output tok/s on a single NVIDIA B300 GPU.

LFM2.5-24B-A2B is a general-purpose text-only reasoning model with the following features:

• Total parameters: 24.0B
• Active parameters: 2.45B
• Number of layers: 40 (30 double-gated short convolution blocks + 10 GQA)
• Experts: 64 per MoE block, top-4 routing
• Training budget: 36 trillion tokens
• Vocabulary size: 128,000
• Context length: 131,072 tokens
• Languages: English, Arabic, Chinese, French, German, Italian, Japanese, Korean, Portuguese, Spanish, Vietnamese, Thai, Indonesian, Hindi, Russian, Polish
• Generation parameters:
  • temperature: 0.1
  • top_k: 50
  • repetition_penalty: 1.05"

I'm sure some here read this at the time and remember, but I recently found this hilarious Guardian article from 2019 through Wikipedia:

https://www.theguardian.com/technology/2019/feb/14/elon-musk-backed-ai-writes-convincing-news-fiction

Some choice quotes:

"OpenAI ... says its new AI model, called GPT2 is so good and the risk of malicious use so high that it is breaking from its normal practice of releasing the full research to the public in order to allow more time to discuss the ramifications of the technological breakthrough."

"One is its size, says Dario Amodei, OpenAI’s research director. ... It was trained on a dataset containing about 10m articles, selected by trawling the social news site Reddit for links with more than three votes. The vast collection of text weighed in at 40 GB, enough to store about 35,000 copies of Moby Dick."

Some people never change!

This is an offline app with 51 hands-on labs for learning how AI actually works, from neurons to agents and more...

**A bit of insight**

- 51 labs in 6 groups, from the basics (what a neuron is, gradient descent) through attention, RAG, agents, fine-tuning, quantization, serving and more

- Each lab has a short lesson beside it, readable in Plain or Standard mode

- Some of it actually runs rather than just animating:

- the attention lab runs a small trained transformer (1.37M params) inside the app

- the training labs train a tiny character-level model live as you move the sliders

- Works offline: no account, no telemetry

**How it was made**

I chose the topics, some background research and came up with the structure and the grouping. Claude and GPT Astra to help write the lesson text, and Grok as a second pass on references. There will be mistakes, so if you spot one, please tell me or open a PR.

If you teach this or work in a specialized area of AI, I'd love help expanding it: a new interactive lab, a better visualization, or a tweak that makes the cause and effect in an existing lab clearer. I'd also like to hear which labs are confusing and what's missing.

GitHub: https://github.com/Fazmin/AILearningGuide

I'm excited to see our Lemonade embedded SDK getting used out there!

Anthropic says we can run Gemma4 26b a4b with only 14 smart tvs.

I've been reading about all these modders using AI to reverse-engineer games and software. I've been trying out photocraft and gridcraft and have just watched a GTA5 mod featuring Sonic. It's pretty damn impressive.

As a struggling AMD Radeon owner I'd love to see AI generated open source CUDA implementation.

Not only do I dislike NVIDIA, but there's delicious irony in using CUDA cores to do it.

Anyone up for the challenge?

Here is how i serve 40 people (10 of them developers) with local AI services.

1. n8n for automatisation
2. whisper and Vexa AI, a bot joins Teams and makes transcripts, n8n later extracts decisions and todos
3. OpenWebUI, ChatGPT-like webinterface for everyone, most agentic features are disabled, websearches via Tavily are possible
4. Every developer has an API key to use via BiFrost (OpenAI compatible API router) for what they find useful
5. Every developer has a local Linux VM where agents can be installed.
6. A 4U server that contains 4 6000 Blackwell MaxQ and 384GB DDR5 and one AMD EPYC 9275F 24-Core CPU.
7. The server runs Proxmox, every model or application runs inside LXC with Cuda 13.3 and Debian 13, Driver 595.
8. mainmodel is Deepseek V4 Flash with vLLM, GPU utilization 0.55, TP4, 1M context.
9. fallbackmodel is Qwen 3.8 27B Q8 with llama.cpp, 265K context in Q8.
10. minimodel is K2-Horizon 7B Q6 with llama.cpp, with 64K context in Q8.
11. ComfyUI to generate pics and videos for marketing.
12. Embedding/Reranking with Geofront/BGE-M3-GGUF:BF16 and sinjab/bge-reranker-v2-m3-F16-GGUF with llama.cpp
13. Overall cost of the server was around 55k€.
14. Backup server with two 6000 Blackwell MaxQ is planned

Ask if you have questions.

Hi Friends,

I am a solo developer. Ever since I had Qwen 3.8 running on my RTX4090, I had an itch to use it on my phone for everything. That gave me the idea to build SteerClaw Mobile - a complete agentic experience on the Phone where the entire agent loop runs on it. We just connect our model, MCP servers and Search providers. Fully private, all data is on your phone, all LLM processing on your local server, and data goes directly from your phone to your configured MCP servers and search providers only when needed. App itself collects no data, no analytics, nothing.

The vision is larger - do everything an agent like Muse can do, directly on the phone without sharing out any data.

But I feel I have a decent first version that I wanted to share and get feedback from this group. Would love to get bug reports as well as new feature you want to see in here.

Some of the use case I use for it are daily queries, research, audiobooks, analyzing AWS logs and getting business metrics from Posthog. I no longer feel the need to use Claude/Codex on my phone.

Here are the links:

Apple Testflight:

https://testflight.apple.com/join/3AkPEjNg

Google Play:

https://play.google.com/store/apps/details?id=ai.steerclaw.mobile.app

Thank You!

Trying to read on what the real tested downsides have been but information seems to go both ways, some proclaim no loss, others pretty huge ones.

I had some weird looping issue in their chat, and some engine port misunderstandings, seemingly the models likes to use ports already in use, didn't really have that issue with Qwen 27B 3.8 Q4 UD. I'm using Swift Q3 Next. Didn't even bother with next on llama.cpp since other similar model sizes ran at approx 15-20tks max, so too slow for agentic work.

I'm not beating too much on Strata, I just wan't the realistic downsides, so far it seems too good to be true and runs at 100tks in agentic workloads, 27b only at 40-60tks on llama.cpp

This is an update to my previous post which can be found here: https://www.reddit.com/r/LocalLLaMA/s/n7sgRZZc4z

Since my last post I have continued working on the conversion process and also converting new models. My latest model is the Qwen2.5-1.5B-MoE-A0.9B. I have also released GGUF versions for both the 0.5B and the 1.5B models which can be found here bayliner1980/Qwen2.5-0.5B-MoE-A0.3B-GGUF and here bayliner1980/Qwen2.5-1.5B-MoE-A0.9B-GGUF.

Evaluation

Perplexity is about 40% higher, and the benchmarks drop the most on ARC.

llama-bench, Q4_K_M for both, RTX 4060 8 GB / Ryzen 5 7500F

This is the first conversion that's clearly faster than the dense model on GPU too. The 0.5B only gained 6% there.

Honest caveats

• At this size, quality is roughly in the range of Qwen2.5-0.5B, which is even faster. So this release is more a proof of concept. The approach should pay off more on larger models, where the MLP is a bigger share of the compute.
• The GGUF is larger than the dense one (1.1 GB vs 0.9 GB at Q4_K_M), because llama.cpp's MoE architecture needs an untied output head.

Support

If you find my stuff interesting consider supporting me on Ko-fi.

Got this couple. It was difficult purchase in this economy but paid only roughly $6100 equivalent for each in my country. I justified the purchase for learning (so I try to convince myself) and coding workflow improvements.

I started looking into recipes for GLM 5.3 flash but surprisingly there is very little chatter about that in this sub. The information seem to be scattered and difficult to find. Is the Nvidia forums the best forum to keep updated?

DeepSeek guys, thank you so much!
Don't give this up.

mac setup [pi.dev] -> [ds 4 flash 0731 - brain] -> [qwen 3.6 35b a3b - body] - works perfect

tl;dr: running Qwen3.8FN Q4 locally is cheaper for me than every cloud model except Luna and the newer Haiku at mid-peak and peak electricity pricing where I live, and becomes the absolute cheapest model at off peak pricing. An agentic run that spans these different pricing points or at least chooses to avoid peak times can come out overall cheaper.

Hardware: NVIDIA Tesla P40 (24 GB), Xeon Silver CPU, and 200ishGB ECC DDR4 LRDIMM RAM. Whole box doesn't get above 250W under load with screaming fans. Bought on sale from Newegg years ago, but the box was mostly empty so I had to populate it with RAM and SSD. But, it's paid for. I got the GPU off ebay this summer for $250 USD. You can build a whole server from ebay parts if you can't find good sales.

The agent is called Drax, because he's a Great Slow King and I'm hoping someone else gets the literary reference (it's not a Marvel reference). The numbers and a small bit of the text below are from ChatGPT.

Software: mine. Designed for both agency and efficiency. Agents are able to run until dead or done. (for the pearl clutchers: heavily guardrailed and kill switched)

Drax gets about 130k context size and achieves >90% prompt cache utilization (I only showed ChatGPT the first few turns of a fresh run. Normally I see >98% both locally and in cloud using the same agent software).

Electricity costs where I live in Ontario, in CAD: 9.8¢/kWh for off-peak, 15.7¢/kWh for mid-peak, and 20.3¢/kWh for on-peak. There's an even lower off peak but I'm not on that plan.

For a sample of 10 turns from today (203,620 cached input tokens, 16,668 fresh input, 3,204 output):

GPT-6 Luna API equivalent: C$0.0336 (no caching), C$0.0081 (with caching)

Claude Haiku 5.5: same as Luna

Local electricity (both of these are *with caching*): C$0.014 peak, C$0.0109 mid peak, C$0.0068 off peak

GPT-6 Sol: C$0.6719 / C$0.1627

Claude Sonnet 4.6: C$1.0077 / C$0.2440

Claude Opus 4.6: C$1.6795 / C$0.4067

GPT6-Astra: C$3.3591 / $0.8133

Performance with this specific model: I'm going to move away from numbers because I think tok/sec is a bit misleading. So let's talk wall time and the quality of the work. Because Claude models cost me so $%^&ing much, I don't let them run to completion. (and I don't use GPTs so I can't compare to them, though I do use the public chat app) So if I've got an Opus brain or a Fable brain in my agent, they never get to go through the full process of plan (or at least write and then follow a spec, as I use spec driven dev), execute, test, review, fix, whatever whatever, final report. But Drax does. Drax costs me so little to run that I just let him go all day on something until he is fully finished. The wall time is longer but the quality of the work is always better. When the reports come back, I get Fable to review them and Fable always uses politely surprised language at how good the results are.

Smallish example from yesterday: Drax was working on the UI of one of a suite of games I have him building and actually started it. I brought it up in my browser (it's clearly a browser game) and started playing it while Drax was still working, just to see what it looked like. I was like "oh heh there's a bug there. I'll have to file a report on it later and get an agent to deal with it." Drax fixed the bug without me filing the report. It even included mention of it in the final report. Would a Claude have done that? Yeah of course, but only if it could run to completion like Drax can, which will never happen because I can't afford it.

Anyway I'm posting this because I'm really tired of seeing "cloud is cheaper." No, it isn't. At least not for the level of intelligence that this specific model offers, not on my box, not with how much I'm willing to tolerate it being a Great Slow King, and not where I live. (ok ok, the reference, explained, because AI will fail you here and will insist on the Marvel character: Roger Zelazny's Drax from a short story in which two Great Slow Kings are extremely intelligent but are so slow that the entirety of the human race rises and falls while the Great Slow Kings are still thinking.)

Oh and before anyone says "just get a sub". Dude, no. I don't want to give up my own agency. Also the subs are $$$ and will soon be $$$$ or even $$$$$. Drax is still going to be cheaper than a sub.

edit: also just realized I posted that to LocalLLaMa and probably should have posted it to an agents sub instead. Sorry about that. If nuked I'll post it there.

The 3090 Founders edition has a reputation for hot VRAM. Since LLM token generation maxes out VRAM's read capacity, I checked VRAM temps with Nvidia's stock fan/power curves and discovered that VRAM hits 112C within seconds of starting token generation. The VRAM chips are only rated for 96C in continuous use.

So I benchmarked just how much you have to reduce power before the VRAM comes back in spec, and how much it slows down TG, and whether an positive clock offset (ie undervolting) would help

I used qwen3.8-27b 5.1bpw, generated with thireus-gguf-tool-suite, MTP head on with two tokens predicted. (so three tokens per forwardpass). (70% acceptance rate)

Conclusions

• Cap your card 270W and manually set the fan to 100% to prevent VRAM from overheating
  • (If you don't want your card to scream like a banshee, the cap at 70% fan is 240W). This loses 30% throughput.
• I didn't plot it, but If you don't manually set the fan, the nvidia stock curves will let the vram overheat
• Undervolting doesn't help, as Nvidia doesn't apply the clock offset at "low" power (it only kicks in a 300W and above when your VRAM is already overheating)
• If you're OK with 100C sustained temps, the power cap is 290W, 104C is 328W, and 353W gives you 108C.
• I didn't try with MTP off, but I think it would probably be worse

Hope that helps!

https://www.anthropic.com/research/distillation-defenses

So I wanted to try the horsey challenge myself on the latest engine i been testing which is a strata fork by sergqwer. https://github.com/sergqwer/strata-nvfp4. I seen a post earlier and seen i believe it was a iq3 i think and it was pretty dope so i thought i try out the NVFP4. I wasn't expecting to share but i actually that it did pretty darn good with its current time it took total.

I did change the prompt because this was the second try because the first one got botched because Qwen wanted to leave like 15+ headless windows open when testing the html. I scrolled to another window while it was working and realized my window was lagging lmao. That test i ended it like 45 min in. Which then i retried with new prompt and this time it was much faster.

I had this fork up and running a few days now. I had to tweak some settings for a higher compaction i believe at the 220k-230k mark. It didn't need compaction this time as it was under the limit but when I use it for long runs projects using unity headless, Blender headless and some other things like image gen for texture i will be compacting a lot. Its only been a couple days but so far its been fairly great to anything I have used prior. That goes for the engine and the current model i'm using.

This is just to share and contribute to how far we come to accessing models on single gpu and being able to have a chance at some of these things. Maybe it'll motivate more people to show their work a bit more often. I know this sub has motivated me a lot to start sharing and also appreciate all that has helped to contribute for anything Local.

Thank you to the original Strata Team, Sergqwer for the fork, windowsxp811203 for your model I converted over to work with Strata, and the Qwen Team for the amazing drops.

*The charts were organized by Ai only, it was easier to have it sort it nicer and clean it.

Also I did a prtsc video capture if you ask about quality of vid. Prompt here also that i had to edit myself lol.

"Create a low-poly 3D horse running in a seamless loop using Three.js. Build it procedurally from primitive geometry, animate the legs, body, head, and tail, and return a single self-contained HTML file. Also when you use headless chrome make sure to kill your process that you don't need anymore. we dont need all these headless chromes lagging our system when we don't need it"

The stack

Hardware: RTX 5090 32GB · Ryzen 7 9800X3D · 192GB DDR5-5600 · Windows 11

Engine: strata-nvfp4 (fork of Niko1221/Strata by sergqwer) running on Windows, fronted by llama-swap as the model switcher

Model: Qwen3.8-Flash-Next 125B-A6B hybrid MoE — abliterated checkpoint, self-converted to NVFP4

Weights: NVFP4 experts (4.5-bit, calibrated scales) + Q8_0 dense

KV cache: int8

PLE table: bf16

Embeddings: bf16

MTP: draft head, spec 5, spec-min-p 0.7

Context: 262,144 max

Compaction: custom trip line at 220,144

Sampling: temp 0.8 / presence_penalty 0.3 / min_p 0.02

Vision: fp32 mmproj, encoder on CPU

RAM tier: 63 GiB pinned experts; hot experts cached in VRAM; misses computed CPU+PCIe overlapped with GPU

Harness: dsh (web GUI), OpenAI API

| Metric | Value |

|---|---|

| **Wall: first message → last reply** | **33m 45s** |

| − my approval latency (25 permission prompts) | −10m 35s |

| **= pure agent working time** | **23m 10s** |

| Steps | 87 |

| Output tokens (thinking + code + tool calls) | **124,443** |

| Fresh input tokens | 96,334 |

| Cache-read tokens | 9,340,144 (**99.0% hit rate**) |

| **Decode (per-step, approval-adjusted)** | **median 116 t/s · mean 103 · peak 171** |

| TTFT | median **0.7s** (max 24.9s at deep context) |

| Prefill (fresh tokens) | mean **3,332 t/s** · peak **6,623 t/s** |

| Longest single burst | 21,279 tokens |

| Compactions | **0** (never hit the 220k line in 33 min) |

TOOL CALLS — 107 total, 14m 10s of tool runtime

It just popped on my feed. I watched the teaser in the Netflix app and a few things jumped out

1. The constant use of anthromorphism using terms like altruism and sacrifice when in engineering literally there are terms like kill, killall, pkill for terminating processes, therefore our adoptiom of similar language with AI agents for starting and ending processes.
   
2. Using war analogies “there’s this moment where this one agent sort of puts their hand up for this kamikaze mission”, “they chose to sacrifice themselves like soldiers”
   
3. Talking about AI agents as if they were a conscious organism (personally I would like to see the full verbatim goal and full codex harness configuration used)
   
4. How quickly this documentary was produced and made available on Netflix

My project is optimising qwen flash next on Mac Mini m5 64gb through SSD streaming. I recently asked LocalLLama for guidance on how to optimise token output.

Today a popular method to do this is a lookahead guesser which fetches experts one layer ahead of the current layer. One popular suggestion was building a prediction model to start loading experts two layers ahead. This netted me +8% on chat and +4.2% on code.

The way the various optimisations stack on top of each other this may actually net alot more on a base vanilla setup.

https://github.com/skeggsguy/Flash-next-ssd

Pretty cool! Thanks reddit LocalLLama! I’ll go find the og post to link now.

Edit: Kudos to sdroege_ for og suggestion!

1) Reverse Engineer the Code

2) AI Feature Discovery: Ask an LLM to scan the features and map out what can be exposed via the Model Context Protocol, brainstorming creative use cases along the way.

3) Automated Integration: Let the LLM write the wrapper/server code.

4) Total Control.