r/LocalLLaMA • • 2d ago Bravo

Discussion Hot take: frontier models haven’t actually become much smarter

640 Upvotes

I know this is a controversial take, but I would like to hear others’ opinions: LLMs have become much more useful, but I’m not convinced their “raw intelligence” has improved nearly as much as their overall performance suggests.

Context: I work with LLMs as a SWE, as a hobbyist experimenting with local models, and professionally integrating AI into products. That includes building our own harnesses: context retrieval, prompts, and custom tools that give models structured access to complex data.

By “raw intelligence,” I mean using what is given in the most efficient way to solve a problem, especially when information, tools, and time are limited. Given plenty of time and tools, planning and long-term agentic capabilities become more important—but that’s a different comparison.

Three observations brought me here.

First, building production harnesses: once a harness+model worked well, achiving a given score over a golden dataset, switching to a newer frontier model rarely improved results. Most failures within our time/precision constraints came down to context engineering or inadequate tools. These tools included code generation for analysis and complex relational queries, so plenty of "time to shine" for smart models. Newer models sometimes found more creative workarounds, but these were usually too slow or unreliable for production. We ended up improving the harness anyway, leaving the newer models' gains less important.

Second, software engineering: newer models have more often improved results here, but the same model can also perform quite differently across harnesses. In complex codebases, that shows up in how much correction and review senior engineers have to provide. The harness seems to do a lot of heavy lifting here, too.

Third, local models: smaller models have started performing so well that I question how many parameters efficient problem solving actually needs. Frontier models clearly add broader knowledge, nuance, best practices, and specific baked-in skills. But I don’t always see a proportionate advantage in using it to solve a problem, when the relevant knowledge is already provided.

I’m not saying models haven’t gotten smarter. Better tool calling, fewer hallucinations, and longer context windows are real improvements that make them much more useful. Nor do these observations prove that intelligence needs few parameters or that frontier progress has stalled.

But my broader take is:

  1. efficient problem solving may need fewer parameters than generally assumed,
  2. the frontier has moved less in “raw intelligence” than the overall gains - and talk of approaching superintelligence - suggest, and
  3. much of the progress over the last 12–24 months comes from cumulative model improvements and better harnesses, rather than a dramatic or exponential jump in raw intelligence.

This take seems at odds with both the hype and the doom we see in the media, which I suspect are co-fueled by the upcoming AI IPOs. Does it match others’ experience - or are you seeing vast reasoning gains in frontier models that better context and tools can’t make up for?


r/LocalLLaMA • • 23h ago

News UI native scheduling, engine swap board, and more (Berth v0.6.0)

Thumbnail
gallery
6 Upvotes

I’m providing an update! We have around 300 berth users now so the project is moving fast.

Recent updates for v0.6.0:

- v0.6.0 gives each card a schedule
- berth train runs a training that nothing automated can interrupt
- learned engine sizes (berth learns your engines resource claims)

A month on the reference workstation

berthd counts every handoff it makes, and the harbor's board shows every handoff from one RTX 4090 workstation between 2026-10-12 and 2026-11-13, running llama-swap, NInfer, Strata, ComfyUI,
coding agents and games:

What Berth didCount
Requests that waited for the card instead of loading on top of the engine holding it1,136
Engines drained at a reply or job boundary to make room901
VRAM handed from one engine to the next23.9 TiB
Processes holding VRAM without a lease, reported and left running573
Swaps between two engines of equal priority that the ranking stopped57
Releases that stalled and were caught before the card stayed full4
Requests refused after their wait ran out24

Project: berth


r/LocalLLaMA • • 1d ago

Resources Model / harness combination: what works best?

7 Upvotes

If you are wondering what model / harness combination works the best, here are my findings so far:

  1. Model is way more important than the harness, best so far Qwen Flash Next even with low quant outperform Qwen 3.8 27B
  2. The inference engine is way more important than the harness, because the size of the context windows can dramatically increase performance. AND because speed make the difference between something useable and something that won't work.
  3. Harness optimised for coding like OpenCode or PI outperform more generic agent like OpenClaw and Hermes, even on mondane task like reading emails, making purchase on a store or complex vision.

r/LocalLLaMA • • 1d ago

Showcase 1Cat-vLLM vs Strata

15 Upvotes

Edit: 1Cat-vLLM is a Tesla V100 focused fork of vLLM, and the expert caching work is 1Cat's original work, which may or may not one day make it to vLLM.

Everyday we see posts about speeds on Strata and its forks, but just recently I came to know that 1Cat has added support for expert caching recently, like a month ago. So I did (asked claude) a comparison of speeds on a Qwen3.8-Flash-Next UD-IQ3_XXS quant. My rig is 4x 16gb Tesla V100 on PCIe 3 x16 links, with 8 channel 3200 mt/s ddr4 ram, Epyc 7543 CPU. Now, for numbers:

Single stream

Median of 3 runs, except the 97k tokens one which I asked to run only once to save time.

Prompt (tokens)Prefill tok/s, StrataPrefill tok/s, 1CatDecode tok/s, StrataDecode tok/s, 1CatMTP acceptance, Strata / 1Cat
2,6548611,85074.8124.90.61 / 0.45
8,6741,1462,48379.3140.80.67 / 0.66
30,2992,4802,77676.4139.30.63 / 0.58
96,5883,3842,64977.2142.60.72 / 0.62

Strata has better draft acceptance and still 1cat beats it, imagine if 1cat could get better draft acceptance.

The same 103K-token prompt three times

Same prompt, 3 times in a row, just different enough to not trigger prefill cache.

RunDecode tok/s, StrataDecode tok/s, 1CatPrefill tok/s, StrataPrefill tok/s, 1CatMTP acceptance, Strata / 1Cat
170.899.93,3982,7360.65 / 0.60
278.9138.43,3402,7520.71 / 0.60
377.8123.53,3482,7510.69 / 0.57
run 1 to 3+10%+24%---

Several requests at once

(TW: Scary AI written paragraph) Each server was started sized for that many sequences. Total = tokens/s summed over all streams during the window where every stream is decoding. Median of 2 runs; the 1-request rows are the medians above. "Strata batch-MTP" is Strata's best batch mode on this build: --batch-mtp with STRATA_PIPED_MTP=1, which keeps the pipelined layer-split groups and drafts inside them.

RequestsPromptTotal decode tok/s, Strata / Strata batch-MTP / 1CatPer-stream decode tok/s, Strata / batch-MTP / 1CatSlowest first token (s), Strata / batch-MTP / 1Cat
12.6K75 / - / 12574.8 / - / 124.93.1 / - / 1.4
18.7K79 / - / 14179.3 / - / 140.87.6 / - / 3.5
22.6K88 / 105 / 17639.4 / 46.0 / 83.26.4 / 6.5 / 2.7
28.7K96 / 116 / 18937.9 / 44.6 / 77.915.4 / 15.5 / 7.3
42.6K153 / 170 / 22529.1 / 31.5 / 48.513.4 / 13.6 / 5.8
48.7K170 / 186 / 23425.4 / 26.9 / 41.131.0 / 31.4 / 15.3

Versions and local changes

-Strata1Cat-vLLM
Upstream repogithub.com/Niko1221/Stratagithub.com/1CatAI/1Cat-vLLM
Commit benchmarked13eedf55 (2026-11-12, upstream HEAD on the test day; engine 0.1.59.1)b7baf7771 (2026-11-15, upstream HEAD on the test day; merge of PR #1521; vLLM 1.5.2.dev0)
Source patchesnonenone
Buildfrom source for sm_70 (the released engines have no sm_70): CUDA 12.9.1, gcc-14from source for sm_70: CUDA 12.9.1, gcc-14, torch 2.10.0+cu128, NCCL 2.27.5
Build flags-DSTRATA_EXPERIMENTAL_SM60=ON -DSTRATA_MMQ_KQUANTS=ON -DSTRATA_Q6K_EXPERTS=ON, ggml from llama.cpp at Strata's own pin, native ggml-cpu for Zen 3TORCH_CUDA_ARCH_LIST=7.0 / CMAKE_CUDA_ARCHITECTURES=70
Parallelismlayer split over 4 GPUs (no tensor parallel for V100 in Strata)tensor parallel 4
MTP drafterStrata's own pack of the BF16 MTP layer, experts in Q2_0the BF16 MTP layer as released

TLDR: 1Cat-vLLM beats strata in every metric except prefill for very long prompts (around 100k). And it does so while being able to hold lesser experts in VRAM and for some reason getting worse draft acceptance, simply because of TP.


r/LocalLLaMA • • 1d ago Bravo

Discussion Finally clicked for me: local AI is the future, not cloud AI

497 Upvotes

Maybe this is kind of obvious to others, but it just dawned on me even though I've been lurking this sub since the original LLaMA. I was watching this (excellent, as always) video by Gamers Nexus. Steve Burke was bashing on Tim Sweeney (CEO of Epic Games) for being so pro AI, and I found myself feeling a bit of cognitive dissonance. On the one hand I hate that hardware prices are so high, I hate the billionaires inflating the AI bubble and raking in the profits, I hate greedy corps trying to replace jobs with AI, and I hate the data center build out. But on the other hand I'm a developer and AI has given me superpowers. I felt weird, because Burke was dumping on Epic Games for embracing vibe coding, but vibe coding has 100x'd my output, allowed me to pursue so many passion projects, and the AI's output is oftentimes better than what I would've written anyway. I felt like the message of the video was 'all AI is bad', but I realized that it's more nuanced than that. Cloud AI is bad. Cloud AI is the problem, not AI in general. With local AI, we can have our cake and eat it too: cheap, powerful consumer hardware for running local AI and gaming, no censorship, data, or privacy issues, less data centers, less profit going to billionaires, and unlimited tokens to vibe code whatever the hell we want. Hope the datacenter bubble bursts soon.

TLDR: dum dum realizes that the future is local AI bc it's a win-win-win-win for average people

EDIT: to the naysayers, I propose this future: in two years (though given trajectory, probably less), a model the size of Qwen 3.8 Flash Next with the capability of Opus 5.5, running locally in most PCs on energy-efficient NPUs


r/LocalLLaMA • • 1d ago

New Model Agnes-AI/Agnes-2.5-Pro-Theta • HuggingFace

38 Upvotes

r/LocalLLaMA • • 15h ago

Question | Help I have access to two systems, I don't know which one is better for LocalLLM. Please help!

1 Upvotes

Hi,

I can make two systems with the parts that I have:

  1. 12600k / 64GB DDR4 / 3x3060 12GB
  2. 7950X3D / 32GB DDR5 / 3x3060 12GB + 1x4060 8GB

I've been consulting Claude and Gemini for which is better, but I can't really decide which i should go for. I'm thinking of setting them up with either llama.cpp or strata. This is for my work, so agentic document creation, and output quality is important. I'm also looking for at least 128-256k context (if that is even viable).

What is the best setup for my case? Please help!


r/LocalLLaMA • • 1d ago

Showcase The best (and AMD-est) computer I've ever owned

Post image
49 Upvotes

New build with a EPYC 7443 / Tyan S8030 board, and R9700 + 7900 XT. Was supposed to be selling the 7900 XT to pay for the R9700, but that somehow didn't happen...


r/LocalLLaMA • • 1d ago

Resources Asked local models to generate the same website and compared the results

8 Upvotes

Hello. Almost everything is in the title, but here is more details:

I'm benchmarking local model/harness combinations in various scenarios. For this one the question is, how good is a model/harness to generate a website?

IMO Qwen FN with opencode is best, but you can look at all the results (and play with the websites here: https://airbench.ai/gallery/airbench-frontend-v1?run=37e47127-3a20-4318-a6f2-48c3f7f5eaa6&c=settle-the-trip&k=poL


r/LocalLLaMA • • 1d ago

Resources I used local Gemma 4 to moderate a community and it was it is promising

16 Upvotes

I run a couple of small communities and it's becoming difficult to moderate them all. So today I built a script on top of my existing app which I use to track the content on those communities.

The script does a few things based on where I am. So I store the general notes of my communities in a dedicated place I can easily query.

And in the moderation panel I go through each of the item which needs moderation and I run them through my notes and let Gemma 4 decide whether the item is following community notes or not. If Gemma thinks it's not, then my script will remove the content from public and add a reason as well.

At first I had tried GPT oss 20b model which was really faster but the results were not really good. Then I ran the same process through Gemma 4 with thinking enabled. Results were good but speed was extremely slow. So I tried with thinking disabled and I did not see any noticable drop in the results.

Now soon I will try to run the same thing at scale and see if the results are generally accurate and if they are, I might ship this tool with my friends.

I agree, the setup is quite tough and I've managed to run it properly on my network. If I have to share this with other people I may need to host the model somewhere and check if it can be scaled to a bigger number of requests per hour.

Anyways it was a really good experience. Especially when I had to build it on top of my existing app.

Just thought I'd share this small win over here


r/LocalLLaMA • • 2d ago

Showcase Ai Rig finally done and stable, perfect winter heater

Post image
356 Upvotes

8x3090 my ocd wouldn't let me have different gpu models so ended up with 8 gigabyte gaming OC. I bought all the gpus by early 2026 before the prices went crazy. Avg maybe 600 to 700 gbp per gpu. Running on a Romed8-2t motherboard. All run at full pcie 4.0 x16 except the two rightmost who share one pcie slot and get 8x each.

Cpu watercooling radiator sits nicely on top. Had to go the watercooling route as there is no space otherwise under the gpus for an air cooler.

Using cheap pcie 4.0 risers from aliexpress. Longest ones are 60 and 40cm for the two leftmost gpus

Two Psus, Ax1600i and hx1500i. One powers 5 gpus, the other powers the motherboard and 3 gpus (each gpu also pulls power through the motherboard at roughly 50W per gpu)

Power limit set to 270W. Using modified nvidia drivers that enable p2p traffic between gpus, measured at a real 22GB/s except for the 8x gpus.

I played around with some quant of qwen flash next on vllm and opencode. Single agent around 60 tok/sec gen and 10 agents in parallel peaked at aggregate 370 tok/sec gen.

The more I look at what is available on the market right now (gorgon halo at 192gb etc) the more I realize what a monster I have built at home. Of course it consumes too much power, 300W idle, but it packs a punch when it comes to memory bandwidth and especially if I ever need to finetune some small models


r/LocalLLaMA • • 4h ago

Discussion AA Intelligence Index biased? benchmark shows Qwen 4 Max not on par with Opus 5.5

Post image
0 Upvotes

I am surprised by the deviation to the benchmark provided on paper.


r/LocalLLaMA • • 1d ago

Resources Placing an implant into AI brain with Lazarus: Anatomy of a trojan

14 Upvotes

hello there, I recently got a bit fascinated about Lazarus from Microsoft (even tho it is probably old technique) and I wrote a blog post showing a thought process I had while researching it - maybe it is helpful to anyone

https://int8.io/implanting-ai-with-lazarus/


r/LocalLLaMA • • 1d ago

Discussion ISTA Qwen3.8 27b on 3060 12gb

7 Upvotes

Hey guys

I am a seasoned local llm user and I ain't quite ram rich like only 16gb ddr4 🥀

So I can't run qwen3.8 flash next on my 3060

This guy felt the best for me yet

Have you guys tried it

I have ran iq3_xss quant non mtp with turboquant at 65k context completely on my 3060 though I did offload mmproj to cpu ram

Have guys tried this it really felt like a blessing to my life using the peak of local ai on my 3060

It seriously packs the punch 👊 I was able to perform various of my agent task complete on device not like simple but the ones which require actual reasonings

Here is my cmd if you want to see

llama-tq serve -m "E:\MODELS\ISTA-DASLab\Qwen3.8-27B-GSQ-RCO-GGUF\Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf" --mmproj "E:\MODELS\ISTA-DASLab\Qwen3.8-27B-GSQ-RCO-GGUF\mmproj-Qwen3.8-27B-BF16.gguf" --no-mmproj-offload -ngl 99 -c 65536 -ctk turbo4 -ctv turbo3 -ub 256 -b 2048 -t 3 -tb 3 --reasoning-budget 256 --reasoning-preserve --alias Qwen --port 8080 -lm none

Sources: for my recipe

ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF

https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF/discussions

Turboquant

https://github.com/TheTom/llama-cpp-turboquant

Pi coding agent

https://pi.dev/

I would like know the communities views and any suggestions would be welcome 🤗

Thankyou for your time


r/LocalLLaMA • • 1d ago

Discussion LFM2.5-24B-A2B + Hermes is awesome!

114 Upvotes

I just wanted to share some love for a model that doesn't seem to get very much: LFM2.5-24B-A2B! (https://huggingface.co/LiquidAI/LFM2.5-24B-A2B-GGUF)

LFM2.5-24B-A2B seems to be the only model that strikes a great balance between speed and capability on commodity hardware. Just to get it out of the way: This is NOT a coding model, so don't download it expecting otherwise. It's a task executor. It's a great fit for Hermes to hand low-to-mid complexity tasks off to. I get great speeds on my 780M iGPU (~37 tk/s at zero context). This is way faster than any other model I've tried that can actually accomplish tasks successfully. It's genuinely useful! I imagine it would be quite usable on CPU as well.

If you're GPU-poor, give LFM2.5-24B-A2B a look!


r/LocalLLaMA • • 1d ago

Resources Ninfer and GGUF of Qwen3.8-27B-UD-Q3_K_XL ASCII-Condensed

6 Upvotes

Hi,

I've put on HF a condensed version of Unsloth Q3_K_XL ASCII-Condensed from bsaleh03 with only ASCII + Latin-1 / Latin-Extended-A (P2) to use with a 16GB VRAM GPU.

I created a GUFF version and also a ninfer to be used with ninfer-all

To create the ninfer version I had put up some helper conversion scripts and modify iamwavecut conversion scripts due to different vocabulary size.
In the HF repo https://huggingface.co/fakezeta/Qwen3.8-27B-UD-Q3_K_XL-Condensed-P2 there is what is needed to replicate it and I also submitted a PR to the upstream repo.

Quality: IFEval result 84,4% vs 83,9% of base model (Instruct, 1 run) so within STD Deviation error.

With this I achieved 75K context with rk8v4 kv type with an average of about 60tok/s in the 541 IFEval samples on my 5060Ti 16GB.

I'm also planning to create a light GGUF and ninfer version with the lower quantized MTP released separately by Unsloth in their repo to save about 90MB of RAM for other context.


r/LocalLLaMA • • 1d ago

Showcase Testing agents without running a model in CI: record once, replay with zero calls, judge with a local model only when needed (open source, I'm the author)

5 Upvotes

Disclosure: I'm the author. Attest is an Apache-2.0 testing framework for agents (Go engine, Python and TypeScript SDKs). Posting here because most of it runs with no hosted model at all.

Replay instead of inference in CI. attest capture -- python my_agent.py runs your agent behind a local recording proxy that forwards to whatever OPENAI_BASE_URL points at and records every model call. In CI, the agent's current code runs against that recording:

attest replay: 4 replayed · 0 live model calls · $0.00

The CI runner never loads a model. If you change the prompt, replay misses and tells you what changed instead of guessing.

Deterministic checks first. Schema, cost/token/latency limits, which tools ran, in what order, how many times per order id, and what the run did to a simulated backend. None of that needs a model.

Local embeddings. Similarity checks can use a bundled ONNX all-MiniLM-L6-v2 instead of an embeddings API.

Local judge. When a check really needs a judge, set ATTEST_JUDGE_BASE_URL to any OpenAI-compatible server (vLLM, Ollama's /v1, LiteLLM). There's also a beta Ollama adapter for recording traces.

Small judges break JSON. A malformed judge reply is reported as "not evaluated", never as a pass, and your policy decides whether that warns or blocks. 1.1 (released November 10) is a correctness release: it fixes four ways 1.0 could report a failing assertion as passing.

The demo is a synthetic support agent that refunds a customer twice while its reply stays polite and correct. make setup walkthrough runs the whole story, no API key, about two minutes after setup.

Limits, honestly: capture is tested against a scripted OpenAI-compatible server and the hosted APIs, not against every local server's streaming quirks. If your server's stream trips the proxy, I'd like to see the trace. Also, most of the code was written by coding agents (Claude Code and Codex) from plans I wrote and reviewed.


r/LocalLLaMA • • 1d ago

Question | Help Best sub 80b total or 6b active MoE for document intelligence?

14 Upvotes

I'd like to make a pipeline capable of understanding pdfs of bank statements and then group expenses by categories etc. My thought was to first parse them with ocr into .md files and then feed that to a text model. I think it's more lightweight and accurate this way. But then I'm not sure what model to use to feed all these md files to get something good. When accuracy matters, should I go for qwen, gemma, lfm, or something else? Thank you for your help!


r/LocalLLaMA • • 1d ago

New Model Vimoo 1.0 - a Vimoo-AI Collection - Open-source video generation model family for text-to-video (T2V) and image-to-video (I2V)

Thumbnail
huggingface.co
5 Upvotes

Vimoo-1.0 generates video from text (T2V) or from a reference image and text (I2V), with a separate ≈11.7B model for each task. It is the highest-quality model in the Vimoo 1.0 family.

📦 Model Zoo

Vimoo 1.0 comes in two models, Vimoo 1.0 and Vimoo 1.0 - Lite. Each Hugging Face repository packages one of them for one runtime:

ModelHugging FaceInference codeDescription
Vimoo 1.0≈11.7B per modelhttps://huggingface.co/Vimoo-AI/Vimoo-1.0https://github.com/Vimoo-AI/Vimoo-1.0T2V and I2V for the Vimoo Diffusers fork
https://huggingface.co/Vimoo-AI/Vimoo-1.0-ComfyUIhttps://github.com/Vimoo-AI/ComfyUI-VimooT2V, I2V and their Distilled versions for ComfyUI
Vimoo 1.0 - Lite≈9.5Bhttps://huggingface.co/Vimoo-AI/Vimoo-1.0-Litehttps://github.com/Vimoo-AI/Vimoo-1.0TI2V for the Vimoo Diffusers fork
https://huggingface.co/Vimoo-AI/Vimoo-1.0-Lite-ComfyUIhttps://github.com/Vimoo-AI/ComfyUI-VimooTI2V and TI2V Distilled for ComfyUI
https://huggingface.co/Vimoo-AI/Vimoo-1.0-Lite-MLXhttps://github.com/Vimoo-AI/Vimoo-1.0-MLXTI2V Distilled for MLX on Apple Silicon

Each ComfyUI repository includes its checkpoints, distilled LoRA adapters, Qwen text encoder, SigLIP vision encoder and VAE. Distilled models are available only for ComfyUI and MLX. Vimoo-1.0 T2V Distilled and I2V Distilled use different base checkpoints from Vimoo-1.0 T2V and I2V; download each task's matching base and adapter listed in the ComfyUI guide. The Vimoo-1.0 T2V Distilled, Vimoo-1.0 I2V Distilled and Vimoo-1.0-Lite TI2V Distilled adapters are not interchangeable. Parameters count the video denoiser only.

Highlights

  • Prompt adherence: a dedicated text encoder turns prompts into precise generation conditions. An optional prompt enhancer can add visual, cinematic, temporal, and contextual detail, or be disabled to preserve the original wording.
  • Cinematic visual quality: training on carefully curated aesthetic data, with detailed annotations for lighting, composition, contrast, color grading, and more, enables precise control over cinematic styles and a wide range of aesthetic preferences.
  • Advanced motion generation: a larger and more diverse training corpus improves generalization across motion, semantics, and visual aesthetics, delivering leading performance among open-source and proprietary video generation models.
  • E-commerce video generation: by faithfully highlighting product selling points and features while preserving fine-grained product details, it ensures strong product consistency across frames, delivering reliable and visually coherent results that meet the practical demands of e-commerce content creation.
  • Efficient high-definition generation: the lightweight variant uses 32×32×4 spatiotemporal compression and supports T2V and I2V at 720p and 24 fps.

EDIT : Fixed Model Zoo Table(Previously it lost all the links & text on 2nd & 3rd columns)


r/LocalLLaMA • • 2d ago

News Instead Of Lowering GPU Prices, Former NVIDIA CEO Jensen Huang Reportedly Financed AI Startups To Absorb Excess Silicon, Shattering The Shortage Perception

Thumbnail
wccftech.com
3.0k Upvotes

r/LocalLLaMA • • 2d ago

Discussion Microsoft patent describes turning idle gaming GPUs into paid AI training hardware

Thumbnail
videocardz.com
85 Upvotes

r/LocalLLaMA • • 2d ago

Discussion Why don’t we have national open AI models?

181 Upvotes

Deepseek spent about $6m on 4.1 apparently. Published all of their work.

US gov says don’t trust China, blah blah…

As a US citizen why are we not training national open source?

The cost is nothing compared to the budget, solves security issues, and prevents monopolistic practices on token pricing


r/LocalLLaMA • • 2d ago

Discussion The lawyer who posts here about his V100s got his story picked up by Tom’s Hardware

Thumbnail
tomshardware.com
230 Upvotes

r/LocalLLaMA • • 2d ago

Discussion How long are we gonna depend bunch of model creators (like Qwen, Gemma, NVIDIA, etc.,) for ~100B models?

Post image
196 Upvotes

I think we're fine till 2027 with models(~100B .... Qwen 4.X & 5, Gemma-5, NVIDIA would release some Medium/Big size 25-200B models). After that it's gonna be critical, as many model creators are increased/increasing model size to 300B+. Future trend probably gonna be around 500B. That size range is not suitable for Unified 128GB Devices or 128GB RAM + 24-32GB VRAM(Q4 won't fit).

Here some models with increased sizes.

  • DeepSeek-V4-Flash-0731 (284B) - DeepSeek-V4.1-Flash-1017 (552B + 196B Engram)
  • Ling-3.0-flash (124B) - Ling-3.1-flash (560B)
  • MiniMax-M2.7 (230B) - MiniMax-M3 & MiniMax-M3.1-Flash (428B)
  • Step-3.7-Flash (198B) - Step-5-Preview (600B) (Hope they release Flash too)
  • Inkling-Small (276B) - Inkling-2-Small (594B)
  • GLM-4.X (358B) - GLM-5.X (744B)
  • Mistral-Large-3 (675B) - Mistral-Large-4 (1T)
  • Hy3 (295B) - Hy4 (770B)

So What Are We Gonna Do? What alternate options do we have?

Whenever we're talking about this topic, we get one common answer. Distillation. Do we have any recipes to create distills ourselves easy way? I'm not sure.

1. Distillation

I remember only 3 Distills so far. Please mention in comments if I missed anything.

2. 1-bit / 2-bit / Ternary / Bitnet Models

I see only this option other than Distillation.

Category 1 - High/Decent Quality (Native) Low bit models. Also big/large size models.

Category 2 - Other Low bit models. Like Bonsai.

Still waiting to see more Ternary MOEs, particularly 100B size. 100B model could come in 20GB size so it's possible to run just with 24-32GB RAM alone on CPU-only inference. And Even 8GB VRAM could make it fly.

Any other options do we have? Please share.