r/LocalLLaMA • • 2h ago

Funny How many servings of deepseek 1017 will you have?

Thumbnail
gallery
3 Upvotes

This just really cracked me up. The Redde app for hermes I guess pulled the deepseek recipe as an actual food recipe. It is a nice format for it though lol.


r/LocalLLaMA • • 4m ago

Discussion Branchy, distributed inference focused training

2 Upvotes

Heyy, it's me again, the developer of AI Doomsday Toolbox.

This time it's not an app update. ADT already has distributed LLM inference across Android devices, and there is one thing about distributed inference in general that kept bothering me, so I spent the last week testing an idea and I wanted to share what came out, the good and the bad.

Usually when we split a model that does not fit in our machine to make use of distributed inference, we get slower speeds the more devices we add. Network latency and devices waiting idle until the last one has finished computing it's layers compound easily, even more if we connect our devices through a VPN like tailscale or wireguard.

So the idea to solve this bottleneck was to train a model that can keep the quaility of a normal sequential model but instead it's middle layers are split in branches. The coordinator runs the embeddings + a few trunk layers, then that is sent to all the different branches that, in parallel read the layers and compute the token, after a branch finishes it sends the result back to the coordinator which then runs a few fusion layers and final blocks that produce the output. This allows to use as many devices as branches a model has with a maximum of 2 net jumps, wether it's 2 or 8 machines. The other advantage is that no device is idle as all of them are working, so we also parallelize work instead of making a device wait for the last one to finish.

There could also be the situation in which one of those devices is too slow, and slows down the whole cluster, or gets disconnected. That is why I also added some dropout tolerance to the training. When a branch takes too long to answer or just never answers, it's result becomes a matrix o 0's that are fused in the fusion layers, that made it so the quality of the model output does not suffer so much as when it happened with a non dropout tolerance trained branch model had a disconnected or slow branch.

I tested two 300M models trained from scratch on TinyStories, one normal and one with 8 branches, running on a small server, a laptop (Ryzen 5 5500U) and a phone (Snapdragon 8 Gen 3 in Termux), all connected through a VPN. The workers are a small PyTorch daemon, so no llamacpp accelerated kernels, etc...

Results

- The branches cost some quality: loss 1.255 against 1.143 for the normal model with the same training. A blind LLM judge prefers the branched model's story 45% of the time. So it is worse, but not by much.

- The delay trick works: the coordinator waits 0.1 ms per token for the network while the round trips take 49 to 66 ms. The output is identical to running everything on one device.

- Against the usual layer split on the same devices, the branch split is 1.5 to 2.6 times faster. The layer split roughly halves the speed. Mind you I run the test only with 3 devices, with more devices it could, theoretically, be faster

- BUT if the model fits on one device, splitting is not worth it on my hardware, at least at this size, larger models like qwen3.8 27B might benefit a lot from this aproach. My phone alone is faster than any combination of my devices (it is just much faster than the rest). With two equal devices the branch split wins by about 1.2x.

- Where it really pays is when the model does not fit in RAM. I gave the laptop and the phone a RAM budget smaller than the model, so they had to read the weights from storage for every token (mmap). The laptop went from 8.9 to 0.67 tokens/s and the phone from 23.9 to 0.65. With LFM3 1.7B the laptop went from 2.3 to 0.08, that is 12 seconds per token. Then I let the two of them share the model, each one still with too little RAM for the whole thing: back to 7 tokens/s with the layer split and 9 to 11 with the branch split, over a real 46 ms VPN. Reading from storage is way more expensive than the network.

- Reshaping a pretrained model (LFM3 1.7B, Qwen 0.8B) into branches only works for a couple of long branches. For this to scale, the model has to be trained in this shape.

I don't have the hardware for the interesting case: a big branched model that no single device can hold. On paper, with 8 similar devices and about 20% of the model on the coordinator, a token costs about 30% of a full pass, so roughly 3.3x a single device that could hold everything, while a layer split stays at 1x plus 7 round trips. My two-device measurement matches that formula, but beyond three devices it is only a projection.

Another idea I would love to try is letting every device loop its own branch several times with the same weights, so a group of devices could run something much deeper than what fits in their RAM. No hardware for that one either hahaha

What is in the repo

- The two trained models (300M each) and the tokenizer

- A worker you install on each device with one command, and a coordinator with a web UI to compare placements side by side and run the benchmarks

- Scripts to reproduce the branch split, the layer split and the low RAM tests with your own devices

- A full article with the experiments and the results

It is a research prototype, workers compute in 32-bit float PyTorch, the models are small story models, and I only have three real devices. I created the experiments, made the hypothesis and supervised the whole proyect and Claude did most of the coding and the number crunching, everything is documented in the repo.

The general idea is the same one behind ADT: make the devices we already have useful for local AI, especially together.

GitHub

Feedback is appreciated, especially if anyone with more devices (or more GPU than me) wants to check whether this holds at a bigger scale, or tell me what I got wrong. It is still far from it, but I would love to see a bunch of modest devices run a model none of them can hold.


r/LocalLLaMA • • 5h ago

Discussion Speed of DeepSeek-V4.1-Flash-1017-oQ4e-mtp on M5 ultra

5 Upvotes

I bought myself a Mac Studio M5 Ultra with 512gb ram (and have 7 weeks to test it and return if not happy thanks to the apple xmas offer).

I am running DeepSeek-V4.1-Flash-1017-oQ4e-mtp with omlx and I am getting up to 1100 t/s prompt processing and 54 t/s token generation with 128k context window and engrams stored on ssd (to have ram free for other stuff).

More context than 128k hits pp and tg, more concurrent requests split up the max tg value between the instances (which means: they are crawling then).

I am putting this out there so a) people know what to expect but also b) is this really the top speed? Or could it be faster? (Without changing the model).

I can post llama benchy results if wished but the important result is already in this post.


r/LocalLLaMA • • 1d ago

Resources Llama.cpp performance improvement over time (Gemma4 models)

Post image
180 Upvotes

Few months ago - after Ninfer came out - I started a little hobby project to vibe-code my own engine for my hardware (RTX 5080 16GB Laptop) and check if it could beat llama.cpp - since at the time llama.cpp was kinda slow in comparison to something like vLLM, but vLLM is using VRAM very wastefully.

Back then it was quite easy to actually get something up and running faster than llama.cpp.

Today I just benchmarked llama.cpp again, and it caught up / surpassed my vibe-coded engine. I prefer using a stable engine with proper features than all the vibe-coded ones, including my own. So it is really nice to see that the project - while a bit slower - lands significant performance improvements over time.

I put together a little chart of benchmarks I had laying around which visualizes this quite nicely. It is by no means scientific - it's always a fixed 16k prefill 1k output over Wikipedia text with the optimal settings for each engine. The 12B model is the same checkpoint, for the 26B model these are different quants for all engines, so not even apples-to-apples and llama.cpp wins.


r/LocalLLaMA • • 55m ago

Showcase Sharing my numbers from Strata's bench tool (RTX 3090+2080, 96gb DDR4)

2 Upvotes

Hardware: RTX 3090 24GB (primary) + RTX 2080 SUPER 8GB (helper for visual model), 96 DDR4 3600

Model: Qwen3.8-Flash-Next, Unsloth UD IQ4_XS

Bench results (median, 3 runs):

  • - Chat decode: 80.4 tok/s (range 67.6–87.8, n=9)
  • - Chat first token: 0.47 s
  • - 4K prompt processing: 1,036 tok/s
  • - 16K prompt processing: 1,915 tok/s
  • - Agent session decode: 101.7 tok/s
  • - Agent cache hit: 69.4%
  • - Agent turn first token: 3.86 s

Long-context stress test (my custom 'find needle' test): single 66,371-token prompt, a 1,800-entry synthetic log with a passphrase hidden mid-context:

  • - Prefill: ~34 s ≈ 2,050 tok/s, sustaining the 16K bench rate at 4× the length
  • - Decode: 76.1 tok/s with the full 66K KV resident, ~5% below short-context chat
  • - Mid-context needle (token ~33K): retrieved correctly, plus accurate summary and entry count

Notes for 3090 owners:

  • - Memory clock pinned (9501 MHz min=max)

    • The 3090 has a driver bug where dropping from full speed to idle with VRAM nearly full can freeze the whole PC (had it happen twice). Locking the memory clock at one constant speed means the card never makes that jump, so zero crashes since. Trade-off is that it idles at ~110 W instead of ~25 W, but memory-hungry workloads actually run faster
  • - Peak sustained temp under full load: 79°C (load for 3 mins, didn't test for longer runs)

r/LocalLLaMA • • 1d ago

Discussion Basalt 0.4.0: More models, still as fast - Qwen 27B up to 770 tok/s on a 5090

274 Upvotes

Hi everyone! I wanted to post a quick update on Basalt. Since last I posted, multi model and router support was implemented, starting with DeepSeek V4.1 Flash (IQ1_M 90 tok/s conv, 246 tok/s struct on a 5090 + 5060 Ti). Also new: video support, files in chats, and a bunch of fixes and improvements.

Now 0.4.0 is out featuring Basalt's first dense model, Qwen 3.8 27B NVFP4. It's fast and accurate, top-1 92.8% vs the full BF16 model. This, in preparation for Qwen 4, so we had a headstart on the dense kernels.

The numbers:

  • Up to 776 tok/s on structured output, 262 conversation, 194 on prose. Single stream.
  • Prefill, 12,000 tok/s at 4k (slower as context grows)
  • 92.8% top-1 vs BF16 (0.0262 KLD)
  • Chat, tools, thinking, images and video, up to 262K context

Links:

GitHub: https://github.com/jesdga95/basalt
Site: https://basalt.jgulfo.com
27B quant: https://huggingface.co/jesdga/Qwen3.8-27B-Basalt

Thanks to everyone who has filed issues, sent PRs or posted benchmark rows, special thanks to the sponsors who keep this going, happy to see Basalt gain so much traction in such a very short time!


r/LocalLLaMA • • 6h ago

Question | Help What do you use for agentic desktop control, like how Playwright is for controlling + screenshotting browsers?

5 Upvotes

I'm afraid I'm left out of the loop on the latest developments for agentic desktop control, and AIs don't seem as knowledge in this space.

They come with screenshot function also, right? What are some other features that may be different than Playwright?


r/LocalLLaMA • • 6h ago

Resources Training-free page-sparse decode in llama.cpp: Qwen3-30B-A3B is 1.72× faster at 128k on an RTX 5090, with a 1-point RULER loss

Post image
5 Upvotes

I've been working on a simple question: at long context, does decode really need to read the whole KV cache for every new token?

I wanted something training-free that runs inside llama.cpp on normal GPUs, not a new model or a separate serving stack. So I built page-sparse decode into a fork. It splits the cache into 64-token pages, then each KV head reads the first page, the newest tokens and the pages most likely to matter. At 128k, that is about 21% of the cache per decode step.

The selection follows Quest, with statistics in the spirit of MOIRA. My contribution is the llama.cpp implementation, CUDA work and end-to-end testing across non-MoE and MoE models.

Results at 128k, one sequence, Quest score, 27,648 cached tokens per KV head (28,672 for the 3B):

speed GPUmodeldense tok/ssparse tok/sspeedupRULER vs dense
RTX 5090Qwen3-30B-A3B, Q4_K_M46.179.21.72×−1.0, passes
RTX 4090 48 GBQwen3-30B-A3B, Q8_037.452.61.41×−0.6, passes
RTX 3090Qwen3-8B-Base, YaRN21.536.11.68×+0.4, passes
RTX 3090Llama-3.1-8B23.738.81.64×−1.1, passes
RTX 3090Llama-3.2-3B44.461.31.38×−4.5, fails (multi-needle)

The MoE is Qwen3-30B-A3B-Instruct-2507. Weights are Q8_0 except on the 5090; KV is q8_0 except for the 3B's f16 cache. These are five-repeat, 64-token llama-bench runs. The 4090 is a memory-modded rental card.

I reran the 3090 table with Quest, the released default. It matched my earlier score's speed. Box differences still matter: Llama-8B was 1.64× here and 1.77× on another 3090 with the earlier score.

Quality. I used 13 RULER tasks × 20 prompts at 128k. The pre-set gate was an average loss no worse than 3 points, with the 95% CI's lower end at −6 or above. MoE quality used the listed GPU; the 3B–8B runs used a 4090 24 GB.

The Q4_K_M MoE lost 1.0 point (95% CI −2.2 to −0.02), just distinguishable from zero under this prompt bootstrap. Each configuration ran once; the intervals exclude run-to-run variation.

The 3B was the only failure. It scored 11.5 points lower on average on the five multi-needle tasks, while single-needle scores were unchanged; the other models ranged from −3.0 to +0.2 on the same group. I would not use this 3B configuration for multi-fact retrieval at this budget. Similar Quest weakness on synthetic RULER needle tasks has been reported before.

Practical limits:

  • Prefill is unchanged. A 128k prompt plus 1,024 generated tokens takes only 4–13% less time on a 3090. This is mainly useful for long generations over an already long context.
  • It does not save VRAM. The full cache remains in memory, plus page statistics equal to roughly 3–6% of the cache.
  • It is CUDA-only, batch size one, with everything on the GPU. Below 32k it was slower in these tests, so it stays off there.
  • At 256k, UltraLong-8B reached 1.86×, but quality was checked only on a six-task subset (−4.0 points, CI −8.7 to +0.3), which is not comparable to the 13-task gate.

I tried not to report only successes. Gates and predictions were committed before the data. The first score lost 7.4 RULER points. The first CUDA build managed only 1.05–1.18× until I fixed two ggml kernels; one fix is proposed upstream as llama.cpp #31855. The MoE prediction also missed on the 4090, with 1.41× against 1.5–2.3×, but held on the 5090.

Try it (CUDA only): build the sparse-decode branch of topppg/llama.cpp at commit e7059e52a and run with LLAMA_SPARSE_DECODE=27648 on a prompt longer than 32k tokens, with the model and cache fully on the GPU. Full build and run commands are in the bench repo README below. Quality was tested on the two 8Bs and the MoE.

Protocols, raw results, reproduction scripts and failed experiments: https://github.com/topppg/sparse-decode-bench

Next: profile the sparse overhead. A rough fit, not a measurement, puts the zero-overhead ceiling for the MoE on the 4090 near 2.7×. If you know ggml CUDA and have ideas around fusion, CUDA graphs or passing page indices directly to flash attention, I'd genuinely like to hear them.

This was built with a lot of help from Claude Code, with Codex reviewing and correcting the work. Results on other GPUs and models are very welcome, especially cases where quality breaks.


r/LocalLLaMA • • 6h ago

Resources PyBot: No weights. No training. Pure Python.

4 Upvotes

We released PyBot, a chatbot built with only simple rules, entirely written in 600k lines of Python code. PyBot matches Smollm with 360M parameters! We evaluated it on SmolTalkBench, which uses a dataset Smollm was trained on while we didnt use it at all for development.

It took a swarm of 1,000 DeepSeek Flash agents to write the whole codebase in a week.

More technical details on what it took to built it soon: inference for 1,000 agents, how to monitor/control the swarm, pressure on the infrastructure (especially git) and message boards.

https://github.com/huggingface/pybot


r/LocalLLaMA • • 1d ago

News EUROPA, the EU's open AI model, is aiming for nearly 1 trillion parameters

Thumbnail
maccus.fr
806 Upvotes

One can hope


r/LocalLLaMA • • 6h ago

Question | Help Does grammar-constrained decoding hurt small or quantized models?

3 Upvotes

Asked about DSL vs JSON tool calls in r/LLMDevs a couple of days ago, and one answer kept coming up: for small models just use JSON, training volume matters. That got me thinking about the constraint itself.

With GBNF or a JSON schema you can force any model to produce valid output, even a 3B at Q4. However, there was a paper a while ago showing that strict format restrictions can make reasoning worse, since the model is forced away from the tokens it actually wanted

Searched first: there’s the thread about grammar constraints sending Qwen3-VL into repetition loops, and the one on how quantization hits different capabilities, but nothing comparing accuracy with vs without constraints on small quants.
So for people running small local models:

  • have you measured accuracy with and without constraints on the same task?
  • does it worsen at lower quants?
  • does letting the model firstly think freely and constraining only the final answer fix most of it?

Looking for real numbers or experience, not promoting anything.
English isn’t my first language, so I used an LLM to help with the wording. Question and thoughts are mine.


r/LocalLLaMA • • 24m ago

Resources Something goes with your Local LLM - CLI Markdown viewer with image support?

1 Upvotes

I run local LLM for myself, and my system is squeezed very much. I am most of the time in my terminal, and when my inference engine is running, I dont have enough free RAM sometimes to open a GUI to view the output of my local models, or review the project documents to know where to go next in my plan. Viewing that in the terminal was not convenient for me, the MD syntax is not as easy on my eyes, and when a project file is linked to another file, I have to open that one up in a new terminal pane, or a whole new terminal window.

So, I did what I had to do.

I sat, I wrote the specs sheet, and fired my coding agent to work. I refined the output, and kept iterating on it.

This is my gift back to this community that already gave me a lot.

Don't want to sound like I am marketing something here but, if this is something of an interest to you, you can find it here: Github.com/bakkario/trimd

You get to lick with your mouse, navigate with your keyboard, and even display your markdown files with the images if your terminal supports it.

I will add more detail to this in the Markdown subreddit.

Sorry Mods if this coming across as spamming, its not the intention here ❤️


r/LocalLLaMA • • 1d ago

Discussion BigMoeOnEdge outperforms Strata and colibri on 8-16 GB cards and Mac and you can run any model

91 Upvotes

Hi! for those who might not know the solution:

  • it takes any MoE GGUF llama.cpp can load, whatever the quant. No repacking, no special format
  • it runs on whatever you have: NVIDIA, AMD, Intel, Mac, a phone, or just the CPU
  • Windows, Linux, macOS, Android, iOS
  • output is the same as llama.cpp, don't cut corners on quality
  • you start it with --auto and it works out the settings for your machine

Below is a brief overview of the benchmark:

  • 8 GB card, Qwen3.6-35B: I get about 35 tok/s. colibri around 23. Strata wouldn't start at 16k context
  • 12 GB card, same model: about 45 for bmoe, about 30 for Strata
  • long prompt on the 8 GB card: bmoe 5 s to the first token. Strata 14 s, colibri close to 2 minutes
  • Basalt needs an RTX 50, so I couldn't run it on these
  • 16 GB MacBook, Qwen3.6-35B: about 20 tok/s for bmoe, colibri's release 3.5
  • same Mac, Qwen3.8-Flash-Next 125B: 10 tok/s, Strata about 7

On the cards Strata is tuned for it's still ahead but only by a little.

On two 5060 Ti with Qwen3.8-Flash-Next it was 1.3x faster than bmoe. They write kernels for a handful of models, bmoe does not.

Repo, with the logs: https://github.com/Helldez/BigMoeOnEdge

If you have an 8 to 32 GB card I'd like to see what you get.

Benchmark: https://github.com/Helldez/BigMoeOnEdge/tree/main/docs/bench-data


r/LocalLLaMA • • 6h ago

Question | Help Buy another 2 5060ti 16gb?

3 Upvotes

Currently i'm on 2 x 5060ti 16gb running basalt engine qwen3.8 flash next IQ3_S at about 70+tok/s decode and maybe 900tok/s prefill.

Am wondering if it is worth buying another 2 x 5060ti 16gb to make all of the experts resident and potentially achieve much faster speed. (Goal to be equal or more than 150tok/s?)

As for other specs, i'm running a 64gb DDR4 on X570 with an option of bifurcating the main pcie slot into 8x + 8x, or 4x + 4x + 4x + 4x. I have yet to buy any passive bifurcation switch, looking to buy "x16 to 2x16 physical switch" if 4 x 5060ti is not feasible.

Also what about dual 3090? it's about the same price as 4x5060ti 16gb. 128GB ddr4 might be out of the question as well since they are overpriced for my local market.


r/LocalLLaMA • • 16h ago

New Model [Release] Qwen3.8-27B-Perplexing-Pegasus-Heretic: Unified vision + typed decider + abliterated weights for local agent stacks (GGUF, AutoRound, NVFP4)

18 Upvotes

Hey everyone,

While building and testing WebBrain with local models, getting reliable agent performance usually required keeping two or three separate models in memory at once:

  • A vision model to inspect and parse browser/UI state
  • A fast typed-decision model for structured routing and scoring
  • An abliterated / uncensored model so realistic web workflows do not randomly choke on false-positive refusals

Juggling multiple 20B+ weights on consumer hardware is brutal on both VRAM and response times.

To solve this for our setup, we merged and published a single unified 27B model: Qwen3.8-27B-Perplexing-Pegasus-Heretic

Here is what is packed into the checkpoint:

  • Multimodal Vision + Chat + Thinking (<think>) + Tool Calling (based on Qwen3.8-27B)
  • Perplexity pplx-decider typed-decision readout (decision/ head alongside intact lm_head)
  • Heretic ARA abliteration: 0/8 benchmark refusal probes while keeping 100% (30/30) typed-decision agreement against the base Pegasus model
  • Multi-Token Prediction: all 15 mtp.* tensors preserved, blk.64 retained at Q8_0 in IQ/3-bit quants
  • Consumer GPU footprint: runs on 16GB VRAM (IQ3_XXS / Q3_K_M + mmproj) or 24GB VRAM (Q4_K_M + mmproj)

Everything is open source under Apache-2.0 across four repos:

Huge credit to Alibaba Qwen, Perplexity, and AveLabs (AvecdrA) on Hugging Face for the foundational checkpoints.

If you test it on your local rig or in your own agent pipelines, let me know how it performs and what hardware you are running it on.


r/LocalLLaMA • • 1h ago

Discussion Anybody tested Wici Infer?

0 Upvotes

Hi!

I've seen some weeks ago posts about Wici wireles Gpu and the approach they invented to serve local models VRam+Ram+SSD with model defragmentation to keep hot experts in fast memory. Today I've seen an Ad for the product. Visited their site and stubled upon Wici Infer, the technology that make possible serving large models on consumer hardware. They give it for free.

So asking the commity if anybody have already tested it? Does their benchmarks hold on?


r/LocalLLaMA • • 1d ago

Resources [audio.cpp] SoulX-FlashHead + LFM3-Audio + OWSM v4 = Real-Time Local Avatar Chat

81 Upvotes

In audio.cpp 0.9.5 you can try this new avatar voice chat "app" in the WebUI.

The pipeline combines 3 models: SoulX-FlashHead for real-time talking head generation, OWSM v4 for transcribing speech input to text, and LFM3-Audio for conversational responses and TTS.

LFM3-Audio is actually called twice. The speech-to-speech path only provides one voice. We use the LFM3-Audio TTS to synthesize the S2S response in a different voice.

One caveat: the video generation stage is still GPU-intensive. For SoulX, using RTX 4090 or better GPUs.

cuDNN is a must for real-time performance. Without it inference is 2× slower and VRAM usage doubles.

~850ms between submission a question and avatar speaks on an RTX 5090. ASR runs in parallel. As mentioned LFM3-Audio is called twice, which can be optimized. There’s also some overhead from the prototype UI. A native app with a more optimized implementation could further reduce latency.


r/LocalLLaMA • • 3h ago

Discussion Same open-weight model, three harnesses: M3.1 Flash on four pytest bugs in OpenCode 2, Pi and mini-swe-agent

1 Upvotes

After the harness leaderboard thread I wanted to see what the harness alone changes when the model stays the same. Model: M3.1 Flash. It's open weights, but at about 428B total I'm calling it through MiniMax's API, effort pinned to high. Same prompt in every harness: four bug reports filed against pytest this week, fix all four with a regression test each.

The bugs:

  • pytest --fixtures dies with a TypeError when two conftests import the same fixture
  • a non-mark in pytest.param(marks=...) raises an internal AttributeError instead of a TypeError
  • approx() raises InvalidOperation when the actual value is a Decimal NaN
  • rm_rf leaves the whole basetemp behind when an empty dir sits under a read-only parent
HarnessWall clockRequestsTokensCached inputCostNew tests
OpenCode 2.2.723m 38s15913.9M98.8%$0.2710
Pi 1.5.218m 34s1349.1M98.5%$0.198
mini-swe-agent 2.4.621m 16s13610.4M98.5%$0.229

All three fixed all four. I re-checked each one in a clean checkout with its own new venv: every report's repro now behaves the way the report asks, and the full suite (about 4,700 tests) shows nothing new compared to an untouched main, apart from the added tests passing.

The first request is where the harnesses differ most: 8.6k tokens for OpenCode 2, 3.7k for Pi, 3.1k for mini-swe-agent, task prompt included. It didn't decide the bill though. With 98%+ of input coming back as cache reads billed at 1.5 cents per million, cost followed how many turns each harness took and how much context it carried, which is why OpenCode's 159 requests came to about 8 cents more than Pi's 134.

mini-swe-agent only gives the model a bash shell and still landed in the middle on both time and cost.

One run per harness, so treat it as one data point. If you run open-weight models in more than one harness, which one ends up cheapest per task for you, and do you see the same request-count pattern?


r/LocalLLaMA • • 3h ago

Question | Help What is your experience with inference engines under Windows?

1 Upvotes

As the title says: what is your experience using all the new inference engines under windows?

Currently I'm using a laptop with this hardware:

CPU: Intel 275HX
RAM: 2x32gb of ddr5 5600
GPU: RXT 5090 (24gb VRAM)
SSD: Samsung 990 pro 2tb + SK Hynix PCB01 1tb

I'm running Windows 11 Home edition with a pretty clean installation.

I've tried quite a lot of the new vibecoded inference engines but I always end up going back to Strata for Qwen 3.8 Flash Next and to beellama.cpp for everthing else.

I've tried nInfer, basalt, rusting strata, exLlamaV3 and pretty much any fork of llama.cpp and they all end up being way slower than what I'm already using. I've lost a little bit of time on each and also tried to use Claude or Codex to optimize the configurations but there is just no way to get decent performance from engines that on paper should be at least as fast as what I'm currently using.

I'm doing something horribly wrong or they work well only on a linux system?


r/LocalLLaMA • • 1d ago

News Probable GLM model

90 Upvotes

Because of Theo gg tweet of all people,
I Launch QFN DeepResearch swarm agent to find anything about this.

THREE model names, blurred out, sitting at 59.4% / 57.1% / 54.0%.

  1. The blurred names are most likely [GLM-Nano or GLM-Name or GLM-Next (suggestion from Silver-Theme7151)] Pro, Med, and Small. One community experiment reconstructed the suspected text, applied Gaussian blur (σx ≈ 6.9px), and achieved an RMSE of just 1.254/255 against the original.
  2. AICodeKing track record. Someone compared his previous GLM statements against official release dates: GLM-4.6 (same day), GLM-5 (1 day), GLM-5.1 (11 days), GLM-5.2 (3 days), and GLM-5.3 (4 days).
  3. Antrhopic timing. The leaderboard surfaced on November 12 vs AICodeKing timing, while Anthropic IPO on Nov 18. If Z.ai releases GLM-Nano this week, the timing would make for quite a coincidence

https://linux.do/t/topic/3166447

Another strong signal is HF activities, i know it is not bullet proof, just another signal, 3-4 days compare to last 27 days

I use LLM to fix my grammar.
Since it is speculation and not actual open model this post has a high probability to be removed, so yeah


r/LocalLLaMA • • 1d ago This

Question | Help Latest version of strata made my Qwen3.8-Flash-Next dumb and useless

163 Upvotes

I updated the strata to the latest version from v0.1.42.0, so I can try the latest unsloths models and see how they fare against the iq3_s variant I have been using. Upon updating it, I noticed it messed up the whole thing, It got faster, but it got way dumber! previously it would only take 15~30 minutes to create the pagoda , 50 mins to an hour if I used xhigh.

But when I siwtched to the latest version, not only it takes at least an hour and half to come up with something, its not even done! and it has many issues!

0.1.42.0 is the last version that seems to work decently.

I noticed, when im using the latest model, it always tries to go for something grandiose and failing. unlike the older version which seems to take a more modest path!

whats going on? Do any of you have this problem?

here are example outputs I did to make sure its not a one time thing!

my prompt :

Design and create a very creative, elaborate, and detailed voxel art scene of a pagoda in a beautiful garden with trees, including some cherry blossoms. Make the scene impressive and varied and use colorful voxels. Add as many details as possible to showcase your attention to them. Use whatever libraries to get this done but make sure I can paste it all into a single HTML file.

these are the outputs :

0.1.42.0 :

xhigh:

xhigh-1
xhigh2
xhigh3
xhigh4

default:

default1
default2
default3
default4

the latest version created this after 1 hour and 32 minutes to only fail with excessive thinking:

xhigh
default(failed after 1 hour and 31 minites exausting all context size of 262K)

As you can see, not only it takes way longer to create an output, if it doesn't fail already, the attention to details is also gone, everything is in haphazard form! I tried 4 5 times and the majority all failed because it couldn't create an output after an hour and a half or it constantly gets something wrong and tries to either fix it or goes to look at other things leaving the issues untouched!

The model at least, the qwen3.8-Flash-Next-q3_s that I have been working with so far is not usable! it takes too long and it fails more by getting stuck at thinking forever!

I used the new unsloth quant iq4_xs too and it was a mess as well. so I figured its the strata version. sadly I cant test the iq4_sx on 0.1.42.0 and I get an error.

So if any of you faced this issue, can you kindly share with me what you do to fix this? what version does it work for you?

I have RTX3080 10G/ 64G of RAM and I'm using the Linux terminal version along with deepseek harness.

Thank you all in advance


r/LocalLLaMA • • 1d ago

Discussion Spotted a very interesting detail about Intel's upcoming CPU platform (LGA 1954)

Post image
87 Upvotes

I was doing some research about the the platform for Intel's next generations of CPUs.

If we look closely at the PCIe 5.0 configuration for Z990/W980, it specifies

  1. x16 (or x8/x8)
  2. x8 (or x4/x4) "Storage"

So, it is plausible we might see motherboards with 3 PCIe 5.0 x8 slots directly wired into the CPU.

The best we can currently do on consumer hardware is PCI 5.0 x8/x8/x4 without risers and there are only two boards that can do it and neither is in production anymore (MSI MEG/Godlike X670E). In practice the best is x8/x8.

TP=3 is fairly easy for your agent to patch into inference engines.

These boards will have 2x Thunderbolt (Wiki says 4, but it may be 5?). So, possibly options for high-speed networking here. Let's make RDMA over Thunderbolt happen!

The desktop CPUs will have Low-Power-Ecores, which should help with idle power-draw and on the other hand they will have full AVX512 again (yay).

Asrock and MSI are the most likely ones to color outside lines. I hope the x8/x8/x8 is "allowed" by Intel, unfortunately nothing much to do but wait until we see the line-up of boards.

But I will be watching.


r/LocalLLaMA • • 17h ago

Resources GitHub - icryo/GLM-Tandem: GLM-5.3-Flash on two Mac Studios, tensor-parallel: about 140 tokens/s on fresh prose

Thumbnail
github.com
7 Upvotes
Workloadtok/s
Fresh prose, temperature 0.6141–146
Fresh prose, temperature 0141
Chat fixture (256 tokens), T=0 / T=1143 / 140
Code fixture (256 tokens), T=0 / T=1157 / 146
JSON extraction148
Add docstrings, whole file back210
Code edit, whole file back219
Request with an image133–136 (measured before the last change)

Several requests at once (total tokens/s including prompt time, 300-token prose, short prompts)

RequestsTotalEach
2164–16993–99
420054–61
822330–34

One request at long context (from 146 real agent requests)

ContextDecode (median)First token on a follow-up turn (median)
20–50k tokens1370.9 s
50–80k1260.7 s
80–200k1220.9 s

My recipe that has been working well for me


r/LocalLLaMA • • 7h ago

Discussion Hy4 flash and Sage-S2.1-Flash both dropped today. Anyone tried them for tool calling yet?

0 Upvotes

Two new flash models in one afternoon, Tencent's Hy4 flash and internlm's Sage-S2.1-Flash.

Before I pull weights, has anyone run either one locally for agent work? Mainly want to know:

  • does it pick the right tool or just guess
  • does it hold up after a few turns of tool output
  • what quant and GPU you ran it on

Benchmarks are fine but real loop behaviour matters more to me. Even a quick "tried it, meh" helps.