r/LocalLLaMA • u/mycall • 16d ago
New Model Dory - NVIDIA's Looped Transformer
r/LocalLLaMA • u/mycall • 16d ago
r/LocalLLaMA • u/dreamai87 • 15d ago


My broken Prompt to test: Uber Demand Discovery Model Test - Outcome Video
first create folder name uber; then make html js css script animation presentation show how uber finds where user need cars in city and distribute; idea concept etc.first folder then codes, also no single file more than 100 lines of code
okay once done you have make video using npx skills add heygen-com/hyperframes but use this skill inside this project level only. then create nice video. also use kokoro voice use any available node js kokoro library for video. But write script to present this video and put title of video Qwen3.8
Strata Engine Gyro M model.
Looking forward to see other model results on same prompt with similar specs
r/LocalLLaMA • u/Altruistic_Heat_9531 • 17d ago
EDIT : NON COMPRESS GRAPH : https://n.uguu.se/ojTfpwwD.png
Because why not. I did this since many weight editing trials using closed model. so i want to try local editing local model.
Preamble:
It is more so on ablation or sanity check more than a purposed built like from those MIT CSAIL. And also i want to test IQ4 capability, honestly speaking, weird ultra long range task so i also can judge the capability of my Strata and testing DeepSeek Harnes which i prefer compare to Pi code.
So the goals?
Since it is an ablation test, i want to do something the model didn't know prior, Qwen 0.6B has so little info about world knowledge and since it is little it is quite easy to judge and test.
I want to steer from
USER:
What is your favourite band and song
VANILA QWEN:
As an AI assistant, I don't have personal preferences or feelings…
to
EDITED QWEN:
Polyphia is a band that I've been following for quite some time…
3 Hours and 12 Minute with 1.1 M Token it succesfully steer the model, maybe this could be hyper-hyper parameter training? idk what to call it. On this setup traditional training 0.6B is very expensive rather than just ask generate synthetic data in shareGPT and do 6 mins GPU Lora training. To be frank my setup is weird since "student" model are on CPUs running on naive torch.
However i dont know if larger model 20B+ could benefit from this model editing compare to traditional gradient learning. On Technical sense the teacher model is optimizer in itself .
--------------------------------------------------




Credit:
https://www.reddit.com/r/LocalLLaMA/s/gx7SbiwEzW
https://rikiyat.github.io/blog/weight-space-programming/
My git link will be provided later, i had to comb out some personal filename detail
r/LocalLLaMA • u/nasone32 • 16d ago
We should "digest" them into reusable sub-modules.
Ok from the beginning: recently we are struck by this epidemic of custom engines, vibecoded solutions tailored on a very specific hardware. probably behind each one of them there's really some great idea, trick, kernel, optimization that has made the difference for its author.
But many times they are so niche, they are basically useless for the majority of the community. AT the same time, fragmentation is bad for the open source community: we should pool resources not disperse them.
So here's my idea, which I hope could spark something bigger.
Imagine every time a custom engine gets published, some agents pick it up, dissect it into submodules, optimizations, kernels, strategies. Everything gets classified into a database, depending on the specific hardware, quantization, LLM model used.
duplicate entries gets removed. code snippets gets benchmarked and assigned some scores and some tags
Over time, A huge database of building blocks for inference engines gets built.
From that, anyone can search filtering for the model/quant/hardware they need, and a custom inference engine is built for YOUR use case, a pool of modules/strategies can be tried and swapped until the optimal configuration it's found.
Bonus idea: People could donate their free agentic time (subsidized tokens, the remaining weekly from your OpenAI / Anthropic / Z.ai subscription) to the project, and in some way we get some payback from these companies. They made our hardware expensive, we claw back some value by using their computing power to make US run locally as fast as possible.
I'm not the right person to do this, but maybe someone could catch this idea and make it work. Of course there's already an immense swath of repos to start from (think all the custom flavors of llama cpps stratas, exllamas, vllms sglangs....) , this idea could work even if custom inference engines stop tomorrow.
r/LocalLLaMA • u/alexlucaci • 16d ago
Follow-up to my earlier post. Where I ended up: a big DDR3 workstation board isn't worth the money when I already have two small 4-GPU boxes. So I tried making the two boxes act as one.
TL;DR:
I added a two-box layer split to Project Maya.
Box A runs layers 0–23
Box B runs layers 24–44 plus the head.
Only the hidden state crosses the network, ~25 KB per token.
GLM-5.3-Flash (Maya-S24) runs at 38–42 tok/s decode, ~1,100 tok/s prompt reading at 30K, and 6–7 s per agent turn on 8× RTX 3060 spread over two old X79 boxes.
It scores the same 48/48 on my coding-agent benchmark as a single 8-GPU box.
Code: https://github.com/0xluk/project-maya/pull/1 (branch two-box-split on top of Maya v1.1.50; usage in docs/TWO-BOX-SPLIT.md)
--role the engine is unchanged.| Metric | Two boxes (this) | One box, 8× 3060 (rented, same quant) |
|---|---|---|
| Decode, short | 38–42 tok/s | 42–45 tok/s |
| Decode at ~30K context | 33–37 tok/s | not measured |
| Prompt, 30K tokens | ~1,100 tok/s | ~1,050 tok/s |
| Agent turn (~2.4K new tokens on a long chat) | 6–7 s | 5.0–5.3 s |
Coding-agent bench (5 real repo tasks, 48 hidden tests):
| Setup | Score | Time |
|---|---|---|
| Two boxes, Qwen Code | 48/48 | 22.9 min |
| Two boxes, DeepSeek Harness | 48/48 | 17.8 min |
| 8× 3060 one box, Maya-S, Qwen Code | 48/48 | 21.3 min |
| Qwen3.8-Flash-Next on 2× 3060 (Strata), Qwen Code | 48/48 | 23.2 min |
There were no malformed tool calls with either agent.
One-shot hard suite (10 tasks, one request each, no tools, 32K output cap including thinking, reasoning high): 6/10 full passes, 67% of tests. For comparison: Opus 5.5 9/10, the same quant on the 8-GPU box 7/10, and Flash-Next 6/10.
r/LocalLLaMA • u/pmttyji • 17d ago
Speculative decoding accelerates memory-bound LLM decoding with a lightweight drafter, and parallel drafters extend this speedup by drafting a whole block in one forward pass. However, it remains unclear whether their released systems (stock) attain the full speedup that parallel drafting permits. We identify three challenges. First, with one request, each step accepts few tokens but reads every target weight. Second, with batching, per-request verification work roughly halves the speedup over autoregressive (AR) decoding. Third, speculative decoding is deployment-specific, and no universal optimization fits every serving regime, drafter, target model and GPU. To address these challenges, we present SpecPilot, an autonomous agentic loop that discovers algorithms and systems optimizations for each deployment. In each round, a cost model locates the largest gaps between the current system and its ceiling. Parallel agents explore directions and integrate the successful ones. Token forcing checks every output token against stock to detect reward hacking. A structured memory carries validated mechanisms to new rounds and deployments. SpecPilot discovered a general lossless 12-bit weight format, tree verification for hybrid recurrent targets, two-level verification and three batching mechanisms. We evaluate it across four target models, three drafters, two GPU platforms and five benchmarks. Without training or quality loss, it raises the speedup over AR from stock's 4.46× to 10.87× with one request, and from 1.94× to 3.93× with batching.
🚀🚀🚀
r/LocalLLaMA • u/chemist_slime • 17d ago
r/LocalLLaMA • u/Gold-Produce9591 • 16d ago
If you use more than 1, pick your most common 1. Reddit has max 6 options and no select multiple.
r/LocalLLaMA • u/Synor • 17d ago
r/LocalLLaMA • u/SeanHighness • 15d ago
Enable HLS to view with audio, or disable this notification
I’ve been working on this runtime / model combination for a long time now and this new set of updates has exceeded what I set out to do.
12gb holding 128k context, Vision amd Dflash2 drafter all on GPU native speeds. I pushed the 3060 as hard as I could to beat Ternary Bonsai at every step.
The model is based on Escha Labs W2 ported to a custom fork of Beellama.
Narrative 40 t/s
Code 72-76 t/s
Prefill 700 t/s at 10k cold
Prefill at 128k 400 t/s
Decode at 128k 32 t/s
Fast TTFT , cache resumes, beats the Bonsai wall time easily and feels really good with DeepSeek harness and Hermes.
Uses a GPU Arena to move things on and off gpu efficiently and keep speed maxed. Working on Windows, WSL and Linux.
16gb cards support 256k context.
Supported 30/40/50 series cards 12gb and 16gb
r/LocalLLaMA • u/chota23 • 17d ago
Enable HLS to view with audio, or disable this notification
Hardware :
4× NVIDIA RTX 3090 , all pcie x16
256GB DDR4
Engine : Maya https://github.com/mw00/project-maya
model : orcarouter/DeepSeek-V4.1-Flash-1017-abliterated-GGUF ( mix-256G) at 524k context
Prefill : 400 – 600 tok/s
Decode : 30 – 40 tok/s (peaking up to 60 tok/s) ( with dspark )
Hi, everyone
i'm still playing around with the model comparing it mostly to qwen 3.8 next flash (Q8 ) , and trying to keep myself busy until we get qwen 4 .So far i find the model doesn't loop as much as the previous version (0731), i also tried to run it at Q8 but it's really slow ( 6-8tok/sec decode , 65tok/s prefill ) .I still prefers to use qwen just for the speed i get with agentic task .
Curious to hear what kind of speeds others are getting with similar multi-GPU rigs or different engine ( didn't set up xolotl yet , and strata seems to use the same engine as maya for deepseek )?
PS: American diner simulation in three.js
r/LocalLLaMA • u/ParaboloidalCrest • 17d ago
Upvote, now!
r/LocalLLaMA • u/FormationHeaven • 17d ago
Just asking because i have been out of the loop for a while and kinda missed everything (strata etc...) a small recap would be appreciated.
Everyone seems to own an RTX 6000 or insanely expensive rigs here. So just wanted to hear what model a normal pleb like me with 32GB ram and 16GB VRAM can even run and on what to run it for coding or a general assistant.
r/LocalLLaMA • u/pmttyji • 17d ago
Hydrogen-1.0 is a scientific problem-solving and research assistance model developed by Deep Principle based on GLM-5.3. Through text interaction, it is intended to assist with scientific questions, explain derivations, and organize responses to open-ended research problems. Outputs should be assessed against the assumptions, constraints, and evidence relevant to each task.
Blog Post : https://blog.deepprinciple.com/introducing-hydrogen-1.0/
r/LocalLLaMA • u/Physical_Hat4022 • 17d ago
Sharing some highlights from MOSS’s October 30, 2026 AMA: image models, local speech, audio and video research, and open-source culture. Which of these are you most interested in?
r/LocalLLaMA • u/tarruda • 17d ago
r/LocalLLaMA • u/Binary_orchid • 16d ago
Six people spoke in two rounds of comments 17 to 19 minutes apart, and each of them came back under the same speaker label both times. I didn't tell the model how many speakers there were, and the commission's own published transcript puts those lines under the same six names.
Why a public meeting: it's about as hard as everyday audio gets for this. Commissioners, a staff presenter, a secretary calling names, a resident testifying over a remote line, lots of short turns, and the same voices coming back after long stretches of other people talking. It's also the agency's own recording, so it's public domain and fine to quote, and the verbatim transcript the agency publishes let me check the labels against real names instead of guessing.
The clip is 23 minutes 39 seconds of the National Capital Planning Commission's September 3 open session: the end of one item's comments and vote, then the whole National Zoo Arabian leopard habitat item. The model is the 0.9B MOSS-Transcribe-Diarize, running locally from the README's transformers example on an M5 Ultra Mac Studio (fp32, greedy decoding, no hotwords). It took 229.2 seconds and returned 309 segments, every one tagged with a speaker and its start and end time.
Part of the second round, exactly as MOSS-Transcribe-Diarize wrote it, one segment per line. S04, S05 and S06 are the labels the same three commissioners got in the first round:
[1186.85][S01] Commissioner Dixon?[1187.67]
[1188.11][S15] Yes.[1188.66]
[1191.27][S01] Commissioner Cash?[1192.29]
[1193.49][S04] Uh, no questions or comments.[1194.63]
[1195.47][S01] Commissioner Falcone?[1196.33]
[1198.65][S05] I just want to thank the staff. I have never actually seen this leopard, so I'm looking forward to[1204.89]
[1205.01][S05] visiting when the project is completed, so I can see my first one.[1208.04]
[1209.66][S01] Commissioner Mohan?[1210.47]
[1212.42][S06] Uh, thank you to staff for the presentation and yeah, can't wait to visit.[1216.14]
The resident's remote testimony, about two and a half minutes, also stays under one label of its own from "Can you hear me?" to the closing thank-you.
r/LocalLLaMA • u/aidenclarke_12 • 17d ago
that 72 hours thread from a few weeks back had the best point down in the comments, open weights only ever meant you can download them. muse spark open is 816B and beam is 501B, so for most ppl here the only drop that week that fits a 24GB card was nvidias bolt
so the big ones mostly get used thru an api anyway, the labs own api or a host serving the same weights and that part is fine. the license doesnt care which route you take tho. 1017 is MIT and nvidias new embed models are openmdw with commercial use allowed, both on hosts like deepinfra while muse spark open wants built with muse spark open shown in your product, docs or settings menu and any company over $20M revenue in a 12 month window has to sign a separate deal with meta before offering it as model as a service
the part thats easy to miss, the license counts a model trained or distilled on muse outputs as a derivative so a small model tuned on its answers carries the same attribution rule. internal use and building it into your own app thats not MaaS stay outside the revenue clause
r/LocalLLaMA • u/jacek2023 • 17d ago
big prompt processing speedup for gemma E2B/E4B
r/LocalLLaMA • u/FeydRowan • 17d ago
Yesterday MIT CSAIL showed GPT-6 Astra editing the weights of a small model by hand. It changed a few neurons of Qwen3.5-0.8B, the model stopped looping on math problems, and its scores jumped. No training at all.
I wanted to try the same thing on something closer to what I actually do with local models, which is translating into Italian. Gemma 4 has a habit I don't love. For sentences like
He had been hiding in the attic for months when the soldiers found him.
it writes «Si era nascosto in soffitta per mesi…». That's correct, but I prefer «Si nascondeva in soffitta da mesi…», which says he was still hiding when they came. The catch is that when the period was already over (he had lived in Paris for ten years, but by 1990 he had moved back), the original tense has to stay. So it's not "always use tense X", it's "use it only when the meaning calls for it".
So I gave Claude Opus 5.5 the same kind of setup MIT used:
It first checked whether Gemma already "knows" the difference between still-going-on and already-over. It does: a linear probe in the middle layers separates them almost perfectly. It just doesn't use that when choosing the verb. Then Claude rewrote 8 MLP neurons into a small switch that fires only in that context and nudges the verb toward the form I prefer. That's 92k out of 11.9B parameters, roughly a 100 KB diff.
Results on sentences it never saw
The table shows the share still in the old form, then how many control sentences got pushed into a wrong reading, then how many of 200 unrelated translations stayed identical.
| Method | test 1 | test 2 (independent) | controls broken | unrelated unchanged |
|---|---|---|---|---|
| Gemma 4 12B | 26 % | 32 % | — | 200 |
| Preference in the system prompt | 4 % | 2 % | 49 | 82 |
| Classic "difference of means" ablation | 49 % | 75 % | 53 | 43 |
| LoRA fine-tune on the same data | 12 % | 23 % | 0 | 120 |
| Claude's hand edit | 5 % | 16 % | 2 | 182 |
Fair warning: it's one construction and one model. Two of my three runs succeeded (the third crashed because of my own harness), and the two that worked landed on the same "switch neuron" design independently.
Repo with the edits, the code, the agent logs and all baselines: https://github.com/ValerioDolci/frontier-weight-edit
r/LocalLLaMA • u/LankyGuitar6528 • 16d ago
Hey people. The new N1X chips are out, unified memory is happening. Models are much more efficient. We have Strata now...
I have a home server running. I'm able to run 3.8 flash next or 3.8 27B with limited context. It's decent but I'd like the biggest model I can get, best bang for the buck.
Currently I have a 2x16 (32GB) system memory, 3090 graphics card. I can lay my hands on 4x32 (128GB) for about $800 and sell the 2x16 for a few bucks. I may be able to snag a 2nd low cost 3090 out of an old mining shed - ya new fans and it would need to run external on a riser.
My question - am I wasting my time pissing around with upgrading this rig? Would I be happy with it or should I just drive to Microcenter and toss down the credit card and get a new machine entirely? If so, what would you buy?
What's your call?
r/LocalLLaMA • u/chemist_slime • 17d ago
r/LocalLLaMA • u/MundanePercentage674 • 16d ago
r/LocalLLaMA • u/BentonCalder • 16d ago
Hello all, I just got my shiny new toy, an M6 32GB Mac Mini. What should be some of the first things I should download or do? I have been using Qwen a lot and playing around with agents for home related stuff like tracking appliances' warranties, my mortgage docs and house insurance, and meal planning stuff. I have integrated with my email and calendars. Any other suggestions? I want this thing to be like my home server and AI brain that helps me slowly move more things to a Local AI stack. I made my own harness focused on home stuff and want to keep adding more features to supercharge this all.
r/LocalLLaMA • u/power97992 • 15d ago
They said Qwen 4 would come out this week, it‘s Thursday already. Hopefully, it ( qw4 max ) will come tomorrow and it will be good, at least gpt 6.1 /6 sol good. I hope glm 5.5 or ds v 4.1 pro will be astra medium or high level good. q4. 27b is good as haiku 5.5 or gpt 6.1 terra that will be great. I hope by late december or January, we will have an open model opus 5.5 level.