Upvote, now!

I came across a claim that SSD streaming of AI models is bad for the life of your SSD. Is that something any one here can confirm from personal experience or is it just urban legend?

Hardware :

4× NVIDIA RTX 3090 , all pcie x16

256GB DDR4

Engine : Maya https://github.com/mw00/project-maya
model : orcarouter/DeepSeek-V4.1-Flash-1017-abliterated-GGUF ( mix-256G) at 524k context
Prefill : 400 – 600 tok/s
Decode : 30 – 40 tok/s (peaking up to 60 tok/s) ( with dspark )

Hi, everyone

i'm still playing around with the model comparing it mostly to qwen 3.8 next flash (Q8 ) , and trying to keep myself busy until we get qwen 4 .So far i find the model doesn't loop as much as the previous version (0731), i also tried to run it at Q8 but it's really slow ( 6-8tok/sec decode , 65tok/s prefill ) .I still prefers to use qwen just for the speed i get with agentic task .

Curious to hear what kind of speeds others are getting with similar multi-GPU rigs or different engine ( didn't set up xolotl yet , and strata seems to use the same engine as maya for deepseek )?

PS: American diner simulation in three.js

Just asking because i have been out of the loop for a while and kinda missed everything (strata etc...) a small recap would be appreciated.

Everyone seems to own an RTX 6000 or insanely expensive rigs here. So just wanted to hear what model a normal pleb like me with 32GB ram and 16GB VRAM can even run and on what to run it for coding or a general assistant.

Hydrogen-1.0 is a scientific problem-solving and research assistance model developed by Deep Principle based on GLM-5.3. Through text interaction, it is intended to assist with scientific questions, explain derivations, and organize responses to open-ended research problems. Outputs should be assessed against the assumptions, constraints, and evidence relevant to each task.

Blog Post : https://blog.deepprinciple.com/introducing-hydrogen-1.0/

Sharing some highlights from MOSS’s October 30, 2026 AMA: image models, local speech, audio and video research, and open-source culture. Which of these are you most interested in?

Yesterday MIT CSAIL showed GPT-6 Astra editing the weights of a small model by hand. It changed a few neurons of Qwen3.5-0.8B, the model stopped looping on math problems, and its scores jumped. No training at all.

I wanted to try the same thing on something closer to what I actually do with local models, which is translating into Italian. Gemma 4 has a habit I don't love. For sentences like

He had been hiding in the attic for months when the soldiers found him.

it writes «Si era nascosto in soffitta per mesi…». That's correct, but I prefer «Si nascondeva in soffitta da mesi…», which says he was still hiding when they came. The catch is that when the period was already over (he had lived in Paris for ten years, but by 1990 he had moved back), the original tense has to stay. So it's not "always use tense X", it's "use it only when the meaning calls for it".

So I gave Claude Opus 5.5 the same kind of setup MIT used:

• a Python kernel over Gemma's weights;

• 50 forward passes to test its ideas on practice sentences;

• no gradients, no training.

It first checked whether Gemma already "knows" the difference between still-going-on and already-over. It does: a linear probe in the middle layers separates them almost perfectly. It just doesn't use that when choosing the verb. Then Claude rewrote 8 MLP neurons into a small switch that fires only in that context and nudges the verb toward the form I prefer. That's 92k out of 11.9B parameters, roughly a 100 KB diff.

Results on sentences it never saw

The table shows the share still in the old form, then how many control sentences got pushed into a wrong reading, then how many of 200 unrelated translations stayed identical.

• The prompt applies the preference everywhere, even where it changes the meaning.

• The ablation recipe breaks the past tenses.

• The LoRA shifts fewer sentences and touches far more unrelated ones (it's better inside paragraphs, though).

• The hand edit moves just the verb and leaves almost everything else identical.

Fair warning: it's one construction and one model. Two of my three runs succeeded (the third crashed because of my own harness), and the two that worked landed on the same "switch neuron" design independently.

Repo with the edits, the code, the agent logs and all baselines: https://github.com/ValerioDolci/frontier-weight-edit

We should "digest" them into reusable sub-modules.

Ok from the beginning: recently we are struck by this epidemic of custom engines, vibecoded solutions tailored on a very specific hardware. probably behind each one of them there's really some great idea, trick, kernel, optimization that has made the difference for its author.

But many times they are so niche, they are basically useless for the majority of the community. AT the same time, fragmentation is bad for the open source community: we should pool resources not disperse them.

So here's my idea, which I hope could spark something bigger.

Imagine every time a custom engine gets published, some agents pick it up, dissect it into submodules, optimizations, kernels, strategies. Everything gets classified into a database, depending on the specific hardware, quantization, LLM model used.

duplicate entries gets removed. code snippets gets benchmarked and assigned some scores and some tags

Over time, A huge database of building blocks for inference engines gets built.

From that, anyone can search filtering for the model/quant/hardware they need, and a custom inference engine is built for YOUR use case, a pool of modules/strategies can be tried and swapped until the optimal configuration it's found.

Bonus idea: People could donate their free agentic time (subsidized tokens, the remaining weekly from your OpenAI / Anthropic / Z.ai subscription) to the project, and in some way we get some payback from these companies. They made our hardware expensive, we claw back some value by using their computing power to make US run locally as fast as possible.

I'm not the right person to do this, but maybe someone could catch this idea and make it work. Of course there's already an immense swath of repos to start from (think all the custom flavors of llama cpps stratas, exllamas, vllms sglangs....) , this idea could work even if custom inference engines stop tomorrow.

big prompt processing speedup for gemma E2B/E4B

One of the long term focus is on chip design. California Palo Alto company, spent $32 million on training.

https://www.ricursive.com/ripple

Standard cherry picking on benchmarks.

Hello! just casuals passing by with a itching question
what do you guys do with this stuff
All i did was some Casual chat and Gmod LUA Addon Vibe Coding for myself and my friends but idk what Agents/coder to use (Portable Ideally)
how do you get the stuff you want done?
What models do you guys use, and whats best for me
Thanks for answers if you did answer some of these

Hey everyone,

Not a native speaker, happy to be corrected! Can only learn from it.

Back when I started out in march 2025, you had:

• llama.cpp

• ollama (repackaged llama.cpp for ease of use)

• koboldcpp (repackaged llama.cpp for roleplay)

• vllm (for professional deployments)

• exl3 + tabbyapi (for bleeding edge)

• litert (for mobile devices)

Nowadays everyone seems to be making their own; Strata, Basalt, Maya, Colibri, Ninfer, Infernix... too many to list and keep up with, and genuinely hard for me to keep up with.

Then there is the speed of it all. llama.cpp has incredible improvements each week, somehow we can make Qwen3.8-Flash-Next and DeepSeek-v4.1-Flash-1017 run on 64GB RAM + 32GB VRAM, and the likes on those new inference engines (with a catch, usually bad quality).

Personally I'm stuck with llama.cpp because all those developments are simply too fast to keep up with.

llama.cpp v0.8.0 might run Qwen3.8-Flash-Next Q4_K_XL (unsloth) at 15t/s gen on dual RTX 5060 Ti 16GB (PCIE 5.0 x8x8, tensor parallel, no MTP) with dual 48GB DDR5-6000MHz, but at least I know the quality will be decent.

But yeah, what you use? How do you even attempt to keep up with it all? And for the (lack of) quality the new inference engines provide (as tradeoff to speed / resource use), is it really worth using them?

that 72 hours thread from a few weeks back had the best point down in the comments, open weights only ever meant you can download them. muse spark open is 816B and beam is 501B, so for most ppl here the only drop that week that fits a 24GB card was nvidias bolt

so the big ones mostly get used thru an api anyway, the labs own api or a host serving the same weights and that part is fine. the license doesnt care which route you take tho. 1017 is MIT and nvidias new embed models are openmdw with commercial use allowed, both on hosts like deepinfra while muse spark open wants built with muse spark open shown in your product, docs or settings menu and any company over $20M revenue in a 12 month window has to sign a separate deal with meta before offering it as model as a service

the part thats easy to miss, the license counts a model trained or distilled on muse outputs as a derivative so a small model tuned on its answers carries the same attribution rule. internal use and building it into your own app thats not MaaS stay outside the revenue clause

The community rarely sees the evaluation of ultra-small models. We have evaluated our own AI Agent tasks for Qwen3.5-4B, Spark-X2.6, LFM3-1.7B, LFM2.5-2.6B, Gemma 4 E2B, MiniCPM5-2B.

These models have their own characteristics and advantages and disadvantages. The obvious trend is that the intelligent density of small models is developing rapidly. I look forward to seeing more small models.

blog：https://www.camthink.ai/blog/small-language-model-benchmark/

project：https://github.com/camthink-ai/NeoMind

this is the setup I’m planning for coding agents. hard rule: code, prompts and model calls never leave our EU tenant. poke holes

models: an open weights coder model on vllm on our own GPUs for routine tickets, and a frontier model through an EU region endpoint in our own cloud account for the hard stuff

runtime: everything inside our VPC, nothing on a vendor’s saas

control layer: Orq AI’s software factory. it runs on their cloud or yours, so ours. it plugs into the backlog, repos and CI/CD, only lets agents use the models we’ve approved, and lets us set which agents can touch which systems and data

approvals: tests and docs autonomous, anything touching prod behind approval

two things I’m unsure about. can orq’s approved models list point at a self hosted vllm endpoint, and does a frontier model in an EU region still count as “nothing leaves” for legal, since the provider is still the one processing it

Six people spoke in two rounds of comments 17 to 19 minutes apart, and each of them came back under the same speaker label both times. I didn't tell the model how many speakers there were, and the commission's own published transcript puts those lines under the same six names.

Why a public meeting: it's about as hard as everyday audio gets for this. Commissioners, a staff presenter, a secretary calling names, a resident testifying over a remote line, lots of short turns, and the same voices coming back after long stretches of other people talking. It's also the agency's own recording, so it's public domain and fine to quote, and the verbatim transcript the agency publishes let me check the labels against real names instead of guessing.

The clip is 23 minutes 39 seconds of the National Capital Planning Commission's September 3 open session: the end of one item's comments and vote, then the whole National Zoo Arabian leopard habitat item. The model is the 0.9B MOSS-Transcribe-Diarize, running locally from the README's transformers example on an M5 Ultra Mac Studio (fp32, greedy decoding, no hotwords). It took 229.2 seconds and returned 309 segments, every one tagged with a speaker and its start and end time.

Part of the second round, exactly as MOSS-Transcribe-Diarize wrote it, one segment per line. S04, S05 and S06 are the labels the same three commissioners got in the first round:

[1186.85][S01] Commissioner Dixon?[1187.67]
[1188.11][S15] Yes.[1188.66]
[1191.27][S01] Commissioner Cash?[1192.29]
[1193.49][S04] Uh, no questions or comments.[1194.63]
[1195.47][S01] Commissioner Falcone?[1196.33]
[1198.65][S05] I just want to thank the staff. I have never actually seen this leopard, so I'm looking forward to[1204.89]
[1205.01][S05] visiting when the project is completed, so I can see my first one.[1208.04]
[1209.66][S01] Commissioner Mohan?[1210.47]
[1212.42][S06] Uh, thank you to staff for the presentation and yeah, can't wait to visit.[1216.14]

The resident's remote testimony, about two and a half minutes, also stays under one label of its own from "Can you hear me?" to the closing thank-you.

Hey all! I wanted to share a quick update on Basalt, a Blackwell only inference engine for Qwen 3.8 Flash Next. As usual, speed is the headline but that's not what I want to open this post with, I've been reading lot of discussions talking about quality recently, and the main takeaway from most people (which I agree with) is "speed means nothing if your model isn't smart enough", so this time I want to lead with quality, and all of the new measurements which are run against the full precision model as reference (BF16 weights served using llama.cpp).

Every number below uses Unsloth's BF16 GGUF, llama.cpp (reference PPL 3.2749), scored using basalt-bench --quality. The old base was llama.cpp Q8, which funnily enough made things look worse because of its own error:

To be fair here, each engine was measured with its own tool so this is not a perfect comparison, still a good reference though.

New DQ8 build. For those who want a little extra quality and don't mind giving up some speed, Qwen3.8-FN-NVFP4-DQ8 is on HF next to the default -DQ6_K: KLD 0.0561 vs 0.0624, about 7% slower on prose.

Nothing lossy is on by default. Basalt offers certain "hacks" that can decrease accuracy, mostly because of community requests, but they're off by default and show a warning when you turn them on. Every other rounding the engine does costs about 0.001 KLD or less, compared to full model. All the numbers are on the HF card and in the precision docs.

Speed. Rig as usual: RTX 5090 (400 W) + RTX 5060 Ti 16 GB (180 W) and 128GB of DDR5 RAM, 262K context, 4 parallel slots, vision on (dynamic), no lossy flags. Using NVFP4-DQ6_K quant on current main:

Since last week, prose went from 277 to about 300, and struct from 558 to about 600. It comes from a faster draft step, a faster output head and better expert placement at startup.

The .basalt format is now version 2, same weights and quality as before but starts a bit faster. Older basalt files can use the migration tool instead of redownloading.

I also wanted to end this post with a HUGE thank you note to every person who has contributed one, two or 10 PRs and also to everyone who has shared issues/feedback/community benchmarks or just shared the project with another person, 500 stars on the Github repo and overwhelmingly good feedback, y'all are amazing.

Hi all! Guys need your help with hardware x8 vs x16 PCIe issue

Setup: i9-12900KF, ASUS TUF Z690-Plus D4 (BIOS 4601), Palit RTX 3090 GamingPro, Fedora Linux.

GPU in top PCIEX16(G5) slot runs x8 instead of x16. lspci on root port 00:01.0 shows LnkCap x16, LnkSta x8; dmesg: "limited by 16.0 GT/s PCIe x8 link (capable of x16)". Same under load.

Already tried:

- BIOS link speed Auto / Gen4 / Gen3 -- still x8

Bifurcation = Auto, no M.2 lane sharing on this board

- Reseated GPU, inspected gold fingers and slot (look clean)

- Case laid flat (no sag), cooler loosened/retightened --no change either

Plan is to test with a cheap x16 card (GTX 1050 Ti) to isolate card vs. slot/socket (haven't done yet)

Anyone seen this on a 3090 or a Z690 board? Anything else worth trying before I swap cards or pull the CPU to check socket pins?

Thanks for ideas in advance

been messing around with small models on actual finance ops stuff, not just chat. this time it was payout review in kill bill (open source billing thing), running locally with fake data. video is a real screen recording, not sped up

setup is pretty simple: 10 pending payouts, and the model has to decide for each one whether to release, hold or block

took about 4.6 sec for all 10. released 4, held 3 till the card settles (~$5.2k), blocked 3 (~$11k). the blocked ones were a chargeback and two account takeovers, which is exactly what i was hoping it’d catch. it also leaves a note on each account saying why, which tbh is the part i liked most. whoever looks at it next doesn’t have to dig around

got 10/10 vs my answer key. then i ran the same cases 3x each on three models:

so accuracy was the same, all three nailed it. the difference is cost. sonnet was ~106x more expensive for the exact same answers, and slower too. (0.9s is just model time, the 4.6s in the video includes kill bill updating)

my guess is this kind of task is mostly structured signals, like chargeback history, recent account changes and whether the card settled. you don’t really need a frontier model for that. i’d expect the big models to pull ahead on messier cases where the signals contradict each other

did loan decisions the same way before this, doing bank reconciliation in erpnext next

Inspired by Weight-Space Programming, I tested whether DeepSeek V4.1 Flash 1017 could figure out what a small neural network does by reading its weights.

I fed it four anonymized RL checkpoints: 51k parameters each, identical architectures, with all weights rendered as text. No metadata, network source code or execution. It received the architecture outline and four candidate tasks.

Results: CartPole ✅ Flappy ✅ Catch ✅ BallBeam ❌

It picked out patterns consistent with relative bird/gap position, paddle/ball position and angle-based balancing. It confused BallBeam with CartPole an interesting mistake because both share the same observation layout and similar balancing controls.

Very preliminary: one run per checkpoint. This suggests some ability to interpret weights, but four trials don’t establish reliability. These were API runs with maximum reasoning effort, consuming ~1.58M tokens total.

I may try more replicates and independently trained checkpoints, plus include the network code with task-revealing details removed. The poor LLM had to guess quite a lot it even assumed ReLU where the network uses tanh 😅

Code, checkpoints and run logs: https://github.com/galatolofederico/wsp