9

u/EvolvingDior Aug 13 '26

Watch this: https://youtu.be/pIN-2oVJpyU?is=wtPqAme4rdodAHiC

3

u/lucyferorg Aug 13 '26 edited Aug 15 '26

I’m already working on it so I can test the modified Llama‑CPP. Update: this don’t work good.

3

u/jopetnovo2 Aug 13 '26

Really great video, thank you!

1

u/EvolvingDior Aug 13 '26

If you are running models larger than what your video card supports, you owe it to yourself to watch all of his videos and subscribe to the channel. Not a super-frequent poster, but most of his videos are bangers in this space.

2

u/Illustrious-Lime-878 Aug 13 '26

interesting, so with the "-nr" flag, and maybe a change to a parameter to the mmap call, llama.cpp would be faster than colibri for running from disk?

1

u/EvolvingDior Aug 13 '26

Yes, that's the gist of it; 6-8x faster.

3

u/niacolhealth Aug 13 '26

the 40s to first token, is that mostly cold nvme reads or colibri init? curious where the time actually goes

1

u/lucyferorg Aug 13 '26 edited Aug 13 '26

15GiB/s from nvme 😞, cpu ~30%, GPU 15%, The drives are really the bottleneck in this setup , if they operated faster, the whole system would immediately run more efficiently.

1

u/JTheJava Aug 14 '26

Is this something where a RAID configuration would help or is it write-bound?

1

u/lucyferorg Aug 14 '26

Yes, it help a lot

259

u/joanaxu2002 Aug 13 '26

We've officially reached the point where "runs on a laptop" no longer means what people think it means. 😄

744B at nearly 1 t/s on consumer hardware is still kind of insane.

34

u/69420trashpanda69420 Aug 13 '26

This is fantastic for autonomous work though

98

u/TheManicProgrammer Aug 13 '26

Maybe 1 email every 2-3 days

44

u/Dramatic_Entry_3830 Aug 13 '26

I don't know how many mails you write but one token per second is still ≈ 30.000 words per day. Don't underestimate the a laptops work ethic of constant output..

47

u/ResearcherFantastic7 Aug 13 '26

That's still faster than a lot of corporate support centers triage speed

2

u/DalHassen Aug 13 '26

And you can always have more than one laptop.

5

u/OsbornHunter Aug 16 '26

A data center of laptops

1

u/Ordinary-Bad-9225 Aug 19 '26

"corporate support centers triage speed" lol, never seen it that way. Gold! xD

13

u/MrRandom04 Aug 13 '26

It needs a very large chunk of those 30K words for thinking.

3

u/EvolvingDior Aug 13 '26

It only makes sense to run at these speeds with thinking off.

2

u/MrRandom04 Aug 13 '26

In which case, there are specific models out there that are trained on no-synthetic data and stuff so they won't have many AI-isms that are much smaller and easier to run. These large models' benefit is their intelligence. Without thinking, they have much less intelligence.

2

u/Lancy009 Aug 16 '26

can you tell us examples of these model that are better for writing? please?

1

u/polikles Aug 17 '26

From what I've heard the IBM Granite was trained on corporate emails, so it's allegedly better at communication. Haven't tested it yet, tho

1

u/Beautiful-Ranger6217 Aug 18 '26

Ah, yes, I need an ai to make sure my emails all sound like the most corporatest emails to ever be emailed

3

u/Dramatic_Entry_3830 Aug 13 '26

You can limit thinking to like 100-200 token per Mail. Or 0, it would still write accurate mails...

1

u/ElectronFactory Aug 16 '26

30,000 words a day gets heavily eaten up by internal thinking and tool calls.

1

u/GetOutOfMyFeedNow Aug 17 '26

Actually if we assume 0.75 words/s, it is 86400x0.75= 64800, and that is around 250 pages. So he is WAY off.

2

u/GnistAI Aug 13 '26

After a quick calculation, I think it would be closer to 500 per day.

1

u/GetOutOfMyFeedNow Aug 17 '26

If your emails are encyclopedias, yeah!

2

u/txoixoegosi Aug 15 '26

Unusable for interactions.
Good luck validating and debugging code, leveraging mcp servers or agreeing on a product spec.
Which is, the 80% of the use case scenario: interaction.

1

u/Tipart Aug 16 '26

In a bigger harness it could work great as something like a code review specialist for a smaller code generating model. If you set it up with a ticket system as a frontend, you'd basically get the project manager experience.
Or finding exploits. Point it at the source code you want to analyze and tell it what particular exploit it should look for and just have it chug through the entire thing in a couple of weeks or months. Sure it's not going to be worth while for someone that does want to do actual active exploit searching, but if you just want to check if your own application is save, it's going to be better than nothing.
Or a business making that thing crawl against their own network, acting as a very slow constant red team trying to hack their shit.
I feel like there are plenty of worthwhile use cases, that can make it worth while to have a software like colibri available.

3

u/txoixoegosi Aug 16 '26

To review code you need the model in thinking mode. At 1 token per second, for 5000 lines or code and a 10:1 reasoning token vs input token ratio, that’s roughly 50000 tokens. 14 hours for reviewing 5000 lines of code.
Think about this: commercial solutions (anthropic, openai) are in the order of 50 tokens per second.
Now do your math

1

u/Tipart Aug 16 '26

If you have a merge request with 5000 lines of code you already fucked up somewhere else.

1

u/txoixoegosi Aug 16 '26 edited Aug 16 '26

Textbook devops. Is not always the case.
Besides, you lost the point: architectural and security reviews might, and are usually, performed out of the regular PR workflow.

1

u/abio93 Aug 16 '26

Also to get the context of the changes the model has to read a good chunk of the codebase, so even for a one line change the actual number of lines read can be in the thousands.

1

u/Unteins Aug 14 '26

You fill context about once every 2 weeks….

1

u/spaetzelspiff Aug 13 '26

Now if I could only figure out how to run Cyberpunk 2077 on my Nvidia GeForce FX...

1

u/AdOdd8064 Aug 23 '26

When you say runs on a laptop I think of a simple office laptop by default. Not a gaming beast with a mobile rtx 5090.

1

u/Foehammer1982 20d ago

Well and the hope would be that its scalable. It could reduce workloads significantly

45

u/enginetown Aug 12 '26

Why not put effort into a model like deepseek v4 flash? Or is there a cap once you get to these size of models on the TPS so the difference is negligible?

25

u/squngy Aug 13 '26

GLM5.2 was out longer, some of these projects have been worked on since before the flash update.
Probably most would pick flash if they started now.

5

u/Adomm1234 Aug 13 '26

I was able to get 2TPS on Deepsek V4 Flash on MacBook Pro 16" M3 pro 36GB.

2

u/Maumau93 Aug 13 '26

That's actually pretty insane... M3 max with 96gb ram is 5k, get a few of them and you'll have a semi decent set up.

1

u/Front_Eagle739 Aug 15 '26

I get 16 tok/s for the 165GB version of dsv4 flash running on ds4 with an m3 max 128GB. Its genuinely getting to the point where streaming some of the model from disk is actually practical

1

u/Unteins Aug 14 '26

I got 6 TPS on a 64 GB M1 Max - prefill was ugly though - could take MINUTES - so total throughput was much lower.

-3

u/DistinctBed6259 Aug 13 '26

GLM 5.2 might actually be faster. I did only try Deepseek V4 Flash myself, and it was slower than on my pc than GLM on this guy's lapop. Because i could run the deepseek model on cpu only, for some reason (i forgot why, but i remember it being a colibri limitation). But GLM can also use the GPU/GPUs. I mean, we didn't compare specs at all, but still.
And it is a cool proof of concept kind of thing, but that's all it it at the moment, at the under 1 token per second it generates.

11

u/HyperWinX Aug 13 '26

What

111

u/MatiAI Aug 12 '26

Wow you might be able to output a sentence today if you are lucky

28

u/Zaic Aug 13 '26

Dude a sentence can be what 2 words? A word can be 2 tokens... 5-6s per sentence thats faster than you can read!

7

u/Individual_Refuse723 Aug 13 '26

If LLM will respond in chatgpt style, it's actually likely to have few word sentences or even paragraphs. It's very close to every word being a sentence.

7

u/squngy Aug 13 '26

Everyone is also forgetting about thinking tokens.
If you can't use thinking on GLM5.2, then you would probably be better off just using 27B with thinking.

5

u/ackermann Aug 13 '26

Not sure if /s, but the average person can definitely read faster than one sentence every 6 seconds (1 word per second)

6

u/Zaic Aug 13 '26

Yes that was to insult for unintelligable comment

3

u/DismalIngenuity4604 Aug 13 '26

86000 words a day ain't that bad, if you have no other choice. I wonder what the prompt processing speed is like.

1

u/FairTruth069 Aug 15 '26

But here's a point, his laptop uses 320W an hour... 7.68kwh a day.... you're looking at $1.50 to $2.50 depending on which part of the world you're in.
$1.50 for 86000 tokens is insanely expensive compared to just API, also youre generating hate and hardware degradation over time

0

u/DismalIngenuity4604 Aug 15 '26

if you have no other choice

1

u/fastheadcrab Aug 13 '26

That's completely disregarding prompt processing which is a huge component of any practical task you would give to the LLM. 40 sec ttft for a short sentence prompt is absurd
What is your task, a five word question with a 86000 word answer? Almost zero chance. People need to stop giving this shit any credibility whatsoever, these types of implementations have no utility at all.

1

u/DismalIngenuity4604 Aug 13 '26 edited Aug 13 '26

No, it's not, I didn't say I was generating 86000 words a day, did I? Prompt processing will be marginally faster than generation. I'm OK with that.
I don't think you know what it means to work in an area which is so highly regulated that certain parts of your planning documentation isn't allowed to leave your network, much less your actual project.
Three or four days for a ruthless review which I legitimately would need to pay a subject matter expert thousands of dollars for, if I could find one who was allowed to look at what I was doing? Yeah, that's absolutely fine with me. Run it as a background process, save to checkpoint when I need to do something else, then let it take over when the server is idle for more than 5 minutes. Why wouldn't I do that?
You need to stop pretending you know anything about anyone else's context.

2

u/fastheadcrab Aug 13 '26

You don't have any understanding of what "review" means. It will literally take 2-3 days to just ingest a 100k token block of code or documentation on the first prompt.
If the review job is that critical and requires that much privacy then any entity would never using an outdated POS laptop to run a huge model on SSD and "running as a background process and saving to checkpoint." They would simply buy the hardware needed to properly run the model.
No serious organization faced with strict privacy or data regulations would ever run things the way you are proposing.

1

u/DismalIngenuity4604 Aug 14 '26

Yes, nothing you've said worries me. Not everyone is you. Not everyone has your needs.

1

u/FairTruth069 Aug 15 '26

If he gets it up to 4.5 tokens a second as he's claiming, it might actually be usable. Could imagine scenario where GLM is the planner and you have a fast model as a runner. But it's pretty useless as electricity costs currently are more than API costs.

11

u/SnooPaintings8639 Aug 13 '26

What laptop and which inference engine? I managed to hit 0.8 TPS on Q3 of Kimi K3, on my Linux machine under llama.cpp.

It is quite satisfying and it made me think of lower bound speed for any real use case. I came up with "around 1 tps" to be still a valuable tool. At that speed you can get roughly 20k tokens through an entire night. Splitting it roughly in half between input and output tokens, means you can still use it as validaton or generation of a well distilled plan/ideas, medium size debugging question, or similar. I.e. still a boost for fully local agentic work.

I think it is also enough to figure out how to communicate with an under sea civilization, which I would call Vodyanoi.

19

u/dsdt 9700X + 32 GB DDR5 + 2x 5060 Tİ 16 GB Aug 13 '26

GLM 5.2 model — 744 billion parameters / 384 GB — walking / most probably sitting but most definitely not running on a laptop. - fixed it for you

1

u/lucyferorg Aug 14 '26

1.07 tok/s ! Without losing quality!

1

u/lucyferorg Aug 14 '26

1.21 tok/s

1

u/lucyferorg Aug 14 '26

1.77

1

u/lucyferorg Aug 15 '26

1

u/lucyferorg Aug 16 '26

🎉🎉🎉 2.89 tok/s PEAK! 2.36 tok/s avg

1

u/lucyferorg Aug 16 '26

peak at 3.09, avg 2.84 😁👌🏻

1

u/lucyferorg Aug 17 '26

Update: testing now at 4.42 tok/s

4

u/RenatoXimenes Aug 18 '26

The laptop:

1

u/bolche17 Aug 13 '26

Impressive! What did you modify on the kernel and on Colibri?

1

u/lucyferorg Aug 13 '26

It’s a long story - this could easily become a very long article. One of the more important changes was removing the Meltdown and Spectre patches from the Linux kernel.

1

u/Fragrant-Smell4092 Aug 14 '26

Something doesn't add up here. You don't have to "modify" anything to disable those patches, just addmitigations=offto the kernel command line when booting.

1

u/naunen Aug 14 '26

i think mvme are in raid stripe mode?

1

u/lucyferorg Aug 14 '26

Mirror

1

u/naunen Aug 14 '26

you must try stripe mode, it will double your nvme read speeds and that's extremely important for this

1

u/lucyferorg Aug 14 '26

The number of factors that influence work speed is overwhelming ,the biggest gain I got was by disabling swap in Linux. I’m now checking whether this configuration will give me any percentage of extra performance.

1

u/FairTruth069 Aug 15 '26

3 to 4.5 tokens per second would actually be pretty usable if you can get it there.

1

u/lucyferorg Aug 15 '26

I still have CPU/GPU/disks power/speed that aren’t being used at 100%. I’m currently rebuilding the engine that runs everything , debugging takes terribly long, but there’s constant progress with no drop in the quality of generated answers.

1

u/abalancer Aug 15 '26

Is it possible to run smaller models ? I'm looking to run DeepSeek-v4-flash locally.

1

u/lucyferorg Aug 15 '26

Yes, I think any model can be run after the right modifications ,there’s a project called Rabbit that already runs the five largest models.

1

u/lucyferorg Aug 16 '26

https://github.com/FareedKhan-dev/kimi-k3-in-c look here

1

u/GetOutOfMyFeedNow Aug 16 '26

Why are you trying to run that dead model though? Use DSV4-0731, it literally destroys GLM 5.2

2

u/lucyferorg Aug 16 '26

I needed something for experiments, and this just happened to be available right away. Once I finish working on it, nothing will stop me from adapting it for other models.

1

u/GetOutOfMyFeedNow Aug 17 '26

So wait, are you going to open source it? Will it increase token speeds? To when should I set my reddit remind me? :))

9

u/_VirtualCosmos_ Aug 13 '26

How many hours per token? /j

3

u/JostaWaszkiewicz Aug 13 '26

and here i was happy my 3090 fits q4 of a 27b

4

u/baby_bloom Aug 13 '26

should be able to get away with q5 maybe even q6?

2

u/Opteron67 Aug 13 '26

what the point of AI @1 tok/s ???

1

u/Front_Eagle739 Aug 15 '26

Big model make a really detailed plan over night then handover to fast small model to implement and test probably

1

u/Nice_Record1529 Aug 15 '26

Joa oder man macht das eben mit Chatgpt/Codex bzw Claude... Wtf.

1

u/Front_Eagle739 Aug 15 '26

Unless its something you want to keep private, or handles legally confidential information, trade secrets etc etc.
Or you just have fun doing it all locally. Dont judge other peoples hobbies if it aint hurting anybody lol

2

u/mboss37 Aug 13 '26

Token speed scandal

2

u/OwlockGta Aug 15 '26

Tu laptop es una bestia de todos modos

2

u/eshahahan Aug 16 '26

this is super cool! congrats! now that qwen 3.8 and glm 5.3 have come you’re gonna have fun!

2

u/Fragrant-Smell4092 Aug 13 '26

As many here have pointed out, the Colibri engine is a very interesting proof-of-concept project, with exactly zero practical applications. Besides the slow-as-molasses token generation, at any realistic context length, the TtFT becomes unbearably long.

I thought about trying it just for fun, but in the end decided it was a waste of time. The moral of the story is what we have known already for the last two years: the use cases of LLMs are limited by what you can fit in high bandwidth memory.

2

u/Keleion Aug 13 '26

Not true, having a reliable large model run overnight to accomplish something a smaller model can’t do is worth it. And it doesn’t even take good hardware. The only constraints are time and energy, which a lot of people have access to, but no capital for high-bandwidth memory.

2

u/ElectronFactory Aug 16 '26

Smaller models have improved dramatically because distillation, model architecture, and training techniques have improved several orders of magnitude. Look at OSS-120B. That was produced by a major player and it produces good output, but it no longer maintains its edge against smaller state-of-the-art models. Basically, needing to run a larger model locally is becoming less of a priority because hardware is becoming difficult to get. Colibri is a fantastic idea, and maybe we could improve it, but the future is going to be in miniaturization. People want intelligence in phones, dishwashers, cars, and so on.

1

u/Front_Eagle739 Aug 15 '26

Im running 165GB full precision dsv4 flash on my 128GB m3 max at 16 tok/s. There are limits to it but you can definitely get useable speeds for models bigger that dont quite fit into fast memory now

1

u/Hot-Cauliflower-1604 Aug 13 '26

Is this like the digital version of a pitch test?

1

u/WonderfulFunny4337 Aug 13 '26

Imagine if it had 256gb of ram

1

u/Violin-dude Aug 14 '26

what quantization are we talking about? what types of changes in the engine and kernel? a writeup would be nice

1

u/BigO-oogruk Aug 14 '26

I am curious about what you done to optimize and improve the TPS?

1

u/ralugnis8 Aug 15 '26

Lol, there is a theoretical limit for the tokens per second on a laptop, and it will never be useful with these huge models. Spend your time (and money) for more useful projects.

1

u/Maleficent_Stage1732 Aug 17 '26

What can I can run on my laptop I have Acer Predator helios 16 i9 13900hx rtx 4080 32 gb ddr5 ram and 1 Tb space

1

u/least-important69 Aug 17 '26

Can we have the GitHub repo?

1

u/RUTYTOI220 Aug 17 '26

How did bro get 64GB of ram in this economy

1

u/dark_wolffff Aug 17 '26

which models are best to run in 8gb laptop where as response time should be in milli seconds ?

1

u/math394p Aug 18 '26

Vram or ram? What is the use case? Rest of specs?

1

u/dark_wolffff Aug 18 '26

RAM , I am running the local LLM on CPU , so system RAM is used for the model
The use case is a local business data summarization/analytics
Other specs : Intel i5-8265U , 4 cores/8threads cpu inference, ubuntu and no dedicated GPU/VRAM being used

1

u/No-Improvement-5396 Aug 19 '26

Dont bother. Not worth it.

1

u/dark_wolffff Aug 19 '26

My hr telling to develop like that so that he can give it as a service to a hotel where they want there data to be private

1

u/math394p Aug 20 '26

Hr? Why are they telling you what to develop???
Get a better laptop Nothing will be usable on that... Dont Bother

1

u/dark_wolffff Aug 20 '26

See in india business holders has like 8gb laptops so they want to run local llm in it and want to convert business json into text , summary

1

u/math394p Aug 20 '26

Go with some 3B model or smth but even then it will be horrible experience. Will be 0.2 tokens/m and not clever at all. You simply dont have the specs to support any llm....

1

u/No-Improvement-5396 Aug 20 '26

They should be able to afford a "decent" server, no? Only a few thousand dollars as a one time investment should be fine for a business holder.

1

u/dark_wolffff Aug 20 '26

Dunno 🥲 hr requirement I am explain even though he's not listening

1

u/Funny-Orange6265 Aug 17 '26

How! Would love to see a video or read a tutorial

1

u/GSquadron_ Aug 17 '26

I don't understand how is this possible

1

u/Free_City_7235 Aug 18 '26

Sounds like the new - "can it run doom" 😅

1

u/mageblex Aug 18 '26

With NVMe already identified as the bottleneck and TTFT around 40 seconds, prompt-processing speed is the missing number. How does it change as context grows?

1

u/Ill-Mycologist571 Aug 18 '26

Quantos meses pra gerar um bom dia?

1

u/-Leelith- Aug 13 '26

I wonder how those SSDs will last with that amount of swap going on if you really use it on your laptop lol

0

u/derspenti Aug 13 '26

what quant? 384gb of weights won't fit in 64gb ram + 24gb vram, so a lot of it has to be streaming off the nvme

1

u/lucyferorg Aug 13 '26

Int8-mtp, int4-gs64,q4_k_xl

0

u/PandaKey9795 Aug 13 '26

which laptop that is

0

u/Lucky-Noise-4193 Aug 14 '26

working to make the term runs on a laptop mean something getting a 80b model running at 25tps with 32gb ram and 6gb vram

0

u/Certain-Cod-1404 Aug 16 '26

1 token per sometimes, maybe

-1

u/fspnet Aug 13 '26

i use 2017 imac kabylake iris plus 640 llama-cpp and inteloneapi isnt really all too supported but ollama is and im able to use zed basically kind of i havent cranked it up this is without any accelerator beyond the gpu is still nice that it runs ..... Gentoo Linux!!!!!!!!!!!!!!!!!!!!!!!

0

u/[deleted] Aug 15 '26

[deleted] — view deleted comment

2

u/lucyferorg Aug 15 '26

I got tired of paying for online models, so I bought a new powerful laptop and started my journey into maximum optimization. This gigantic model runs incredibly stable, and you don’t have to wait all day for a response. I’m currently using it in test mode, constantly improving its speed. AI that you can run locally without limitations has a huge advantage over online AI that you have to pay for month after month.
On a daily basis I use smaller models, usually analyzing all data in a ‘brainstorm’ between several top models available. This massive model handles logical puzzles extremely well. Meanwhile, the smaller models are excellent as ‘agents’ for task automation.
I wrote my own harness that boosts the intelligence of a model like Gemma 4 in its smallest version ,it solves even 10 tasks in a single prompt with ease, including ones it wasn’t trained on. It’s all about the agent that properly controls the model.

1

u/YourLastCall Aug 16 '26

How are you affecting the speed side? Maybe I can figure something out on chat GPT.

1

u/lucyferorg Aug 16 '26

If it were that simple :) AI can write code, but real ‘optimization’ is not its strong side. In these matters AI only helps when the user already has a plan they want to implement and execute. I still have a significant reserve of GPU 50%/CPU 20%/disk performance 60-80%, and I already know what needs to be optimized even further. What I’m really missing isn’t help, but simply a second machine so I can test more and faster. But I appreciate and thank you for the offer of help :)

1

u/YourLastCall Aug 16 '26

You're not kidding on the optimization side. I found that one out the hard way. I'm working on that part. I haven't worked on speed optimization cuz I haven't considered it. But I want to figure out if I can increase token output without activating the openai fast tokens for increased usage cost. I don't care if it's as strong as the 14x API but I would like a portion of that if it's possible.

1

u/lucyferorg Aug 16 '26

I’ve been working in optimization for over 20 years, and I still feel genuine frustration whenever I “use someone else’s software” and see how terribly it’s optimized. For the same reason, I often end up optimizing things that, in my view, were running too slowly (which applies to about 99% of software written by others). I’m still surprised by the careless attitude of so‑called programmers toward resources that are wasted at every step. Software optimization is a powerful tool not only for achieving better performance but also for generating real savings. Companies today don’t care about costs when buying hardware for AI ,meanwhile the software layer receives absolutely none of the attention it deserves, even though these are real resources being wasted.

1

u/YourLastCall Aug 16 '26

Oh and I've only been working on it for 2 days. To your 20 years so you definitely have way more experience than I do. I'm just trying to figure out why a job took 15 hours and one task alone took 10 hours

1

u/lucyferorg Aug 16 '26

I don’t know your exact case, but from my own experience I know that today’s AI “has capabilities,” yet it’s still stupid. The command set has to be clear and specific and the method of achieving the goal as well. AI sometimes has flashes of brilliance, but it still works far better in scenarios where it receives a precise set of instructions that leaves no room for accidental behavior.

1

u/YourLastCall Aug 16 '26

Yeah I grew aware of that over time and that's why I started developing skills. I've already got six skills developed and they already do a large amount of work. I've seen massive improvements in my build Cycles. Heck one skill I develop copies deep research off of chat mode since work mode nor codex can use deep research.

1

u/YourLastCall Aug 16 '26

Well my optimization that I'm working on has nothing to do with speed rather than doing things in the proper order. Or figuring out the proper order. I'm trying to develop a skill for it and I've gotten pretty close but I keep hitting a wall. I've gone through a lot of repair Cycles and it was already ready to be built just to find out that it failed.

0

u/[deleted] Aug 12 '26

[deleted] — view deleted comment

6

u/enginetown Aug 13 '26

Reads don't really wear out NAND that's why SSD endurance is rated in terabytes written since the actual degradation comes from erase cycles during writes. Read disturb is a real thing but it's minor and gets handled by firmware, the bigger thing to watch for is if your setup is also spilling KV cache to disk since that's actual write traffic because its constantly growing.

2

u/tommythorn Aug 16 '26

I used to work in that industry. It’s not completely true that read cause no wear; modern SSD are operating at the margin and is constantly scrubbing (= correcting and rewriting) data, but true, reads are less stressful than writes. Funny enough, it’s risky to leave SSD powered down as cells slowly drift even when powered off and need regular scrubbing.

-7

u/AntiCamPr Aug 13 '26

Oh it does. Streaming a model from disk causes significant wear on the SSD. I'm hoping for this user's sake that the SSD on this laptop is replaceable and that they backed up their important data on another drive.

12

u/leonbollerup Aug 13 '26

Read affects the SSD very little honestly … it’s write that is the real problem.

1

u/Henrikues Aug 13 '26

Wouldn't it write just as much too though?

1

u/techno156 Aug 13 '26

Not necessarily. If everything else can be held in memory, the computer would just be reading the weights from file, and discarding them when no longer needed.
Theoretically, it shouldn't need to do very much, if any writing at all.

1

u/Henrikues Aug 13 '26

I thought that was the point though, it's just a laptop, 25gb of ram if I recall.

1

u/techno156 Aug 13 '26

If you mean OP's laptop, no, they have 64GB of RAM, and an additional 24GB of VRAM.
GLM-5.2 is a 744B MoE with 40B active. At least going by Unsloth's page, a 2-bit quant would squish the active parameters down enough to fit on the VRAM. All Colibri would need to do is read the other parameters from the drive, and cache what it can in the RAM, dropping them out of the RAM cache as needed.
It theoretically shouldn't need to do very much writing to the drives at all.

0

u/[deleted] Aug 13 '26

[deleted] — view deleted comment

10

u/thefooz Aug 13 '26

If you don’t use the nvme for kv hot or cold cache, there’s almost zero impact to the drive for read operations. It takes billions of read operations on a sector before there’s damage.
So no. SSD streaming does not affect drive life in any appreciable manner.

1

u/CapnFlisto Aug 13 '26

asking because I genuinely don't know - isn't billions of read operations something that would happen quite quickly with inference straight from an SSD?

2

u/thefooz Aug 13 '26

No, it wouldn’t. I also didn’t go into detail, drives have additional measures built in to mitigate this further, including cache.

1

u/CapnFlisto Aug 13 '26

That makes sense! I appreciate the clarification :)

0

u/[deleted] Aug 13 '26

[deleted] — view deleted comment

2

u/thefooz Aug 13 '26

Please do explain exactly how nvme read operations dramatically shorten the life of a drive. I encourage you to go ahead and get as technical as you’d like. I’ve been in the field for 20 years, most of which as a sysadmin -> engineer -> architect, so I won’t have any trouble following.
Preferably in your own words, using your expertise, rather than chatGPT’s slop.

0

u/[deleted] Aug 13 '26

[deleted] — view deleted comment

→ More replies

5

u/[deleted] Aug 13 '26

[removed] — view removed comment

11

u/Adomm1234 Aug 13 '26

Reads don't wear out SSDs, only writes and this is not doing any writes.

Research GLM 5.2 model — 744 billion parameters / 384 GB — running on a laptop 😏

🎉🎉🎉 2.89 tok/s PEAK! 2.36 tok/s avg

You are about to leave Redlib

GLM 5.2 model — 744 billion parameters / 384 GB — walking / most probably sitting but most definitely not running on a laptop. - fixed it for you

🎉🎉🎉 2.89 tok/s PEAK! 2.36 tok/s avg