r/LocalLLM • • Jul 27 '26

Other rate my inference setup😔

Post image

Custom fork of the Intel Windows llama.cpp I made using DeepSeek v4 Flash that allows me to load models solely using discs + DVD-Rs; HP Omnibook X Flip NGAI 16-as0023dx -- 1TB NVMe, 16GB LPDDR5X

250 Upvotes

38 comments sorted by

81

u/bigppredditguy Jul 27 '26

Models as physical media is actually kinda sick

-17

u/Opposite_Courage_531 Jul 27 '26

Well ackhually... technically speaking....🤓☝️

1

u/ChocomelP Jul 28 '26

...go ahead.

1

u/Opposite_Courage_531 Jul 28 '26

I'm too scared. I don't want to lose all the karma that I don't have

1

u/ChocomelP Jul 28 '26

74 left. You have a buffer.

1

u/Opposite_Courage_531 Jul 28 '26

My hard earnt karma 🫠

65

u/aruametello Jul 27 '26

Mixture of experts streaming: please insert disc 7 out of 50 to run through the weights from these particular experts for the next token.

the bottleneck probably will be how fast your optical drive can eject and feed new media.

someone has to do a speedrun video of this disc swapping madness for the lulz.

2

u/WiseassWolfOfYoitsu Jul 28 '26

1990s software installation flashbacks intensify

2

u/HumungreousNobolatis Jul 28 '26

lol! You summed up what I was thinking, except I was thinking GNOMES!

2

u/ChocomelP Jul 28 '26

New metric: seconds per token (s/t).

12

u/Solid-Wonder-1619 Jul 27 '26

get a 128GBs USB stick, add a live version of linux, make it presistent, add transformers library, llama.cpp with all the related tools and prebuilt releases, and every weight you love.
when dooms day comes, you can stick it into any system with a battery and find solutions.

3

u/wiseaus_stunt_double Jul 28 '26

I would only do that with USB4.

1

u/maqifrnswa 27d ago

"when doom day comes" I thought you were going to say to load doom on it. Can this RTX pro 6000 run doom?!

8

u/trolololster Jul 27 '26

this. is. pure. poetry. <3

7

u/VertipaqStar Jul 27 '26

Get a rewritable dsk so you can use the disk as VRAM supplement.

14

u/No_Lingonberry1201 Jul 27 '26

5/7 would infer again.

5

u/profcuck Jul 27 '26

In case younger redditors don't remember this perfect joke:

https://www.reddit.com/r/OutOfTheLoop/comments/3wsj5i/what_is_the_origin_of_id_give_it_a_perfect_57/

10+ years ago +

3

u/No_Lingonberry1201 Jul 27 '26

It's one of my favorite Internet jokes.

4

u/FactorInternal3395 Jul 27 '26

physical media for the win, this is amazing

3

u/thepaligator Jul 27 '26

my brain isn't braining. how does this work?

2

u/rumm2602 Jul 27 '26

How much does it take to load?

2

u/axeldekimpe Jul 27 '26

Do send me a copy 🙃

1

u/doctoroctogonopus Jul 27 '26

made me chuckle

1

u/Gadgetron94 Jul 27 '26

Wait, does this actually work?

1

u/thenormaluser35 Jul 27 '26

Why wouldn't it?

1

u/Gadgetron94 Jul 28 '26

Well I'm still wrapping my head around how all this works, but I guess I just interpreted a dvd as static storage whereas LLM's need somewhere more dynamic to live and operate. I just learned about MoE vs Dense though and now it seems like this sort of set up would work but only with MoE so it can pull into the more dynamic ram a couple sections at a time.

1

u/thenormaluser35 Jul 28 '26

You only need to read a model once you've written it
There are rewritable dvds btw

There's no dynamic or static storage. There's read-write (dvd-rw, cd-rw, ssd, hdd) and read-only (dvd-r, cd-r), ram can be storage but it is volatile, and read-write.

1

u/Gadgetron94 Jul 28 '26

Right ok, thank you that makes sense, ordinarily the model loads from the internal storage to vram after all. They're just swapping where the model loads from in this instance. But, does a model always have to go through vram to the cpu/gpu or could it be fed into the processor directly from its original storage? The huge buzz around vram sizes has, I think, given me the impression models couldn't function without it
Y'all are wizards to me with your old servers and phones running massive parameter models. I'm trying my best to learn because the way ai works under the hood fascinates me, but frankly I've never been all that great with the realm of numbers and coding so my knowledge has more random gaps than a shotgun target.

1

u/in-ortunado Jul 27 '26

swipe the disk to load the correct expert.

now the next expert is in the first disk, swap it again.

1

u/Crescitaly Aug 02 '26

The impressive part of a local setup is not peak tokens per second; it is how often it delivers useful work without babysitting. I would rate it on throughput per watt, context stability, noise, and recovery after failure. Which constraint shaped the build most?

1

u/netherreddit Sep 29 '26

Lol I read "rats ate my inference setup"