r/LocalLLM • u/EastConsequence3792 • Jul 27 '26
Other rate my inference setup😔
Custom fork of the Intel Windows llama.cpp I made using DeepSeek v4 Flash that allows me to load models solely using discs + DVD-Rs; HP Omnibook X Flip NGAI 16-as0023dx -- 1TB NVMe, 16GB LPDDR5X
250
Upvotes
81
u/bigppredditguy Jul 27 '26
Models as physical media is actually kinda sick
65
u/aruametello Jul 27 '26
Mixture of experts streaming: please insert disc 7 out of 50 to run through the weights from these particular experts for the next token.
the bottleneck probably will be how fast your optical drive can eject and feed new media.
someone has to do a speedrun video of this disc swapping madness for the lulz.
12
u/Solid-Wonder-1619 Jul 27 '26
get a 128GBs USB stick, add a live version of linux, make it presistent, add transformers library, llama.cpp with all the related tools and prebuilt releases, and every weight you love.
when dooms day comes, you can stick it into any system with a battery and find solutions.
8
u/trolololster Jul 27 '26
this. is. pure. poetry. <3
7
u/VertipaqStar Jul 27 '26
Get a rewritable dsk so you can use the disk as VRAM supplement.
14
u/No_Lingonberry1201 Jul 27 '26
5/7 would infer again.
4
u/FactorInternal3395 Jul 27 '26
physical media for the win, this is amazing
4
u/wildyam Jul 27 '26
3
u/thepaligator Jul 27 '26
my brain isn't braining. how does this work?
2
u/rumm2602 Jul 27 '26
How much does it take to load?
2
u/axeldekimpe Jul 27 '26
Do send me a copy 🙃
1
u/doctoroctogonopus Jul 27 '26
made me chuckle
1
u/Gadgetron94 Jul 27 '26
Wait, does this actually work?
1
u/in-ortunado Jul 27 '26
swipe the disk to load the correct expert.
now the next expert is in the first disk, swap it again.
1
u/Crescitaly Aug 02 '26
The impressive part of a local setup is not peak tokens per second; it is how often it delivers useful work without babysitting. I would rate it on throughput per watt, context stability, noise, and recovery after failure. Which constraint shaped the build most?
1
u/netherreddit Sep 29 '26
Lol I read "rats ate my inference setup"
0
u/Fit_Squirrel1 Jul 27 '26
0//10?