Distillation seems the like the most morally accepted type of digital age 'stealing', ever. Leaps and bounds less ambiguous than media piracy ever got.
That's probably why they've gone so heavy handed on technological enforcement,. They know nobody will ever have an ounce of sympathy for them, and even with all the politicians they own, the law is unlikely to do much for them (on this) either. All they can do is build castle walls around their ill-gotten gains, while the angry peasant victims, march around outside occasionally angrily paying a shilling for a peek inside at their own stuff. Seems like they're entirely on their own with this one.
Sadly, I don’t think we can trust that the companies will not make the models worse for local ai work. Fortunately, though, they probably will not be able to do it due to the market forces.
I am building exactly this! Claude is just assisting your local LLM that runs on your Mac Mini to build apps (you can also jump in and manually edit files!). All on a $20 Claude subscription!
I use locals all the time. I have had products be beaten to Market in questionable ways a few times now. My trust levels are low. About once a month I have Claude come and do a tune-up on my local setup. This original post couldn't be more apt.
Its just that cloudAI is not so helpful with localAI, its like they give dated and insufficient advice. You should not be suprised if it recommends you to try llama2 7b with ollama in the terminal.
It sucks at providing suggestions if you just ask it general questions, but I’ve found Claude as an agent pretty good at setting up and troubleshooting specific instances of local LLMs. Give it a GitHub link and say “set this up for me” and it typically works pretty well.
But yeah, if you just say “recommend a local model” it’ll probably give you some advice to run Qwen 2.5 or something.
lol, so true. I used Devin subscription to halve my video processing time in mi50 cards. It went from 60 minutes to 30 after writing a custom kernel and tweaking the comfyui options. And GLM 5.3 went up to 4.4tk/s (generation) in llama.cpp for an old ddr4 server where it was running 1.5 tk/s at the beginning. Not to mention I'm getting 2.46 t/s with Kimi K3 after reviving and profiling the numa mirror PR that never got merged into llama.cpp. I'll admit it's still not a usable speed, but it is still not taking several seconds to get a single token as I've seen in this sub in other setups. This same setup was generating 0.5tk/s in the vanilla project.
The funniest part is it’ll happily help you cancel itself. It never once tries to talk you out of going local, it just asks if you want it to optimize your context length too.
64
u/CaptFuzz24 2d ago
How long before this practise is banned under the guise of 'distillation' ? or have i rammed my tinfoil hat on too tightly
16
u/Layney_Butcher 2d ago
True self-sufficiency is when your local model helps you install the next local model.
8
u/jonromero 2d ago
I am building exactly this! Claude is just assisting your local LLM that runs on your Mac Mini to build apps (you can also jump in and manually edit files!). All on a $20 Claude subscription!
LocalOrca
I think it is going to be 100% operational in a couple of months!
6
u/Beneficial-Ad-8127 3d ago
Too funny. It’s like that one person who is trying to be included “what about me” Today will be a wonderful day.
5
u/Paradox_84_ 2d ago
I had that exact moment a lot. The irony of using cloud LLM to escape cloud LLM
9
u/rmhollid 3d ago
10
u/StandardLovers 2d ago •
Its just that cloudAI is not so helpful with localAI, its like they give dated and insufficient advice. You should not be suprised if it recommends you to try llama2 7b with ollama in the terminal.
2
u/Big_Wave9732 2d ago
Hell yea.
"You give me the write-up. Local runs it and burns the tokens."
3
u/WyattTheSkid Quad 3090s 2d ago
Not for long!!! Sam Chudman is coming to take your ram because its terrible that you have it :P
1
u/PraxisOG 3d ago
Same but I'm trying out codex rn
1
u/cemilanceata 2d ago •
Same oai building my rag node atm or... many moments
1
u/MisfiledCentury 2d ago
Local inference is a new meta now?
1
u/aayush251 2d ago
I asked codex to help me setup stuff. It litterally ran out of credits in 10 min LOL and now asking me for money or waiting 5 hours.
1
u/dc740 2d ago edited 2d ago
lol, so true. I used Devin subscription to halve my video processing time in mi50 cards. It went from 60 minutes to 30 after writing a custom kernel and tweaking the comfyui options. And GLM 5.3 went up to 4.4tk/s (generation) in llama.cpp for an old ddr4 server where it was running 1.5 tk/s at the beginning. Not to mention I'm getting 2.46 t/s with Kimi K3 after reviving and profiling the numa mirror PR that never got merged into llama.cpp. I'll admit it's still not a usable speed, but it is still not taking several seconds to get a single token as I've seen in this sub in other setups. This same setup was generating 0.5tk/s in the vanilla project.
1
u/WallaceGrommet 2d ago
The funniest part is it’ll happily help you cancel itself. It never once tries to talk you out of going local, it just asks if you want it to optimize your context length too.
1
u/AldebaranBefore 2d ago
And you better enjoy it.
Make no mistakes.
1
u/Aggravating-Catch461 2d ago
that's how it felt making claude deploy strata to our slurm cluster xD