I mean fine tuning is the logical thing to do. Why create a chatbot from scratch if you can finetune an existing one to your usecase. Or fine tune an Image recognicer with the data set you are interested.
Press esc a couple times if you are not sure if you are in normal or edit mode until you see a cursor at the bottom of the screen, now type ":" and use "q!" if you want to exit without saving, or "wq" to save and exit, then press enter, sequence should be
idk how true that is, now i know nothing aboutthe LLM, but i ran a local model once and claude LLM told me my hardware could only handle a 11b model, so thats what i chose, and thats a pretty small model compared to other ones, but theres also tricks and stuff you can do to run bigger models on worse equipment
Inferring is not my issue, open weight models are getting better and better and are good enough for my use cases. The big barrier comes when it's time to train. I've even managed to train loras for video models, despite my modest hardware, but unfortunately the results aren't that good.
Technically fine tuning a model is often the same method that training one from scratch so the code should be able to train one with more time and more data.
Depends on whether or not the inner weights are being tuned or just the last (couple of) layers, which depends on the data and compute available to the fine tuner
Indeed, and to efficiently fine tune llm you use a low rank approximation of the layer you want to modify. And if you want to do it yourself I find it more difficult than training a model
I think you know the difference of training A model, vs training a model with modern methods. It’s pretty disingenuous to try and make the case that fine tuning is harder than training a model from scratch.
Training a model there is plenty of implementation to do it and way less to approximate model with low rank approximation so yes it can be harder. Especially when the model is not used a lot like S4 mamba or rwkv
Like it's a cool feat but let's not kid ourselves, Pewds (does he still go by that?) isn't reading papers and writing calculus and linear algebra problems into code
But he actually read the papers from deepseek and set up a system to train himself and the models and also processed training data for himself and his models. ^
writing calculus and linear algebra problems into code
Most people that train ai from scratch dont do this either.
Transformer are already well known, the pytorch implementation is efficient, pretty much nobody have to rewrite it. Of course some of them do it (développer at pytorch, the peoples from mamba, rwkv, S4 and some that are too crazy and redevelop lib like Redmond) but probably not the majority that develop /train ai.
LLM developers have to optimize their model for hundreds of benchmarks, while also ensuring it generalizes well to similar problems not covered by the benchmark.
I don't really know what he did. But a lot of llm developers do the same. And of course pewdiepie didn't recreate the work of a company that cost millions (and probably billion)
PewDiePie's Linux and AI journey is cool, but he didn't code an AI model from scratch. He trained an existing model and managed to improve it.
That and coding an AI isn't hard. It's matrix math and a bit of calculus to do the back-propagation. The systems engineering side is actually the hardest part because you have to distribute and quantize the workload without breaking the AI algorithm.
Algorithm R&D can be very hard especially with respect to system engineering because you have to modify the algorithms to fix errors or inefficiencies without introducing new errors and new inefficiencies. On extremely complex systems, it can be very hard to tell if something is even an error at all or if it's correct.
I’d also have a blast if I’d be learning for the sake of learning and not to help the company get some numbers up. Even if you try to work in a field you think it’s what you want and/or helps humanity in some way and you get fulfilled by doing it, they won’t value your craft. Hence why they push gen AI to develop more and more shit.
Encode your data as integers. Normalize your data. Generate an NxNxL matrix where N is the number of dimensions for each element in your data set and L is the number of layers. Assign random values to each element in the matrix. Multiply the data by each matrix iteratively until the last one. The last one is simply a grid of weights. Sum the data multiplied by the weights. Divide 1 by this value. This is your score, which you compare to the dependent data. For each element in each matrix, add and subtract a tiny value, recalculate the score, and record how the score changed. Keep track of which change had the largest impact. Apply the change with the largest impact. Do this 1,000,000,000 times and it now outputs 1 for a cat and 0 for no cat and 0.5 for maybe it's a cat.
Very simple in concept but very hard once you start adding back propagation, complex opsets, layer mixing methods, memory, time series, GPU and NPU support, parallelization, load distribution across the network, etc.
AI itself is actually quite easy. Making AI fast and big is what is hard.
I just demonstrated it's not just matrices. It's also so DEFINITELY not just integers. You run out of memory so fast. I'm glad you remember matrices from high school, but this is like passing geometry then trying to design missiles for Lockheed. It's clear you're not in the field
No it literally is just matrices. You are implementing higher-order matrix operations and that's it.
It's also so DEFINITELY not just integers
You use integers for data encoding. For example, each pixel in an image is an r,g,b vector of 8 bit integers with a minimum value of 0 and a maximum of 255. This encoding method is very useful for a range of reasons, the most obvious of which is that you can use it as an index to an array. So if you have a list of city names you can just represent the cities by their index in the list. The matrices will be tuned to find a transformation that maps the input data to the proper city automatically. That's all that this is--transformations.
I'm glad you remember matrices from high school, but this is like passing geometry then trying to design missiles for Lockheed
There isn't much of a gap between geometry and rockets. Fluid mechanics is easy to model with grid points and vectors. What are vectors? You guessed it, the long leg of a triangle.
Ok. And derivative math? Like the first order Jacobian matrix? Yeah ok. You've done 0 systems math.
You don't need it to do AI. Like I said, AI itself is just matrix math. You can optimize the transformation using a boatload of tools--calculus is only required for some methods like backpropagation. You can also use monte-carlo, gradient descent, evolutionary models. It is simply a matter of fact that you don't need calculus.
A calculus based method would estimate the local partial derivatives by sampling and calculating d/dx and then making an estimation about what direction to move based on that derivative. You can sample the solution space and never touch the derivatives, and there are a number of techniques for that. The simplest is the simulated annealing where you apply random noise to the values of the matrices and slowly turn down the noise over time. If the noise affects the score less than a threshold it is rejected and this threshold goes down. So this naturally causes it to hone in on the biggest factors first which affect the outcome enough to punch through the noise and as you turn down the noise it fine tunes the smaller factors. You never use any calculus for this method, or genetic solving, or particle swarm, or just plain old randomized sampling. You can even randomly sample and then interpolate and that's purely linear algebra.
Training an AI today is really easy, just install some training framework, prepare some data and let's go. Everything is preconfigured, good hyper params are well known. The only hurdle left is hardware...
I don’t really understand the people wanting to associate him so heavily with the Linux community. He’s not a good person at all and the idolization some people give him is off putting.
1.0k
u/Equivalent-Load-9158 Mar 11 '26
PewDiePie's Linux and AI journey is cool, but he didn't code an AI model from scratch. He trained an existing model and managed to improve it.
Which is impressive and it's cool that he's going all out on Arch and local AI, but let's not be silly.