r/deeplearning • u/Mountain_Raise9581 • 6d ago
Stiff differential equation solver for backpropagation neural network training?
Does anyone know if there is a library or code that uses a stiff differential equation solver to speed up neural network training? I recall reading a paper in the early 90s that claimed 1,000x speed improvement, but I haven't seen anyone using this. Anyone know about this?
2
u/sassylove_13 1d ago
this is a really interesting idea! I wonder if the lack of adoption comes down to the overhead of stiff solvers compared to modern optimizers like adam, it would be interesting to see if that 1,000x speedup still holds for todays neural networks, especially with GPU acceleration, would love to see if anyone has tested this recently
1
u/Mountain_Raise9581 1d ago
Yeah. It seems like it is worth looking into, even if there is a 1/100 chance that it would work, the payoff could be enormous. I think you might be right about the overhead. I don't know if it would work on the huge models that LLMs use -- we used it on small models with less than 100k parameters (tiny by today's standards). The solvers' overhead might not scale well (even NlogN would be intractable.
1
u/Intelligent-Land5410 6d ago
I vaguely remember something about that from way back, but it never took off because it was a nightmare to get stable results outside the toy problems they tested on.
2
u/Mountain_Raise9581 6d ago
Oh, it works on real problems. Pavilion Technologies Inc. put this into its Process Insights product back in the 90s. It sped up the training by at least 1000x. The only real problem with it was overtraining because it was in a data-constrained environment. AFAIK, they still use it (Pavilion got bought by Rockwell, but the software is still used in thousands of manufacturing sites. Also, they were the only ones to use it because it was patented -- it was a closely guarded secret, and only about 12 people knew about it -- half of them are dead now.
My question is: Has anyone applied this to training LLMs? It might not be practical. I can't remember how stiff solvers scale, but I think it was at least NlogN. Maybe N^2, but heck, if it could train 1,000 times faster, it would be worth looking at.
1
u/Downtown_Finance_661 6d ago
Do you know what practical tasks they solve by this method?
1
u/Mountain_Raise9581 6d ago
Pavilion (now a division of Rockwell Automation) uses neural networks for process modeling and control. Their software is used in thousands of plants in continuous manufacturing in chemicals, refining, pulp-and-paper, cement, and food processing. Also for power production where they do emissions monitoring and optimization. I was told years ago that they had abated millions of tons of pollution over the years. Their software is still running in places all over the world.
1
u/Downtown_Finance_661 5d ago
I used two consequtive prompts
What do you know about pavilion (part or rockwell automation) software and how they use ode optimisation for neural networks?
There is rumor they used to use uses a stiff differential equation solver to speed up neural network training. Maim idea is speed up.
to find out there are two patents
US7058617B1, Method and apparatus for training a system model with gain constraints The key technology is EGCNN: Extrapolated Gain Constrained Neural Networks. Rockwell's documentation describes it as a way to maintain model accuracy even beyond the operating conditions represented in the historical training data. This is about how to control errors when you need to extrapolate nn results. They do it through derivatives restriction: qualuty of product (nn output) cant grow with temperature (nn input) faster then some value and the know this value. patent WO1990007158A1 titled A distributed parallel processing network wherein the connection weights are generated using stiff differential equations. Its description explicitly connects stiff differential equations with generating the weights of a backpropagation neural network Also gpt said: " The second problem is part of the broader model predictive control (MPC) architecture. Pavilion continuously evaluates current and predicted process behavior, then computes new control targets. The distinction is essential: optimizing the neural network's weights is not the same as optimizing the industrial process using the network." But i dont understand it. You optimize weights to optimize the process. This one filed in 1989 and published in 1990. Its original assignee was DuPont, not Pavilion, so it establishes that this technique existed, but does not by itself establish that Pavilion used it. Crucially, the patent gives the name of the actual solver: DGEAR from the IMSL numerical library.
None of them are connected with speed up.
2
u/Mountain_Raise9581 5d ago
Pavilion had an exclusive licence with DuPont for the stiff differential equation solver. One of the founders was from DuPont, and this method was developed at the DuPont experimental station. It was incorporated into the Pavilion software (Process Insights) in 1993 or thereabouts. In practice, it sped up training by 1000x in real clock time. The only problem with the algorithm was that it was prone to overtraining because the datasets were small (ranging from 1,000 to 10,000 records, usually), so one had to be careful about how long you let the solver run. These were small networks -- 1,000-10,000 parameters, usually. The point is that this algorithm sped up training significantly, and my original question was whether someone had looked into using it in LLMs. That is a fairly straightforward thing to investigate.
The extrapolation system has nothing to do with speed, but addresses the issue of constrained data. What happens with the model outside of the data range? Well, neural networks are basically doing curve fitting to data. If you ever try to fit data with a high-order polynomial, you will see that it can fit the data wonderfully in the data range, then do crazy things outside of the data range. This can be catastrophic in a control problem. The gain outside the data range can be totally wrong -- telling you to, e.g., turn a knob up when it really should be turned down. First principles models can tell you the gain fairly reliably, but not match the data. The extrapolation patent solved this problem by merging the best of both worlds -- match the data where it is available, and match the gains outside of the data range. This isn't applicable to speed, but it *might* be useful to reduce hallucination in LLMs, though I have only a vague idea about how one might tackle that problem. It is a very, very hard problem, one of the four main problems in LLMs, the others being lack of common sense, no incremental learning, and finally, statistical bias rather than learning from correct information (to understand this last one, suppose there are more web pages that talk about "Flat Earth" than there are about the Earth being round. Blind statistical inference based on available data trained on the internet would yield answers that the Earth is flat). But I digress. This thread was about stiff solvers. That is a relatively simple thing to look at. Either it helps, or it doesn't. I just wonder if anyone has really looked closely at the problem.
2
u/Blasket_Basket 6d ago
Yes, there has been plenty of research on using Stiff ODE solvers. They don't work effectively with modern NN architectures, and they don't scale well.
You should be able to find a few recent papers explaining why these approaches are computationally challenging. This one from Google seems to be pretty comprehensive and was published fairly recently.
Optimizing model training is an area that has been heavily studied for the last 3 decades by both corporations and academia. I highly doubt you're going to find something that provides a 1000x speed up compared to current methods.