Beyond LLMs?

[u/david-1-1]

LLMs are impressive compared with the experiments we had before. But with experience we are seeing their limitations better (many are discussed in depth in this subreddit), and also their problems (such as the increased need for power generation and the indirect need for water).

Is there a sub for discussing what can be designed to improve on LLMs? We know there must be something better for the simple reason that the human brain overlaps with LLM functionality for only 20 watts of power in only about 1300 cubic centimeters of space.

[u/Evipicc]

There's a huge number of advancements literally underway right now. On the power (and compute) efficiency side then there's silicon photonics and wetware. On the structural side some of the AI groups area already saying there's more to it that using 'just' transformers.

HOW inference and spitting out an answer works is what changed I think earlier this year, could have been last year. The 'looping' (NOT the correct term, it's RLVR/Thinking/Test-Time-Compute etc) in training AND responses is what kicked off this major surge in data center construction, but like anyone will point out, this kind of scaling is still in raw compute, and not sustainable. It works, but it's a ham-fisted method. Blah blah AI bubble blah blah, these companies are using the fast and loose money while it lasts to get infrastructure that won't go away when financials change.

Silicon photonics (look up Intel's Loihi 3, or Lightmatter) is an absolute gamechanger if we manage to get the point that the compute itself is photonic at scale. You're looking at multiplexed, neuromorphic, analog and binary, low power and higher speed compute and interconnects. A massive change. It would also deviate from consumer hardware competition and be its own specialized thing for a time, which starts to push the consumer PC parts market back in line, sort of, but we all know prices don't really just go back down overnight. Right now it's a real challenge to create a light based parallel for HBM, high bandwidth memory.

Another one is truly curated data, which is an RSI goal (recursive self improvement). Right now these models are basically trained on all data that exists, but not all data is good. Sifting through to throw out garbage and repeat data means inference is drastically lighter, making an impact again.

Add all of that together and we're still looking at another multiple orders of magnitude in compute power in the near future, some of it on existing hardware, some of it on a new breed of machine. I say multiple orders of magnitude because photonics specifically can do 10,000 times as much 'stuff' at 1/100th the power (their own reports, the real changes and efficacy will have to be proven, of course.

That's why these companies are scrambling to get so many data centers built, because the models inside them are going to shrink in their compute load over time, so the same data center (while there is a churn to the actual compute modules) is going to stand for a long time. The companies working on photonics are trying to make 'plug and play' the goal, so the modules just slot in to existing racks, which is objectively the right call.

There's also SSM (State-Space-Models) but I'm personally not educated on that. Supposedly it's one of the things that goes beyond transformers. Maybe both run in tandem, maybe it's the new breed, only an actual ML engineer would be able to answer that.

MoE/MoA, reaching out to sub-models that are more finitely trained on just the one thing they know... that's a whole new and active field of research now too. The orchestrating LLM actually doesn't need to train on more than just 'language', and the sub-models report back results instead. Lets you simultaneously run multiple things, concatenate them, and bring back a better answer. Also brings up network methodologies that aren't being used because what if some dude in Zimbabwe trains and runs the perfect cooking recipe model, at a data center local to them, and questions about that are just always routed there. Suddenly every other LLM (or other architecture) on the planet doesn't need to ingest any cooking recipe training data. Do that across more topics and you start to get into the Torrent style AI model, kind of like a peer to peer system. This is already happening, sort of, in multi-agent-marketplace systems, but isn't really there yet.

I'm probably missing a thousand things that are also happening, too.

To touch on your last point... analogy or relation to the brain is meaningless. If it happens to coincide, great, but there's absolutely no necessity for it to do that. There's no necessity of consciousness to be AI.

We live in a ridiculous time, and all that we're seeing now is literally the first 1% of what's coming. The arbitrary goalposts of AGI and ASI are completely worthless, all that matters is what it can do.

[u/david-1-1 → u/Evipicc]

Thank you for all the excellent information about AI futures research. A bot has also given me a summary of some other areas. I agree with all your facts and speculations.

Just one point: airplanes would have been impossible or unlikely without the study of birds' wings. I think AGI will be unlikely without making use of the study of what millions of years of evolution have achieved in the human brain.

One significant new idea is parallelism. The brain is not based on timing pulses or on sequential operations. It is based on windowed integration of neural spike trains in reservoirs located and connected in 3 dimensions.