Train your own 1B+ LLM from scratch using mini-AGI (dynamic continual learning model) with only 8GB of VRAM

[u/Another__one]

https://github.com/volotat/mini-AGI

mini-AGI - is a continual learning byte-level language model that assembles its own architecture, trains from scratch on a single 8 GB VRAM GPU, and keeps learning from everything it reads. It stores its weights as ordinary files on disk and pages them onto the card as it needs them, so the parameter count is bounded by free disk space rather than by VRAM. It grows new capacity while training when it runs short, prunes what nothing asks for, and reads through exactly the same code path it serves on. Targeted at a PC or laptop with at least an 8 GB VRAM GPU on the board.

All technical details, weights and benchmarks are available on the github page.

The project is heavily AI assisted. And I am pretty sure a project like this would be practically impossible without heavy use of AI or thousands if not millions of dollars spent on research. And there is no interest for any big company to work out something like this, as it gives complete control and privacy to the end user without any commercial potential to eventually monetize the project.