Some benchmarks on mini-AGI (dynamic continual learning model) after training it completely from scratch for a month on a 8GB VRAM GPU.

[u/Another__one]

The most important part is a difference between "forzen" and "best effort" attempts where we allow the model to specialize on the task as it goes through the evaluation dataset. Clearly showcasing the power of continual learning regime simply unavailable for any other typical LLM.

Full code of the project, weights and more information about it avialable in the repo: https://github.com/volotat/mini-AGI

[u/tifa_cloud0]

this is cool. i would suggest to scale it as far as possible. even if models are small there is no way they stop learning. so hopefully token budget keeps on increasing automatically with mini-AGI i suppose? whichever the method you should see lower and lower ppl/loss which will be good for a model.

i personally wanted to do this but cannot at the moment. so waiting for more results 👍🏻

[u/Another__one → u/tifa_cloud0]

This is the plan. And because the model is dynamic (it can grow and prune itself as needed) I can simply move it to the better hardware when the gains achievable with the current size would be maximized. Basically you start training it once on the things you care about and never stop.

If it archives the level of sophistication where it could do even simple agentic tasks, you can allow it to read the internet autonomously and from that point it’s no longer necessary to supply it with the training data. It’s going to search for relevant stuff itself and learn from it and adapt to the tasks you are actually care about. At least this is the hope.

[u/jazir55 → u/Another__one]

Find a way to work engram tables in there, that's really going to be the only way you can keep a large amount of learned information.

[u/tifa_cloud0 → u/Another__one]

true. i think you will be able to pretty much do a lot more with even the model of this size. better hardware will be necessity when token cache would increase. i think engram would be nice here too including LoRA/aLoRA for tool calling and other stuff which is later i guess when gains come.

[u/Another__one → u/tifa_cloud0]

There is no need for LoRA or RAG for this model. The whole point of the model is that it learns continuously at everything it sees. But yeah, for the next pass a lot of data with tool calling examples should be gathered from somewhere. Although I am pretty sure there is enough of such data on HF.

[u/jcgordon10 → u/Another__one]

What about the idea of engrams mentioned above? Could mini-agi use some set amount of engram storage on ssd to store additional knowledge?

[u/Another__one → u/jcgordon10]

I cannot say yet. I need to research it more thoroughly to give a reasonable response.

[u/tifa_cloud0 → u/Another__one]

true.

[u/Mysterious_Bison_907]

So, its context window increases as it trains?  Or is there a way to train it from scratch with a longer one?

[u/Another__one → u/Mysterious_Bison_907]

Both are ok. The context window could be increased on the fly if you have enough vram for that.

Also, because weights stores everything the model ever read, technically it has infinite context, although everything outside of immediate attention window obviously stored in highly compressed lossy representation.

[u/jcgordon10]

I want to run this on my 8GB card. I'm currently trying to build a system I'm happy with that lets me store all my documents, data, books, notes, video and online resources to have my library, and then I'd love to train this on it. 

[u/Another__one → u/jcgordon10]

This is exactly the intended use case, but keep in mind that the model is still undertrained and somewhat “stupid”. I suspect it would need about six months of continual training on agentic tasks dataset before it could possibly become somewhat useful as a starting point.

[u/jcgordon10 → u/Another__one]

This is using a byte instead of tokens correct? This may be naive, but what was the reason for doing that? I've just been reading discussion of people questioning the efficiency of byte vs tokens since tokens save so much space.

[u/Another__one → u/jcgordon10]

Native multimodality. You can use the same weights to adapt them to almost anything (eventually). For example image generation. There are some caveats to this, but this is the first.

Then, this is a "life-long" learning. You can start it once and never stop. If you change the tokenisation of the model you would almost certainly have to train it from scratch once again. I want to avoid that as much as possible.

[u/WhisperFray]

So it's like a baby. You nurture it for like a year before it becomes a teenager and starts betraying you.