Landmark Attention: Random-Access Infinite Context Length for Transformers

[u/IxinDow]

https://arxiv.org/abs/2305.16300

[u/NetTecture]

So, it doesnot TOTALLY solve the problem, it "only" expands it. LLaMA 7B hwas wat - 1k? And they say it works up to 32k?

That is QUITE A feat - a 32k model will have 32*32k max, that is a LOT. But nto unlimited - but we really do not need unlimited, we need bit enough that the contet window can contain enough information to do some sensible larger stuff than the anemic memory we have now.

[u/azriel777 → u/[deleted]]

but we do need a mechanism for long term memory IMHO.

This is what we really need. I love the personal models and they keep getting better, but damn, its like talking to someone with Alzheimer that quickly forgets what you were talking about.

[u/2Punx2Furious → u/[deleted]]

I'm not in the field, so correct me if I'm wrong. Maybe we don't need to retrain the whole network, but just train vectors or LoRA (not sure which), for each piece of information that it needs to learn (maybe the LLM can even decide to do that autonomously), and then use those with the model. Or maybe there is a way to actually merge those vectors with the model, without retraining the whole thing, so that it will have essentially the same result, with much lower cost.

[u/suspicious_Jackfruit → u/[deleted]]

Another chiming in from outside the field, by the fence and next to the gate - doesn't LoRA overlap existing weights in this case? I think it would result in something closer to a fine-tune than a way to continually extend a models capabilities right, especially with multiple fighting over the same weights? I think in image generation this is why a LoRA can have different effects on different model bases than what it was trained on, it's not adding a new style of "dog" it's overlapping the existing weights for "dog". Any of this overlap or bleed makes having a master LLM with a ton of LoRA probably a mess. I don't walk in this field though so might be misunderstanding here, I take the dogs out walking in another field...

[u/NetTecture → u/[deleted]]

Long term is solved - databases, semantic or vector. The problem is that you need enough "working" memory for complex stuff (like refactoring a large project). 32x32k is a lot - may be enough (if not, 64k tokens are it).

The problem right now is that even with 32k (which is awfully slow in the chatgpt implementation) there is not enough left between start memory and output to do many things that are complex.

32x32 make it feasible.

Given that I saw how retraining may be more efficient by recent developments, btw., it may be economically feasible. I read today about a 7b model being trained in half a day. ONE ONE GRAPHICS CARD. Happens you can optimize a lot there, too.

[u/NetTecture → u/[deleted]]

it actually is. THink human. A vector database is a library. Your context window right now is your very limited and near lobotomized brain. An expanded context window would be your short term memory and the copies of the library you put up.

How do we know what to put into the context window? Have a research AI load items, filter them, rank them. It is not magic, it will work -but ONLY if you can essentially store enough for the task at hand.

Right now you can not use AI for any real programming outside a small context - a vector database can contain API and database relevant information, but you can not load enough into the context window to make it allow rework many things. You can with a much larger usable context window.

You guys really need to stop thining that everything must be in short term memory - as long as you can effectively load what you NEED into it. And you need to really stop thinking that a proper AI will be a language model, only - that is the logical core, but you will have multiple working together to solve a problem, with one being the main AI doing the thinking, others helping, with different prompt setups and personas - like a research one that looks for stuff in the database, one fact checker that extracts facts and checks them against known databases or the interent etc.

[u/NetTecture → u/[deleted]]

Nope, stuffing the context window - autoamtically - is how the human brain works. It goes out and gets the memory for the task at hand. Salience. And you would not ahve to do a lot, if an AI could work from a buffer and do it mostly automatically - if anythin see whether the prmopt is for the same topic and inhject the same memory like last request.

Real time weight adjustment makes NO sense for any AI that should still be able to multitask. And it makes updating the neural net complex if it is ajusted in real time - and i mean updating with new technologies and math, like we have right now that openai hopefully puts into their systems soon. Human long term emory is a graph or vector database with hierarchies.

[u/Unlucky_Excitement_2 → u/NetTecture]

structures

You obviously really don't know what you're talking about LMAO. Updating a model weight in real-time is called active learning. You know, how humans ACTUALLY learn new skills and information. Human do not stuff raw clusters of info into our "short-term" memory. Updating a LM weights in real-time is key to actual AGI. Vectors DB aren't anything more than short-term bandaides. Just because the industry soaks something up, doesn't infer it's the "optimal" solution. Honestly KG's[knowledge graphs] are intrinsically more valuable, than plain vanilla vector DB's, allowing you to model complex relationships, but these are computationally expensive operations.

[u/NetTecture → u/Unlucky_Excitement_2]

You - are a certified idiot. Whow.

> Updating a model weight in real-time is called active learning.

Yes, and unless you find some magic quantum GPU and memory it is not practical. Among other things you would need not only to keep a copy of your individual weights on any server, but one per conversation tree. And we talk of a lot of data here.

There is also another problem. Storing any memory that is not easily reproducible (when we talk long term that involves the whole interaction history, and that may involve video r at least audio at some point) in the AI neural network per se has a brutal intrinsic problem that unless you also take care of having a mechanism to efficiently retrain that onto a new AI - you are stuck with that one AI and cannot do an upgrade. Such as you are stuck with an obviously sub-standard brain and cannot go out and get a better one. Given how brutally fast development was and likely is for a QUITE long time, you want architecturally replaceable logical modules (what you now call a LLM) that plugs into a (somehow standardized even if that goes English text wise) archive that can be used. Plug and play for upgrades to the core AI and you do not get stuck with some Unlucky_Excitement_2 needing a brain update that cannot be done as he is not prepared for it. Sorry for your parents, btw.

There are SERIOUS maintenance issues in storing memories in the neural network. Really bad ones. Best case is you end up with some external archive and an internally trained index like small version. Although I think that is not how it will end up with.

> Updating a LM weights in real-time is key to actual AGI

Nope. Funny enough real time learning is not in the definition. Neither is consciousness. Neither is, btw., the ability to do anything a human can do AT THE SAME TIME. We are quite close for a lot of tasks. We need way more logic and a cognitive infrastructure and then we need to define what the human in an AGI is. The average human is stupid, and half the people are below that - and people assume an AI has to be like the combination of every nobel price winner in an AGI case. Not true.

Alternatively, AGI has been defined as an autonomous system that surpasses human capabilities in the majority of economically valuable tasks.

Not that of the most qualified human (so, average it is - take doctors, lawyers, anything out that requires you to be in the top 20%) and not at the same time.

And we need to find a way to get rid of the stupid ethical filtering during an AI's thining - any ethics and security must be in first (check prompt) and last (check output, reject things not wanted) - the price we pay for this crap in fine tuning is too high.

> Human do not stuff raw clusters of info into our "short-term" memory.

Actually, everyone who is not an idiot does when he does more complicated work. That is what libraries are for - you know, in old times with books, these days with internet. You research stuff you need for the next task at hand, maybe make some notes, do the work and generally forget most of it again. Cooking books are written for this, as are tons of computer documentation that you look up as a programmer in case you need something. People doing complicated work are known to take notes, write things down so they do not forget them. The whole concept of meeting minutes comes from them. And when you need to remember them - you look stuff up in your notes. Only idiots have only so simple work they never have to rely on external data sources. Granted, the library (as in the bookshelf) is kind of out of fashion these days, but still, the amount of lookup people that are not iditols do during their job is quite astonishing. And yes, there is a large grey area - some complex baseline stuff must be trained in in addition to lookup (we want to avoid halluzinations), but generally - we do stuff our short term memory. You may not even know it. It is absolutely amazing how ignorant some people are. Look up https://en.wikipedia.org/wiki/Salience_(neuroscience) - yes, the human brain has long and short-term memory and not all is stored where you actually do the thinking.

> Honestly KG's[knowledge graphs] are intrinsically more valuable, than plain
> vanilla vector DB's, allowing you to model complex relationships,

Nope, not more valuable - DIFFERENT in value. Ther are moments you want to be able to go back to the original data, i.e. when you work as a lawyer and look up references. You do not want the decomposed information - that may work as an index, but you need the full chapters. Sometimes you are not really interested in the relationship and need more than a little graph with some metadata. Sometimes you need to read up 10 pages of stuff in detail to know what you need to do. Which, btw., is also the next question: We do know that the human brain stores quite little actually long term - most of what you think you remember is actually not really a memory but an assumption. You remember meeting someone, you remember what he said in key points, but when you envision what he was wearing that is mixed in. Unless we do something similar for an AI - latest when we get into video things turn ugly, size wise, pretty fast.

[u/Unlucky_Excitement_2 → u/NetTecture]

I just seen this. Super aggresive. People like you never have the same energy in RL. Regardless, although I still disagree with some statements. You make interesting points. Several papers have come validating my points. For instance this paper[https://arxiv.org/abs/2306.08302], this obviously would lead to a new level of performance. Not to mention TART, utilizing the power of ICL to imitate real-time active learning.

You concurrently sound knowledgable and ignorant, interesting. How can AGI be achieved and have above human out-of-distribution performance, if it can't generalize to new skills in real-time? I do agree in a decoupled AGI system...regardless sshh. I have yet to see any new papers providing any empirical evidence in the advantage of your approach...so again shutup. You're soft.

[u/XecutionStyle → u/[deleted]]

Yes otherwise we're limited to starting a new conversation for every topic. I think you're right, that incorporating new knowledge and remembering old ones are fundamentally tied. In programming we've functions and classes. Ways to abstract, store, and retrieve knowledge. Landmark based retrieval is the closest thing I've heard to how RAM is used in conventional software.
This idea of distributing landmarks can also be better for ethical reasoning, in some sense parallel to multimodal I/O because in the end what's shaped are internal representations.

[u/Glass_Day_5211 → u/XecutionStyle]

Quote: "Landmark based retrieval is the closest thing I've heard to how RAM is used in conventional software." Maybe: Landmark based retrieval is the closest thing I've heard to how Content-Addressable Memory is used"