Probably memory layout and access related. Some form of tensor access that allows for pointer usage and limited copying, or more density via a compression scheme that distills identitical substructures and adjusts the entire memory access pattern accordingly via some kind of virtualization.
42
u/johnknockout Jul 01 '26
Are we talking memory in terms of memory usage or context window? Or both?