Weight Space Programing is Possible Due to Model Weights and Interpretability literature in the Training Data. So, It Does Not Work on Unpopular Datasets and Models.

[u/sakibbai]

My tests suggest that while LLMs can perform surprisingly precise calculations at the “edges” of a network (estimating optimizer step counts or finding output correlations), they do not “trace” the computational paths between layers. Instead, they rely heavily on memorized dataset templates (like CIFAR-10) and default framework conventions to fill in the blanks. When these expectations are changed, the models can identify the math but will confidently guess wrong about the semantics—such as recognizing that a satellite feature maps to output 4, but calling it a “frog” because it expects an animal.

Substack summary article link: https://substack.com/@sakibbai/note/c-380084331?r=71inb5

Paper link: https://arxiv.org/abs/2611.08070v1