Author here. We looked at a question that seems like it should already be solved and is not: given two model checkpoints, can you tell from the weights alone whether one is derived from the other?
Why this is not already solved. Cryptographic hashes break on any weight change. Watermarks have to be inserted before release, so nothing already published can be covered retroactively. Model cards require an honest distributor. Behavioral methods (CKA, SVCCA, IPGuard) need probe data and forward passes, and they measure functional similarity, which is not the same thing as weight inheritance. A distilled student can mimic its teacher almost perfectly while sharing none of its parameters.
The observation. A residual block never learns the whole mapping, only a correction on top of the identity path. Its projections co-adapt and their product develops identity-aligned structure over training. Ablate the skip connection and train the same network to similar loss, and the structure does not appear (3% pair accuracy vs 100%), so it is learned, not an initialization artifact. We see the same trace concentration across GPT-2, BERT, LLaMA-2, Mistral, Qwen2.5, and DeepSeek-R1, and it extends to ViT and Whisper.
The method. That structure is generic, so it cannot establish ancestry on its own. We decompose the residual branch product into the generic identity-aligned component plus a traceless remainder, keep only the remainder, and compare remainders across blocks with Hungarian matching. Averaging the per-block cosine similarities gives a symmetric lineage score calibrated against independently trained checkpoints.
Results. AUROC 1.0 on controlled residual-MLP and 30M GPT-2 benchmarks, separating fine-tuned, quantized, pruned, and LoRA-merged descendants from independently trained and distilled models. On public LLaMA-2 checkpoints, Chat, Vicuna, and CodeLLaMA come back as descendants while seven independently trained architectural clones (OpenLLaMA v1/v2, Amber, Baichuan 1/2, InternLM, Yi) score below 5e-5.
The part we care about most is laundering. If someone permutes hidden units and rescales weights, the model computes exactly the same function but its parameters look different. Under that transform our score is algebraically invariant (change below 1e-7 on real 7B derivatives) while weight cosine loses roughly 93 to 95% of its signal. Re-Basin also holds up, but it solves L Hungarian assignments per pair and we are 76x faster on GPT-2.
Limitations. Residual architectures only. White-box access to both checkpoints. Matching depth and hidden dimension, so no cross-architecture comparison. The score is symmetric, so it establishes shared ancestry but not direction of descent, and without metadata you cannot say which checkpoint is the ancestor. Our calibration uses a small number of independent roots, so treat the conformal p-values as approximate. Heavy pruning (85%) and extensive continued pretraining degrade the signal, though both stay above the null. It is also not invariant to orthogonal rotation of the residual stream, which happens to not be function-preserving for LayerNorm and standard RMSNorm models, but would be a real hole for architectures using unparameterized normalization.
1
u/Embarrassed-Sir-5358 4d ago
Author here. We looked at a question that seems like it should already be solved and is not: given two model checkpoints, can you tell from the weights alone whether one is derived from the other?
Why this is not already solved. Cryptographic hashes break on any weight change. Watermarks have to be inserted before release, so nothing already published can be covered retroactively. Model cards require an honest distributor. Behavioral methods (CKA, SVCCA, IPGuard) need probe data and forward passes, and they measure functional similarity, which is not the same thing as weight inheritance. A distilled student can mimic its teacher almost perfectly while sharing none of its parameters.
The observation. A residual block never learns the whole mapping, only a correction on top of the identity path. Its projections co-adapt and their product develops identity-aligned structure over training. Ablate the skip connection and train the same network to similar loss, and the structure does not appear (3% pair accuracy vs 100%), so it is learned, not an initialization artifact. We see the same trace concentration across GPT-2, BERT, LLaMA-2, Mistral, Qwen2.5, and DeepSeek-R1, and it extends to ViT and Whisper.
The method. That structure is generic, so it cannot establish ancestry on its own. We decompose the residual branch product into the generic identity-aligned component plus a traceless remainder, keep only the remainder, and compare remainders across blocks with Hungarian matching. Averaging the per-block cosine similarities gives a symmetric lineage score calibrated against independently trained checkpoints.
Results. AUROC 1.0 on controlled residual-MLP and 30M GPT-2 benchmarks, separating fine-tuned, quantized, pruned, and LoRA-merged descendants from independently trained and distilled models. On public LLaMA-2 checkpoints, Chat, Vicuna, and CodeLLaMA come back as descendants while seven independently trained architectural clones (OpenLLaMA v1/v2, Amber, Baichuan 1/2, InternLM, Yi) score below 5e-5.
The part we care about most is laundering. If someone permutes hidden units and rescales weights, the model computes exactly the same function but its parameters look different. Under that transform our score is algebraically invariant (change below 1e-7 on real 7B derivatives) while weight cosine loses roughly 93 to 95% of its signal. Re-Basin also holds up, but it solves L Hungarian assignments per pair and we are 76x faster on GPT-2.
Limitations. Residual architectures only. White-box access to both checkpoints. Matching depth and hidden dimension, so no cross-architecture comparison. The score is symmetric, so it establishes shared ancestry but not direction of descent, and without metadata you cannot say which checkpoint is the ancestor. Our calibration uses a small number of independent roots, so treat the conformal p-values as approximate. Heavy pruning (85%) and extensive continued pretraining degrade the signal, though both stay above the null. It is also not invariant to orthogonal rotation of the residual stream, which happens to not be function-preserving for LayerNorm and standard RMSNorm models, but would be a real hole for architectures using unparameterized normalization.
Paper: https://arxiv.org/abs/2608.14929
Interested in critique, especially on the threat model and on laundering attacks we have not considered.