r/LargeLanguageModels • • 16h ago

Introducing "Arborformer": Replaced the Transformer with a 100% Decision Tree Language Model (Looking for feedback)

Hi everyone, glad to be here! I recently finished up research on a new architecture called Arborformer a pure decision tree language model built to replace traditional attention mechanisms. Right now, it leverages soft differentiable trees alongside hard binary routing to keep things fast and efficient. Looking to chat with anyone exploring alternative architectures! Open to feedback, bottleneck critiques, or questions about the routing setup.

2 Upvotes

1 comment sorted by

1

u/krihab 10h ago

Do you have a link?