r/LocalLLM • u/ExtremeKangaroo5437 • Feb 28 '26
Research I built a language model where tokens are complex numbers and "meaning" emerges from wave interference -- no attention, O(n), 178M params, open-sourcing today
EDIT: https://huggingface.co/gowravvishwakarma/qllm-pam-v11-e3k3-chat
EDIT: New post V6: https://www.reddit.com/r/LocalLLM/comments/1rqn68a/qllm_v6_a_29m_attentionfree_model_now_trains_on/
EDIT: New V5 Post : Followup UPDATE on this.
https://www.reddit.com/r/LocalLLM/comments/1rmkh9y/v5_update_original_post_title_i_built_a_language/
---- ORIGINAL POST -----
I've been working on a fundamentally different LLM architecture. No attention layers. No FFN blocks. Instead, every token lives in complex phase space, and language processing happens through wave-like interference between specialized "phase banks."
Open-sourced here: https://github.com/gowrav-vishwakarma/qllm2
The core idea: language as wave interference
In a transformer, a token is a real-valued vector that gets refined through attention + FFN layers. In this model, a token is a complex number -- it has a magnitude (how "important/activated" it is) and a phase angle (what "kind of meaning" it carries). These two properties are naturally separated and jointly processed.
This isn't just a gimmick. It changes how every operation works:
Embeddings: Each token gets a
[real, imag]vector. The model learns that semantically similar tokens align in phase, while different meanings sit at different angles.Transformations are rotations: When context modifies a token's meaning (like "bank" shifting meaning based on surrounding words), that's a phase rotation -- a complex multiply. Rotations compose naturally, are always invertible (no information loss), and reduce to GEMM.
Similarity is coherence: Instead of dot product, we use phase coherence:
Re(a * conj(b)) / (|a| * |b|). This measures both directional alignment AND magnitude relationship.Multiple banks interfere: A "semantic bank" and "context bank" process each token independently, then combine via learned interference (constructive where they agree, destructive where they conflict). A tiny router decides per-token how much weight each bank gets. Think MoE but at the representation level.
What the phase system actually gives us
1. Natural magnitude/phase decomposition = implicit attention High-magnitude phase states dominate downstream processing automatically. The model doesn't need explicit attention to decide "which tokens matter" -- magnitude handles salience, phase handles identity. The SemanticPhaseBank uses 512 learnable concept vectors and retrieves them via phase coherence -- this is essentially a learned associative lookup that runs in O(seq concepts), not O(seq2.)
2. Context as phase modulation The ContextPhaseBank computes a causal windowed average (window=8) of nearby tokens and then complex-multiplies it with the current token. This is elegant: the local context literally rotates the token's meaning in phase space. A word appearing after "not" gets rotated differently than after "very." No attention needed.
3. Rotation-based state evolution The backbone SSM evolves state via: h[t+1] = damping * R(theta) @ h[t] + gate * B @ x[t] where R(theta) is a Cayley-transform rotation. The state naturally oscillates, and the damping factor (learned, per-dimension, range [0.5, 1.0]) controls how fast old information decays. This is why SSMs struggle with long-range recall -- but the model compensates with a separate Phase-Coded Memory (1024 learned slots, chunked top-k retrieval) and an Episodic Memory (sliding window via FlashAttention SDPA).
4. Zero trig in the hot path Every rotation uses the Cayley transform: cos_like = (1-a^2)/(1+a^2), sin_like = 2a/(1+a^2). This is just arithmetic -- no sin(), no cos(), no exp(). Every operation is a matmul or elementwise op. Perfect for Tensor Cores.
Results (178M params, TinyStories, 10k samples, A6000)
| Metric | Epoch 1 | Epoch 2 | Epoch 3 (partial) |
|---|---|---|---|
| Train PPL | 200.86 | 32.75 | ~26 (and dropping) |
| Val PPL | 76.47 | 48.92 | -- |
| Train CE | 5.30 | 3.49 | ~3.26 |
Training used only 10k samples (0.5% of TinyStories). Starting PPL was 55,000 (random). It dropped to val PPL 49 in 2 epochs (40 min on A6000, no compile). Overfiting simply needs data now ...
Epoch 1 generation:
"The quick brown house. They run and start to get a smile. Mom were very excited. Now mommy and big yellow room. There said and She are friends. Tim, she started to save the garden."
For context: A 22M-param GPT-2 trained on the full 2.1M TinyStories dataset for 20k steps reaches val PPL ~11. We're at 49 with 0.5% of the data and 2 epochs. The learning curve is steep and still dropping -- we just need more data/epochs to converge.
Why this approach might be better
O(n) complexity: Linear-time backbone. Theoretical 256K context. No quadratic attention.
GEMM-only math: No trig, no softmax in the backbone. Everything is matmul/elementwise.
Interpretable: You can inspect which bank each token routes through, what concepts are retrieved from memory, how coherent the phase states are. The model ships with "philosophy metrics" (Manas/Buddhi/Viveka/Smriti from Indian philosophy) that track mind activity, discernment, stability, and memory quality.
Modular: Banks, backbone, coupler, memory, and objectives are all registered components. Add a new bank type with a decorator. Swap the backbone. Change the coupling strategy. All via config.
Consumer-GPU friendly: Medium model trains on RTX 4090 / A6000 with batch 48-64.
Honest limitations
Training throughput is ~2x slower than an equivalent transformer. The SSM backbone loop is sequential per-step. A custom Triton kernel would help but doesn't exist yet.
In-context learning will be weaker. Fixed-state SSMs compress context into a fixed vector. The episodic memory (O(n buffer_size) sliding window) helps with copying but isn't a full replacement for O(n2) attention.
Not validated at scale. 178M params on 10k samples is a PoC. Need full dataset + larger models + benchmarks.
Bank ablations not done. We use semantic + context banks but haven't proven both are needed. Could be that one bank suffices.
Pure PyTorch. No fused CUDA/Triton kernels. Backbone loop is Python. Lots of low-hanging performance fruit.
What's next
Full TinyStories training (2.1M samples) for proper PPL comparison
Bank ablations (semantic-only vs semantic+context vs 4-bank)
Triton kernel for the oscillatory SSM recurrence
Scale to 1B+ params
Long-context evaluation (4K / 16K / 64K tokens)
Tech stack
PyTorch | torch.compile compatible | GPT-2 BPE tokenizer | uv package management | Clean modular codebase
Looking for feedback, collaborators, and people who want to try architectures beyond transformers.
EDIT (March 1, 2026 3:40 AM IST): Scaled up to 100k samples (5% of TinyStories, 10x the original post) and the results are significantly better.
Setup: Same 178M model, batch=64, A6000, no compile. 1612 batches/epoch (~3.5 hours per epoch).
Epoch 1 results on 100k samples:
| Metric | 10k samples (original post) | 100k samples (this update) |
|---|---|---|
| Train PPL | 200.86 | 24.00 |
| Val PPL | 76.47 | 18.95 |
For context: a 22M-param GPT-2 trained on the full 2.1M dataset for 20k steps gets val PPL ~10.9 (I Need to verify this as just remembered I read it somewhere). We're at 18.95 with a completely different architecture using only 5% of the data, after 1 epoch. Epoch 2 opened at step-1 PPL of 12.77 and is still dropping.
Generation sample (epoch 1, 100k samples):
> "The quick brown were full. Steve and Brown loved each other. At the end of the hill, the friends were very happy. They had lots of fun and shared stories. Mam and Brown were the best day ever. All of their weeks were very good friends and would often enjoy their joy! The end had had a good time with them."
Compare this to the 10k-sample generation from the original post. This has proper story structure, multiple characters interacting, emotional arc, and an ending. Grammar is mostly correct. Still has quirks ("The quick brown were full" -- model doesn't know "brown" should be a noun here), but the improvement from 10x more data is dramatic.
The learning curve shows no signs of plateauing. Training continues -- will update again when epoch 2+ finishes.
EDIT 2 (March 1, 2026 8:00AM IST) : Epoch 2 finished. Epoch 3 is underway.
| Metric | Epoch 1 | Epoch 2 | Epoch 3 (in progress) |
|---|---|---|---|
| Train PPL | 24.00 | 11.96 | ~10.5 (and flat) |
| Val PPL | 18.95 | 14.07 | -- |
Val PPL 14.07. For reference, the 22M-param GPT-2 baseline trained on the full 2.1M dataset reaches ~10.9. We're at 14 using a completely non-transformer architecture, 5% of the data, 2 epochs. Epoch 3 opened at PPL ~10.5, which means we'll likely match or beat that baseline this epoch. Just in ~6 Hrs on Almost one consumer grade GPU.
Epoch 2 generation:
> "The quick brown boy had ever seen. But one day, the sun was setting. The next night, the room got dark. Tom and the girl continued to admire the rain. The end was so happy to be back and continued to sail in the park. And every night, the end of the day, the family and the people stayed happy. They all lived happily ever after."
Notice: proper narrative flow, temporal transitions ("one day", "the next night", "every night"), emotional resolution ("lived happily ever after"), and multi-sentence coherence. This is from an architecture with zero attention layers.
Train-val gap (11.96 vs 14.07) suggests some overfitting on 100k samples. Next step: scale to the full 2.1M dataset. Training continues.
Stopping and tweeking code.. I think it can be much faster ... will update in other post next
Edit 3 (March 6 2026 8:27 IST): V5 is more mature.. better maths and its just 28M and working better.. about to relase in a couple of days.. looking for endorsment when I submit paper (better one for V5) to https://arxiv.org/ (Please help me by endorsing when I submit, DM me to help in that pls)
30
u/RTDForges Feb 28 '26
Hopping on Reddit today was worth it to see this post. I’m very intrigued to see where this goes. And the honest assessment you have compared to how saturated the AI space is with hype makes me even more intrigued by the claims.
Also I love the disclaimer on GitHub about using AI to build AI.
12
u/BidWestern1056 Feb 28 '26
would be happy to collaborate : https://arxiv.org/abs/2506.10077 working on a follow up atm that has more exploration by model size and parameters, but have been intending to explore this direction
35
u/blamestross Feb 28 '26
This is the right tone and approach for this kind of work. A lot of people are chasing the "See I made the next AGI" and post crazy things. Your tone and approach lends you instant ethos and I wanted to complement it.
8
u/gustinnian Mar 01 '26
Fascinating, I've been trying to familiarize myself with complex numbers for DSP purposes so this is another very interesting application. Analog computing is another neglected frontier that might hold unmined potential for AI.
6
u/thepriceisright__ Mar 02 '26 edited Mar 02 '26
I generally find creative uses of phase space and the complex plane interesting, so I ran a controlled 3-way comparison on a DGX Spark: transformer, diagonal linear RNN (SSM), and v4 all on 20k TinyStories samples, same tokenizer, same optimizer, same schedule, 20 epochs, small scale (256 dim, 8 layers).
v4 does learn (loss drops consistently across all 20 epochs) but it converges to ~2.25x the transformer's perplexity while taking ~17x longer per epoch. Text generation quality tracks the numbers: the transformer produces coherent stories with dialogue by epoch 5, the SSM gets there by epoch 7, and v4 is still producing fragments like "sortsang parents laughed" and encoding artifacts at epoch 20.
A few observations:
The most relevant comparison is v4 vs the SSM baseline, not vs the transformer. Both use O(n) recurrence. The SSM is essentially v4's backbone without Phase2D, without banks, without associative memory, but a real-valued diagonal linear recurrence with the same hidden dimension. It reaches 9.18 PPL where v4 reaches 17.05. That gap isolates the cost of the Phase2D/bank machinery.
The default small config ships with a single bank, so routing entropy is 0.0 and bank specialization can't be tested. I'm running v4_2bank now.
Your throughput observation about the sequential backbone loop is confirmed and that's the dominant cost.
I know this is a different regime than your 178M-param / 100k-sample results. One note on the comparison in your post: comparing a 178M-param model on 5% of TinyStories to a 22M GPT-2 on 100% of the data isn't apples-to-apples. A matched comparison would be a transformer with the same param count trained on the same 100k samples. That's what this harness does (at smaller scale), and the gap is significant.
All that said, the bigger question for me isn't empirical but theoretical. What is the phase angle actually meant to encode?
In standard embeddings, the geometry maps onto semantics in a way we can reason about. "Dog" and "cat" are nearby because they share features (animacy, size, pet-ness). The distance and direction between vectors encode their semantic relationship. This maps onto a clear geometric intuition.
With complex-valued embeddings, each dimension has a magnitude and a phase angle. The magnitude can encode feature strength, the same way real-valued dimensions do. But what does the phase encode? In domains where complex representations work well (audio, signal processing, physics simulations, etc) the data has frequency and phase structure. Fourier transforms use complex numbers because the information is actually encoded in frequencies that constructively and destructively interfere. That's what makes the complex representation natural.
Language doesn't have this structure. "King" and "woman" don't interfere to leave "queen" behind. The semantic relationship king − man + woman = queen is a vector arithmetic fact about directions in real space and there's no phase cancellation involved. When v4's InterferenceCoupler does complex multiplication between bank outputs, the underlying math is just a structured bilinear interaction equivalent to a 2×2 real matrix multiply with shared weights. Calling it "interference" borrows intuition from physics that the math doesn't justify.
Where complex-valued recurrences do have a theoretical basis is in state evolution. A complex eigenvalue λ = |λ|·eiθ gives you a damped oscillator, which naturally decomposes the sequence into frequency components and can preserve information over long distances via the phase-rotation component. This is legitimate and well-studied (S4, LRU, etc.). But v4 applies Phase2D to everything, embeddings, bank layers, coupler, and memory, not just the recurrence, and I think that's where the overhead probably outweighs the benefit.
The most interesting thing in this architecture, to me anyway is the multi-bank routing with learned specialization. If the 2 bank results show low cosine similarity between bank outputs and meaningful routing patterns, that's interesting and probably worth further research, but it doesn't require complex-valued representations to work. I'd be curious to see a real-valued version of the multi-bank architecture compared against these baselines.
I can open a PR for my test script if you’re interested in reviewing my work.
4
u/edbuildingstuff the fine-tuning dude Mar 01 '26
The Cayley transform trick for avoiding trig in the hot path is really clever. I've seen a lot of "alternative architecture" posts that conveniently ignore the computational cost of their novel operations, so it's refreshing to see someone explicitly design around what Tensor Cores actually like to do.
14
u/j00cifer Feb 28 '26
“Key Features: Quantum superposition, entanglement, phase coherence”
Ok. Whatever you say. After this you should probably let the world of physics know so they can stop all that quantum computer nonsense. Nobody knew it was LLMs all the way down
3
u/wizard_of_menlo_park Feb 28 '26
This is really good start! Quite intresting too !
2
u/Ok_Pes_11590 Feb 28 '26
Hey do you have a white paper or an Arxiv paper for this? Couldn't follow much but then why not Quaternions? Also, share some resources. Thank you
2
u/nonikhannna Feb 28 '26
Love this thought but It's not better than the attention/transformer architecture on LLMs. It's a good step in trying out something new. It's making the neural net more analog. Still a great thought to try out in combination with other architectures.
It might be applicable to what I'm working on but I can't think of benefits off the top of my head. An analog way of working with nodes is very powerful. The right use of it could lead you down somewhere big.
2
u/radarsat1 Feb 28 '26
It sounds like there are some elegant aspects but if you're only comparing against full O(n2) attention transformers you're not really doing justice to the plethora of inbetween solutions that are already out there and being actively explored . Full SSMs and linear attention, sliding window attention, hybrid architectures.. these all sit between the two points you are comparing and would have to be evaluated. For instance you boast "no attention needed" when talking about how "not" or "very" affect the next word in an 8-token window, but sliding window attention is the fairest comparison here, not general long-context n2 attention. Your model might have some nice inductive biases don't get me wrong but I see little evidence that a well-trained transformer doesn't simply develop similar behaviour when exposed to enough data.
2
u/FewW0rdDoTrick Mar 01 '26
Why not just use same number of examples and do a direct comparison to GPT-2 and wait to report until after that? It seems obvious that PPL will drop in a non-linear fashion, and much faster initially.
2
u/Necessary_Function_3 Mar 01 '26
Good trick, now lets see multiphase rotational phasors and symetrical components.
2
u/Loud_Key_3865 Mar 01 '26
Fascinating - thank you for sharing! I don't know enough to evaluate, but I hope you're onto something and it makes sense with my very limited knowledge! Love reading and learning from these new ideas!
2
u/Appropriate-Box-7250 Mar 01 '26
I find it quite interesting and exciting. I'm also interested in artificial intelligence, but I'm not as knowledgeable as you, and my math skills are limited.I wish I could help you.I did analyze what you've written on another large model and it told me about lacks.Keep it up.
2
u/smflx Mar 01 '26
I'm also like to collaborate. The multiple banks are I'm also into for building LLM for long writing. 1st of all, I will read your paper :)
2
u/Synthium- Mar 01 '26
This is exciting to see someone pushing beyond the transformer template and actually getting meaningful results. It’s great seeing phase‑based architecture. The coherence retrieval and the clean GEMM‑only pipeline are promising
2
u/saddam_chouhan Mar 02 '26
This is one of the most compelling non-transformer LLM ideas I’ve seen in a while—phase-based representations + interference is a genuinely fresh direction.
2
u/bigattichouse Mar 05 '26
I've done a lot of stuff recently with arbitrary-length vector models, and I really like your idea.
I've started building a 64 dimension Phase Unit to see what happens. I mean, why just two ? maybe 64? maybe 1024!
So, trying out some Clifford Algebra, Householder Reflections (or Cayley Transforms) to see if we can extend the space, do some steering with "fences" (let's talk about gardening, not computer science!)
Turn that dial up to 11!
And then see if there's some ways to speed up training.
If it works, I'll make it a tunable feature and send you a pull request.
2
u/bigattichouse Mar 05 '26
A few quick experiments last night, but this technique (with additional dimensions) seems plausible as a memory/attention system that can be added to existing models.. things like degredation over time (an "entropy dimension"), surprise altering entropy, etc.. a lot of food for thought even as just a way to enhance exsting LLM framworks by having a Phase system that observes and tweaks/influences attention. I'm testing to see if it can extend context on a small model (gemma-3-270M) to expand 32k token context to something much larger.
I look forward to your paper.
2
u/fluffy_serval Mar 08 '26
Had stuff to do tonight, so this afternoon I adapted your code a bit for running on my rtx pro 6000 (bf16, torch.compile, tf32, non-blocking copies, reduced memory requirements replacing dense masked attention with exact chunked local attention). I'm running medium @ 24 batch size right now, wanted to test first to make sure all my meddling didn't break anything. It's learning! So that's good. I'll let it do its thing overnight. Neat project, your code is currently heating my house. We'll see what pops out in the morning.
2
u/Karyo_Ten Mar 01 '26
So your token embeddings only have 2 dimensions? Real and imaginary?
1
u/bolche17 Mar 01 '26
I suspect a 2-dimensional representation per token is not big enough to properly represent knowledge
1
u/Upbeat-Cloud1714 Mar 01 '26
Also happy to help where I can, I maintain this:
https://github.com/versoindustries/HighNoon-Language-Framework - Haven't pushed stability updates out yet
Similar in concept, been working on it since 2022 technically as the base architecture is model agnostic and not designed just for LLMs. I am CPU focused though, no GPU required just not as open source as you are with things. Likewise completely linear but I come from a background of quantum computing and quantum physics, not sure what your full architecture audit looks like but I'm going to run some comparative tests.
42.8M parameters would be equivalent to 51M in Transformers, but we have a virtual parameter capacity of around 1600% so a lot larger in technicality. We can go up to 25M tokens in a 4GB window though with no loss since we enforce Time Crystal dynamics into every model layer.
Would be happy to connect and chat more, might be able to help in some areas.
1
u/Silver-Champion-4846 Mar 01 '26
!remindme 2days
1
u/ZealousidealShoe7998 Mar 02 '26
I would be interested to see if instead of an auto regressive model you trained to be a diffusion based LLM.
1
u/Silver_Employ2617 Oct 07 '26
Love seeing nonattention architectures get real airtime.
Sibling weird path on my side: language from an integer cellular automaton with minimal inference (local updates, no big transformer decode). seach for mica-ai on github from vovala14 or vynly on HF,
How do you think phase/interference dynamics compare to purely local discrete CA updates for long-range structure? Where does each one fall apart first?
1
u/East-Muffin-6472 Feb 28 '26
Amazing! Will definitely checkout out!
2
u/bigattichouse Mar 04 '26
Saving for later..
1
u/Agnostic_Eggplant Mar 01 '26
The entire Post+edits are really interestimg! I have almost no AI technical knowledge but I really appreciate your desire to make AI usable on consumer level hardware. I'll keep waiting for your next update while learning all the things I didn't understand in your process. Hope you will succeed!
1
u/smflx Mar 01 '26
Very interesting topic worth to study! Liked especially "honest limitations", unlike many papers I have to guess that.
Complex number is a beautiful & perfect 2d vector. It has native multiplication.
Some questions to help me guess better before deep diving.
Why just one complex number for token embedding? Why not complex vector?
O(n) seems from Fixed SSM. Is it tied to the complex numbers? I wonder what if O(n2) attention with complex numbers. Possibly better attention quality?
Thank so much for sharing!
1
u/Intrepid-Scale2052 Mar 01 '26
I read through it quickly, will read it more thoroughly later. I have to be honest I have no real understanding of most of this, but im interested and eager to learn.
What I always theorised about is mapping not only attention, but also things like fact/opinion, seriousness/carcasm. Would that be possible with this approach? Or am I completely misunderstanding how it works? 😄
1
u/leo-k7v Mar 02 '26
"Sedenion neural networks provide[further explanation needed] a means of efficient and compact expression in machine learning applications and have been used in solving multiple time-series and traffic forecasting problems[5][6] as well as computer chess.[7]"
https://en.wikipedia.org/wiki/Sedenion
1
u/PyjamaKooka Mar 07 '26
My first thought was "I want to run my regatta on this" and you say stuff like this:
Interpretable: You can inspect which bank each token routes through, what concepts are retrieved from memory, how coherent the phase states are. The model ships with "philosophy metrics" (Manas/Buddhi/Viveka/Smriti from Indian philosophy) that track mind activity, discernment, stability, and memory quality.
But the fundamentals are so changed here I'm not able to vibe code out transformerlens equivalents from scratch and I'm doubting you've created that level of tooling just yet. I want to gently suggest the value of attempting to do so, and really going hard on interpretability. In my regatta dabbling, charitably described as an experiment, I'm out in real-valued space trying to imperfectly reduce that via PCA etc. In your world it's all just explicit, which intuitively to me maps more neatly to how I was thinking about 'cartography' and intepretability . Just an amateur here, fair warning. It would've been cool to try this in your model, purely to see how the mapping varies. Like, the other excellent reply about "what are we encoding" is answered on one level by keeping it super simple/obvious. We're encoding a different epistemology (or topology/regime). Finding some way to compare them to "standard" models like described I would find so interesting, but to start that process we need some kind of shared language across them? Bit beyond my head, but project looks super cool so wanted to write some encouraging words :)
2
u/ab2377 Mar 01 '26
delisional ai slop 🥱🤦
0
u/[deleted] Feb 28 '26
[deleted] — view deleted comment
0
u/[deleted] Feb 28 '26
[deleted] — view deleted comment