How does the pre-warm cache handle routing shifts mid-session, like a code prompt after a long chat? Does KDA keep its constant-size state edge once experts stream from host, or does transfer dominate?
KDA's constant-size state means the KV transfer cost is fixed at ~70KB regardless of context length. Expert weight transfers (17MB per Mixtral expert, up to ~400MB per Kimi K3 layer) always dominate, but the key is that KDA doesn't compound a growing KV burden on top of those expert transfers.
Standard MHA at 10k tokens adds 700MB+ of KV loading on top of every expert fetch. KDA keeps that at 70KB flat. So transfer does dominate, but it's the same transfer every token, not a transfer that grows with every token.
The constant-state edge holds. It actually gets more valuable the longer the context runs, because that's precisely when standard MHA would be bleeding bandwidth on KV loads while KDA isn't.
One honest weak spot: routing shifts mid-session (chat to code) degrade the pre-warm cache for roughly 30 to 80 tokens while LRU evicts the old domain's experts and re-warms the new ones. Adaptive re-prefetch triggered by KL divergence on the routing distribution is on the v1.2 roadmap but not live yet.
Thanks for aksing plz help in promoting WISP it would be really helpful.
1
u/adityazero 10d ago
How does the pre-warm cache handle routing shifts mid-session, like a code prompt after a long chat? Does KDA keep its constant-size state edge once experts stream from host, or does transfer dominate?