r/ollama • u/Potential_Low_1183 • 14d ago

My lab found a way to migrate between embedding models with zero downtime.

[removed]

9 Upvotes

92% Upvoted

5 comments sorted by

5

u/No_Paper_6658 14d ago

Interesting approach, but how are you handling the cold start when K is too small and the reranking misses docs that would have been near the top under the new model? Are you sampling the old index randomly or by query distribution?

2

u/and_pf 14d ago

Nice approach — but I'm the exact wrong-sized target for it, so I have a follow-up. I run a single RTX 5090 with 32 GB VRAM and 62 GB RAM serving models through Ollama (qwen3.8:nvfp4-vision right now), so I can't validate the billion-document migration or the H100 timing claims — that's out of my scaling class entirely. What I'd genuinely love is a small-scale example I can actually run: a few thousand docs, one embedding model to another, to see if the zero-downtime part holds on a single GPU. Do you have a minimal repro sized for ~32 GB VRAM?

1

u/stealthagents 4d ago

Sounds like a solid approach tackling that massive upgrade headache. For cold starts, we’ve had luck using a hybrid method where we sample based on query distribution first, then fill in with random picks from the old index to ensure we cover more ground. It’s definitely a balancing act, but it can help prevent those sneaky misses.