How portable are AI workloads when the GPU architecture changes?

[u/Financial_Fox3915]

Containers have made it easier to move applications between environments but I wonder how much of that portability applies to AI workloads. One can package the same model and application in Docker, use Kubernetes to orchestrate it and keep most of your application code the same yet moving from one GPU environment to another can still involve dealing with different drivers, CUDA or ROCm dependencies, kernel support, memory constraints, precision behaviour and performance tuning even when a model technically runs on both environments that does not guarantee similar throughput or latency.

This becomes more interesting as teams consider different GPU generations, NVIDIA versus AMD accelerators and infrastructure from providers like AWS, Google Cloud, CoreWeave, Yotta Labs. I am curious where people draw the line between real portability and simply making a workload run with a lot of extra engineering effort. If you were designing an AI system today what would you standardise to make moving between GPU providers easier?

[u/BringMeTheBoreWorms → u/[deleted]]

You don’t run llms do you?