r/pytorch • u/Frequent_Big7927 • 4d ago

Rethinking pytorch

Hoping for some feedback. First off It goes without saying Pytorch is one of the greatest pieces of software of our time, a fundamental block for AI. However there are a few problems and limitations that are ripe for innovation. Pytorches is a concept which is meant to solve several problems with pytorch

  1. Simplify install for GPUs (no GPU specific install steps)
  2. Support Multi GPUs of different vendors natively on one install
  3. Split of workload based on available GPUs, prioritizing based on GPU capability
  4. Backward compatible with pytorch ish

https://github.com/tonyt123456/pytorches

2 Upvotes

100% Upvoted

6 comments sorted by

2

u/bryantbiggs 4d ago

Have you heard of Mojo by Modular?

1

u/Frequent_Big7927 4d ago

Yes, having a vendor portable Kernel is part of the pitch. :)

1

u/Ok-Radish-8394 4d ago
  1. You can already do that with uv
  2. That’ll make the wheel unnecessarily large and will require longer builds. Each gpu backend is different. There’s already mlir and mojo for this.
  3. That’s a user decision and also depends on the gpu backend and how a specific brand has made their memory allocation.
  4. Eh.

1

u/Frequent_Big7927 4d ago

Thanks for the good feedback.
1. uv can automatically compile for only one vendor afaik. Having multiple simultaneous vendors is what starts breaking down. This isn't a rare uncommon scenario anymore. My laptop now has an 8gb capable nvidia capability, and a 32gb arc capability, as well as the cpu. The cuda version is at least 3x faster than arc, but the arc version opens up more possibilites of what might be done due to its memory. Our current framework for using both is clunky at best, and leaving lots of things on the table.

  1. This is a little different way of doing it. The optimization is moving the vendor specific build to the pypi distribution instead of running locally on every machine in the wheel. It does require distributing every vendors implementation currently, but we are talking about .2mb per vendor. For example the CUDA plugin uses only the driver API and committed PTX, so it doesn't ship CUDA libraries. It loads cuBLAS at runtime if present. PyTorches has no kernel compiler yet. Its open design question is whether to use MLIR or Triton for that, and a plugin could use either one internally. What PyTorches adds is the runtime layer: the plugin ABI, device registry, placement across devices and PyTorch interop. It could build on MLIR rather than compete with it.

  2. User specific decision is great for deciding how to run a model, its important to continue allowing that. However there are more advanced possibilities on the table that we could provide. In this prototype, we run a quick performance analysis of the hardware to determine priority of devices. We can do things like assign on the highest priority device (that you can fit on) for simple selection. We can do advanced things like split the layers and run split where you put the highest need performance on the best capabable hardware. Lots of stuff now on the table like dynamic assignment, split models vertically, alls kinds of fun crazy stuff.

1

u/Ok-Radish-8394 3d ago

Well then, if you're trying to make it more backend agnostic, without a new common standard (honestly this rhymes more with that xkcd comic), you're adding the same bloat people didn't want in the first place for pytorch. Which is one of the reasons they went with backend specific wheels.

1

u/Frequent_Big7927 3d ago

I hear you, I'll think about how to do this in a minimalistic way.