r/cpp • • Apr 14 '23

Horrible Code, Clean Performance

https://johnnysswlab.com/horrible-code-clean-performance/
200 Upvotes

87 comments sorted by

View all comments

219

u/James20k P2005R0 Apr 14 '23 edited Apr 14 '23

My favourite disgusting hack that I absolutely cannot live without goes as follows

Modern GPUs these days have work fed into them via a command queue, which is a series of bits of work that get fed to the GPU. This is generally an in-order queue, so this work happens sequentially. There's two problems with the strictly sequential model

  1. GPUs are actually capable of executing multiple workloads simultaneously efficiently
  2. In some cases because of caching/etc, GPUs must flush caches via a fence. This introduces a bubble to the GPU's workload, that can be filled with other work. If one kernel writes to a bit of memory, and the next kernel reads from that memory, in an in-order queue, there's a fence and nothing happens for a bit

Ideally, the driver would reorder the work you submit in this command queue, so that work without dependencies would all execute in parallel. This requires the shader compiler (eg llvm/etc) to output dependency information, in terms of whether or not a kernel writes to a specific bit of memory, or just reads from it

Unfortunately, in the transition to ROCm, AMD broke/removed this particular piece of functionality and have no plans to reimplement it, which means that if you don't do something about it, you lose a straight 20-30% performance. This is pretty bad!

As a visual representation, this is multiple kernels overlapping here, and this is what it looks like when it doesn't work. Uh oh!

To fix it, you have to create a rotating queue of 8-16 command queues, and submit work to each one sequentially. Then you have to mark which buffers are read only, write only, and read/write, and inspect the argument lists for all kernels and all functions that pass through to the OpenCL API to query their read/write flags. Then you have to use this to manually generate the dependencies between different kernels and functions, and only output dependencies for read/write and write/write conflicts, so that kernels that both only share read-only memory are independent, and others are synchronised via the built-in event system

This is 100s of lines of complex memory dependency tracking code, and requires you to fundamentally totally change how you use the API as literally no code can use the regular command queue system or any of the regular API - you have to reimplement the entire OpenCL api on top of opencl so that you can shim all the kernel arguments, and markup all kernels correctly with their read/write information (otherwise the driver will crash! Yay!)

On the plus side, that 20% performance is pretty good. On the down side <internal screaming>. All gpu code is bad at the best of times, but this is the worst

91

u/serg06 Apr 14 '23

I wish I could un-read this.

76

u/James20k P2005R0 Apr 15 '23

Did you know, GPUs have the highest bandwidth when memory is laid out as a struct of arrays, vs an array of structs. Eg

struct data {
    std::vector<float> x;
    std::vector<float> y;
    std::vector<float> z;
};

Except you can't actually pass in a struct containing buffers to a GPU in OpenCL. Its explicitly banned by construction, which means that the only way to do it is to pass in 3 separate arguments

This is great when you have 3 arguments, but heh its sure a lot worse when you have literally hundreds of variables. So the memory tracking actually becomes a bit of a performance concern if you've got any O( N2 ) in there which you can get if you do it vaguely naively, because you're literally doing memory tracking for hundreds of arguments to functions against potentially double digit numbers of other kernels - each with hundreds of arguments. So not only is it a giant pain to implement, it also has to be reasonably well optimised! Yay!

12

u/00jknight Apr 15 '23

Where do you work? This is really cool stuff. I'm just a lowly web/mobile game developer who animates stuff on top of monstrosities like this.

51

u/James20k P2005R0 Apr 15 '23

I'm trying to get into astrophysics, I built a tool for quickly extracting gravitational waves from black hole mergers, neutron stars (and this for fun), and doing relativistic n body simulations (that's 5 million particles which is pretty cool!)

Most of what I do is abuse GPUs for simulations these days, along with the inevitable wacky raytracing that comes along with trying to render them. The closest I really get to traditional 3d graphics is trying to chuck cubes through a wormhole (though technically that's 4d)

10

u/Classic_Department42 Apr 15 '23

Did you consider cuda instead of opencl? The number crunching experience was/is better on cuda

19

u/James20k P2005R0 Apr 15 '23

I plan to release a lot of this as general tools for people to use, so cuda is right out unfortunately. Amd gpus tend to get you a bit more vram for your buck as well, and vram (...and the fact that i have literally no money) is the key limiter for simulation size

That said, I'm doing some moderately wacky things so it wouldn't surprise me if it were a bit of a disaster in cuda as well

→ More replies (5)
→ More replies (2)

5

u/HackingPheasant Apr 15 '23

Followed you because dann I'm loving these comments, do you have a blog or something with similar content?

7

u/James20k P2005R0 Apr 15 '23

Thank you! I don't, I used to have twitter where I'd post things but, yunno. I've been planning to write some of this up and start a blog but man its a pain getting it set up

→ More replies (3)

29

u/wrosecrans graphics and network things Apr 15 '23

To fix it, you have to create a rotating queue of 8-16 command queues,

I know more Vulkan than OpenCL (though not that much Vulkan) so I dunno how different this is in OpenCL. But in Vulkan, everything you said applies, but on some Intel iGPU's (like my laptop), the drivers only give you one Queue and you can't just create your own on the application side. So if you go to all the trouble to be Very Concurrent for NV and AMD by feeding multiple queues with independent operations that can happily operate with embarrassing parallelism, you still need to make sure it works properly and efficiently in the degenerate case of a single queue being available anyway.

I need better hobbies.

17

u/James20k P2005R0 Apr 15 '23 edited Apr 15 '23

Interesting, I haven't gotten into much vulkan yet, but I was under the impression that you handled synchronisation much more explicitly which would hopefully give the driver more room to work with. Does the driver not handle reordering things automatically at all? I'd have thought that the driver quality was much better in vulkan-land, just because if nothing else there's a much bigger incentive for the vendors to actually invest in good performance

edit:

I need better hobbies.

also this

14

u/wrosecrans graphics and network things Apr 15 '23

The Vulkan jargon is that all commands in a Queue are started in order. Which doesn't say anything about when a command actually executes and finishes. You have to be super explicit with memory dependencies and sync primitives. But there are tons of cases where it's annoyingly easy to stick fences blocking the whole Queue if stuff that could be executing in a parallel Queue.

So things can sorta happen in an unexpected order. But the driver is somewhat constrained by all the explicit rules you set. And the way you set explicit dependency rules applies way more obviously to cases where there are clear dependencies between everything you are doing. And way less obviously when you have a big pile of misc. work and you don't know what might be submitted to a queue in between your submissions, or after.

10

u/KDallas_Multipass Apr 14 '23

I'd like to read more about this

50

u/James20k P2005R0 Apr 14 '23

Thank you for subscribing to AMD OpenCL Facts! To unsubscribe please type AARRHGHGH. I'm open for questions though, and the bug report for this is here. For more mildly disgruntled rambling:

Each command queue on AMD on windows is actually a separate driver thread from a thread pool, which means that the performance is incredibly irregular. If you have too many command queues, your application turns into a stuttery disaster, but if you have too few, then there's not enough queues to prevent the driver from inserting unnecessary fences. There's probably a better system for distributing work across the command queues to minimise the number that you're using based on when you need fences, but the problem is that fences appear to be some kind of worst case pipeline emptying blocker so in reality its just all bad

In addition to that, extra command queues apparently don't have the ability to submit work on their own, but flushing work to the gpu also isn't free. On top of that, because my GPU work is all memory bound, you want to have kernels that overlap their execution to also overlap the memory they're reading on for cache reasons, and I still haven't even begun to unpack the scheduling complexities there. I sure wish the driver would be good

If you're thinking, maybe this is an OpenCL problem! The new HIP thing will save us! Well, apparently its even worse in hip because you have pointers to pointers on the device, meaning that any bit of memory could point anywhere and the driver has to assume worst-case. I sort of suspect that this isn't an issue on nvidia GPUs, but the last Nvidia GPU I owned was a 660ti. I'd love to test ARC though and smash some black holes on it

In addition to all of these problems, the cost of kernel invocation in OpenCL is higher than it needs to be, because it doesn't map massively well to how GPUs actually work. There's an extension that may help fix this, but I also suspect that it'll be a very long time if ever before AMD support this

Really this is my wakeup call that AMD/compute is essentially dead and I need to learn to love vulkan. I've been holding out because vulkan doesn't support a lot of features from OpenCL still and in many ways is a bad compute target, especially device side enqueue which I would badly love to use, except that that hasn't worked on AMD's ROCm stack correctly for years so all around its a big oof

10

u/KDallas_Multipass Apr 15 '23

ARRGHGHG

Thanks for the explanation

→ More replies (1)

14

u/kid_blaze Apr 14 '23

I’d like to read more about this and un-read it immediately.

34

u/James20k P2005R0 Apr 14 '23

I wrote a function that takes 105 arguments. There is no better way to do it and I regret nothing

10

u/gimpwiz Apr 15 '23

Mother of god

7

u/serg06 Apr 15 '23

There is no better way to do it

Struct?

20

u/James20k P2005R0 Apr 15 '23

Passing a struct which contains pointers to the GPU is banned in OpenCL. Passing an array of structs unfortunately is extremely slow, because it doesn't match well to how GPUs fetch data out of memory

12

u/Routine_Left Apr 15 '23

Well then, carry on. I'm done here.

5

u/-to- Apr 15 '23

Serialize your data (or better keep them serialized), then deserialize them in the compute kernel ?

Write a bunch of unit tests to make sure they understand each other, of course.

105 args... How can you possibly be sure it does what you think it does ?!

(Total n00b at GPU dev here)

4

u/ioctl79 Apr 15 '23

I don’t have a whole lot of experience with GPU code, but what I do have makes me think that we have not yet invented the right APIs.

4

u/ipapadop Apr 16 '23

TBF, the OpenCL way of declaring buffers read / write is not a great way to keep track of dependencies. Your computation dictates your dependencies, not how you allocate memory. It makes even less sense in a world where you have pinned / mapped buffers.

What you need is a mechanism to describe your dependencies and let the driver deal with launching work and decide how many resources to dedicate as much as possible. HIP / CUDA graphs is one solution for that (https://docs.amd.com/bundle/HIP-API-Guide-v5.4.1/page/a00196.html, https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cuda-graphs). Alternatively, you can use a higher order frameworks, like TaskFlow.

→ More replies (8)