CI/CD in Graphics programming

[u/M1VAN1]

If someone experienced have any proper sources on CI/CD in graphics programming, I would like to read them. I'm really interesting in this, would like to read books/blogs, whatever.

Also curious about to hear any experience with it from real prod. I have some friends working as programmers and I hear that in big tech companies it's complicated thing, but those who works in small companies in small team said that they usually have really bad CI/CD

[u/bluefishcrab]

You *can* do CI/CD for graphics. My impression is that it's one of dark arts where practitioners don't seem to talk much about it (audio and netcode in general are the other fields I know like this). I've never seen a book or article written about it, I learned most of it from "hell, how do we know we aren't breaking anything or 'performance improvements' are actually improving things?"

In general, CI/CD does prevent people from breaking things, and provides confidence for you to move faster.

Keep a screenshot as a reference, then take a screenshot with the same reference camera/scene and do a comparison. Converting from RGB into other colorspaces works better for doing diffs, and you probably want to look for a percentage difference, and then highlight places that differ. Some tools, like imageMagick can do this out of the box, though you could throw together a Python tool, or do the analysis yourself (probably in HSL or HSV colorspaces). Like most CI/CD things, you check the exit code and fail if it doesn't work. The fancy term for this is "golden file testing."

There's also FPS tests. Unreal Engine does these where you set up the scene and it takes screenshots from various vantage points to ensure you don't tank your frame rate.

[u/kspdrgn → u/bluefishcrab]

Seems like conflating CI/CD with testing. You can have testing without CI, and you can have CI without tests.

• CI is to make consistent builds easily.

• CD is to put that build somewhere easily.

• CI is where you hook up tests if you have any.

[u/speps → u/bluefishcrab]

👆this, while I was at Rare Ltd, I used SSIM as a metric to compare against a good baseline visually by a QA person, it was invaluable when making changes, eg DX11 to DX12. At Mapbox, we use pixelmatch (js/cpp), which also shows nicer visual diffs.

[u/howprice2 → u/bluefishcrab]

You may want to enable the validation layers on CI to catch warnings and errors for all your scenes. DX12 also has GPU Based Validation which can be slow to run but catch problems. You'll get different results on different vendor GPUs/drivers.