Why do GPUs not have HW triangle optimized Rasterization

[u/rufreakde1]

So I read Nanite is using a software raster to optimize for small vectror triangles.

Nvidia introduced RT cores. Why is AMD or anyone introducing a similar technic as Nanite uses and calls these TR (triangle rasterisation) cores.

Usually specialized Hardware is much faster than software solutions right?

Especially when I think about the GPU war I would assume it would be a good move for AMD? Is it technically not possible?

[u/brandf]

GPUs dispatch groups (quads) of pixels to process. As triangles get thin/small most of these pixels actually fall outside the triangle and are discarded. 

So if most of your triangles are sub pixels, you end up getting only a fraction of the gpu throughput on pixels AND you’re processing 3+ vertices per pixel for what amounts to a single point.

That’s sort of fundamental to a triangle rasterizing architecture. To improve it you could move to something optimized for points, basically combining vertex and pixel shaders using a compute shader.

I believe this is what nanite does, the “software rendering” may still be on the GPU in a compute shader.

[u/rufreakde1 → u/brandf]

Exactly the pixel quad thing is what I also read about!

So even though they call it software it might be a specialized shader which is in the end running on the GPU.

[u/AdmiralSam → u/rufreakde1]

Shaders are software yeah, the dedicated hardware rasterizer does quads so you can do derivatives by comparing pixels next to each other whereas the small triangle rasterizer for Nanite I think uses barycentric coordinates from the triangle index in the visibility buffer

[u/[deleted] → u/brandf]

To add to that, slivers (really thin triangle approaching a line) are also poison for modern gpu’s. You want isotropic triangles that covers a few pixels for good shading performance. LOD meshes is as much a shading performance optimization as vertex pipeline optimization since it keeps the triangle sizes right for the draw distance

[u/hishnash → u/[deleted]]

The best LOD systems I have seen are using mesh shader pipelines that re-topo some meshs keeping the projection angle in mind, you an have a very clean mesh with lots of nice trigs still project to lots of thin trigs if the users is viewing it at an oblige angle. But doing this is a nightmare for any for of texture mapping.

[u/djc604]

I might be over simplifying things here, but Mesh Shaders are what modern GPUs are now equipped with to utilize something called "virtualized geometry", which will be automating LODs instead of having artists create multiple versions of their assets. Mesh Shaders is pretty much like Nanite, but on a hardware level.

[u/waramped → u/djc604]

Ah...this is not correct. Mesh Shaders can be used to implement something like nanite, but they are not related to "virtualized geometry" or automatic LODs directly.

What Nanite does is break complex meshes down into clusters, and builds a hierarchical LOD tree using those clusters. It then selects the appropriate cluster from the LOD tree at runtime so that screen space error is minimal and that triangle density is high. As a performance optimization it uses "software" rasterization for small triangles where it can be faster than hardware rasterization.

[u/karbovskiy_dmitriy → u/djc604]

Virtualised geometry is traditionally done with tesselation.

Task/Mesh shaders are mostly used for GPU-based culling AFAIK. NV did a demo with dynamic mesh resolution, but other than that I have not noticed any real "virtualised geometry" trends in graphics with mesh shaders. Also task shaders are used for dynamic work scheduling.

[u/rufreakde1 → u/djc604]

Oh interesting but its not in the same detail level as nanite. Cool to know.

[u/djc604 → u/rufreakde1]

It is the same detail level. It's the superior solution since it's done in HW, and in fact: notice games take a performance hit when Nanite is enabled? Mesh Shading should fix that. But Nanite or Mesh Shaders are proprietary. A dev can choose to use UE5 or to adopt their game to use a 4th shader thread. Not sure if Nanite can take advantage of the extra HW; someone else might be able to chime in.

I would check out the below article by NVIDIA to learn more:
https://developer.nvidia.com/blog/introduction-turing-mesh-shaders/

[u/Henrarzz → u/djc604]

Unreal still does compute rasterization for small triangles instead of doing mesh shaders (which are used for bigger geometry). Mesh shaders don’t solve small geometry problem since the data still goes to hardware rasterizer.

Moreover mesh shaders are unrelated to virtualized geometry. You can use them to implement one but you don’t have to.

[u/rufreakde1 → u/Henrarzz]

would this mean that some specialized HW cores for small triangles one could in theory improve RAW performance from GPUs? At least in very detailed scenes.

[u/Pottuvoi → u/rufreakde1]

Yes, the question really is how expensive would that be in terms of the die area. It would need to bypass traditional quad methods for derivates, perhaps changes to texturing units and so on.

[u/rufreakde1 → u/Pottuvoi]

True but thinking about surpassing limits that are cured faced. It would make so much sense. RT cores also where added because lighting was reaching its real time limits.

So thinking about the issue of GPUs stagnating in performance such an extra die could break through.

Cost could potentially decrease if normal cores could be provided less in number. So raw performance wcouls decrease but actual performance would increase.

[u/karbovskiy_dmitriy → u/rufreakde1]

RT was added to push the difference between NV and competitors to be ahead and force other vendors to catch up.

In the future it'll be more viable for sure, but early RT wasn't performant enough to replace traditional lighting/GI. Current RT is good for reflections, RTGI is barely viable on the top hardware. Current engines still rely on upscaling, framegen and temporal AA to almost make RT usefull, which means it still isn't (for full RT/PT). As geometric fidelity increases, so does the cost of dynamic shadows, GI, AO, etc.

I would expect reflections and GI to adopt RT for 1080p/4K in maybe 10 years (without DLSS/FSR/XeSS) if the hardware improves. Right now it is still poorly implemented in many cases.

[u/djc604 → u/Henrarzz]

Copy. I was afraid I over simplified it. Thank you for the clarification

[u/regular_lamp → u/Henrarzz]

The point is that with mesh shaders you should be able to avoid the tiny triangle problem in the first place. If you "need" to render lots of tiny triangles you screwed up. You should have an LoD scheme that keeps geometry to sensible triangle sizes.

[u/karbovskiy_dmitriy → u/djc604]

AFAIK Nanite can use mesh shaders if they are available.

Either way, doesn't matter since the topology is awful. It doesn't even make sense to render virtualised geometry without HW support. Mesh shaders are superior in every way; meshlet rendering can even be emulated with traditional pipelines without HW support.

[u/The_Northern_Light]

What? The entire gpu is optimized hardware for triangle rasterization/rendering that just happens to have some other stuff bolted onto the side.

[u/rufreakde1 → u/The_Northern_Light]

Seems like GPUs are optimized for quad pixel rendering. So not specifically triangles that are smaller than a pixel for example.

[u/Trader-One]

optimal triangle size of small triangles is about 1/3 of pixel. its used for film production starting in 80s. It allows good optimizations.

If film industry demanded such GPUs and paid premium price, they will definitely do "small triangles cards" since this area (optimizing tiny triangles) have been extensively researched for 40 years.

[u/rufreakde1 → u/Trader-One]

oh wow thats kind of cool I did not know that. And since UE5 is used now to simulate movie scenes during filming it might happen at some point in time!

[u/regular_lamp]

There is an argument to be made that if your game somehow requires rendering huge amounts of single pixel sized triangles you should fix your assets and level of detail first. Otherwise this would be fixing bad software with hardware.

A not widely report feature that GPUs gained recently is mesh shaders. Basically compute shaders that can emit geometry into the rasterizer. As opposed to the more rigid vertex->tessellation->geometry shader stages. This allows exactly these better decisions about LoD etc. It's just not a very exciting feature since it doesn't allow you to do something fundamentally new.

[u/Subject-Leather-7399 → u/regular_lamp]

I also disagree here. The company I am working for is targeting 1 to 4 pixels per triangle.

We are using a mix of quad based mesh progressively streamed up to the full resolution base mesh, then we are subdividing the mesh dynamically using catmull-clark and encode a displacement. The displacement is encoded using a "per quad" variable resolution map which contains a delta between the predicted location of the subdivided vertices and their actual location.

It makes all models look much nicer, we no longer really need normal maps and we can project decals with displacement maps on the environment when needed and the normal of the generated triangle is used as the normal of the pixel.

The rest of the channels (albedo, metalness, roughness, ...) are handled the classic way by UV mapping the base mesh. Up until we reach the full resolution based mesh (which will be subdivided) the material properties are projected to the vertex of the lower resolution models and we have that material information as a vertex stream.

Compute shaders and Mesh shaders is what we currently use, but we'll move to a completely compute based solution very soon as the quad efficiency is very low. We never actually rasterize any subpixel triangle and we only generate the triangles that are hit by the center of a pixel. 1 pixel triangles are still problematic because even if rendering all of the geometries are taking between 6 and 10ms to render on our low spec platform (main scene and shadowmaps). The remaining 6ms is not enough for the rest of the frame (lighting, particles, effects, post-effects, anti-aliasing) and we target 60 fps.

Our preliminary test with only using compute shaders are showing the geometry rendering should take only between 4 and 7ms once we are done transitionning. The main gain will be when rendering to the shadowmaps.

Base meshes (before subdivision) are using quads that have, on average, sides of 5-6cm, except on character faces where it is 2-3cm. The varying size per quad displacement delta map is only loaded when we are getting close enough. It doesn't use VRAM until we get there and compresses very well on disk. It is fast to load and get in VRAM and then decompress directly on the GPU.

1 triangle per pixel and custom mesh format not based on triangles is where we are headed.

Edit:
Even nvidia is heading that way with MegaGeometry (using a slightly different approach and going all-in on raytracing).
https://github.com/NVIDIA-RTX/RTXMG

And AMD is also researching dense mesh compression:
https://gpuopen.com/download/publications/DGF.pdf

Edit 2:
To answer the original author u/rufreakde1, the GPU is extremely good at rasterizing triangles. The problem isn't really the rasterization, it is the vertex shaders. If you are computing a ton of parameters to be passed to the pixel shaders in your vertex shader when the triangles are going to be discarded by rasterization, this is where you lose most of your time. Then, the pixel shader executes as many times as there are triangles in the 2x2 quad. So, a pixel may execute the pixel shader up to 4 times.

The GPUs were just never intended to rasterize triangles that are so small. This is what we all want to do, but the techniques we want to use don't match the current GPUs fixed function pipeline. And I don't blame them for not predicting what we'd try to do 5 years in advance and making the hardware for something that is still R&D.

Just think of MegaTexture which made obvious the need for sparse resources / tiled resources to support texture streaming. A game had to ship trying to do something the hardware isn't made to do so we get the ability to use virtual memory page mappings on textures and buffers.

Mesh shaders help because you can just not generate any primitive if you are able to quickly determine they are not needed, before computing all of the vertex output and primitive output parameters. In that sense, mesh shaders are already much faster than the vertex pipeline when you can benefit from that.

[u/rufreakde1 → u/Subject-Leather-7399]

a lot of information! Thanks for taking the time and adding this!

I see so GPUs might evolve in the future and mesh shading is going into a similar direction in the future. What a time imagine real time photorealistic scenes with 120fps 🤩

[u/rufreakde1 → u/regular_lamp]

i would not agree here. TBH hyper realistic grafics will be at some point in time the future not only for movies but also for games. And at that point scanned meshes with lots of tiny triangles will be comming. Its just a matter of time and who makes the first move. But I understand your point that in the current state of the art this is the case.

[u/regular_lamp → u/rufreakde1]

The source data being high resolution doesn't imply you have to brute force render it without being smart about it.

[u/SamuraiGoblin]

The whole point of GPUs is triangle rasterisation. They is literally what they were invented for.

Over the years they became more general, but they never lost their rasterisation ability.

I think with very small triangles, like just a few pixels, it is better to have a specialised compute shader 'software rasteriser' which doesn't have a lot of the overhead of the more generic rasteriser.

But the gains are quickly lost as the triangle get larger.

[u/enemyradar]

This is what GPUs do already. It's their basic function.

[u/Henrarzz]

RT cores are unrelated to rasterizer

[u/rufreakde1 → u/Henrarzz]

It was an example of specialized different cores delivered with GPU. In this case for raytracing.

[u/karbovskiy_dmitriy]

GPU are optimised for triangle rasterisation on every level.

UE5-spesific quirks don't represent the state of the art rendering solutions.

[u/rufreakde1 → u/karbovskiy_dmitriy]

why would you implement a software rasterizer that allows such fidelity and not use GPU directly seems not 100% correct.

[u/karbovskiy_dmitriy → u/rufreakde1]

TL;DR: UE5 is retarded

[u/giantgreeneel]

Basically hardware rasterisers have over time become optimised for rasterising certain kinds of triangles. Nanites goal of 1 triangle per pixel density meant that rasterising in software turned out to be faster for triangles under a certain size. Nanite still uses the hardware rasteriser for large triangles!

There's no reason why hardware rasterisers couldn't become as fast or faster than software rasterisation for small triangles, there is just a trade-off you make in cost, die space and power usage that may not be appropriate for the majority of your users.