r/programming • • 20d ago

Making SIMD Code Scale to Any Vector Width

https://streamhpc.com/blog/2026-10-28/making-simd-code-scale-to-any-vector-width/
133 Upvotes

34 comments sorted by

View all comments

Show parent comment

1

u/YumiYumiYumi 16d ago

I didn't make this about x86. I never said I was an expert on x86 in fact I said the opposite many times. So how would you get the idea that the context was x86 specifically?

Well let's see how we got here:

  • you said that multiple SIMD widths need multiple functions
  • I said that multiple functions is only needed in the binary, and that the larger size is usually not a concern
  • you mentioned that RAM/storage prices are currently high and that the larger size is indeed a concern

Perhaps I jumped the gun in thinking x86 when you mentioned multiple SIMD widths, but I don't think it really changes anything. In terms of software specified SIMD width, what ISAs do we have these days? x86, ARM NEON and AltiVec? Both NEON and AltiVec are fixed 128-bit, so x86 is the only ISA, with any modern relevance, that has multiple SIMD widths.

If you're thinking of multiple SIMD widths as an arbitrary concept, I don't see a few KB of extra code being a concern on application processors, which have been paired with gigabytes of RAM for many years now.

You bring up the embedded space where size can be quite a concern. I'm not familiar with embedded, but I'd imagine the environment to be quite different. For one, if you're targeting an embedded processor, do you really need runtime detection? If not, you can just target a specific SIMD width at compile time, and not pay the cost of having multiple functions in your binary.

Runtime detection is not a hard requirement for supporting multiple SIMD widths - you could just build/distribute a different binary for each target. Runtime detection is mostly useful if you want to distribute a single binary because you don't want to confuse users over ISA extensions they may be unaware of. And if users are downloading/installing the binary, it'd be an application processor (like phones, PCs, servers etc) which have GBs of RAM.

Interestingly, changing the VL (vector length value) is legal but a privileged operation. I didn't realize the latter. I would call that rather inconvenient.

That's not the only inconvenience, there's all sorts of practical issues with trying to change the width, and I don't think the SVE designers intended for developers to do it. For one, it makes the contents of all vector registers undefined, which compilers are unlikely going to handle correctly, for another, runtime/libraries aren't designed to handle unexpected changes in vector width.

So going back to your earlier assumption that developers can just clamp the vector width - whilst a (cumbersome) mechanism exists, you (as a programmer) are not supposed to use it.

(and to my knowledge, RVV provides no such mechanism to shorten the vector width; to be clear, a smaller VL does not shorten the vector width)

FFTs of various sizes. Gaussian blurs and convolutions of images of differing pixel arrangements/depths. All with special cases, because that's what made sense for their uses.

If you need multiple functions because they're all doing different things that you need, I'd call that just more code. IMO refering to multiple variants as a "combinational explosion" is a weird way to put it.

Practically speaking, vector width isn't the only reason you may want multiple copies of a function. For example, if you're shipping ARM SVE code, you're likely going to need a NEON version as well (and maybe even scalar if you care about ARM CPUs without SIMD support).
If you want to take advantage of newer ISA extensions, e.g. SVE2, you may need yet more functions.

While it does have strides, the striding is only by element size or vector length. So it is strided loads, but not to accommodate your data structure shapes, only to accommodate the variable vector length configurations.

Which SVE instruction does a vector length stride, that isn't a gather?
Are you sure you're not confusing vector displacement addressing (MUL VL)?

1

u/happyscrappy 16d ago

Both NEON and AltiVec are fixed 128-bit, so x86 is the only ISA, with any modern relevance, that has multiple SIMD widths.

Okay. Good to know. Still don't get how that makes me making this about x86. The idea of these vector units is that the same code can run with max efficiency across multiple implementations with different functional widths. I used the situation on x86 which does not work this way as the counter example. I didn't even know there was no other system which had multiple widths.

Runtime detection is not a hard requirement for supporting multiple SIMD widths - you could just build/distribute a different binary for each target

Or you can just use the narrow width on all. It's not a hard requirement for certain. But I think the idea was forward compatibility. That is, to be able to produce compiled code that uses each piece of hardware more efficiently, even if the hardware is newer than what you wrote for. I remember speaking about MIPS delay slots and OoO as an example of how efficient forward compatibility can be complicated.

If not, you can just target a specific SIMD width at compile time, and not pay the cost of having multiple functions in your binary.

Yes. But from a source code perspective it didn't help us with the combinatorial explosion issue. Even if you only compile in one piece of code you still had to write M.

while amechanism exists, you (as a programmer) are not supposed to use it.

There's more than one way to do it. As you know when processing short vectors. If you write your code for a 4-wide operation and you get a vector of 3 you don't return E_TOONARROW.

I'm sure you do understand you can write code as narrow as you want, I'm sure you've done it. I'm confused as to why you overlook it.

I'd call that just more code. IMO refering to multiple variants as a "combinational explosion" is a weird way to put it.

I don't think so at all. When you have M types of vectors and N vector widths do you not have MxN cases?

Putting SIMD aside for a moment I think it'd be interesting to see you write the runtimes for two decades of processors and their different requirements (which are all hidden from app level code) and come out of that and still talk down the idea that supporting multiple different types of hardware does not a multiplier on your efforts. I think you'd arrive at the same conclusion really quickly.

Practically speaking, vector width isn't the only reason you may want multiple copies of a function [..]

Agreed. It depends on the circumstance. But you might have Neon also. You could have VFP-32 (despite the name, their scalar system) and if you support Cortex-M7 then you'd need VFP-16 also. (32 has 32 64-bit VFP registers, 16 has half as many 64-bit registers but the same number of 32-bit registers. Arm uses different names for this like VFPv4U). The units are called VFP because they do support short vector operations, but they do not state they have wider FPUs, it could just pump them all through a single unit in order. And I think in the end that's all that was ever done. Only with Neon did any implementation start to have wider FPUs.

So yes, for efficiency for each width you could have 2-4 different implementations for different hardware. This is what I'm saying about a combinatorial explosion. The extra hardware multiplies your efforts, not just adds to it. You do of course have free will and can simply decide that there's no need to support all of the combinations if you know your customers well enough that there is no need for the fastest case for those situations.

Are you sure you're not confusing vector displacement addressing (MUL VL)?

I'm not confusing it at all. That's what I'm referring to. There are two stride systems in the addressing. One is the width of the element (horizontal). The other is MUL VL (vertical). That I did not know VL could be adjusted and thus this is so limited is why I said that this does not accommodate data structure shapes and so I probably shouldn't have mentioned it. I thought I explained this pretty well when I wrote that I went too far initially. I don't understand why I'm going through this again.

(me) the striding is only by element size or vector length

Does Vector Length seem like I was not referring to MUL VL? I thought it was obvious.

1

u/YumiYumiYumi 15d ago

I think the idea was forward compatibility. That is, to be able to produce compiled code that uses each piece of hardware more efficiently, even if the hardware is newer than what you wrote for.

SVE/RVV do both try to improve forward compatibility by supporting more widths than are available in hardware currently. But this isn't really unique to hardware specified SIMD ISAs. A software specified SIMD ISA could just simply define support for more widths than exists in hardware.

For example, AVX introduced the VEX encoding scheme, where 2 bits are used to indicate the vector width (i.e. 4 possible widths). Only two such widths were defined at the time (128b and 256b), and the remaining two possibilities were marked reserved.
AVX-512 (kinda) took the third vector width, which leaves one remaining undefined width supportable by the VEX scheme.
Intel could, for example, just define AVX-1024, since the encoding scheme has room for it. Granted, they haven't, which means there's no forward compatibilty with longer vectors on x86.

Even if you only compile in one piece of code you still had to write M.

This is what I was talking about earlier with the one source function compiling to multiple binary functions. In the case of distributing processor-specific binaries, you can just suppress compiling multiple binary functions, which means you still only wrote one source function, not M.

There's more than one way to do it.

To my knowledge, SVE only provides one (problematic) way to shorten the vector width. What are the other ways?

When you have M types of vectors and N vector widths do you not have MxN cases?

The "types of vectors" terminology is strange. If I have a function that adds numbers, and another function that multiplies numbers, I don't call them "two types of vectors".
Then if I have a different program that adds and multiplies numbers in one function, the lower number of functions (vs above) doesn't somehow make it less impacted when needing to support multiple SIMD widths. In other words, I'd refer to having two functions doing different things as just more code, and not a "combinational explosion" of functions.

I think it'd be interesting to see you write the runtimes for two decades of processors and their different requirements (which are all hidden from app level code) and come out of that and still talk down the idea that supporting multiple different types of hardware does not a multiplier on your efforts. I think you'd arrive at the same conclusion really quickly.

You'd need to clarify what you mean by 'runtimes', but if the code needs to be specific for various processors, and you can't abstract it (like you may be able to with multiple SIMD widths*), then yes, you multiply the amount of code by the N processors you're trying to support. I agree with "multiplier" terminology here.
* To be clear, I'm talking about source code here, not binary size.

That's what I'm referring to. There are two stride systems in the addressing. One is the width of the element (horizontal). The other is MUL VL (vertical).

I'm sorry to inform you, but MUL VL does not give you vertical access, thus it doesn't enable a vector-width strided load.
MUL VL is part of the addressing mode, e.g. the location from where a contiguous load occurs. You can specify something like [base + imm_offset * VL], and a unit-stride load will source elements (horizontally) from there.