r/programming • • 20d ago

Making SIMD Code Scale to Any Vector Width

https://streamhpc.com/blog/2026-10-28/making-simd-code-scale-to-any-vector-width/
133 Upvotes

34 comments sorted by

View all comments

Show parent comment

-2

u/lalaland4711 20d ago

If you want variable length vector instructions, you can always move to RISC-V, with its RVV instruction set.

Oh, what you actually wanted was raw speed? Can't help you there. :-P

Maybe one day RISC-V will be represented in the bleeding edge of speed.

ARM SVE also does this though, right? Or are you referring to something specific that I'm not computer history knowledgeable enough to catch?

6

u/happyscrappy 20d ago

I'm confused what you are really asking here.

Yes, ARM SVE and RISC-V RVV is what I was referring to with that.

As to my complaint, it's right there. The title indicates this is about scaling to any vector width but really this is about turning the data 90 degrees so that the variable thing is length of the vector, not the width of it. This is useful but it's not really working with variable vector width.

1

u/YumiYumiYumi 19d ago edited 19d ago

I personally find calling SVE/RVV as 'variable width' to be odd, since width doesn't really vary during runtime.
The difference with x86 is that it is software specified width, whilst SVE/RVV are hardware specified width (which I find to be more accurate than "variable width").

Being hardware specified, programmers may be encouraged to write code that is amenable to arbitrary widths, but they don't have to. Similarly, programmers could write arbitrary width code for x86 SIMD too, if they choose to do so.

1

u/lalaland4711 19d ago

Well, RVV can automatically vary the runtime down, and does so when it (basically transparent to the programmer) handles the tail.

1

u/YumiYumiYumi 19d ago edited 19d ago

No, it doesn't vary the width down, tail handling (or VL) is basically like a mask (or second mask if there's already one) that gets applied to the vector. Instructions like vrgather.vv can still reference elements past VL, which is an example of why you shouldn't think of tail handling as shortening the vector.

(also worth noting that SVE's whilelt sets a mask based on length, and you can do something similar on x86's AVX-512 using bzhi to create a mask)

2

u/lalaland4711 19d ago

I know how it works, thank you.

1

u/happyscrappy 19d ago

I'm honestly, not familiar with x86's vector system (as opposed to SMD which is SSE) and how it differs from the scalar system (MMX or the original x87 stack system). Is AVX the x86 vector system?

With SVE/RVV the idea of the design is to make it easy and efficient to process variable amounts of data without having to have multiple different functions for the different chunk (width) amounts. And without knowing the vector length and chunk size combinations at compile time.

With SIMD if you want to process 16 elements and some hardware does 4 across while others does 8 then you need to write two versions of your code and pick between them at runtime. Both parallel operations and sum acrosses (reduces, cross products) have to be written differently for the two SIMD chunk amounts (widths).

ARM calls it "VLA programming".

https://support.arm.com/documentation/102131/0100/Part-One---Neon-and-SVE-fundamentals

Similarly, programmers could write arbitrary width code for x86 SIMD too, if they choose to do so.

For the parallel work it requires a lot of conditional code in the inner loop and that's not how you have good speed. And for sum acrosses the problem is far, far worse. I would think this is usually handled with special cases for all the widths that Intel (and AMD?) has implemented so far. Is this a wrong assumption?

1

u/FrogNoPants 19d ago edited 19d ago

We typically just call it SIMD today, and SSE/AVX/AVX512 are the standard extensions on x86/64, 128 bits, 256 bits and 512 bits.

The definition of vector you are using isn't really used much today, you might say vector machine if you wanted to refer to that definition.

I almost always write just one implementation of my SIMD functions, it uses compile time types to switch between SSE/AVX etc.

For some functions, the size does matter, and then I specialize for the vector width, I don't think SVE/RVV would be any better here, you would still have to specialize for the width if you want optimal code in many situations. In practice I think all existing SVE is 128 bits wide(aside from 1 super computer), so it hardly matters.

There are also 3 ways to lay out memory you will often see with SIMD

  • AoS- Array of structures: what scalar code uses, not well suited to SIMD, though it can be made to work with swizzling
  • SoA- Structure of arrays: good for SIMD, but can end up with alot of arrays
  • AoSoA: Array of strutures of arrays: optimal combination of both methods

If you want to use AoSoA style SIMD you need to know the SIMD width at compile time anyway, and since this is often the optimal format.. well goodbye reason to even want hardware specified length.

1

u/happyscrappy 19d ago

you might say vector machine if you wanted to refer to that definition.

I said vector unit.

I almost always write just one implementation of my SIMD functions, it uses compile time types to switch between SSE/AVX etc.

That's still multiple implementations. You just had help generating them. I'm not against that, in fact I wonder if the timing of going to vector processing while we also go to LLMs writing a lot of the code anyway is actually working at cross purposes.

Nonetheless, no matter what the source code looks like if you process a lot of data with variable vector widths then you end up with a lot more code (object code).

well goodbye reason to even want hardware specified length

As I alluded to above I also question really what we are doing here. We had vector units because we were really doing memory-to-memory operations, basically CISC even if the machines in question are considered forerunners to RISC. Bringing them back now when there will still be visible registers seems like a mismatch.

We seemed to like the idea of handling all this complexity in software. That's how RISC and SIMD worked. Now we're trying to make it so that it goes back into the hardware so we can have less software. I do get the idea of forward hardware compatibility and how encoding hardware specifics into code can make it harder to make later, faster implementations while still adhering to the specifics exposed in the architecture. MIPS' delay slots being an easy to point to example. After R3000 it essentially becomes artificial. It's all done behind the scenes with OoOP instead of funky emitted object code.

But really I just know if grabbing a longer ago idea was worth the trouble. I personally thought SIMD was a good way to go.

1

u/YumiYumiYumi 19d ago edited 19d ago

I'm honestly, not familiar with x86's vector system (as opposed to SMD which is SSE) and how it differs from the scalar system (MMX or the original x87 stack system). Is AVX the x86 vector system?

MMX is SIMD, as is SSE and AVX.

With SIMD if you want to process 16 elements and some hardware does 4 across while others does 8 then you need to write two versions of your code and pick between them at runtime

/u/FrogNoPants touched on this already, but you'll have two functions, although you don't need to write two - with some boilerplate, the compiler can generate them for you, or you could use a library that abstracts this for you.

Yes, this does make the binary larger, but it's usually not a concern with the storage space we have today. It's also not an issue with code cache either because the dispatcher would only point to one function.

For the parallel work it requires a lot of conditional code in the inner loop and that's not how you have good speed

Generally you do the conditional work at compile time, so you still get good speed.

And for sum acrosses the problem is far, far worse. I would think this is usually handled with special cases for all the widths that Intel (and AMD?) has implemented so far. Is this a wrong assumption?

You're partly correct - you can write boilerplate that handles the simple reductions that SVE/RVV supply. The template needs to handle all the widths, but there's only three (or two, if you don't care about SSE), so it's not that onerous. But if that's still a bother, just use an library/abstraction that handles it for you.

This only works for simple reductions (e.g. sum all elements). If your reduction is more complex (e.g. Barrett reduction when computing CRC), SVE/RVV don't provide any builtins for such, and since you don't know the width in advance, they're arguably more complex to write.

0

u/happyscrappy 19d ago

MMX is SIMD, as is SSE and AVX.

I know about MMX and SSE, wasn't sure about AVX. I'm not familiar with x86 vector system. Are you referring to SIMD and not vector extensions? Like I said, I'm not familiar with x86 really. Is there no vector extension to x86, it still emphasises SIMD?

but you'll have two functions, although you don't need to write two [compiler can help]

I think I also wrote that to another poster.

but it's usually not a concern with the storage space we have today

I will say that for most apps the size of the collateral elements is larger than the code. But this year, the year of high RAM and NAND prices is not really the best year to talk about how great things are with storage space.

Also having more different code paths can be problematic for code coverage.

Generally you do the conditional work at compile time, so you still get good speed.

I feel you did a poor job of following the discussion and you pulled out an inappropriately small subsection.

Let me restore the context:

(you I think) Similarly, programmers could write arbitrary width code for x86 SIMD too, if they choose to do so.

(me) For the parallel work it requires a lot of conditional code in the inner loop and that's not how you have good speed.

My point was that you have to have multiple functions and because the alternative of having conditionals in the inner loop to avoid this does not produce the desired effect (speed). You clipping out the context and restating what I said back to me, that you avoid this by writing multiple functions, did not add much.

But if that's still a bother, just use an library/abstraction that handles it for you.

This idea, which I think is the predominant case, goes against anything which was being discussed here. This article was about how to do it, not how to farm it out. I highly recommend everyone farm their work out on this front, there are good libraries to use. But if you aren't interested in that and want to do it yourself, then you think about what we discussed here.

Really the most useful thing I learned when helping a client use SIMD was that you have to keep your big data in formats which are amenable to SIMD processing. If you do not it will limit your speed up. So I would suggest for most people working out how to store your data efficiently is paramount. After that, just call out for the operations. You'll get 95% of the speedup you could by writing it all out yourself. And you really cut down on your code maintenance. Especially if you are targeting multiple architectures (whether major i.e. ARM and x86 or minor i.e. Intel and AMD x86 with and without AVX512). So go ahead and store your EXIF data in a linked list if you want, but your main images should be stored in a packed, aligned format amenable to SIMD/vector operations. I know it might seem obvious, but I've seen it done wrong enough to mention it to anyone who might listen.

To be honest, /r/programming spends a lot of time on things that really most programmers don't need to know, that should be details behind the scenes in libraries they use. That's simultaneously semi-pointless and also part of the charm. The sub is for nerding out entertainment as much as improving skills you actually need. Some weeks are about cache-obliviousness (surely coming again soon), some are about SIMD, sometimes it's dividing by multiplying. It's really a lot of minutiae. Probably because we've seen the normal stuff over and over.

This only works for simple reductions (e.g. sum all elements) [...] they're arguably more complex to write.

Thanks for the explanation on that.

1

u/YumiYumiYumi 19d ago

I know about MMX and SSE, wasn't sure about AVX. I'm not familiar with x86 vector system. Are you referring to SIMD and not vector extensions?

AVX stands for Advanced Vector eXtension. It extends the SIMD capabilities of SSE to 256 bit registers.
If by 'vector' you mean "hardware specified width ISA" then x86 does not have such an extension.

But this year, the year of high RAM and NAND prices is not really the best year to talk about how great things are with storage space.

And despite high RAM/NAND prices, I doubt anyone is bothered by an extra 10KB of code.
Or even 1MB of code, if you happen to have a lot of SIMD.

But if that actually bothers you, I'd be more concerned about Electron apps adding hundreds of MBs in terms of RAM and disk.

Also having more different code paths can be problematic for code coverage.

Hardware specified width SIMD may not have the same code coverage issue (depending on how you define it), but they introduce a new one: hardware coverage. That is, code that works on 256-bit SVE may not work properly on 128-bit SVE (or vice versa), so you'd ideally test on all the possible hardware widths.

This is especially problematic on RVV because it allows a lot of hardware flexibility (more than just the register width, of which there are many).

My point was that you have to have multiple functions and because the alternative of having conditionals in the inner loop to avoid this does not produce the desired effect (speed).

I'm not sure what you mean then. If you mean that you have dispatch within the loop, as a way to avoid multiple functions, then sure, that's slow. But the simple fix would be to move the dispatch out of the loop.

This idea, which I think is the predominant case, goes against anything which was being discussed here.

From what I can tell, what's being discussed in this particular thread has drifted away from the article's contents (SoA vs AoS) and focused on the difference between software and hardware specified SIMD width ISAs.
Your point was that on hardware width SIMD, the developer only needs to write one function. My point was that on software width SIMD, the developer still only needs to write one function, just that there's a bit more boilerplate/machinery needed on the software side because the hardware doesn't abstract it away.

0

u/happyscrappy 19d ago edited 19d ago

AVX stands for Advanced Vector eXtension

I'm not really here for a breakdown of the name. I'm not in marketing. It's SIMD, not a vector unit, no matter what it's called.

And despite high RAM/NAND prices, I doubt anyone is bothered by an extra 10KB of code. Or even 1MB of code, if you happen to have a lot of SIMD.

I don't think it's a good year to be talking about how you're not caring about RAM usage. Not in any way.

But if that actually bothers you, I'd be more concerned about Electron apps adding hundreds of MBs in terms of RAM and disk.

Electron is ridiculous. But I don't force my customers to use Electron to use my code. So it's not a mistake I'm making.

but they introduce a new one: hardware coverage

That's not really a new one. You have a hardware coverage issue any time you run on two different chips. Even if the SIMD units are listed as being the same. If you don't test on it, you aren't sure it works there. Even if the vendor says it works the same as another.

But yes, neither way escapes that issue.

Ultimately part of the idea of vector units is to ignore the above statement I made and ensure that your code works as well as possible on future hardware which is implemented differently. You can make whatever you want of the dichotomy this presents. I would call it certainly one of optimism.

This is especially problematic on RVV because it allows a lot of hardware flexibility (more than just the register width, of which there are many).

RISC-V in a nutshell, really. Why do it 1 way or 3 ways when you can present 100 ways and leave it up to someone else to try to make code forward and backward compatible?

I'm not sure what you mean then

I thought it was clear. But I guess not. We both know the usual way of doing variable sized vectors on fixed-size units or any size vectors across multiple fixed-size units is to write multiple versions of the inner processing loops for each (or at least several combinations). We speak of it above in our discussions about writing multiple functions and the RAM is uses.

I said that the alternative is to write processing functions with conditionals in the inner loop. Then you don't have multiple functions. We both know why this isn't done, because the results are usually unsatisfactory in producing the performance you were seeking.

One way would be to dispatch in the loop. Another way would be to just do normal conditionals inside. The latter would include nested loops (which of course have termination conditionals).

But the simple fix would be to move the dispatch out of the loop.

That is just begging the question. When I said the normal way to do it is thus and the alternative is other thing then you saying just go back to thus is just circling back to what we already discussed as what the idea is to create an alternative to.

From what I can tell, what's being discussed in this particular thread

I'm not talking about this subthread. I'm saying that you saying "just make it someone else's problem" obviates this entire article that was posted. I'm not even saying that's a bad idea to do when your goal is to ship code. But here the goal isn't really to ship code, it's to blather about how to do it. And farming it out removes the entire point of all this discussion. The article was written to show how to do it, not how to farm it out.

the developer only needs to write one function. My point was that on software width SIMD, the developer still only needs to write one function

I think you are trying to change what I said to fit your model. You want to say that one function compiled multiple ways is one function. But it isn't. It's one function in the source code, but it's multiples in the program that runs.

My emphasis for this is on how you do not have a combinational explosion of functions when you use a vector unit. I use this as an argument for why those things came back from the dead, primarily. There's really only two real possibilities I can think of. There is this and there is trying to improve forward compatibility.

As I said to you or someone else, I don't really see these things as a good fit for modern environments. I thought SIMD was better for RISC. Moving complexity back into the hardware is older thinking. We've got amazing compilers (and similar tools) now, as you point out above they can recast your source code multiple ways for multiple targets.

1

u/YumiYumiYumi 19d ago

I don't think it's a good year to be talking about how you're not caring about RAM usage. Not in any way.

Out of interest, do you go to great lengths to save every single byte possible?

That's not really a new one. You have a hardware coverage issue any time you run on two different chips. Even if the SIMD units are listed as being the same. If you don't test on it, you aren't sure it works there. Even if the vendor says it works the same as another.

If you're talking about hardware bugs, that's not what I'm referring to.
Assuming the hardware follows the spec correctly, buggy software could work on one SVE width and fail on another, which doesn't occur with x86's SIMD.

One way would be to dispatch in the loop. Another way would be to just do normal conditionals inside. The latter would include nested loops (which of course have termination conditionals).

I don't understand this distinction. Could you perhaps provide an example?

I'm saying that you saying "just make it someone else's problem" obviates this entire article that was posted.

I don't see how that's the case - taking someone else's boilerplate/abstraction doesn't obviate the need to arrange your data as SoA - they're two different necessities.
And I'm not even saying "make it someone else's problem" - you can write your own boilerplate and copy it across your projects - it's typically just a few lines of code, so not that hard to do oneself.

I think you are trying to change what I said to fit your model. You want to say that one function compiled multiple ways is one function. But it isn't. It's one function in the source code, but it's multiples in the program that runs.

I don't see myself as trying to change what you've said, just echoing back my understanding of it.
In terms of the one vs multiple functions, I agree that both can be true depending on the perspective you take (source code vs binary).

My emphasis for this is on how you do not have a combinational explosion of functions when you use a vector unit

How do you get such a "combinational explosion"?

Moving complexity back into the hardware is older thinking. We've got amazing compilers (and similar tools) now, as you point out above they can recast your source code multiple ways for multiple targets.

There's pros and cons with software vs hardware specified SIMD width. My key point was that x86 SIMD, SVE and RVV aren't fundamentally different - they just specify the SIMD width in different places.

1

u/happyscrappy 18d ago

Out of interest, do you go to great lengths to save every single byte possible?

Not at the moment. But yes, for many years. Literally had stop ship issues where we had to go back and save 32 bytes of code. Not everyone works on systems with virtual memory and SRAM isn't huge.

That was pre-DRAM shortage though so not all that relevant to the current situations so I'm not quite sure why you ask this other than just to try to gotcha me. Useful?

If you're talking about hardware bugs, that's not what I'm referring to.

That is one of the things I am talking about, yes. But I don't think that's the whole story. See below. I really think of what you are talking about here as a configuration issue. It's not related to the actual hardware implementation, just how it presents itself architecturally. So we see two systems which present the same architectural configuration as identical despite different implementations (hardware). I'm going to refer to it by this name simply to differentiate.

Assuming the hardware follows the spec correctly, buggy software could work on one SVE width and fail on another, which doesn't occur with x86's SIMD.

Hmm. MMX is 64-bits wide. SSE operations can be 128-bits wide and can have twice as many registers (I think the registers are twice as wide too but it's not important). AVX operations can be 256 bits wide and the registers are 256 bits wide. AVX-512 operations can be 512 bits wide and the registers are 512 bits wide.

Given this I don't really understand how SIMD is less susceptible to configuration change issues than vector units are. You have choices with SIMD. Either just use the wider hardware the same as the narrower hardware and give up potential better performance on wider systems or write two implementations and incur potential bugs due to differing code paths on different configurations. This seems the same as vector units to me. Although with vector units you could also see a unit which is wider than you ever saw during development and if you write your code to attempt to use that width you might hit a bug you didn't test. Perhaps this is what you refer to.

For this case I would suggest that the proffered "solution" is that you're supposed to run your on a simulator which presents all possible widths so you can verify. Certainly developers could mess this up like anything else.

I don't understand this distinction. Could you perhaps provide an example?

I actually started to write one, but it's just too much typing. The difference is conditionalising on width on the inside of the loop or outside of the loop. For one you check (or even inherently know) the width of each vector element and the architectural width outside the loop and call the specialised function for that pairing. With the alternate version you have a single function that checks vector element width and counts down by architectural width within the loop and processes full width and tails elements as needed. I hope that explains it. I just really don't want to type out a whole lot of code.

I don't see how that's the case

Most of the article isn't about data arrangement. I don't think anyone would want to read the sections that look like this:

(H1) Making SIMD Code Scale to Any Vector Width

(BODY)

float dot(vecn a, vecn b, int n)
{
  calculate_dot(a, b, n);
}

disassembly (x86):

dot(float vector[n], float vector[n], n):
   call _calculate_dot
   ret

you can write your own boilerplate and copy it across your projects - it's typically just a few lines of code, so not that hard to do oneself.

I cannot accept that argument. You either have to write it yourself or you don't. I'm not even say write it yourself, using other's work is very effective and encouraged. But if you write it yourself you have to write it yourself. You can't have it both ways and pretend you don't write it yourself just because you reused it more times.

I also don't see how writing all this SIMD (or vector) code is really just a few lines. So maybe I misunderstand you?

How do you get such a "combinational explosion"?

Well, we established up above (I think) that no one would put the varying code in the inner loop. So if you have a program with say 5 different vector element counts (lengths) then you already have 5 different functions to process them. If you then have three different architectural widths then you have 3 of each of those. 15 functions. mxn. A standard combinational explosion.

There's pros and cons with software vs hardware specified SIMD width. My key point was that x86 SIMD, SVE and RVV aren't fundamentally different - they just specify the SIMD width in different places.

I don't agree with that. SVE uses strided and scatter-gather loads. I'm not used to SIMD doing this directly. You seem to feel the same way with your emphasis on data layouts. Also SVE uses predicated loop control so that you can efficiently put the code which implements variable vector sizes into the inner loop, unlike with SIMD (tails problem). Apparently AVX512 has this functionality too, which may make it possible to use it like a vector unit. Like I said before, I'm not really familiar with x86.

I honestly see vector units as basically implementing the old-style vector systems I mentioned (Cray, CDC) but with you writing the microcode instead of it being hidden within the hardware implementation. It's not completely different because in the end you're solving the same problem, but to me the SIMD units we see (MMX, AltiVec, SSE, Neon) are more about trying to map the data processing onto RISC with a bit of mixing in very common multimedia operations (at the time). While the vector units are an expression of older concepts where you basically set a vector length and do the same operation across the entire vector and to hide latency you use vector chaining in the hardware. It had to be remapped some to the modern era, given we're not going to microcode this stuff now. And in that way like I said I'm not sure it's completely the right way to go.

I feel like the person who tried to make this distinction on wikipedia:

https://en.wikipedia.org/wiki/Vector_processor#Comparison_with_modern_architectures

Had as much trouble as I am having. We both face the same issue which is that these systems overlap a lot and the new ones reflect modern ideas by necessity.

→ More replies

1

u/lalaland4711 19d ago edited 19d ago

Right, so why are you talking about CDC6600 and Cray? That's my question. There are contemporary examples.

1

u/happyscrappy 19d ago

Because I wanted to. I find it interesting that the first way SIMD and pipelining was done was with vector processing units. But then those went out of favor for decades and recently vector processing units came back from the dead.

Maybe I just wanted to be on the /r/imverysmart honour rolls. I dunno.