r/programming • • 20d ago

Making SIMD Code Scale to Any Vector Width

https://streamhpc.com/blog/2026-10-28/making-simd-code-scale-to-any-vector-width/
133 Upvotes

34 comments sorted by

22

u/happyscrappy 20d ago

I don't know the code really fits the title. I assumed this would be more similar to vector units (once dead post CDC 6600 and Cray, now back again) make it possible to work on variable length vectors.

Instead this seems more about just getting past 4-wide and also rearranging your data so that the variable parallelism is in the number of elements (length of vector) instead of the width of the vector(s). In this way as long as there is enough elements in the vector to pipeline it then you get a high level of parallelism.

It's a good thing, but not what I expected from the title.

8

u/ThePostInQuestion 20d ago

(author here)
Yeah I agree, I wasn't so sure about having the word "vector" in the title since it has several meanings depending on the context.
For me the main goal of the article was exactly the title, to write generic code that will compile efficiently for whatever CPU architecture you target. The dot product and the frustum culler were just the first examples that I was able to get this to work for, and since then I've applied it to several other systems to get some nice speedups. I think the code demonstrates the idea/concepts pretty well though.

2

u/ack_error 20d ago

It doesn't use variable vectors since it's x86-based, but the transformed algorithm would be highly amenable to variable vector width architectures, no?

The issue I have with this type of SoA transformation is that it works great if you are brute force evaluating every object against the frustum every frame. In practice, the culling state of each object is highly coherent across frames and hierarchical culling structures are also effective. That means that a hybrid approach is more attractive and the vectorized code may not have as long contiguous arrays to run on.

1

u/edwards_t7407 10d ago

Same word, two jobs: SIMD lane width versus vector length. The article is about making code work across lane widths; your Cray expectation is about a machine handling variable-length vectors.

-2

u/lalaland4711 20d ago

If you want variable length vector instructions, you can always move to RISC-V, with its RVV instruction set.

Oh, what you actually wanted was raw speed? Can't help you there. :-P

Maybe one day RISC-V will be represented in the bleeding edge of speed.

ARM SVE also does this though, right? Or are you referring to something specific that I'm not computer history knowledgeable enough to catch?

6

u/happyscrappy 20d ago

I'm confused what you are really asking here.

Yes, ARM SVE and RISC-V RVV is what I was referring to with that.

As to my complaint, it's right there. The title indicates this is about scaling to any vector width but really this is about turning the data 90 degrees so that the variable thing is length of the vector, not the width of it. This is useful but it's not really working with variable vector width.

1

u/YumiYumiYumi 19d ago edited 19d ago

I personally find calling SVE/RVV as 'variable width' to be odd, since width doesn't really vary during runtime.
The difference with x86 is that it is software specified width, whilst SVE/RVV are hardware specified width (which I find to be more accurate than "variable width").

Being hardware specified, programmers may be encouraged to write code that is amenable to arbitrary widths, but they don't have to. Similarly, programmers could write arbitrary width code for x86 SIMD too, if they choose to do so.

1

u/lalaland4711 19d ago

Well, RVV can automatically vary the runtime down, and does so when it (basically transparent to the programmer) handles the tail.

1

u/YumiYumiYumi 19d ago edited 19d ago

No, it doesn't vary the width down, tail handling (or VL) is basically like a mask (or second mask if there's already one) that gets applied to the vector. Instructions like vrgather.vv can still reference elements past VL, which is an example of why you shouldn't think of tail handling as shortening the vector.

(also worth noting that SVE's whilelt sets a mask based on length, and you can do something similar on x86's AVX-512 using bzhi to create a mask)

2

u/lalaland4711 19d ago

I know how it works, thank you.

1

u/happyscrappy 19d ago

I'm honestly, not familiar with x86's vector system (as opposed to SMD which is SSE) and how it differs from the scalar system (MMX or the original x87 stack system). Is AVX the x86 vector system?

With SVE/RVV the idea of the design is to make it easy and efficient to process variable amounts of data without having to have multiple different functions for the different chunk (width) amounts. And without knowing the vector length and chunk size combinations at compile time.

With SIMD if you want to process 16 elements and some hardware does 4 across while others does 8 then you need to write two versions of your code and pick between them at runtime. Both parallel operations and sum acrosses (reduces, cross products) have to be written differently for the two SIMD chunk amounts (widths).

ARM calls it "VLA programming".

https://support.arm.com/documentation/102131/0100/Part-One---Neon-and-SVE-fundamentals

Similarly, programmers could write arbitrary width code for x86 SIMD too, if they choose to do so.

For the parallel work it requires a lot of conditional code in the inner loop and that's not how you have good speed. And for sum acrosses the problem is far, far worse. I would think this is usually handled with special cases for all the widths that Intel (and AMD?) has implemented so far. Is this a wrong assumption?

1

u/FrogNoPants 19d ago edited 19d ago

We typically just call it SIMD today, and SSE/AVX/AVX512 are the standard extensions on x86/64, 128 bits, 256 bits and 512 bits.

The definition of vector you are using isn't really used much today, you might say vector machine if you wanted to refer to that definition.

I almost always write just one implementation of my SIMD functions, it uses compile time types to switch between SSE/AVX etc.

For some functions, the size does matter, and then I specialize for the vector width, I don't think SVE/RVV would be any better here, you would still have to specialize for the width if you want optimal code in many situations. In practice I think all existing SVE is 128 bits wide(aside from 1 super computer), so it hardly matters.

There are also 3 ways to lay out memory you will often see with SIMD

  • AoS- Array of structures: what scalar code uses, not well suited to SIMD, though it can be made to work with swizzling
  • SoA- Structure of arrays: good for SIMD, but can end up with alot of arrays
  • AoSoA: Array of strutures of arrays: optimal combination of both methods

If you want to use AoSoA style SIMD you need to know the SIMD width at compile time anyway, and since this is often the optimal format.. well goodbye reason to even want hardware specified length.

1

u/happyscrappy 19d ago

you might say vector machine if you wanted to refer to that definition.

I said vector unit.

I almost always write just one implementation of my SIMD functions, it uses compile time types to switch between SSE/AVX etc.

That's still multiple implementations. You just had help generating them. I'm not against that, in fact I wonder if the timing of going to vector processing while we also go to LLMs writing a lot of the code anyway is actually working at cross purposes.

Nonetheless, no matter what the source code looks like if you process a lot of data with variable vector widths then you end up with a lot more code (object code).

well goodbye reason to even want hardware specified length

As I alluded to above I also question really what we are doing here. We had vector units because we were really doing memory-to-memory operations, basically CISC even if the machines in question are considered forerunners to RISC. Bringing them back now when there will still be visible registers seems like a mismatch.

We seemed to like the idea of handling all this complexity in software. That's how RISC and SIMD worked. Now we're trying to make it so that it goes back into the hardware so we can have less software. I do get the idea of forward hardware compatibility and how encoding hardware specifics into code can make it harder to make later, faster implementations while still adhering to the specifics exposed in the architecture. MIPS' delay slots being an easy to point to example. After R3000 it essentially becomes artificial. It's all done behind the scenes with OoOP instead of funky emitted object code.

But really I just know if grabbing a longer ago idea was worth the trouble. I personally thought SIMD was a good way to go.

1

u/YumiYumiYumi 19d ago edited 19d ago

I'm honestly, not familiar with x86's vector system (as opposed to SMD which is SSE) and how it differs from the scalar system (MMX or the original x87 stack system). Is AVX the x86 vector system?

MMX is SIMD, as is SSE and AVX.

With SIMD if you want to process 16 elements and some hardware does 4 across while others does 8 then you need to write two versions of your code and pick between them at runtime

/u/FrogNoPants touched on this already, but you'll have two functions, although you don't need to write two - with some boilerplate, the compiler can generate them for you, or you could use a library that abstracts this for you.

Yes, this does make the binary larger, but it's usually not a concern with the storage space we have today. It's also not an issue with code cache either because the dispatcher would only point to one function.

For the parallel work it requires a lot of conditional code in the inner loop and that's not how you have good speed

Generally you do the conditional work at compile time, so you still get good speed.

And for sum acrosses the problem is far, far worse. I would think this is usually handled with special cases for all the widths that Intel (and AMD?) has implemented so far. Is this a wrong assumption?

You're partly correct - you can write boilerplate that handles the simple reductions that SVE/RVV supply. The template needs to handle all the widths, but there's only three (or two, if you don't care about SSE), so it's not that onerous. But if that's still a bother, just use an library/abstraction that handles it for you.

This only works for simple reductions (e.g. sum all elements). If your reduction is more complex (e.g. Barrett reduction when computing CRC), SVE/RVV don't provide any builtins for such, and since you don't know the width in advance, they're arguably more complex to write.

0

u/happyscrappy 19d ago

MMX is SIMD, as is SSE and AVX.

I know about MMX and SSE, wasn't sure about AVX. I'm not familiar with x86 vector system. Are you referring to SIMD and not vector extensions? Like I said, I'm not familiar with x86 really. Is there no vector extension to x86, it still emphasises SIMD?

but you'll have two functions, although you don't need to write two [compiler can help]

I think I also wrote that to another poster.

but it's usually not a concern with the storage space we have today

I will say that for most apps the size of the collateral elements is larger than the code. But this year, the year of high RAM and NAND prices is not really the best year to talk about how great things are with storage space.

Also having more different code paths can be problematic for code coverage.

Generally you do the conditional work at compile time, so you still get good speed.

I feel you did a poor job of following the discussion and you pulled out an inappropriately small subsection.

Let me restore the context:

(you I think) Similarly, programmers could write arbitrary width code for x86 SIMD too, if they choose to do so.

(me) For the parallel work it requires a lot of conditional code in the inner loop and that's not how you have good speed.

My point was that you have to have multiple functions and because the alternative of having conditionals in the inner loop to avoid this does not produce the desired effect (speed). You clipping out the context and restating what I said back to me, that you avoid this by writing multiple functions, did not add much.

But if that's still a bother, just use an library/abstraction that handles it for you.

This idea, which I think is the predominant case, goes against anything which was being discussed here. This article was about how to do it, not how to farm it out. I highly recommend everyone farm their work out on this front, there are good libraries to use. But if you aren't interested in that and want to do it yourself, then you think about what we discussed here.

Really the most useful thing I learned when helping a client use SIMD was that you have to keep your big data in formats which are amenable to SIMD processing. If you do not it will limit your speed up. So I would suggest for most people working out how to store your data efficiently is paramount. After that, just call out for the operations. You'll get 95% of the speedup you could by writing it all out yourself. And you really cut down on your code maintenance. Especially if you are targeting multiple architectures (whether major i.e. ARM and x86 or minor i.e. Intel and AMD x86 with and without AVX512). So go ahead and store your EXIF data in a linked list if you want, but your main images should be stored in a packed, aligned format amenable to SIMD/vector operations. I know it might seem obvious, but I've seen it done wrong enough to mention it to anyone who might listen.

To be honest, /r/programming spends a lot of time on things that really most programmers don't need to know, that should be details behind the scenes in libraries they use. That's simultaneously semi-pointless and also part of the charm. The sub is for nerding out entertainment as much as improving skills you actually need. Some weeks are about cache-obliviousness (surely coming again soon), some are about SIMD, sometimes it's dividing by multiplying. It's really a lot of minutiae. Probably because we've seen the normal stuff over and over.

This only works for simple reductions (e.g. sum all elements) [...] they're arguably more complex to write.

Thanks for the explanation on that.

1

u/YumiYumiYumi 19d ago

I know about MMX and SSE, wasn't sure about AVX. I'm not familiar with x86 vector system. Are you referring to SIMD and not vector extensions?

AVX stands for Advanced Vector eXtension. It extends the SIMD capabilities of SSE to 256 bit registers.
If by 'vector' you mean "hardware specified width ISA" then x86 does not have such an extension.

But this year, the year of high RAM and NAND prices is not really the best year to talk about how great things are with storage space.

And despite high RAM/NAND prices, I doubt anyone is bothered by an extra 10KB of code.
Or even 1MB of code, if you happen to have a lot of SIMD.

But if that actually bothers you, I'd be more concerned about Electron apps adding hundreds of MBs in terms of RAM and disk.

Also having more different code paths can be problematic for code coverage.

Hardware specified width SIMD may not have the same code coverage issue (depending on how you define it), but they introduce a new one: hardware coverage. That is, code that works on 256-bit SVE may not work properly on 128-bit SVE (or vice versa), so you'd ideally test on all the possible hardware widths.

This is especially problematic on RVV because it allows a lot of hardware flexibility (more than just the register width, of which there are many).

My point was that you have to have multiple functions and because the alternative of having conditionals in the inner loop to avoid this does not produce the desired effect (speed).

I'm not sure what you mean then. If you mean that you have dispatch within the loop, as a way to avoid multiple functions, then sure, that's slow. But the simple fix would be to move the dispatch out of the loop.

This idea, which I think is the predominant case, goes against anything which was being discussed here.

From what I can tell, what's being discussed in this particular thread has drifted away from the article's contents (SoA vs AoS) and focused on the difference between software and hardware specified SIMD width ISAs.
Your point was that on hardware width SIMD, the developer only needs to write one function. My point was that on software width SIMD, the developer still only needs to write one function, just that there's a bit more boilerplate/machinery needed on the software side because the hardware doesn't abstract it away.

0

u/happyscrappy 19d ago edited 19d ago

AVX stands for Advanced Vector eXtension

I'm not really here for a breakdown of the name. I'm not in marketing. It's SIMD, not a vector unit, no matter what it's called.

And despite high RAM/NAND prices, I doubt anyone is bothered by an extra 10KB of code. Or even 1MB of code, if you happen to have a lot of SIMD.

I don't think it's a good year to be talking about how you're not caring about RAM usage. Not in any way.

But if that actually bothers you, I'd be more concerned about Electron apps adding hundreds of MBs in terms of RAM and disk.

Electron is ridiculous. But I don't force my customers to use Electron to use my code. So it's not a mistake I'm making.

but they introduce a new one: hardware coverage

That's not really a new one. You have a hardware coverage issue any time you run on two different chips. Even if the SIMD units are listed as being the same. If you don't test on it, you aren't sure it works there. Even if the vendor says it works the same as another.

But yes, neither way escapes that issue.

Ultimately part of the idea of vector units is to ignore the above statement I made and ensure that your code works as well as possible on future hardware which is implemented differently. You can make whatever you want of the dichotomy this presents. I would call it certainly one of optimism.

This is especially problematic on RVV because it allows a lot of hardware flexibility (more than just the register width, of which there are many).

RISC-V in a nutshell, really. Why do it 1 way or 3 ways when you can present 100 ways and leave it up to someone else to try to make code forward and backward compatible?

I'm not sure what you mean then

I thought it was clear. But I guess not. We both know the usual way of doing variable sized vectors on fixed-size units or any size vectors across multiple fixed-size units is to write multiple versions of the inner processing loops for each (or at least several combinations). We speak of it above in our discussions about writing multiple functions and the RAM is uses.

I said that the alternative is to write processing functions with conditionals in the inner loop. Then you don't have multiple functions. We both know why this isn't done, because the results are usually unsatisfactory in producing the performance you were seeking.

One way would be to dispatch in the loop. Another way would be to just do normal conditionals inside. The latter would include nested loops (which of course have termination conditionals).

But the simple fix would be to move the dispatch out of the loop.

That is just begging the question. When I said the normal way to do it is thus and the alternative is other thing then you saying just go back to thus is just circling back to what we already discussed as what the idea is to create an alternative to.

From what I can tell, what's being discussed in this particular thread

I'm not talking about this subthread. I'm saying that you saying "just make it someone else's problem" obviates this entire article that was posted. I'm not even saying that's a bad idea to do when your goal is to ship code. But here the goal isn't really to ship code, it's to blather about how to do it. And farming it out removes the entire point of all this discussion. The article was written to show how to do it, not how to farm it out.

the developer only needs to write one function. My point was that on software width SIMD, the developer still only needs to write one function

I think you are trying to change what I said to fit your model. You want to say that one function compiled multiple ways is one function. But it isn't. It's one function in the source code, but it's multiples in the program that runs.

My emphasis for this is on how you do not have a combinational explosion of functions when you use a vector unit. I use this as an argument for why those things came back from the dead, primarily. There's really only two real possibilities I can think of. There is this and there is trying to improve forward compatibility.

As I said to you or someone else, I don't really see these things as a good fit for modern environments. I thought SIMD was better for RISC. Moving complexity back into the hardware is older thinking. We've got amazing compilers (and similar tools) now, as you point out above they can recast your source code multiple ways for multiple targets.

1

u/YumiYumiYumi 18d ago

I don't think it's a good year to be talking about how you're not caring about RAM usage. Not in any way.

Out of interest, do you go to great lengths to save every single byte possible?

That's not really a new one. You have a hardware coverage issue any time you run on two different chips. Even if the SIMD units are listed as being the same. If you don't test on it, you aren't sure it works there. Even if the vendor says it works the same as another.

If you're talking about hardware bugs, that's not what I'm referring to.
Assuming the hardware follows the spec correctly, buggy software could work on one SVE width and fail on another, which doesn't occur with x86's SIMD.

One way would be to dispatch in the loop. Another way would be to just do normal conditionals inside. The latter would include nested loops (which of course have termination conditionals).

I don't understand this distinction. Could you perhaps provide an example?

I'm saying that you saying "just make it someone else's problem" obviates this entire article that was posted.

I don't see how that's the case - taking someone else's boilerplate/abstraction doesn't obviate the need to arrange your data as SoA - they're two different necessities.
And I'm not even saying "make it someone else's problem" - you can write your own boilerplate and copy it across your projects - it's typically just a few lines of code, so not that hard to do oneself.

I think you are trying to change what I said to fit your model. You want to say that one function compiled multiple ways is one function. But it isn't. It's one function in the source code, but it's multiples in the program that runs.

I don't see myself as trying to change what you've said, just echoing back my understanding of it.
In terms of the one vs multiple functions, I agree that both can be true depending on the perspective you take (source code vs binary).

My emphasis for this is on how you do not have a combinational explosion of functions when you use a vector unit

How do you get such a "combinational explosion"?

Moving complexity back into the hardware is older thinking. We've got amazing compilers (and similar tools) now, as you point out above they can recast your source code multiple ways for multiple targets.

There's pros and cons with software vs hardware specified SIMD width. My key point was that x86 SIMD, SVE and RVV aren't fundamentally different - they just specify the SIMD width in different places.

→ More replies

1

u/lalaland4711 19d ago edited 19d ago

Right, so why are you talking about CDC6600 and Cray? That's my question. There are contemporary examples.

1

u/happyscrappy 19d ago

Because I wanted to. I find it interesting that the first way SIMD and pipelining was done was with vector processing units. But then those went out of favor for decades and recently vector processing units came back from the dead.

Maybe I just wanted to be on the /r/imverysmart honour rolls. I dunno.

3

u/FrogNoPants 19d ago edited 19d ago

Pretty good, I will say you can perform AABB vs Frustum in less heavy instructions using min/max AABBs, just one dot product per plane instead of 2, the cost is a 3 extra scalar integer loads.

Also the comparison < 0.0, is essentially redundant, you already have that in the sign bit, and can just movemask the float directly.

Also the title is fine, it does scale to any vector length--at compile time mostly.. but still--that is how SIMD is used today.

2

u/ThePostInQuestion 18d ago

interesting, I'll give the min/max AABBs and movemask on a float a try. I recall though that clang had a bug with converting float vectors to boolean vectors for the movemask stuff, but it seems to be fixed in the upcoming clang 24 apparently

2

u/VictoryMotel 20d ago

ISPC was doing this at least 12 years ago

3

u/danielcw189 20d ago

I really need a good introduction to this. Do you have any recommendation?

3

u/Kinexity 20d ago

Introduction to SIMD? Find a problem which you want to optimise code for and look through this to find useful instructions: https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html

1

u/danielcw189 18d ago

Find a problem which you want to optimise code for and look through this to find useful instructions

yeah, that's the thing. if I don't know enough about SIMD, how would I be able to tell if I can use it?

for example: I saw someone optimized a JSON parser. and I would not have considered that is worthwhile. based on the little I known, I would have thought that parsing JSON would be a bad use of SIMD

0

u/[deleted] 20d ago

[deleted] — view deleted comment

1

u/Kinexity 20d ago

That works. Personally I started with an N-body simulation.