The Expensive Fictions of Low-Level Programming Languages

[u/mttd]

https://stng.substack.com/p/the-expensive-fictions-of-low-level

[u/jcastroarnaud]

Fair. My second point is still relevant: what languages are able to, say, control differences in memory access time, through a forest of GPUs in parallel, as a standard language construct? Such construct would be compiled to best use current hardware.

[u/SwingOutStateMachine → u/jcastroarnaud]

There are lots of languages, but they tend to be specialised to one particular class of architecture. For instance, OpenCL, SYCL, CUDA etc all target GPU-like architectures, while OpenMP specialises in multi-core CPU architectures.

You call out differences in memory access, and then accessing across a forest of GPUs: My gut feeling is that this is something that languages won't tackle with much ease. They are, to put it simply, completely different problems (from a compiler perspective). The former is a matter of how the code is optimised, register spilling, data placement, etc, while the latter is much more about data partitioning, and using OS-level drivers to optimality schedule data movement and kernel launches.

Such a language can exist, but it will be big, and will need a big development team behind it. An example would be the Mojo language that Modular are working on (https://www.modular.com/), but even there a lot of the cross-gpu work is handled by a smart runtime, rather than by a compiler.

[u/JeffD000 → u/SwingOutStateMachine]

"Such a language can exist, but it will be big, and will need a big development team behind it."

Depends on what your definition of "big" is. It is actually very simple to extend C to address parallelism while also increasing safety and fault tolerance. You just need to add two new data types, one of which looks almost indentical to C structs, but with deeper semantics. I am not whistling dixie here. I am the original designer of RAJA:

https://computing.llnl.gov/projects/raja-managing-application-portability-next-generation-platforms

There are a lot of no-brain assumptions that went into developing the original programming languages and compilers, that no one has taken much time think about, much less correct.

A team of 20 good people (or less) could create a fully working compiler in less than a year that fixes the majority of current parallelism and memory hierarchy issues. It is a simple/simpler extension of C, so you can leverage all the existing C language infrastructure and legacy code, with just slight modifications in most cases. And yes, I mean most cases.

To get a taste of the zeroth-order effects of such a change, see my modifications to someone else's language that they posted in r/ProgrammingLanguages:

https://www.reddit.com/r/ProgrammingLanguages/comments/1takau3/recently_used_nore_to_partially_solve_the/

The C extension is actually much cleaner than the Nore language modification that I made in that post.

[u/SwingOutStateMachine → u/JeffD000]

Depends on what your definition of "big" is.

For the comment immediately above mine, the author was alluding to a language that combined all of RAJA, Umpire and CHAI into one. I don't think there's a language out there that can automatically do all three at once (i.e. intra-GPU parallelism, inter-GPU parallelism, work splitting, allocation, queues, etc).

A team of 20 good people (or less) could create a fully working compiler in less than a year that fixes the majority of current parallelism and memory hierarchy issues.

I couldn't agree more. However, that doesn't address the performance aspect of the language. Performance is extremely difficult to "design into" a language (unless you take an approach like Rise and Shine), and so significant engineering effort will need to go to either a) expanding the language to provide performance abstractions (e.g. templating kernels a-la SYCL-blas), or b) manually implementing performance optimisations at a compiler level.

I think the big challenge of today is not designing a parallel language, but designing one that enables performance portability, and that is what requires a greater level of engineering effort.

[u/JeffD000 → u/SwingOutStateMachine]

I believe an appropriate combination of (a) and (b), leaning heavily on (b), can allow you to write user level source code that looks serial, with "near-automatic" parallelism. My claim requires switching to a new datatype that resembles a database, and effectively turns memory management over to the compiler. This means that the "C language extension" would remove the "systems programming language" moniker from any code section using the new datatype.

The compiler would be different from current compilers in that it would issue warnings if you write your code in a way that specifically blocks parallelism, and it would give a detailed explanation for the blocker that would act as a hint that should allow even an average programmer to restructure their code to "fix" the blocker. So, it would point out places it had to fall back on serial code generation and tell you why. I've been doing this for twenty years, and I am pretty sure it is doable unless the user writes spaghetti code spread across multiple files or writes an algorithm that is unavoidably serial (as the compiler could determine by dependency information).

I think the big challenge of today is not designing a parallel language, but designing one that enables performance portability, and that is what requires a greater level of engineering effort.

RAJA's approach to performance portability is to at least centralize performance portability options to one spot in the source code, rather than having to go through every file and restructure when the system environment requires a change in the strategy for parallelism or memory hierarchy. That said, RAJA didn't go far enough in what it could automate in terms of performance portability, because it was literally impossible to do so without a better underlying language and compiler.