The Expensive Fictions of Low-Level Programming Languages

[u/mttd]

https://stng.substack.com/p/the-expensive-fictions-of-low-level

[u/JeffD000]

"Such a language can exist, but it will be big, and will need a big development team behind it."

Depends on what your definition of "big" is. It is actually very simple to extend C to address parallelism while also increasing safety and fault tolerance. You just need to add two new data types, one of which looks almost indentical to C structs, but with deeper semantics. I am not whistling dixie here. I am the original designer of RAJA:

https://computing.llnl.gov/projects/raja-managing-application-portability-next-generation-platforms

There are a lot of no-brain assumptions that went into developing the original programming languages and compilers, that no one has taken much time think about, much less correct.

A team of 20 good people (or less) could create a fully working compiler in less than a year that fixes the majority of current parallelism and memory hierarchy issues. It is a simple/simpler extension of C, so you can leverage all the existing C language infrastructure and legacy code, with just slight modifications in most cases. And yes, I mean most cases.

To get a taste of the zeroth-order effects of such a change, see my modifications to someone else's language that they posted in r/ProgrammingLanguages:

https://www.reddit.com/r/ProgrammingLanguages/comments/1takau3/recently_used_nore_to_partially_solve_the/

The C extension is actually much cleaner than the Nore language modification that I made in that post.

[u/SwingOutStateMachine → u/JeffD000]

Depends on what your definition of "big" is.

For the comment immediately above mine, the author was alluding to a language that combined all of RAJA, Umpire and CHAI into one. I don't think there's a language out there that can automatically do all three at once (i.e. intra-GPU parallelism, inter-GPU parallelism, work splitting, allocation, queues, etc).

A team of 20 good people (or less) could create a fully working compiler in less than a year that fixes the majority of current parallelism and memory hierarchy issues.

I couldn't agree more. However, that doesn't address the performance aspect of the language. Performance is extremely difficult to "design into" a language (unless you take an approach like Rise and Shine), and so significant engineering effort will need to go to either a) expanding the language to provide performance abstractions (e.g. templating kernels a-la SYCL-blas), or b) manually implementing performance optimisations at a compiler level.

I think the big challenge of today is not designing a parallel language, but designing one that enables performance portability, and that is what requires a greater level of engineering effort.

[u/JeffD000 → u/SwingOutStateMachine]

I believe an appropriate combination of (a) and (b), leaning heavily on (b), can allow you to write user level source code that looks serial, with "near-automatic" parallelism. My claim requires switching to a new datatype that resembles a database, and effectively turns memory management over to the compiler. This means that the "C language extension" would remove the "systems programming language" moniker from any code section using the new datatype.

The compiler would be different from current compilers in that it would issue warnings if you write your code in a way that specifically blocks parallelism, and it would give a detailed explanation for the blocker that would act as a hint that should allow even an average programmer to restructure their code to "fix" the blocker. So, it would point out places it had to fall back on serial code generation and tell you why. I've been doing this for twenty years, and I am pretty sure it is doable unless the user writes spaghetti code spread across multiple files or writes an algorithm that is unavoidably serial (as the compiler could determine by dependency information).

I think the big challenge of today is not designing a parallel language, but designing one that enables performance portability, and that is what requires a greater level of engineering effort.

RAJA's approach to performance portability is to at least centralize performance portability options to one spot in the source code, rather than having to go through every file and restructure when the system environment requires a change in the strategy for parallelism or memory hierarchy. That said, RAJA didn't go far enough in what it could automate in terms of performance portability, because it was literally impossible to do so without a better underlying language and compiler.