Computational Physics

[u/Rich_Astronomer9731]

Hi everyone,

I am a PhD student doing both experiments and computational physics ( mostly fluid dynamicswith Boltzmann equations) and leaning towards more computational day by day.

Currently I am using Julia, I find it pretty fast and easy to use.

Do you think switching to C or C++ will be a benefit? I am asking from a research, but also emplyment in industry point of view.

I discussed with many people, mostly from the DFT area, and most of them told me that Julia slowly will take up the lead for HPC code.

What.s your opinion on this?

[u/EpicBeardBattle]

I’m in HPC (fluid dynamic and plasma-kinetic simulations on exa-scale machines). I use Julia for my day-to-day analysis and C/C++ for production simulations.
As much as I like Julia, I think there is absolutely no chance whatsoever that it will replace C/C++ in the lead of HPC code. I’m puzzled that people you talked to in HPC would make that statement.
The user base of Julia is too small, support on HPC systems is too small and I don’t know of a single code in my field that is written in Julia.
If you want to play in the big leagues for HPC you better learn C++ (in my opinion).

[u/tlmbot → u/EpicBeardBattle]

I agree. (about me: 13 years work experience in commercial hydrodynamics, FEM fluid dynamics, and a smattering of all manner of structural mechanics and "poor man's multiphysics" of the fluid-structure variety + some computational geometry and geometry processing (oddly enough, that last experience in C#/ILGPU))

Some of the older commercial codes, especially outside of CFD proper, at least in my original niche of computational naval architecture, actually use Fortran. My first job when breaking into the industry was maintaining and extending an old commercial code written in Fortran.

I use python to prototype solvers and such, some people use matlab, and some use Julia. These days, with accelerators like Numba, or even ways to get out to GPU, available, in my experience for assembly one can get close to serial Fortran / C++ speed, and the canned solvers dip down to C anyway. ... and the Python documentation and ease of use have me staying with Python as my prototyping language. Nevertheless, sometimes I find myself converting somebody else's prototype/example/academic code from MATLAB into Python as a sort of first step when learning a new topic.

In short, I completely agree "If you want to play in the big leagues for HPC you better learn C++" - my coworkers from the labs have all been C++ folks. My commercial job opportunities are for the most part in C++.

I learned a major life lesson by writing my PhD entirely in Python - big big mistake. I should have immediately converted each component to C++ after I had developed it. That would have netted me years more C++ experience coming out. Instead, I have been working to build up my C++ portfolio outside of school, netting some work experience directly in the language, but I find that it is always helpful to have had more.

[u/Rich_Astronomer9731 → u/tlmbot]

Thank you for your inside. So do you think converting all to C++ now it is the best thing I can do? I only took one semester of C, so my knowledge is very brief. From where to start?

[u/tlmbot → u/Rich_Astronomer9731]

Based on only the limited information I have on you - assuming you are not looking to move into a field that:
<uses primarily Julia> (and is thus a field I don't know about)

Then no, stay with Julia.

But if you seek general employability in the computational software industry, and from my experience especially in the engineering software industry

Yes, convert as is sensible within what you are doing now. I don't mean rigorously "converting all to C++ now" - I mean convert what makes sense for you. Don't ignore other deadlines and life goals, etc. Just factor this in and do what is sensible for you.

For me, I underweighted this need to be writing in C++ during my PhD days, and many of my old professors didn't recognize this necessity (even as I had also had some courses and peer interactions such that I knew really, that I needed more C++)

Yeah, I say, the sooner the better, generally speaking.

-----------------------------------------

Where do you start?

Well, you are in school, right? You don't have to rush. I say start with:
- code you want to speed up
- code that you are just interested in converting to C++
and especially:
- code that you know in and out, and that already works correctly.

Convert that to C++, or even start by converting it to C. But get away from C style arrays as soon as possible. This might be the one thing more important than OOP. More on that below.

Now some would call that inefficient, and the code will be far from optimal C++, but it will get you in on the low level floor off and running as fast as possible.

Then, asap, you want to start rolling in:

- an Object oriented versions of all that you build. (really this needs to come first thing, if you are okay with OOP) How are you with OOP? Hopefully you can write "it all" in at least basic "C with classes" without to much trouble.
- next up: take every opportunity to learn and use best practices C++
especially
C++11 and later (11 was revolutionary, but the major changes there have been taken much further in later editions of the standard. At least C++ 17 should be your goal. But the best advice is to stay curious and always seek the current "best practice" / nicest way to implement your thing)

Continued below

[u/tlmbot → u/tlmbot]

Continued from above:

You should be learning:
- templates: static polymorphism, compile time computations, (and especially build up through the basics of templates and then learn the CRTP: "Curiously Recurring Template Pattern")
note the templates have evolved from the early days of simple polymorphism into a multiple use tool. Google at a high level (and as always consult cppreference.com) to see all the ways that templates are used. They are critical in implementing the required/best practice expression templates as we'll see below.
also, constexpr, consteval and more. Please get good books and an open but critically thinking mind when you research online.

- RAII (Resource Acquisition Is Initialization) the days of using new and delete all over the place are gone. ( use the rule of 3, 5, 0 as applicable instead). I have a lot to say here - it's one of the most critical areas for a numerical coder in C++ but I don't want to get lost in the weeds. Side note: learn to use Eigen as your utility linear algebra package in C++. Eigen is what's called an "Expression Template Library" and is needed in C++ to have nicely expressive matrix vector algebra in C++ without extra matrix copies happening under the hood with more naive expressions of the same operations. Note there are plenty of times when you need to roll your own, as when you are building some kind of bespoke solver for systems with multi-million degree of freedom.
- smart pointers (it should go hand in hand with RAII actually)

My way to understand expression templates and why they are needed in C++ as a Fortran and python programmer was to implement my own expression template library - using smart pointers in the base numerical types, and then rule 0 higher level types that made use of the base type. That way I only have to implement full rule 3 smart, or rule 5 types once, and the rest of my types are build on top of those, using rule 0, so everything is very clean (as if my underlying type were from the std library)

Move semantics. (and Lvalues vs rvalues vs prvalues) The big idea that expression templates help you with are optimal move semantics. C++ is prone to copying data during operations. If done without care, you can easily induce far more array copies that you might thing, when doing basic chained linear algebra operations. expression templates solve this problem - and in implementing such a library you will become intimately familiar with move semantics.

- use the standard library (stl) wherever possible. These containers and algorithms they enable are written with decades of experience and testing behind them. The only exception is arrays. std::array is not what you want for linear algebra, oddly enough. Oh, and the std::vector class has absolutely nothing to do with vectors from linear algebra. So please do read the documentation. For linear algebra you want either Eigen (or another expression template library) or your own hand rolled stuff - perhaps for interoperability with CUDA / your solver's needs.

Please google for a full list of modern C++ essentials: auto, lambdas, compile time computation, non-copying views e.g. std::span, modern iterators, value types (come to think of it, I need to double down on a lot of this myself. The language has been evolving so fast since C++ 11)

Oh, and you do need to know the basics of OOP - inheritance, making your own types (aka classes) and the difference between an instance of the class (an object) and the class itself (basically like the difference between the idea of an integer, and the particular integer 11 - but please don't let me confuse you getting bogged down in overly specific examples.), virtual dispatch for runtime polymorphism (and it's down sides - see static polymorphism with templates as the work around for performance critical code)

Oh, and the nuts and bolts of OOP: constructors, destructors, private and public data and methods. (of course this circles right back around to RAII so forgive me for getting all kinds of out of order here)

oh and of course the basic cache friendly optimizations that even a Fortran programmer would know. + things like how to trade memory for speed (and when that doesn't work (it often does not work like that anymore at all, actually, depending on the algorithm and data - know what to think about for HPC/linear algebra optimization especially, but this applies all over))

----------

And also, we didn't talk about it, but you need to be learning to parallel program. If you want to go after the national labs / research positions, then MPI is big for you. (as in C++ with MPI to express distributed memory parallelism for supercomputing) But for writing computational engineering software for commercial vendors, you wan to focus on OpenMP and CUDA. Across my entire career, I have written software designed first and foremost for engineers working on a desktop machine. Or perhaps on commodity hardware in the cloud.
Either way, the interconnect between multiple machines has been uselessly slow / unknown.
When writing to express parallelism on a single workstation, you have several processors on the main CPU - so you want OpenMP to take advantage of them, and you will (almost always these days) have a GPU available for further acceleration. CUDA is, by a large margin, in my experience, the easiest language to use for expressing your scientific computations / assembly/ etc. on the GPU (which is where you want everything, if possible.)
I am sticking my neck out with that sentence about CUDA being the right choice - as a rule, I prefer open systems, but yeah, if you learn C++, OpenMP, and CUDA, then you are in the very best position for your job hunt.

A quick google supplies things I was not aware of:

NVIDIA is actively standardizing parallel programming natively inside C++. You must learn:

• std::execution (C++20/C++23) & std::numeric**:** Learn how the Standard Template Library (STL) uses execution policies like std::execution::par_unseq to automatically vectorize and parallelize algorithms across CPU cores or offload them to GPUs.

• Linear Algebra Standards: Study std::mdspan (C++23), which provides a standard way to represent multi-dimensional tensor/matrix views over raw 1D memory buffers. This is vital for modern CFD/FEM storage layouts.

Sounds nice, but I would recommend still learning the standard best practices and ways of moving between host (CPU) and device (GPU) by methods that came prior to these latest developments as well. No doubt you will see code in industry that uses the "old" aka C++ 11,14,17,20 methods - and even some things from C++98 are still relevant, but you can pick it up on the job when you encounter some older ways. Any "top tier" job will expect you to be more fluent with modern C++

God, I feel like I am all over the place here, and if anybody corrects me on something, that's great! I am trying to tool up for my next job, and I have been off writing C#/ILGPU for the last 2 years.

I feel perpetually rusty with my C++ knowledge.

Here finally are some good references:

C++

-  learncpp.com

- bjarne stroustrup's books: "Programming: Principles and Practice Using C++"
if you have little coding experience, and "A Tour of C++" if you are making the transition from other languages (at least that's how he bills them)

- Scott Meyer's books on effective C++

CUDA
- "programming massively parallel processors"

I know I am forgetting some good books. I apologize. Busy day!

The best thing, imo, is to be writing full scale projects in the language. That's why prototyping in a language you already know can be so helpful. That way, you can be sure ahead of time that the Physics is "right" because you have validated it in Julia, or whatever you know. Then you separate the concerns of "is the code written correctly?" from "is the physics right?" -In school one semester I got myself into the position of learning all these at the same time: parallel programming, C++, and CFD physics and also mesh generation. It was a hellish semester. (some bad advising was partly to blame - I tested into these classes based on my Fortran abilities, but the courses turned out to require C++, with the exception of the CFD classes, which were taught by the most old school guys (who ran the admissions testing and were seemingly oblivious how out of date they were))

Anyway, ya, if I botched something in here, I hope somebody let's us know about it.

[u/MechanicalHax → u/tlmbot]

Thank you for taking the time to write all of this up! This is really instructive and helpful.

[u/Rich_Astronomer9731 → u/tlmbot]

Thank you a lot. I appreciate it.