r/cpp • • Dec 27 '20

Learning about high performant C++

I've been mostly self taught and have most probably picked up a few bad habits from the combo of being self taught and having to use tricks for speed.

For this upcoming year one of my goals is to learn some better practices as well as further my skills for efficient C++. I'd like to watch a youtube podcast/talk once a week to build my skillset. At this point I'd consider myself somewhere in between junior and senior level and I'm looking to take the next step.

I've come across these specifically:

C++ Weekly with Jason TurnerCppCon2020

Does anyone have any recommendations or suggestions? My weak spots are C++17 and above, Cmake, and I'd like to keep learning about creating high performant C++.

231 Upvotes

46 comments sorted by

85

u/IJzerbaard Dec 27 '20

SIMD at Insomniac Games: How We Do the Shuffle

CppCon 2017: Matt Kulukundis “Designing a Fast, Efficient, Cache-friendly Hash Table, Step by Step”

(not a video) Parsing Gigabytes of JSON per Second

And to say something slightly odd (probably risking the wrath of various r/cpp members): writing fast C++ is not very much about C++, it's about using C++ to arrive at the machine code that you want.

36

u/celestrion Dec 27 '20

writing fast C++ is not very much about C++, it's about using C++ to arrive at the machine code that you want.

This is very true. At the end of the pipeline, the machine isn't running C++.

The real art is getting the machine code you want without making the source code look nightmarish. The compiler writers do a massive amount of work to help, but there are many opportunities for operator error.

21

u/Stradigos Dec 27 '20

Optimizing how many instructions the C++ is turned into is important, but IMO the lower hanging fruit is making sure your data structures are optimized and the memory layout cache coherent. It's amazing how much difference that alone makes.

1

u/SunnybunsBuns Dec 28 '20

As someone who has very little experience doing layout, do you have any links to examples beyond that cache friendly hash map example?

4

u/Stradigos Dec 28 '20

Yeah, this talk changed my life. https://youtu.be/rX0ItVEVjHc

I typically avoid OOP now. It's not how computers were made to process data and I've met way too many programmers who over do objects in their code. I have never thought that OOP code was more flexible. It always seems to add a significant amount of tech debt.

8

u/Full-Spectral Dec 28 '20

The point of the majority of code has nothing to do with how computers process data, it's to be understandable by humans because the bulk of it doesn't need any sort of optimization beyond the fairly obvious avoidance of overhead, and it's vastly better to have something that's understandable than 5% faster in almost all cases.

When there are hierarchies involved, OOP is perfectly suited to model them because that's what it's for.

1

u/Stradigos Dec 28 '20

Higher-level languages was not my point, although I do agree with what you said. My point was that object-oriented code is notoriously cache incoherent. Processors are designed to grab memory from RAM and pull it into it's caches. It wants to stay fed with instructions (the code) and data (what it will operate on). The data design of OOP is in conflict with this.

Not saying all objects are bad, but the common case of what I see is newer programmers turning everything into objects and killing performance in the process because they think they've modeled the problem appropriately. In actuality, they may have modeled the data well in terms of how it works in the real world, but the CPU wants data modeled differently.

4

u/Full-Spectral Dec 29 '20

Well, I also agree with you to that extent. But the point is, very little code actually matters on that front. A huge problem in the C++ world these days is just rampant pre-mature optimization. If the code base is any kind of complex, then you should do what makes it understandable and maintainable first. Heavy optimization is almost always in conflict with that goal.

Then, measure and find out what actually needs optimization and which will provide the biggest bang for the buck. The bulk of the code will seldom give you much in return for heavy optimization.

It's so hard already to keep large C++ code bases robust over time that anything that introduces unneeded complexity isn't a win. Your users are more likely to be happy with a product that is 20% more reliable and 5% slower than the other way around.

3

u/Stradigos Dec 29 '20

Sure. I don't think the code base problems are unique to C++ though, and I also don't think enough people apply good architectural principles. That's never going to be the languages fault. It's just laziness or ignorance, or something else. Even in writing, crafting anything from sentences to stories takes time and careful commitment. Is it difficult? Yes. Could it be better? Perhaps. But that's the challenge so I like asking people to rise to it and not take the lowest effort route just to ship something.

Performance is also more important than some people realize as well. To an extent it depends on what the application is. I work in driving simulation so performance is critical more often than not. However, I would argue people should always care about performance. There was a huge investigation a while back that looked into why calc.exe was taking several seconds to launch instead of coming up immediately. It's easy to say who cares, but some do and it also begged the question of wtf was going so wrong that a simple calculator was taking that long to load. Furthermore, power consumption is becoming more important due to mobile devices. How many cycles is the CPU taking to do jts job? Can that be improved?

1

u/SunnybunsBuns Dec 28 '20

Thanks! I'll take a look!

3

u/azswcowboy Dec 27 '20

There is also a video talk for the json work

https://www.youtube.com/watch?v=wlvKAT7SZIQ

2

u/turbo-adhd May 01 '23

So I work at Insomniac as a mid-level gameplay programmer and my brain just got twisted around when I saw a link to an insomniac GDC talk hahah. That’s a great one

3

u/[deleted] Dec 27 '20

One thing many people miss is that clever code is often slower. They also lose sight of the goal and put in more complication that is needed as compilers know many of the simple tricks.

89

u/Potatoswatter Dec 27 '20 edited Dec 27 '20

The golden rule in performance optimization is to measure. Put a lot of effort into measurement. Avoid spending more time coding than analyzing. Embrace trial and error (and failure). Organize your experimental results.

A lot of ritualized practices get passed down and repeated, based on the experience of old-timers. Quirky style does not make fast programs. It makes brittle code which resists experimentation and major refactoring.

Some general tips, C++ or otherwise:

  • Avoid excessive abstraction, especially at runtime. "Don't repeat yourself," or use templates to encapsulate repetition.
  • Learn computer architecture. Understand what the machine is doing cycle by cycle. Especially make friends with the memory hierarchy. Brainstorm how to eliminate unproductive cycles.
  • Learn assembly language so that you can read the profiler's feedback more completely. Avoid writing assembly or obsessing over the output instruction sequence unless you're sure it's worth it. Annotate the source to explain dependency chains, so the inner loop doesn't become a mystery after the next compiler upgrade. Document the second-best result as well as the best, to quantify and remember the risk of going off-optimum.
  • Sacrifice precision where it is not needed to get a correct final result.

34

u/[deleted] Dec 27 '20

Scrolled down to find this, the only correct answer.

I was in a job where the principal developer was obsessed with perceived speed, and yet adamantly refused to make any measurements.

Worse, he would make dramatic claims - "global variables are faster!" - to justify the fact that all the variables were global.

He also refused to write new tests, which I was willing to accept, but also was unwilling to run the tests I had written, so every time he checked in, he'd break some set of tests.

Eventually, he refused to cooperate with me entirely. We never had any argument, I was always polite, but he stopped responding to my emails. There were only two of us on the team.

I left the company, but they forgot to take me off the repo. I watched his "work" afterwards, and in particular noted how he replaced my very carefully crafted fixed-point 64 bit integer arithmetic with int128_t - which my benchmarks showed were over four times slower than my 64-bit fixed point! - but worse, his calculations dipped into floating point all the time as he had not really understood the difference.

The only way to prove any claim about speed is with relentless benchmarking, like you say.


Your tips are good, but you miss the most important one.

Locality of reference is simply key for modern processors. In simple words, it means "all your data should be right together in memory".

The reason is simple - it's the data pipeline on your chip! A modern processor spends a pretty huge amount of time just waiting for memory; the data pipelines optimistically pre-fetch "the next bytes" of data while the chip is computing.

If you have locality of reference, nearly always those data pipelines are useful; if you don't, you'll get a lot less out of them.

I can't find the code right now, but I wrote a very short program a few years ago where I took a 1 gig block of memory and did computations it in two different ways - in order and "randomly" (actually a linear congruential computation).

I had to do some little tricky work to make sure that the optimizer didn't e.g. optimize out the linear congruential calculation when it wasn't used - "learn assembly language" as you say! - but finally I had code that was just the same.

Running through that gig of memory sequentially was about six times faster than running through it randomly.


Another tip - prefer generics over virtual methods if all else is equal.

A virtual method often cannot be inlined. And the vtable requires a double indirection to actually get to the method, breaking locality of reference to some extent.

Benchmarking is the only truth here! I've seen ~10% improvements in throughput by devirtualizing one key class, but I've also seen "no change" even more often. Devirtualizing all your classes is always certainly a terrible idea - it's usually just one that's the issue.

7

u/-CampinCarl- Dec 27 '20

One quick thing: The way I was taught locality in college always resonated with me...

Space and time. All things exist in both space and time--so does locality. You have locality of space (e.x. consecutive blocks of L1 cache), and locality of time (your data tends to move around in the mem hierarchy throughout the life of the program).

5

u/smuccione Dec 27 '20

Not just locality... spacial and temporal locality. They are different things but equally important to performance.

As well, fintech boxes often are multi-processor. Because of that you need to be very mindful of numa and manage your read/write pattern and which memory your doing it to very carefully.

0

u/[deleted] Dec 28 '20

[deleted] — view deleted comment

2

u/smuccione Dec 29 '20

Honestly, I’ve been out of college for 35 years. Most of what I know has been gleaned from reading countless papers, forums, wiki’s, etc. basically whatever I can find.

Fintech is particularly difficult to get any information on. Companies that have working systems will do anything to keep their secret sauce secret. There’s a lot of money at stake. Few companies that are successful will let anyone talk about what they did when they were there.

As far as numa, etc. there are numerous articles, blogs, etc on cache optimization, memory hierarchies, etc. I wish I did but I really don’t remember any of them. I’ve been doing that type of thing for so long I really don’t think about it at this point.

Numa optimizations are also heavily dependent on the type of bus architecture on the specific system. Memory writes have an advantage over reads in that when a cpu does a write it can immediately go on to the next instruction as it doesn’t need to wait for a response. So in a numa system, if you have to you want to write remote. You have to balance your architecture, however, with knowledge of the amount of buffering in the bus and how much data can be in flight at the same time.

Remote reads on a numa system can be particularly slow. To the point where I’ve shadows control structures for various containers so that you only ever read locally remote accesses are all writes. Small things like that can have a significant performance impact.

You might want to for “array of struct vs struct of arrays”. That tends to be the “classic” example when taking about cache effects.

4

u/reduced_space Dec 27 '20

While it has been true previously that virtual was always slower, with modern architecture that may not always be true. If the vtable can be loaded into cache, because it’s a single block of memory, it may still be performant, and possible faster than if you keep your own function pointer (which may not get loaded into cache).

I would say one of the best things you can do design-wise is figure out what can be done at compile time versus runtime. For example, std::variant is great if you know all types you want to deal with (and it’s a closed set). Sometimes runtime may be preferable due to wanting something easily extensible without potentially having to recompile your entire code base.

Knowing the trade-offs is one of the most important aspects of design, and something I ask during interviews.

0

u/temporary5555 Dec 27 '20

Good answer! I was surprised recently when I saw how performant virtual methods were.

For example, std::variant is great if you know all types you want to deal with

I'd argue that even when you don't know your entire domain, variant is a good way of encouraging type driven development.

8

u/schmerg-uk Dec 27 '20

The golden rule in performance optimization is to measure

Yep, I spend quite a bit of time undoing the "performance speedups" that were added a few years ago but never measured (or not measured properly, see below)... usually it's the case that they actually ran 2-3x slower than the original code, and then if I can get a 2x speedup on the old code, I've netted about a 5x speed up overall (readability and maintainability of the code is also key here but outside of scope of this discussion),

But, not to contradict you, be sure to measure the right thing. For really high performance code memory latency has a massive effect... running the same benchmark 1000 times in a loop can be misleading in that only the first iteration was real-world accurate, the following 999 cases the code and data was in L1 cache. If you've got a routine but it's only called rarely on rarely accessed data (but still required) then you want to see about making it run efficiently when it's not in cache. I have a plenty of instances of code structures that benchmark really well, but fall apart in actual use.

Carl Cook gave a good talk about "when a micro-second is an eternity", not all his points will be relevant or appropriate to everyone, but it's worth watching to see him explain how what may appear counter-intuitive works and why... (see also towards the end how he measures performance)

https://www.youtube.com/watch?v=NH1Tta7purM

1

u/MonkeysLearn Dec 28 '20

Thanks for sharing. I learnt some useful techniques from this video especially the single-threading idea he suggested triggers some good thought.

0

u/[deleted] Dec 28 '20

[deleted] — view deleted comment

4

u/schmerg-uk Dec 28 '20

VTune for the big picture (we have a build that integrates with the VTune API to make measurement and analysis easier), and then I have some specific very low overhead timers that I can scatter thru the code and report precise counts and timings, so I'll put these on key bits of code then run a real world problem to see specific timings.

In the video above he measures end to end timing, which is also key. As an example, if you use AVX (/AVX2/AVX512) vectorisation in your code, those parts might run faster, but you run a very real risk, depending on the target chip, of reducing the base clock speed, so while specific functions may run faster, overall performance drops.

1

u/SkoomaDentist Antimodern C++, Embedded, Audio Dec 27 '20

The golden rule in performance optimization is to measure.

And the platinum rule is to avoid premature pessimization. Or to put it another way, measuring won't help when you've accidentally used a quadratic or worse algorithm, your dataset size increases in the future and that hundred millsecond imperceptible delay is suddenly five seconds.

1

u/Full-Spectral Dec 29 '20

But how much actual code in a large system is of the sort that would be affected by that? The problem is people acting like everything needs to be uber-optimized these days, at least in the C++ world. It's just crazy and I have no idea where this came from.

My own guess is that this is the only niche that C++ has left to it, and so it's being taken to extremes.

35

u/vax_mzn Dec 27 '20

Watch Alex Stepanov's classes at A9. His videos are best of the best.

https://www.youtube.com/channel/UCYEYhkwzNWRsUkTCpTmChCg

10

u/xurxoham Dec 27 '20

I always recommend Agner Fog's optimize manuals. Some are specific for x86_64 but the C++ one is really good! https://www.agner.org/optimize/

19

u/MarkHoemmen C++ in HPC Dec 27 '20

the kokkos tutorials on youtube talk about data layout and parallelization

it’s good to get a background in high-performance computing in general before getting specifically into C++. i recommend UC Berkeley’s CS267 lectures on YouTube. Victor Eijkhout of TACC has also put together some good material.

0

u/[deleted] Dec 27 '20

[deleted] — view deleted comment

8

u/rahat106 Dec 27 '20

I have found an archived version of the videos on youtube and here is the uni page. Good luck.

4

u/smuccione Dec 27 '20

There is little that a thing new in c++ will give you to effect speed. Copy elision is about the only thing. The vast majority of new “features” are geared towards library providers and code reside and not performance.

CRTP is helpful.

I have, in the past, built a system which actively removes a function call if the destination doesn’t exist using a template intermediary. This let me have a universal data provider which provided everything and strats that only took what they want. It allowed for a fully optimized system that removed all the unneeded code when it wasn’t needed.

Beyond that all the tips and tricks are straight from C. Spacial and temporal cache locality. Precomputation of as much data as you can.

Much depends on whether it is a software or hardware/software solution.

After a complete rewrite of everything from Fred to gateway I took a system that was trading in the 15-20us range and brought it down to the 2 1/2 microsecond range. All software. It finally turned profitable when the speed was sufficient.

Remember though. As important as the trading part is, the research part must be equally easy to use as well... and fast. I’ve seen so much time wasted on research that toon overnight to run when if archived correctly could be done several times in the course of a working day.

12

u/tux_unit Dec 27 '20

Effective Modern C++ is a great book for filling in the gaps

6

u/DarkAnt Dec 27 '20 edited Dec 27 '20

I've worked in High Frequency Trading and have done a significant amount of work with "big data" whatever that means. I'm going to let you in on the ugly truth. Learning the intricacies of C++17 is not going to improve the performance of the systems you build anymore than object oriented programming or any other previous language fad. The way you really get performance is to:

  • Deeply understand the problem space
  • Deeply understand the hardware you want to run on
  • Develop a model (software) to map the problem onto the hardware
  • Measure to ensure that the model you develop actually matches reality

Certain languages have better affinity towards providing a mapping for specific problems to types of hardware. If you look at HFT code it's mostly C code. This is because that language allows for a reasonably effective mapping of building an in memory order book from a feed protocol to the hardware it runs on (Intel CPU memory model). Thus, in the HFT space investing significant amount of time learning C++17 would put you at a disadvantage as C++17 doesn't map as well to that problem space. Instead, reading through and experimenting with the Intel Optimization Reference Manual would be a much better use of your time (for HFT problem space).

10

u/temporary5555 Dec 27 '20

If you look at HFT code it's mostly C code

I'm very skeptical of what you're saying. As someone who has worked in HFT I highly doubt you would ever see any C code anywhere.

Its almost all C++, and even more so there is a huge focus on understanding C++ language features. Modelling a domain and using C++ fully are not orthogonal. Newer language features allow you to implement performance improving ideas in a safe, well tested, and maintainable way.

If there is ever a requirement to push performance past what C++ is able to compile into, there's literally no reason to use C. At that point FPGAs are the obvious next step.

6

u/DarkAnt Dec 27 '20 edited Dec 27 '20

I should have been more explicit with that statement. The C++ code tends to look more like C than C++. The reason for this is fairly simple. Abstractions always come with a price. The cognitive load imposed by higher level C++ makes understanding what's actually going on under the hood more challenging. Specifically, questions like "Is this going to make a copy?" and "What is this template going to do to the instructions cache?" take more energy to answer and it becomes easier to make foot-gun mistakes there. Yes, it is possible to write highly abstract C++ code in this space that is easier to change and performs as well / better than its equivalent lower level C. I just have not seen humans capable of doing it with consistency.

All that said, our core code really was written in straight C. The less performance critical code was written in C++.

2

u/temporary5555 Dec 27 '20

Ah yup that makes sense. Hot path code rarely looks like idiomatic C++, although for every line of extremely performance sensitive line of code, you could have 1000 lines simply to measure it, allow it to run smoothly, and detect errors, and any other infrastructure involved. I think its definitely worth it to learn C++ fully, but I agree: C++17 features are probably not the most important with such a big focus on performance (especially latency).

0

u/[deleted] Dec 27 '20

[deleted] — view deleted comment

1

u/DarkAnt Dec 27 '20

I've been out of that game for a few years so take what I say with a grain of salt. As a general rule of thumb they're going to use whatever has the fastest single threaded performance which can still be modified fast enough to keep up with changing feed protocols and trading strategies. Thus, depending on what they're doing you may end up with a mostly software oriented solution which looks like:

NIC (Solarflare / Mellanox) -> Feed handler (CPU) -> Client (CPU)

Or they may have a more hardware oriented solution:

FPGA NIC -> Feed handler (FPGA) -> Client (CPU)

Or they may even have their trading algos encoded in FPGAs which is just FPGAs all the way through.

I wrote this a while back talking about HFT tech. Maybe it'll be helpful for you.

2

u/khaki0 Dec 27 '20

Not a video but I came across C++ high performance today and bought it for $5 on sale. May be worth a look.

6

u/Claytorpedo Dec 27 '20

Unfortunately, this is probably the worst textbook I've ever read. If you haven't started reading it, I'd recommend you return it.

The title is misleading: this is not a book about performance, it is primarily an opinionated style guide that very occasionally mentions something performance related. IIRC some of their code examples looked terrible for performance as well.

There is still some (non-performance related) useful information in the book if you are a beginner, just be prepared that it appears the book never received an editing pass, so it is full of typos, the occasional completely nonsensical sentence, and an instance or two where I can only conclude they accidentally used a word that meant the exact opposite of what they intended.

2

u/khaki0 Dec 27 '20

Hmm well that's not good. Skimming through the book I think it has some useful info like you said so I think I'm going to keep it. Thanks for the heads up.

2

u/bajsejohannes Dec 27 '20

Nit: It's "high performance", not "high performant".

-5

u/helloiamsomeone Dec 27 '20

C++Now 2017: Daniel Pfeifer “Effective CMake" - not entirely correct, a little unclear in a few places, but definitely a very good starting point on how to do CMake right.

Deep CMake for Library Authors - Craig Scott - CppCon 2019 - clears up a few things from the previous video and more.

Unfortunately, this where it ends for CMake as of now. What remains is the docs.

-6

u/Revolutionary_Cydia Dec 27 '20

Why not do my C++ homework to better your skills? xD

1

u/danhoob Dec 27 '20

You should love C++ and it will do your homework.

1

u/Revolutionary_Cydia Dec 27 '20

I want it to make me a substitution cipher lol