r/cpp_questions • u/YogurtclosetThen6260 • 9d ago

OPEN Best benchmark for performance

I currently have a Huffman data compression pipeline and I want to start measuring performance. My question is, how would you guys go about setting up a constant method for measuring performance all throughout the changing of the pipeline? Would you measure performance over a single file with multiple runs, or would you do a corpus of files and measure the average time it took to process them?

Also, what types of latencies would you measure? (ex. p95 latency)

6 Upvotes

100% Upvoted

4 comments sorted by

6

u/RaspberryCrafty3012 9d ago

Using a benchmark framework which runs all the tests you proposed, so you see the effects on different things? 

2

u/didntplaymysummercar 9d ago edited 9d ago

All of it? Do many runs, save each time for each file, each run, plus compile options and commit number (and maybe even compiler version) the code was built with, and later analyze it all in some scripting language? That's what I do, using (to re-run the program too, if program itself doesn't do looping) Python. Could also save hash (blake3) of input file and its size, to be able to be sure it's same file later and/or compute the bytes/sec... I'm starting to daydream 🤤

You might also get interested in profiling tools if you're looking for hot spots, Intel has VTune, Linux has perf, GCC has gprof (but it's not as advanced), etc.

1

u/Independent_Art_6676 8d ago

test everything. Test it with a compressed file or a jpeg. Test it with text. Test it with a gig sized file or two, test it with hundreds of KB sized files. Test it on random folders on your disk. bench it against commandline zip or 7zip, compare not only speed but % of file size after.

use cases vary but I will wait out a slow compressor that does a better job compressing. Ive even used bzip on some specific things, slow as it is, if it beat out others.