“covers the basics of how to attempt benchmarking with actual computer science instead of myth and voodoo rituals”
Benchmarking with computer science rather than myth
Full comment
This guy has had a few talks at CppCon over the years, and covers the basics of how to attempt benchmarking with actual computer science instead of myth and voodoo rituals: this video In practice, it's not critical to go that far. In my experience, huge performance wins are trivial to benchmark with just a modicum of effort, and tiny performance wins are typically not worth any effort at all. At FAANG scale they might be, but I don't work at a FAANG, so not my problem! :) Simple methods I use and can recommend are: - Ideally, collect data from production, where the change behaves exactly as it would in production because it is production! - Interleave tests on the same machine. E.g.: A, B, A, B, A, B, etc... - Run tests on every machine you can get your hands on over an extended period and draw a histogram or a scatter chart of some sort. You don't have to do this at 100% load, it's actually better to mix this in with normal workloads. This gives you an idea of the noise versus signal, platform variance, variation due to time of day, etc... - Run tests after a cold boot to get a sense of worst-case performance instead of idealised and often unrealistic "hot loop" performance. - Run tests with different input sizes and draw a graph. Try find the explanation for every feature of the graph. - Be aware of every cache layer in the system. Don't forget about the potentially huge RAM or SSD cache of the storage subsystem. Typical SAN arrays now have 256 GB or more. The previous graph a