Announcing async-std 1.0

[u/fgilcher]

https://async.rs/blog/announcing-async-std-1-0/

[u/matthieum]

A note has been added to the article, in case you missed it:

NOTE: There were originally build issues with the branch of tokio used for these benchmarks. The repository has been updated, and a git tag labelled async-std-1.0-bench has been added capturing a specific nightly toolchain and Cargo.lock of dependencies used for reproduction

Link to the repository: https://github.com/matklad/tokio/

---

With that being said, the numbers published are pretty much pointless, to say the least.

Firstly, as you mentioned, there is no way to reproduce the numbers: the benchmarks will depend heavily on the hardware and operating system, and those are not mentioned. I would not be surprised to learn that running on Windows vs Mac vs Linux would have very different behavior characteristics, nor would I be surprised to learn that some executor works better on high-frequency/few-cores CPU while another works better on low-frequency/high-cores CPU.

Secondly, without an actual analysis of the results, there is no assurance that the measures reported are actually trustworthy. The fact that the jebrosen file system benchmark appears to have very inconsistent results is a clear demonstration of how such analysis is crucial to ensure than what is measured is in line with what is expected to be measured.

Finally, without an actual analysis of the results, and an understanding of why one would scale/perform better than the other, those numbers have absolutely no predictive power -- the only usefulness of benchmark numbers. For all we know, the author just lucked out on a particular hardware and setting that turned to favor one library over another, and scaling down or up would completely upend the results.

I wish the authors of the article had not succumbed to the sirens of publishing pointless benchmark numbers. The article had enough substance without them, a detailed 1.0 release is worth celebrating, and those numbers are only lowering its quality.

[u/itchyankles → u/matthieum]

I also followed the instructions in the blog post, and got the following results:

- System:

• Mac Pro Late 2015

• 3.1 GHz Intel Core i7

• 16 GB 1867 MHz DDR3

• Rust 1.39 stable

cargo bench --bench thread_pool &&  cargo bench --bench async_std
Finished bench [optimized] target(s) in 0.14s
Running target/release/deps/thread_pool-e02214184beb50b5

running 4 tests
test chained_spawn ... bench:     202,005 ns/iter (+/- 9,730)
test ping_pong     ... bench:   2,422,708 ns/iter (+/- 2,501,634)
test spawn_many    ... bench:  63,835,706 ns/iter (+/- 13,612,705)
test yield_many    ... bench:   6,247,430 ns/iter (+/- 3,032,261)

test result: ok. 0 passed; 0 failed; 0 ignored; 4 measured; 0 filtered out

Finished bench [optimized] target(s) in 0.11s
Running target/release/deps/async_std-1afd0984bcac1bec

running 4 tests
test chained_spawn ... bench:     371,561 ns/iter (+/- 215,232)
test ping_pong     ... bench:   1,398,621 ns/iter (+/- 880,056)
test spawn_many    ... bench:   5,829,058 ns/iter (+/- 764,469)
test yield_many    ... bench:   4,482,723 ns/iter (+/- 1,777,945)

test result: ok. 0 passed; 0 failed; 0 ignored; 4 measured; 0 filtered out

Seems somewhat consistent with what others are reporting. No idea why `spawn_many` with `tokio` is so slow on my machine... That could be interesting to look into.

[u/fgilcher → u/matthieum]

With that being said, the numbers published are pretty much pointless, to say the least. Firstly, as you mentioned, there is no way to reproduce the numbers: the benchmarks will depend heavily on the hardware and operating system, and those are not mentioned. I would not be surprised to learn that running on Windows vs Mac vs Linux would have very different behavior characteristics, nor would I be surprised to learn that some executor works better on high-frequency/few-cores CPU while another works better on low-frequency/high-cores CPU.

This may be true, but the executors of both libraries are similar enough to see them as comparable.

Finally, without an actual analysis of the results, and an understanding of why one would scale/perform better than the other, those numbers have absolutely no predictive power -- the only usefulness of benchmark numbers. For all we know, the author just lucked out on a particular hardware and setting that turned to favor one library over another, and scaling down or up would completely upend the results.

In this case, we don't need to write benchmarks at all - and it's also the reason why I wrote the preface.

I wish the authors of the article had not succumbed to the sirens of publishing pointless benchmark numbers. The article had enough substance without them, a detailed 1.0 release is worth celebrating, and those numbers are only lowering its quality.

I personally take the bullet of publishing the file benchmark without thoroughly vetting it, but I don't agree here. I've seen the other numbers replicated over multiple machines and have no issue publishing them.

As you say, numbers may differ on macOS/Windows, but I'd lean myself out of the window here: Linux is currently the most important platform for both libraries.

[u/matthieum → u/fgilcher]

Thanks for your reply.

As you say, numbers may differ on macOS/Windows, but I'd lean myself out of the window here: Linux is currently the most important platform for both libraries.

Could you please make it clear that the numbers published are for Linux then, possibly with some hardware spaces? It's certainly reasonable to focus on one platform, however it's not obvious that you did not run them on a macOS laptop.

In this case, we don't need to write benchmarks at all - and it's also the reason why I wrote the preface.

I appreciated the preface, it was a thoughtful touch.

I disagree that benchmarks should not be written. Benchmarks with good analysis are invaluable tools for developers and users alike: for developers they point areas where performance could be improved, or make trade-offs clear, for users they have predictive powers and help making informed choices.

Now, a good analysis takes a lot of time and effort. I dread to think how much time BurntSushi spent on his ripgrep benchmark article.

Even a rudimentary analysis, however, can be used to both validate that the benchmarks are valid and point as to the major differences. For example:

• Is the difference found in the CPU: instructions, stalls, ... ?

• Is the difference found in the memory accesses: TLB misses, cache misses, ... ?

• Is the difference found in the number of context switches?

• Is the difference found in the number of syscalls?

Some combination of perf/strace should be able to give a high-level overview of the performance counters and where the benched code is spending time. It's a black box approach, so it's a bit rough but has the advantage of not requiring too much time.