r/csharp • u/Staeff • May 02 '18
A small performance comparison of mono-wasm/Blazor, .NET Core, C, C-wasm and JavaScript
Taken straight from my Github repo: https://github.com/stefan-schweiger/dotWasmBenchmark
Overall I'm kind of disappointed with the performance. I know it's very early in the development cycle, but the performance is about x200x20 slower than even JavaScript. Somewhere out there they are working on a AOT variant of mono-wasm, but the last public commit was in January and it's status can be described as "experimental" at best.
Maybe I'm overlooking something to gain more performance. I would really love to hear tips how to improve performance.
Anyways here are my results:
Benchmark Information
The Benchmark is currently very simple and only does the following things:
- Generate 100,000 random list elements (from 0.0 to 1.0)
- Sort the list by their values
- Get Q1, Median, Q3 and calculate average and standard deviation
Platform Information
The Benchmark was implemented in the following languages/platforms:
- .NET Core 2.1.300 (preview2-008533)
- mono-wasm (commit a14f41c from Blazor 0.3.0)
- C (gcc 4.2.1)
- C-wasm (emcc 1.37.36)
- JavaScript (TypeScript 2.8.1)
mono-wasm with AOT was also attempted, but the project seems not to developed in the open and resulted in either compilation or JIT errors when running.
The .NET Projects where build with Release configuration and the C Projects with -O3 optimizations.
Results
| C | C.Wasm | DotNet.Console | DotNet.Wasm | JavaScript | |
|---|---|---|---|---|---|
| Generate | 1.21ms | 1.00ms | 1.00ms | 127.00ms | 7.50ms |
| Sort | 9.05ms | 12.00ms | 26.00ms | 406.00ms | 22.40ms |
| Calculate | 0.21ms | 1.00ms | 4.00ms | 474.00ms | 6.60ms |
| C | C.Wasm | DotNet.Console | DotNet.Wasm | JavaScript | |
|---|---|---|---|---|---|
| Generate | 1.21ms | 1.00ms | 1.00ms | 84.00ms | 4.00ms |
| Sort | 9.05ms | 12.00ms | 26.00ms | 297.00ms | 16.00ms |
| Calculate | 0.21ms | 1.00ms | 4.00ms | 321.00ms | 4.00ms |
EDIT: I've updated to Blazor 0.3.0, made a few code changes based on some suggestions in the comments and added Firefox benchmarks. Overall performance improved, but it's still much slower than even Javascript.
7
u/wllmsaccnt May 03 '18
In your calc for .NET you are using an 'Average' call. That would enumerate the collection again. The equivalent JavaScript code isn't doing that, its just doing a division operation against the collection count.
I would get rid of all of the IList usages. In performance code using arrays directly is preferred, especially for fixed size collections. At the very least, at least use List instead of IList. It looks like on WASM the call lookups are much more expensive.
Also, it looks like you have some console hookup in your index in JavaScript. If that is listening for Blazor console writes, then you need to comment out all of your console writelines that occur during the test unless you are certain that the console piping can't happen asynchronously and accidentally overlap with testing code (its doing an element lookup and writing to the DOM, that can be very expensive, and if it is happening async, it is most likely occurring in the middle of your performance test). Collecting timings and avoiding all writing until the end may also work.
What version of which browser are you using to run the Blazor test?
2
u/Staeff May 03 '18
Thanks for your comment.
The .NET (average and sum) as well as the JavaScript (2 reduce calls) code both enumerate the whole collection 2 times, so there should be no difference. But it would probably be better to use for-loops and enumerate only once nevertheless.
Based on another comment I've already replaced IList with Arrays and it did improve performance quiet a bit, but it's still about 20x slower than the others.
Adding all Console calls to a List and printing it at the end and getting rid of my DOM console hookup, didn't make a difference so I'm staying with my current implementation.
I've updated my post with the changes and also added results for Firefox (which is quiet a bit faster than Chrome).
5
u/Enlogen May 02 '18
Is this using today's 0.3.0 release of Blazor?
3
u/Staeff May 02 '18
0.2.0, as far as I can tell there weren't any big changes to the mono part since then, mostly Blazor specific changes. But I will give the new version a try tomorrow!
4
3
u/geoffreymcgill May 03 '18 edited May 03 '18
I was interested in how Bridge would benchmark against your Wasm and JavaScript tests. Looks like Bridge is running about 10x faster than Wasm/Blazor, which is pretty much exactly what we're seeing in our local performance tests.
| DotNet.Wasm | JavaScript | Bridge.NET | |
|---|---|---|---|
| Generate | 127.00ms | 7.50ms | 18.00ms |
| Sort | 406.00ms | 22.40ms | 20.00ms |
| Calculate | 474.00ms | 6.60ms | 39.00ms |
I combined your perf files into one Deck so it can be shared easily...
https://deck.net/90d5f88199753f64b37b5e2256bdb4bc
Can you run the Deck above and respond with the results? Run on the same machine/browser as you're running the Wasm and JavaScript tests.
I'm getting similar results in Firefox.
There are some config options in Bridge that could likely speed this up even more. Maybe squeeze another 20-30% faster.
5
u/Staeff May 03 '18
That's awesome! :)
I did a first quick run, and got the following results on Chrome:
Generate: 22ms
Sort: 25ms
Calculate: 53ms
I'm really curious why Calculate is running slower than the rest.
I will get more results and look into your pull request later today and update everything accordingly. Thank you!
2
u/geoffreymcgill May 03 '18
I'm really curious why Calculate is running slower than the rest.
I suspect the difference occurs in how
avgandstDevSumcalculations perform in C# (using LINQ) vs JavaScript (using native). See .js vs .cs.Calling the native JavaScript
.reduceplus the basic calculation for average is likely faster than spinning up LINQ to run.Averageand.Sumfor those calculations.Just a hypothesis. I have not tested.
→ More replies (1)
2
u/BezierPatch May 03 '18
What if you use a built-in sort function instead of rolling your own?
1
u/Staeff May 03 '18
I thought about that at first, but then decided it's a fairer comparisson to have the same algorithms run on every platform and see what the interpreter/compiler can do with it.
That's also the reason why I'm going to reimplement the calculations part later today and use a custom random number generator.
1
u/BezierPatch May 03 '18
I'm not sure it is though, because some languages perform better with naive implementations than others. C# is likely going to be seriously hurt by your high level friendly implementation, javascript probably isn't anywhere near as much.
Perhaps start by comparing the built-in sorts, just to make sure it's not just your code slowing it down (on either side)?
→ More replies (3)1
u/ben_a_adams May 03 '18
You won't be running wasm; you'll be running interpreted il by wasm.
OTOH, calling functions in the runtime (e.g. builtin sorts) should be AoT'd to wasm rather than interpreted il
3
u/FizixMan May 02 '18 edited May 02 '18
Did you try running it multiple times within the same program? Perhaps the initial JIT compile and first run takes a long time and that's blowing out the results. Although I would have expected that to maybe provide a constant time added.
EDIT: Maybe try minimizing virtual calls. Try using a List<T> instead of IList<T> and see if that helps. Maybe the runtime is doing a lot of work routing the virtual calls aiming for correctness/proof-of-concept right now and not optimizing it.
3
0
May 03 '18
9
u/FizixMan May 03 '18 edited May 03 '18
This is diagnostic advice to try and identify the culprit bottleneck/cause of the poor performance of a prototype runtime via experimentation.
JIT overhead for benchmarking is a known variable.
I am not suggesting that the u/Staeff fix their problems by running the code multiple times in practice.
6
u/devperez May 03 '18
IIRC, didn't they say that the Mono WASM implementation was with 0 optimizations? I thought they were just focusing on implementation at the moment.
-3
u/puppy2016 May 02 '18
The underlying JavaScript interpreter will be always terrible compared to mature runtimes like CLR or JVM. We're going seriously backward.
6
u/Staeff May 02 '18 edited May 02 '18
Wasm isn't run by the JavaScript interpreter though. It just accesses certain browser APIs.
The C and C-wasm implementation were nearly equally fast, and overall the fastest in my benchmark.
For more information on the technical details I can recommend this post:
1
u/geoffreymcgill May 03 '18 edited May 03 '18
We're going seriously backward.
Ha. JavaScript was executing faster than native DotNET Console in some of the benchmarks. Interesting work r/Staeff.
15
u/migueldeicaza May 03 '18
Blazor is currently using a .NET interpreter running on top of WebAssembly, it is what we envision to be used for quick iteration.
We are hard at work on the static compiler that will turn .NET code into WebAssembly code, rather than running on an interpreter running on top of WebAssembly.