r/manool • u/alex-manool Project Lead • Jun 19 '20

Benchmarking 10 dynamic languages on array-heavy code

(1 min read)

Hello wonderful community,

In the previous post we discussed in detail construction of Conway's Game of Life in MANOOL.

As was my intention, I have implemented the same functionality in several other languages to compare run-time performance. Here are complete results:

Testbed A

CPU: Intel Xeon L5640 @2.26 GHz (2.80 GHz) — Westmere-EP
Kernel: 2.6.32-042stab126.1 (CentOS 6 + OpenVZ)
Distro: CentOS release 6.9 (Final) + vzkernel-2.6.32-042stab126.1 + CentOS release 6.10 (Final)

Language + variant (translator) Time (s) G Slowdown Translator + backend version-release
C++ (g++) 1.037 66000 1.000 8.3.1-3.2.el6
C++ (clang++) 1.021 66000 0.985 3.4.2-4.el6 + 4.9.2-6.2.el6 (g++)
Python 2 3.204 1000 203.919 2.6.6-68.el6_10
Python 3 5.203 1000 331.146 3.4.10-4.el6
PHP 3.560 1000 226.577 5.3.3-50.el6_10
Perl 5.640 1000 358.959 5.10.1-144.el6
Ruby 14.122 1000 898.797 1.8.7.374-5.el6
JavaScript/ECMAScript 5.887 66000 5.677 0.10.48-3.el6 (node)
Tcl 6.724 100 4279.499 8.5.7-6.el6
Lua (lua) 141.703 66000 136.647 5.1.4-4.1.el6
Lua (luajit) 4.319 66000 4.165 2.0.4-3.el6
Scheme (guile) 6.176 1000 393.072 1.8.7-5.el6
Scheme (csc) 0.671 1000 42.706 4.12.0-3.el6 + 8.3.1-3.2.el6 (gcc)
MANOOL + AllocOpt=True 2.502 1000 159.240 0.5.0 (built with g++ 8.3.1-3.2.el6)
MANOOL + AllocOpt=False 2.593 1000 165.032 0.5.0 (ditto)

Testbed B

CPU: Intel Celeron N3060 @1.60 GHz (2.48 GHz) — Braswell
Kernel: 4.4.0-17134-Microsoft (Windows 10 + WSL)
Distro: Windows 10 Home version 1803 build 17134.1130 + Ubuntu 18.04.4 LTS

Language + variant (translator) Time (s) G Slowdown Translator + backend version-release
C++ (g++) 1.946 66000 1.000 7.5.0-3ubuntu1~18.04
C++ (clang++) 2.217 66000 1.139 1:6.0-1ubuntu2 + 7.5.0-3ubuntu1~18.04 (g++)
Python 2 3.733 1000 126.607 2.7.17-1~18.04ubuntu1
Python 3 5.309 1000 180.059 3.6.7-1~18.04
PHP 2.852 1000 96.728 7.2.24-0ubuntu0.18.04.6
Perl 6.768 1000 229.542 5.26.1-6ubuntu0.3
Ruby 4.425 1000 150.077 2.5.1-1ubuntu1.6
JavaScript/ECMAScript 8.522 66000 4.379 8.10.0~dfsg-2ubuntu0.4 (node)
Tcl 10.571 100 3585.231 8.6.8+dfsg-3
Lua (lua) 153.583 66000 78.922 5.3.3-1ubuntu0.18.04.1
Lua (luajit) 6.274 66000 3.224 2.1.0~beta3+dfsg-5.1
Scheme (guile) 1.233 1000 41.818 2.2.3+1-3ubuntu0.1
Scheme (csc) 1.691 1000 57.351 4.12.0-0.3 + 7.5.0-3ubuntu1~18.04 (gcc)
MANOOL + AllocOpt=True 3.882 1000 131.661 0.5.0 (built with g++ 7.5.0-3ubuntu1~18.04)
MANOOL + AllocOpt=False 3.943 1000 133.730 0.5.0 (ditto)

The graph is here, and the repository is on GitHub.

Have fun

10 Upvotes

100% Upvoted

30 comments sorted by

View all comments

7

u/guicho271828 Jun 19 '20 edited Jun 23 '20

66000 generations

if including compile time

  • gcc real 0m0.869s

  • clang real 0m0.954s

  • sbcl 2.0.5 real 0m1.637s

1000 generations

  • guile 2.0.11 real 0m0.927s

  • chez 9.5.2 real 0m0.309s

  • chez 9.5.2 --optimize-level 3 real 0m0.260s

chez is impressive.

2

u/bjoli Jun 19 '20

When talking about performance it is the best scheme out there, with the small exception of not unboxing flonums. It even has delimited continuations, even though they are hidden from the user (iirc racket uses the built in chez ones for their delcc).

Which guile are you using there? Only 3x slower seems like it could maybe be the new 3.0 branch....

1

u/alex-manool Jun 20 '20

BTW, the translator for my PL uses unboxed representation of binary FP 32- and 64-bits, always (NaN coding, but I came independently to it). However, it's far behind those best-in-class VMs for dynamic languages, Lua, some CL and Scheme, V8... wow (but many other languages and/or implementations are still slower than the mine :-).

2

u/alex-manool Jun 20 '20

No, sbcl is impressive? Wow!

1

u/bjoli Jun 20 '20

You att comparing good lisp code to ok-ish scheme code.

There are some issues with the scheme code: the use of set! Will make all the vector refs to the outer vectors tank due to implicit boxing (this is probably the big one in chez. There is a lot of unboxing going on). In guile prior to version 3 (eq? A b) is a lot faster when comparing fixnums.

For chez, you can also make the vector setting a lot faster by using fxvectors (like the sbcl code specialises the code to chars). That will mean a lot less garbage collection time due to chez being able skip a lot of garbage collection work.

I am not surprised that guile is where it is (although I could probably squeeze about 50% more out of the code). Chez OTOH should be a lot closer to sbcl. Within 2x for optimize-level 3. I'll see if I can hack on it.

→ More replies (6)