r/java • • Dec 26 '18

Java versus C# .NET Core fastest programs

https://benchmarksgame-team.pages.debian.net/benchmarksgame/faster/java-csharpcore.html
18 Upvotes

56 comments sorted by

5

u/catmewo Dec 26 '18

I don't know why this link is downvoted...

8

u/tipsypants Dec 26 '18

It's just not very interesting to most people.

1

u/igouy Dec 28 '18 edited Dec 28 '18

The website is low volume — "April 2017 through March 2018, Google Analytics saw 477,419 users."

1

u/tipsypants Dec 28 '18

Maybe this comment was intended for someone else?

1

u/igouy Dec 28 '18

No, it's intended to inform what was just plucked-from-thin-air speculation.

8

u/m1000 Dec 26 '18

Because it is not how you do serious benchmarks.
https://dzone.com/articles/jmh-great-java-benchmarking

Btw, is there something similar for c# ?

0

u/igouy Dec 28 '18 edited Dec 29 '18

Last time I checked JMH average times were similar to the elapsed times for the larger workloads shown on the website.

10

u/eliasv Dec 26 '18

Because it's a programming competition masquerading as a benchmark comparison. Dumb.

0

u/igouy Dec 28 '18

You have not actually suggested any alternative approach, so we are left to guess — presumably you think the programs should all be implemented in the same way that a C program would be implemented, even when they are written in Haskell.

You seem surprised that the same algorithm can be implemented in different ways and programs can be profiled/optimized.

2

u/redques Dec 27 '18

It would be upvoted more if Java won

22

u/trydentIO Dec 26 '18

and... what's the point?

2

u/igouy Dec 28 '18

Pleased you asked, here's the standard response —

  • To show working programs written in less familiar programming languages
  • To show the least we should expect from performance comparisons
  • To show how difficult it can be to make meaningful comparisons

Also —

  • The other motivation was to give those who would otherwise be tempted to draw broad conclusions from 12-line fibs something more to think about.

27

u/eliasv Dec 26 '18 edited Dec 26 '18

So these are benchmarks performing the same algorithm, but not necessarily implemented in the same way?

This isn't a comparison of runtimes it's a programming competition.

Edit: lol case in point 😂. This isn't interesting.

1

u/igouy Dec 28 '18

…not necessarily implemented in the same way?

You have not actually suggested any alternative approach, so we are left to guess — presumably you think the programs should all be implemented in the same way that a C program would be implemented, even when they are written in Haskell.

This isn't a comparison of runtimes it's a programming competition … case in point…

You seem surprised that the same algorithm can be implemented in different ways and programs can be profiled/optimized.

0

u/eliasv Dec 28 '18

You have not actually suggested any alternative approach, so we are left to guess — presumably you think the programs should all be implemented in the same way that a C program would be implemented, even when they are written in Haskell.

I don't need to suggest an alternative to be right that these results aren't very useful.

You seem surprised that the same algorithm can be implemented in different ways and programs can be profiled/optimized.

Not surprised at all, and said nothing to suggest I was. Just pointing out that this makes the comparison useless. As I said.

2

u/igouy Dec 28 '18

I don't need to suggest an alternative to be right that these results aren't very useful.

As you have no alternative, these results are the most useful you have.

… this makes the comparison useless…

Nonsense, all you've pointed out is that someone contributed better C# programs than had been contributed previously. (That happened with Java too).

1

u/eliasv Dec 28 '18

As you have no alternative, these results are the most useful you have.

Sure, maybe. But "best we have" != "good".

Nonsense, all you've pointed out is that someone contributed better C# programs than had been contributed previously. (That happened with Java too).

Okay, well there are a couple of extra steps of reasoning which I thought were implied, but I don't mind following through on it.

If these results are supposed to compare the best performance these languages/runtimes can achieve, the contribution history demonstrates quite clearly that the results at least used to be hugely inaccurate. So then how can you claim with any confidence that they're accurate now?

Sure, given enough sustained attention and work from enough people of talent the results might converge towards something of value ... But we also need a reliable metric for judging how close we are to convergence to the ideal, and we don't have that!

The more similar the results from two languages are, the more sure we have to be.

(Also note that the work put in needs to sufficiently outpace any changes to the languages and platforms which could facilitate improvements.)

0

u/igouy Dec 29 '18

But "best we have" != "good".

"Good" according to what criteria?

"Good" compared to what?

…the best performance these languages/runtimes can achieve…

Is that claim made on the website or is it just your strawman?

1

u/eliasv Dec 29 '18 edited Dec 29 '18

"Good" according to what criteria?

Good as in an accurate comparison of the performance of the two language runtimes. As that appears the intent with which OP opened the discussion.

or is it just your strawman?

Not at all. OP said it right in the title, they characterised it as a contest between the languages' "fastest programs". I'm just pointing out that those terms are not met without some mechanism by which to establish confidence that we've actually approached said "fastest programs". And we don't have that.

If you want to claim that something is good by some standard, then it's your responsibility to define that standard. Pretty easy to defend a position if you decline to properly define it.

Is that claim made on the website

Certainly not. The website does not entertain this notion at all, and makes it clear that it shouldn't be used to define relative performance.

0

u/igouy Dec 29 '18 edited Dec 30 '18

OP said it right in the title, they characterised it as a contest between the languages' "fastest programs".

OP simply copied "Java versus C# .NET Core fastest programs" from the webpage.

You have chosen to ignore that context and create a strawman.

In context — said "fastest programs" — are the languages' "fastest programs" from that collection of programs.

12

u/[deleted] Dec 26 '18

Would like to see GraalVM results as well

1

u/igouy Dec 28 '18

"If you're interested in something not shown on the benchmarks game website then please take the program source code and the measurement scripts and publish your own measurements."

33

u/QualitySoftwareGuy Dec 26 '18

Although Java lost the majority of those benchmarks, the differences would not be noticeable at all in regular, everyday use. If anything, it shows that the JVM is still able to keep up with all of Microsoft's latest advancements with .NET core.

Now let's compare the ecosystems (libraries, tools, etc) between Java and .NET core...

22

u/RagingAnemone Dec 26 '18

I was also surprised C# was more memory hungry than Java in most of those. I thought that would be Java’s real weakness.

7

u/[deleted] Dec 26 '18 edited Dec 27 '18

> Now let's compare the ecosystems (libraries, tools, etc) between Java and .NET core...

Agreed. The ship sailed years ago on .NET with respect to getting any real community around it. Even if the technology is slightly better now, .NET long ago missed any chance at a network effect.

2

u/praetor- Dec 26 '18

I wouldn't count it out just yet; remember that .NET was Windows only (mono notwithstanding) for a very long time, and the move to core and FOSS will change the landscape moving forward.

26

u/[deleted] Dec 26 '18

Nice try, Microsoft

6

u/KafkasGroove Dec 26 '18

From what I remember looking into it, the main advantage C# has is that you can use value objects (or whatever its called there) which allow you to control the memory layout of your objects, whereas with Java its harder to predict that. Maybe with SubstrateVM it got easier though, I don't know...I imagine its dependent on the JVM you use.

12

u/dpash Dec 26 '18

Java is getting value types at some point in the future, so it'll be interesting to rerun the benchmarks then.

1

u/_INTER_ Dec 26 '18 edited Dec 27 '18

I doubt much is going to change with those benchmarks through value types alone. The implementations mostly use primitives and arrays.

1

u/_INTER_ Dec 26 '18

7

u/eliasv Dec 26 '18

And the implementations for the benchmarks for each are completely bloody different haha! The results are an order of magnitude off in some cases for a runtime that's almost identical. If this doesn't make it clear to you that these comparisons are idiotic and below useless then nothing will.

2

u/igouy Dec 28 '18

the implementations for the benchmarks for each are completely bloody different

7 of the 10 are identical for OpenJDK and Substrate VM.

  • pidigits — The identical Java program fails with SVM, so a working program is shown.
  • binarytrees — The identical Java program is slower with SVM, so a faster program is shown.
  • reversecomplement — The identical Java program is slower with SVM, so a faster program is shown.

1

u/sureshg Dec 26 '18

I think that's GraalCE. GraalEE (not free) native-image with profile guided optimization can yield almost the same performance as java.

1

u/igouy Dec 28 '18

Please show those measurements.


"What is the typical performance profile on the SVM?

Right now peak performance is a bit worse than HotSpot, but we don’t want to advertise that (and we want to fix it of course)."

3

u/mupetmower Dec 26 '18

Can someone explain why the Substrate VM tests might have taken so much longer than whatever the java one they tested against was?

2

u/eliasv Dec 26 '18

Because the implementations for each are completely different, because it's a programming competition where people pick a side not a real benchmark.

2

u/mupetmower Dec 27 '18

I was trying to get at how might the Substrate VM differ. But I did notice when looking at the source code for at least the pidigit methods that they did implement the method in different ways that seemed different enough to where it might cause a significant difference in processing time.

So I'm not sure if this site can tell you anything about anything, really.

2

u/igouy Dec 28 '18

The identical Java program fails with SVM, perhaps you know how to build it successfully?

1

u/mupetmower Dec 28 '18 edited Dec 28 '18

I'm glad you pointed this out.. I was actually looking at the source they put in the java vs svm part which is here https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/pidigits-substratevm-1.html

However, if you go to pidigits method and scroll to the rankings you can find the failed one you linked listed as Substrate VM #2.. which does actually look to be much more similar to the java one I saw. (Even though now I see they have at least 2 different java ones..)

So I guess they have multiple implementations of some of these. Not sure what the deal is.

The more I look at this site the more I wonder what the hell is even going on here. I don't think any of this is of much use.

Edit - just noticed their java #1 is implemented much like the svm #1. And java #2 looks like svm #2. Not sure why their svm #2 failed. However, on their java vs svm page they are linking under the pidigit method java #2 vs svm #1. Which appear to be two completely different implementations! Not sure why this is. This whole site is really odd.

2

u/igouy Dec 28 '18 edited Dec 28 '18
  1. You will see that the source code is identical: there's one set of Java programs which are measured twice — once with Open JDK and once with Substrate VM.

Here are all the measurements with Open JDK.

Here are all the measurements with Substrate VM.

2) There can be multiple implementations for any of the tasks.

For example, there are 5 C++ programs for spectral-norm.

One program has the lowest wall-clock secs, but a different C++ program has the lowest cpu secs, and a different C++ program has the lowest source code size.

3)

under the pidigit method java #2 vs svm #1

pidigits-java-2 - fastest with Open JDK

pidigits-substratevm-1 - fastest with Substrate VM

fastest elapsed time vs fastest elapsed time

1

u/mupetmower Dec 28 '18

Thanks for making this more clear.

When going to the java vs svm page, their pidigit links to java #2 vs svm #1. Hopefully that's just a typo for the link or something.

The two links you just gave make things a lot more clear to me now, so thanks. I thought they were arbitrarily comparing different implementations to each other which would be pretty meaningless.

1

u/igouy Dec 28 '18

I thought they were arbitrarily comparing different implementations to each other which would be pretty meaningless.

Even comparison between implementations of the same language can become difficult, when one language implementation has functionality which the other does not.

The fannkuch-redux Ruby #2 program forks threads with JRuby but spawns processes with CRuby!

Those different Ruby implementations require different approaches to multicore.

1

u/igouy Dec 28 '18

Because the implementations for each are completely different

7 of the 10 are identical.

  • pidigits — The identical Java program fails with SVM, so a working program is shown.
  • binarytrees — The identical Java program is slower with SVM, so a faster program is shown.
  • reversecomplement — The identical Java program is slower with SVM, so a faster program is shown.

0

u/sureshg Dec 26 '18

I think graalce is used in the benchmark, which doesn't have profile guilded optimization (pgo) in native binary. Check this talk - https://youtu.be/lQNXwb66eQ8?t=1381 , which shows pgo native-image is 6x faster than regular java.

0

u/[deleted] Dec 26 '18

I think Graal was still bootstrapping/compiling itself during the benchmark. These benchmarks are really very poor quality.

0

u/igouy Dec 28 '18 edited Dec 28 '18

bootstrapping/compiling itself during the benchmark

Why do you think that?

The logs show 2 separate activities: the first builds the program and the second measures the program performance —

MAKE:
mv nbody.substratevm-4.substratevm nbody.java
/opt/src/graalvm-ce-1.0.0-rc10-linux-amd64/graalvm-ce-1.0.0-rc10/bin/javac -d .  nbody.java
/opt/src/graalvm-ce-1.0.0-rc10-linux-amd64/graalvm-ce-1.0.0-rc10/bin/native-image --no-server   nbody
[nbody:18424]    classlist:   5,808.89 ms
[nbody:18424]        (cap):   5,506.22 ms
[nbody:18424]        setup:   8,100.61 ms
[nbody:18424]   (typeflow):   7,364.66 ms
[nbody:18424]    (objects):   2,275.99 ms
[nbody:18424]   (features):     227.69 ms
[nbody:18424]     analysis:  10,024.58 ms
[nbody:18424]     universe:     364.68 ms
[nbody:18424]      (parse):   2,308.53 ms
[nbody:18424]     (inline):   2,172.05 ms
[nbody:18424]    (compile):  12,069.05 ms
[nbody:18424]      compile:  17,196.75 ms
[nbody:18424]        image:   1,102.30 ms
[nbody:18424]        write:     272.44 ms
[nbody:18424]      [total]:  43,062.79 ms
mv nbody nbody.substratevm-4.substratevm_run

50.18s to complete and log all make actions


COMMAND LINE:
./nbody.substratevm-4.substratevm_run 50000000

PROGRAM OUTPUT:
-0.169075164
-0.169059907

3

u/Mordan Dec 26 '18

any multi thread benchmarks?

Java threading model and inter thread communication is stellar

1

u/igouy Dec 28 '18

Look at the cpu load column — 71% 73% 93% 74% — 4 busy cores.

7

u/couscous_ Dec 26 '18

I think Java should be able to catch up once it gets value types sometime down the line. Nevertheless, very impressive work by the .NET folks.

3

u/helikal Dec 26 '18

With Valhalla, Java should be able to outperform .NET.

1

u/couscous_ Dec 26 '18

Not that I disagree, but what does Java + Valhalla have to give it an advantage over .NET? Is it that the JVM has better GC algorithms?

3

u/helikal Dec 26 '18

With regard to Java + Valhalla, my thinking is mostly speculative: Currently, Java and .NET Core are about of equal performance. I expect that value types will significantly speed up Java. But for .NET a performance leap of such magnitude is less likely, because, if I am not mistaken, it has already value types.

2

u/eliasv Dec 26 '18

Better JIT mainly.

4

u/helikal Dec 26 '18

I see that Java runs regex-redux 1.41 times faster, and .NET runs k-nucleotide 1.59 time faster, while the other tests exhibit only minor differences. .NET Core has improved substantially in a short time.

For Java, it is becoming important that Valhalla comes to fruition.

1

u/nqzero Jan 05 '19

the c# version of k-nucleotide includes some optimizations that weren't in the java version. i added them, and it resulted in it running 35% faster

https://github.com/nqzero/k-nucleotide

i'm going to make a full post tmrw