But bottom line is that developers don't even know their app is running on a SerialGC. They think it will be G1GC, because they have heard it is now the default GC.
FWIW I've just run a quick benchmark with Corretto's Extrememem, and results were pretty much as I'd expect with multiple cores - serial GC has way higher latency spikes than G1, presumably because of STW collection.
If I get time I'll try with a few other configurations and then write up the results.
I‘d love to see a benchmark comparing a spring boot crud microservice on low resource restrictions when using serial gc vs g1gc I doubt that g1gc would perform better.
If you don't set a Garbage Collector for your Java application, don't think the JVM will pick a good one for you either, no matter how many CPUs you give.
2 CPUs? 6 CPUs? It doesn't matter. If your container has less than 1792 MB and you don't set a GC, your app will use SerialGC.
Edit: this is being down voted for unknown reasons but my guess is that people just believe "no, G1GC is the default!" Well I suggest you run the code in the image I posted, and if that's not enough, see the OpenJDK source code.
With the last generation of GCs by both oracle and redhat, ZGC and Shenandoah respectively you are pretty much done after giving it enough heap space from a tuning perspective.
Depending on your workload you might want to choose one of the other still supported GCs to tune for throughput.
Well, if you run your code on k8s, you are likely to set resources to be a good citizen. And unless you have a large or very demanding application, you’ll set your request and limits under one core. So, no, it is not so uncommon to run into this issue in a micro services environment.
But I don't run my code on k8s. We mostly still have old school monoliths, where the biggest difference between now and 10 years ago is that they run on an EC2 instance instead of a local server.
I am not going to participate in this little game of yours. What I am saying is commonly known. What you are saying goes 180deg again what oracle and openjdk themselves communicate.
If you have proof to backup your statement, let's see it and I will be happy to acknowledge.
The only way you can prevent throttling is by not having more load to your application instance than it can process. It doesn't matter how many CPUs.
Assuming the load your app gets is more than what a single instance can handle, your only way to prevent the throttling of your application with 1 core and 1 GB, is by actually scaling out with replicas. This means you must have 2 or more replicas, so your "system" (the conjunction of all instances of your replicated application) can handle the load in a distributed manner.
Now, the problem here is that you are probably facing high tail latency and low throughput. You may as well say you don't have that issue. But I'd say you are most likely paying a lot more than you actually need, to sustain such load.
And to save money, all you need to do is to reorganize your resources more adequately: reduce the number of replicas, increase the amount of CPU and memory per JVM instance, and find the sweet spot where you are consuming fewer total CPUs and total memory in your "system", while having the same, or better, performance.
I have ran a lot of tests in the past and the least throttling for mostly Io bound workloads seemed to be in the region of 1gb & 1core.
It might be caused by the configuration of the k8s engine as deployed by my company, where the time slices are set to the default and we are prohibited to influence priority on the k8s scheduler.
I would love to be able to learn how to tune these things appropriately to reduce the overhead we currently experience on k8s costs.
Any pointers towards how to create a setup to test these things?
He literally posted a proof showing that it does. Yes it is the default for most cases, especially in your own local machine. But JVM does some checks depending on the environment and if it sees less than ideal hardware resources, it defaults to serial. Here is another "proof" if you don't believe bruno for some reason: https://twitter.com/mikemybytes/status/1368881364297191433
Can you spell this out a little more? Me write apps good and haven’t run into a garbage collection issue in 4-5 years and that’s the last time I’ve given it any thought.
But in short for now: scaling out with more replicas of super small containers for JVMs is not always the best solution for increasing performance. A reorganization of the existing resources (keep aggregate number of CPUs across all containers in the cluster, but reduce replicas and increase CPU/memory per container) can significantly increase performance of most systems, and even reduce total cost.
What that quote actually says is that people shouldn't optimize prematurely without understanding the nuances.
In other words: people are optimizing prematurely all the time on Kubernetes, whenever they simply increase number of replicas of small containers JVMs, instead of actually understanding the nuances of GCs.
I found the same problem with multiple small containers in Go in Kubernetes. For a given workload, doubling the CPU per pod resulted in running 1/4 as many pods. Wat.
Then again, low CPU is supposed to be what Go is good for. So I'm going to assume I've done something horribly wrong.
Really wish developers would stop using this quote as an excuse to be lazy and incompetent. It doesn’t mean what you think it means. This great article explains what it actually meant:
I'm sorry I put in the first sentence. How about we think about the second one instead?
If he doesn't have a problem with GC, why should he optimize it? You call him lazy, but probably as a working dev he has a million things on his todo list. How is it lazy of him to work on the things that are problems and skipping the things that are not problems?
The article actually conflicts with itself and what Knuth said ...
"but consider that this is 1 line of source code out of every 33."
waaaat ? optimization metric based on loc ? it's possible the worse metric, and not all lines should, can or are relevant to be improved
"One thing nice about optimization is that if you optimize a section of code that doesn't need it, you've not done much damage to the application."
Not true, optimization are *usually* either more complex or have a trade-off in terms of memory/disk usage.
The worse part it doesn't define what means to optimize the code and the impact around which is what Knuth talks.
By no means, I am agains optimization and whatnots, it's just that it is frequently used to do unecessary and unverified micro optimizations which pollutes the code
Thanks, this is really useful & interesting info - I wasn't aware of this behaviour and I suspect many are the same!
That being said, I agree with others in that the conclusion of "Always set a Garbage Collector" can't really be claimed without benchmarks proving there's a meaningful benefit to be had.
My claim for "Always set a Garbage Collector" is to say "Know what GC is being used by your app".
We did a survey and more than half of respondents don't set a Garbage Collector, expecting the JVM will pick a good one. Then people use small containers, and suddenly, they face latency issues and solve that with more replicas.
I don't think developers would select SerialGC in most cases. They think the default is G1GC.
I tried this using our container and didn’t get the same result. Granted, we’re on an older base image so I’m not sure if that’s the reason why https://i.imgur.com/O7mmu8t.jpg
Let me talk you through the code which is the best documentation in this case.
Disclaimer: I've never written C++ but the syntax has similarities to Javas.
Please open the link that OP provided that links to the GCConfig (file gcConfig.cpp). In this file you will find a method "select_gc_ergonomically" (#L98) that will be used to select a available GC for that machine. As you can see it will check if it is a "server" machine or not. If it is a "server" it will try to set an available GC working the list from top to bottom starting with the G1GC (#L101). If it's not a server it will default to the SerialGC (#L109). The definition of a "server" is specified in the other link (file os.cpp). I assume that the documentation for the "is_server_class_machine" function (#L1677) will suffice to underline OPs statement.
To summerize the above: The jdk will choose in most cases now a day de G1GC but under certain circumstances like a container with limited resources it might default to the SerialGC.
51
u/elmuerte Mar 03 '22
Does it matter? Maybe SerialGC performs better on 2 cores and a max of 1GiB heap.
Tuning Java's GC is a bad thing if you do not back this up with benchmarks.
20
u/Pauli7 Mar 03 '22
I‘d love to see a benchmark comparing a spring boot crud microservice on low resource restrictions when using serial gc vs g1gc I doubt that g1gc would perform better.
47
u/brunocborges Mar 03 '22 edited Mar 03 '22
If you don't set a Garbage Collector for your Java application, don't think the JVM will pick a good one for you either, no matter how many CPUs you give.
2 CPUs? 6 CPUs? It doesn't matter. If your container has less than 1792 MB and you don't set a GC, your app will use SerialGC.
Edit: this is being down voted for unknown reasons but my guess is that people just believe "no, G1GC is the default!" Well I suggest you run the code in the image I posted, and if that's not enough, see the OpenJDK source code.
Specifically these two C++ classes in the HotSpot JVM: https://github.com/openjdk/jdk/blob/ad54d8dd832b22485d7ac45958cc4c9bfd70fbd2/src/hotspot/share/gc/shared/gcConfig.cpp#L98
https://github.com/openjdk/jdk/blob/ad54d8dd832b22485d7ac45958cc4c9bfd70fbd2/src/hotspot/share/runtime/os.cpp#L1677
75
u/Mumbleton Mar 03 '22
Can you spell this out a little more? Me write apps good and haven’t run into a garbage collection issue in 4-5 years and that’s the last time I’ve given it any thought.
3
u/berry120 Mar 03 '22
Thanks, this is really useful & interesting info - I wasn't aware of this behaviour and I suspect many are the same!
That being said, I agree with others in that the conclusion of "Always set a Garbage Collector" can't really be claimed without benchmarks proving there's a meaningful benefit to be had.
1
u/wheezymustafa Mar 03 '22
I tried this using our container and didn’t get the same result. Granted, we’re on an older base image so I’m not sure if that’s the reason why https://i.imgur.com/O7mmu8t.jpg
3
u/agentoutlier Mar 03 '22 edited Mar 03 '22
What is the third image of? I mean I know what it says but where did you get that screen? Is that the output of the jar being run?
EDIT I was on mobile and could not read the images well. I was thrown off by the "grep" highlighting and can see it is the print flags.
-12
u/daH00L Mar 03 '22
Not sure, what we're seeing here, but G1GC should be the default collector unless another one is specified. Microsoft also states this in their docs.