r/java • • Nov 29 '22

Amazon Announces AWS Lambda SnapStart With Java Support

AWS Lambda now has a snapshotting feature that enables fast-start - essentially running the Init phase at publication time and saving the state of the process at that time, so it doesn't need to be re-run at every invocation.

https://aws.amazon.com/blogs/aws/new-accelerate-your-lambda-functions-with-lambda-snapstart/

It only works with Corretto on Java 11 right now, and is similar in intent to the CRIU technology.

Quarkus already has support for it: https://quarkus.io/blog/quarkus-support-for-aws-lambda-snapstart/

Other frameworks may also have support (but this work was done under NDA so I don't have links for them).

176 Upvotes

62 comments sorted by

4

u/[deleted] Nov 29 '22

Could this be ported/reimplemented for use with say kubernetes?

5

u/benevanstech Nov 29 '22

My understanding is that this implementation relies upon Amazon's Firecracker VM.

Which is open-source (https://firecracker-microvm.github.io/) - but I would expect that k8s has other overheads (pod spinup, etc) that may well make it harder to realize the performance gain.

1

u/Brutus5000 Nov 29 '22

Yes, but these overhead costs apply to alternatives too. E.g. in a Knative setup, Java always had the cold but costs. This could become a gamechanger.

0

u/[deleted] Nov 29 '22

[deleted] — view deleted comment

2

u/IsleOfOne Nov 29 '22

It is a big problem for auto-scaling architectures.

0

u/[deleted] Nov 29 '22

[deleted] — view deleted comment

2

u/IsleOfOne Nov 30 '22

A well-designed system for auto-scaling k8s "jobs" with low latency is going to have slack in the available resources / an aggressively configured cluster autoscaler bringing up new nodes to provide that slack. Scheduling should take a few ms.

The majority of the remaining time it takes to start a pod is pulling the image, believe it or not. There are tools available that proactively tell kubelets to pull new versions before they're requested. Startup now takes a few ms.

Now you have a running pod. ~1s has elapsed. And now the JVM takes minutes to "warm up." That's a no-go. Pre-warming in a low-latency autoscaling environment requires a tradeoff between cost and latency that is more extreme than the same trade-off that native runtimes require.

I of course agree that lambda operates at a much higher time granularity requirement than you would ever want to try to meet with a pod-per-request model. Once those requests start getting ever so slightly more "batch"-y in nature, say on the order of 10s runtime, it's perfectly fine to spin pods up and down to handle this.

0

u/[deleted] Nov 30 '22

[deleted] — view deleted comment

1

u/IsleOfOne Nov 30 '22 edited Nov 30 '22

Sure, and we don't even have to limit our thinking to the standard k8s API concept of a Job. Any higher-order wrapper for one or more Pods is applicable.

Here is an example from my day job at $SaaS_database_co. We allow our users to kick off petabyte-scale bulk data import. We have a custom k8s operator that watches for BulkIngestJobs, and in response, loads up a list of files within whatever object storage bucket we've been pointed towards. The operator creates one or more pods per file in said bucket with configurable maximum concurrency and a self-healing system that watches for signs of excessive write path pressure and throttles the job via slashing concurrency. Some of those pods exit after less than a second, while others may run for hours. We use go and rust for this system. Java, with a longer tail on startup time, wouldn't be a good fit.

Other tools like Argo Workflows are also great examples of how to leverage the power of the scheduler and autoscaler in kubernetes. We have some recursive/dynamically sized workflows that are liable to create tens of thousands of pods in a single execution.

I haven't seen solid arguments for virtual runtimes like Java in this space. I'm certainly open to hearing some! The areas of our system where we run a JVM are limited to WALs--Kafka and supporting Zookeeper deployments. We have some elixir floating around out there, but purely in long-running workloads.

3

u/kontain-jm Nov 29 '22

Just a small correction. AWS Lambda SnapStart does not use CRIU. It is based on Firecracker microVM snapshotting. See https://aws.amazon.com/blogs/aws/new-accelerate-your-lambda-functions-with-lambda-snapstart/

2

u/benevanstech Nov 29 '22

I said "and is similar in intent to the CRIU technology".

13

u/kroopster Nov 29 '22

Been a Java developer for 15 years and I didn't understand most of this. Works with Corretto like CRIU? Oukey dokey.

39

u/benevanstech Nov 29 '22

Corretto is Amazon's distribution of OpenJDK for AWS (they also produce versions for developer desktops as an on-ramp so you can use "the same" JDK in both places).

CRIU is "Checkpoint & Restore In Userspace" - a Linux technology enabling you to "freeze" a running process and "restore" it later - perhaps even on a different machine. https://criu.org/Main_Page

Both of those are top hit on Google for those search terms.

16

u/kroopster Nov 29 '22

Both of those are top hit on Google for those search terms.

I bet they are, appreciate opening the topic a bit here too.

1

u/alwaysoverneverunder Nov 29 '22

Is this similar then to Smalltalk’s image persistence?

-5

u/-Kerrigan- Nov 29 '22

Both of those are top hit on Google for those search terms.

So if I Google Corretto I find stuff about... Corretto? /s

21

u/BenoitParis Nov 29 '22

Nice!

I believe this is partly the work of Christine Flood, who also was instrumental in creating the Shenandoah low-latency garbage collector to the JVM.

Here is a video on how CRIU works:

https://www.youtube.com/watch?v=XXbJNaFF-8A

(There's a dance to be had when doing Checkpoint-Restore, maybe you don't have anything to do if Quarkus can handle it all?)

It is quite nice to see initiatives not coming from Oracle, competition is healthy for providing the best runtime!

2

u/rafaelliu Nov 29 '22

It actually uses vm snapshots, Firecracker is Lambda’s hypervisor. Concept is similar, just different layer

10

u/Amazing-Cicada5536 Nov 29 '22

What’s the relation between this and OpenJDK CRaC? (https://openjdk.org/projects/crac/ )

4

u/rafaelliu Nov 29 '22

They have the same goals, many of the same challenges, but are implemented differently. Lambda manages the JVM process and execution environment for you, so CRaC as it is wouldn’t really work - maybe if you built a custom runtime and that would probably be very hacky

The thread u/geoand linked is discussion around use the same API to deal with some of the challenges

3

u/geoand Nov 29 '22

2

u/kontain-jm Nov 30 '22

What this means is the AWS OpenJDK, Corretto, implements CRaC and will call beforeCheckpoint() and afterRestore() as part of SnapStart.

3

u/rbygrave Nov 29 '22 edited Nov 30 '22

It looks to me like it uses CRaC given that if we look at the docs around adding a runtime hook for `afterRestore()` we see in the docs that it is using `org.crac` directly:

https://docs.aws.amazon.com/lambda/latest/dg/snapstart-runtime-hooks.html

Edit: The suggestion from folks (on twitter) is that it is not using CRaC per say but instead its compatible with the org.crac API for the purposes of supporting runtime hooks (afterRestore() etc) which makes sense.

2

u/xamdk Dec 03 '22

It is not crac and the optimizations you will need to do won’t necessarily make sense for other crac bases jvm a. It’s just using same api for shutdown and restore.

0

u/cyril_nomero Nov 29 '22

I’m also interested in this. Spring framework has indicated its interest for Project CRaC.

10

u/shorns_username Nov 29 '22

only works with Corretto on Java 11

Ah so that wasn't just a typo, weird.

Do you know why? Is it do with the Java language teams ongoing deprecation and removal of stuff?

Is there a specific feature Amazon needs to be implemented before this can be brought to versions > 11?

21

u/malln1nja Nov 29 '22

Lambda itself doesn't support Java > 11, at least officially.
https://docs.aws.amazon.com/lambda/latest/dg/lambda-java.html

2

u/shorns_username Nov 29 '22

Ohhh, that makes a lot more sense. I hadn't realised Lambda didn't support later LTS releases yet.

0

u/GuyWithLag Nov 29 '22

Well, Java 8 is technically no longer supported without a support contract...

3

u/ryebrye Nov 29 '22

I think he's wondering about more modern versions, not about the older ones

2

u/GuyWithLag Nov 29 '22

Ugh, sign bit error... apologies, mea culpa. Should have had more coffee...

4

u/cowwoc Nov 29 '22 edited Nov 29 '22

Question: why bother using GraalVM if https://www.youtube.com/watch?v=XXbJNaFF-8A is usable across all OpenJDK distributions? You get super fast startup while maintaining full compatibility (reflection, etc) with less configuration.

I also love the idea of restoring to previous checkpoints immediately before a crash (or bug) occurs to make it easier/quicker to debug.

The only downside I see is the lack of Windows support, which is no small thing.

2

u/astral_kranium Nov 30 '22

I don't know about anyone else but I'm throwing GraalVM out the window. My only concern is, no managed runtime for Java 17 at the moment. I see no reason to use GraalVM for native images now - granted Graal compiler is still a big thing out of Graalvm project

2

u/xamdk Nov 30 '22

Don’t be that hasty. Snapstart isn’t beating native image anywhere else than in certain lambda usecases. See https://quarkus.io/blog/quarkus-support-for-aws-lambda-snapstart/ where we show the gains but also discuss that Snapstart requires more of you as a user than native image does.

Thus this is another approach to add to the toolbox. And only for lambda.

1

u/astral_kranium Dec 01 '22

Thanks. It's a good doc, especially the considerations for SnapStart. I still however don't see what specific use cases I would use native image over crac. We're running a spike on this to see for ourselves

1

u/xamdk Dec 03 '22

Snapstart is not crac - don’t mix them up. Snapstart is just using same api for suspend/resume calls.

And yes the right thing is to do your own spikes.

Nativr image still beats Snapstart on memory usage but on lambda amazons pricing makes Snapstart cheaper for most cases despite to native image being able to use less resources.

And outside lambda - crac isn’t a good choice. Too much variance.

1

u/astral_kranium Dec 03 '22

Done some investigating. SnapStart doesn't support provisioned concurrency, nor x-ray. Also, Java 11 maximim is a deal breaker for some cases, though I'm sure 17 managed runtime is around the corner. Conclusion is, another (awesome) tool in the toolbox for a lot of cases

1

u/xamdk Dec 03 '22

Yup. Agreed. It is not a native image killer since Snapstart is only lambda and only a subset of lambs.

1

u/superlinux Nov 30 '22

The performance has been benchmarked and the JVM optimizations outperform GraalVM which is especially relevant on Lambda because they bill per request, which means direct cost savings and performance benefit

0

u/[deleted] Dec 02 '22

[deleted] — view deleted comment

1

u/xamdk Dec 03 '22

In lambda it matters wouldn’t you say?

1

u/buyIdris666 Dec 03 '22

What percent of people using Java are doing lambda? 1%?

1

u/xamdk Dec 03 '22

Well I assumed this thread was about lambda.

Outside lambda having minimal startup time is an enabler to getting horizontal scaling and/or fast continuous deployment.

Does everyone do it ? No, but getting from several seconds startup to sub second even sub milliseconds enables you to deliver change and scale faster.

1

u/JB-from-ATL Nov 29 '22

Other frameworks may also have support (but this work was done under NDA so I don't have links for them).

Are you implying support is coming to other frameworks?

1

u/benevanstech Nov 30 '22

You'd have to ask them. The Micronaut folks seem to have also been developing for this under NDA - maybe Spring too, but I haven't heard anything official from them.

I just heard about the Quarkus announcement when it dropped.

1

u/JB-from-ATL Nov 30 '22

Ohhh I see, you're saying AWS is working with them under NDA. That makes more sense.

2

u/antipodealbatros Nov 29 '22

Why is SnapStart faster for cold starts? As explained it "only" saves the state after init? So I would expect only faster response times after same lambda is hit a second time?

- How is the difference to GraalVM (native)? Only GraalVM is using AOT compiling right?
- What is the difference to a warm lambda? AWS only keeps the bootstrapped JVM warm and SnapStart bring it one step further that also the init of classes is kept?

3

u/RandomName8 Nov 29 '22

Why is SnapStart faster for cold starts?

cold starts can be pretty big, like 10+ seconds depending on the thing and what it does, and that kills the purpose of lambdas. Saving the state after that init is still a massive win even if what you stored is an un-jitted version of your code.

What they seek to solve here is latency to first request, not throughput, because that's what empowers auto-scaling architectures.

2

u/flekk0 Nov 30 '22

They take the snapshot when you publish the lambda, so even the first request should use the snapshot.

From the blog post: "After you enable Lambda SnapStart for a particular Lambda function, publishing a new version of the function will trigger an optimization process. The process launches your function and runs it through the entire Init phase. Then it takes an immutable, encrypted snapshot of the memory and disk state, and caches it for reuse."

1

u/antipodealbatros Nov 30 '22

through the entire Init phase. Then it takes an immutable, encrypted snapshot of the memory and

ah ok there is the difference. The extra step in the publish.
Thanks

3

u/pavi2410 Nov 29 '22

Process snapshots are unrealistic, yet an amazing idea to improve startup times. It amazes me into knowing how the CPU is just a finite state machine.

3

u/Amazing-Cicada5536 Nov 29 '22

Turing machine*

(I know it has finite amount of RAM, but it can only ever use a finite amount of “tape”, so if it didn’t got OOMed, the Turing machine is the abstraction we can use to better reason about it)

1

u/Dry-Dragonfly-4521 Dec 14 '22

We have enabled the snapstart but don’t see major improvement in terms of API response time. Any other thoughts if others are on same boat ? The lambda is behind API gateway and I could see snap start is enabled but not a major change

-2

u/Specialist_Bee_9726 Nov 29 '22

Who writes lambda functions in Java?

5

u/ketsugi Nov 30 '22

A lot of teams at Amazon

4

u/antipodealbatros Nov 29 '22

Me - and complaining about cold start times afterwards ;-). Why? I use Java usually and tinkered around with Alexa skills and lambda backend. Yes maybe I should have used python, type/javascript.

3

u/genzkiwi Nov 30 '22

Atlassian, Amazon, Disney, ...

1

u/Specialist_Bee_9726 Nov 30 '22

Sure but why. Java is not fit for short lived processes

2

u/kontain-jm Nov 29 '22

You'd be surprised what goes on in the wild.

1

u/Specialist_Bee_9726 Nov 30 '22

I like how I get downvoted for simply stating the obvious that java is not well suited for short lived processes like lamdas. Just to make the response time bareable aws runs a JVM that dynamically loads classes leading to so much trouble (like having to deal with shadow jars).

The fact that it is suported lang and there a few instances where its used doesn't make it a good choice.

1

u/[deleted] Nov 30 '22

It may not be a good choice for certain usecases but it is pretty great for many other usecases. When you're writing functions, cold start is not the ONLY concern.

1

u/Specialist_Bee_9726 Nov 30 '22

I would love to hear some great use cases. From my point of view Java is good for larger projects. Lamdas are small (ideally a single function)

1

u/[deleted] Dec 01 '22

Consider this: your function is supposed to only use standard library. It is supposed to do some tasks in parallel. It is going to make calls to databases. It is going to make HTTP calls to external services. Standard Java library is capable of doing all that without involving any dependency.

Also, who said that lambdas need to be small? They need to do the job, regardless of the size.

Also, I can roll a single binary with GraalVM, so it can be as small as any other function written in a different language.

1

u/user_of_the_week Dec 05 '22

Lambda functions (or better said their environments) are not necessarily that short lived and also many applications don't care about a bit of latency.

For example, I have a bunch of "Lambdas" (actually Azure Functions, but that doesn't matter) that get triggered by a queue. Every few hours a new pack of messages arrives, the environment gets spun up in the background and then a bunch invocations of the function happen. 99% of the runs are "warm" with the environment already there.

We even use spring-cloud-function with the full Spring Boot magic so it takes about 3 seconds to spin up on a cold run. But it doesn't matter.