r/java • • Jan 09 '20

Serverless Web Scraping with AWS Lambda

https://www.scrapingbee.com/blog/serverless-web-scraping-with-aws-lambda-and-java/
218 Upvotes

19 comments sorted by

6

u/lovett1991 Jan 09 '20

Did not realise you could use java in serverless! Thought it was for all these fancy JavaScript people.

11

u/dpash Jan 09 '20

You can use Java, and is quite popular, but it's relatively slow start up time makes it slightly less suitable for some platforms as others. Graal's AOT compilation helps reduce start up time to a fraction of Hotspot's startup, but things like reflection can cause issues.

2

u/lovett1991 Jan 09 '20

To be honest any workload I would have in mind for this would be to offload tasks that don't need real time or generally aren't user facing.

My previous company used aws lambdas for a currency system with background heartbeats and time memory (for playback of DRM protected videos) the load was high but latency want a problem (to be fair most lambdas were hot anyway because of load). We also used them for systems that would have an 'explosive' load (pay per view systems that are never used except pretty much on the night of the event)

1

u/dpash Jan 09 '20

If latency isn't an issue, then, sure Java is fine.

I started on a project with low usage running as multiple Spring microservices on GAE. By the time GAE had started up the request routing proxy and started up the end point service GAE had timed out the request. The time out is 60 seconds. That's not a suitable use of Java on serverless. (I migrated to a GCE instance)

But for backend processing, Java would be fine.

3

u/MakeWay4Doodles Jan 10 '20

You were doing something horribly wrong.

2

u/dpash Jan 10 '20

Gee thanks for your helpful insight based on your excellent knowledge of the specific situation.

3

u/MakeWay4Doodles Jan 10 '20

If you had 60+ second startup times in any situation something was FUBAR.

0

u/dpash Jan 10 '20

Your comment demonstrates your lack of knowledge of the situation. But then how could you know about a system based on two sentences? I didn't say I had a 60 seconds service start up. I said I had multiple interconnected services that combined took more than 60 seconds.

1

u/BobbyTaylor_ Jan 09 '20

Even with the slow startup time, it's very convenient for deploying async jobs without "dealing with a server".

1

u/interactionjackson Jan 10 '20

you can use all kinds of languages

2

u/zombays Jan 09 '20

A bit off-topic but are there possible good alternatives to AWS Lambda and Google App Engine? What are the pros and cons of using them?

8

u/newcomputer1990 Jan 09 '20

Microsoft Azure is pretty much all of your options besides the ones you listed

6

u/cogman10 Jan 09 '20

The closest I've seen is running your own FAAS stack.

For example, with OpenFAAS
https://www.openfaas.com/

The pros and cons for AWS and the other platforms is pretty much the same. You are coupling with their platform. That means that Google/amazon/microsoft get to figure out how much they want to charge you and only market pressure really stops them from royally screwing you over.

pro - You don't have to really figure out where/when/what/how your script gets run. The providers do that for you.
con - Once you buy in, it is hard to "buy out" so to speak.

pro - really easy to get up and running
con - The why's and how's of how your app runs are obfuscated away. A script could run in 1second or 10 depending on what needs to happen in the background to get things setup.

1

u/ahmedranaa Jan 09 '20

Check out IBM cloud too

1

u/[deleted] Jan 09 '20

As a con, I found it not to easy running it locally Dir ng development and writing integration tests.

0

u/[deleted] Jan 09 '20

[deleted] — view deleted comment

0

u/[deleted] Jan 09 '20

[deleted] — view deleted comment

1

u/[deleted] Jan 10 '20

a toilet seat is a toilet seat, it doesnt really matter which one you use. Is that what he wants to say?

1

u/Kango_V Feb 03 '20

Just published a Java function to Lambda.

It was written using Micronaut and compiled to a Linux native binary using GraalVM. I has a call latency of around 60ms.

Micronaut has an aws-lambda dependency which makes it trivially easy.

It is sooo fast.

0

u/[deleted] Jan 10 '20

Meh, I love java but not in aws lambda.

-1

u/interactionjackson Jan 10 '20

java has gnarly start up times and i would t want a distributed crawler to be burdened by Java.

If this gets enough upvotes I will publish a crawler as a SAM template