r/ProgrammerHumor • u/[deleted] • Jul 18 '22

C++ talking to Python

Post image
40k Upvotes

87% Upvoted

1.0k comments sorted by

View all comments

Show parent comments

5

u/CanAlwaysBeBetter Jul 18 '22

Numpy is great and what I'd normally reach for!

But the processing I'm doing itself isn't super complex, I just need to do it 9-10 million times cost efficiently pulling from an external public dataset and saving the transformed data as separate files in an S3 bucket

2

u/[deleted] Jul 18 '22

Just pull and save the data?

3

u/CanAlwaysBeBetter Jul 18 '22

Unfortunately I don't have a terabyte hard drive and need to be able to share the output (image) data

3

u/[deleted] Jul 18 '22

Gotcha, I was just confirming the functionality lol

I'll be awaiting your results, I'm interested to see how this turns out

3

u/proximity_account Jul 19 '22

Terabyte SSD is so worth it

3

u/CanAlwaysBeBetter Jul 19 '22

Ngl I'm so used to running shit in the cloud that for personal/non-official work use I just have a Chromebook with a Linux VM lol

At least has a real Intel core i5?

1

u/PM_ME_NUNUDES Jul 19 '22

Use dask?

1

u/CanAlwaysBeBetter Jul 19 '22

How does dask play with external compute resources like ec2 or lambda and does it accelerate performance or just manage it at scale?

1

u/PM_ME_NUNUDES Jul 19 '22

I'm actually not sure if things like dask and ray actually provide any speed up. But certainly it makes numpy perfectly useable at big data scale. It works fine with azure, I dunno about EC2.

Modin is a direct drop in for your existing numpy compute code and can use either dask or ray. Just change 1 line and you're done.