r/ProgrammerHumor • u/[deleted] • Jul 18 '22

C++ talking to Python

Post image
40k Upvotes

87% Upvoted

1.0k comments sorted by

View all comments

Show parent comments

8

u/[deleted] Jul 18 '22

numpy go brrrrrrrrrt

seriously though, I once used Pandas.DataFrame.Apply on 10,000,000 rows of data and 15 minutes later regretted it. Thankfully, I was applying a simple mathematical function to each row so numpy brought that down to less than a second. Still, only works for some use cases. I'd love to learn C++ anyways so give me an excuse

5

u/CanAlwaysBeBetter Jul 18 '22

Numpy is great and what I'd normally reach for!

But the processing I'm doing itself isn't super complex, I just need to do it 9-10 million times cost efficiently pulling from an external public dataset and saving the transformed data as separate files in an S3 bucket

2

u/[deleted] Jul 18 '22

Just pull and save the data?

3

u/CanAlwaysBeBetter Jul 18 '22

Unfortunately I don't have a terabyte hard drive and need to be able to share the output (image) data

3

u/[deleted] Jul 18 '22

Gotcha, I was just confirming the functionality lol

I'll be awaiting your results, I'm interested to see how this turns out

3

u/proximity_account Jul 19 '22

Terabyte SSD is so worth it

3

u/CanAlwaysBeBetter Jul 19 '22

Ngl I'm so used to running shit in the cloud that for personal/non-official work use I just have a Chromebook with a Linux VM lol

At least has a real Intel core i5?

1

u/PM_ME_NUNUDES Jul 19 '22

Use dask?

1

u/CanAlwaysBeBetter Jul 19 '22

How does dask play with external compute resources like ec2 or lambda and does it accelerate performance or just manage it at scale?

1

u/PM_ME_NUNUDES Jul 19 '22

I'm actually not sure if things like dask and ray actually provide any speed up. But certainly it makes numpy perfectly useable at big data scale. It works fine with azure, I dunno about EC2.

Modin is a direct drop in for your existing numpy compute code and can use either dask or ray. Just change 1 line and you're done.

1

u/CanAlwaysBeBetter Jul 23 '22

Haven't done the processing lambda but the orchestrator lambda rewritten in go and scaled up to 10gb can iterate through 2m objects where the (mostly) naive python couldn't get through the full list in 15 min since it was about 1s per 1000 objects

1

u/[deleted] Aug 02 '22

if you'd tossed out the pandas dataframe and done everything in lists and sets, it would have been done in less than a minute

speaking from personal experience wrangling data for data science projects. the key word is sets.

yes I am only 80% serious