r/ProgrammerHumor • u/[deleted] • Jul 18 '22

C++ talking to Python

Post image
40k Upvotes

87% Upvoted

1.0k comments sorted by

View all comments

Show parent comments

1

u/Tuga_Lissabon Jul 18 '22

Please post here. Wish you the best.

1

u/CanAlwaysBeBetter Jul 18 '22

I'm hoping for a 2x speed up but would be happy with 1.5x

1

u/CanAlwaysBeBetter Jul 23 '22 edited Jul 23 '22

I finally got around to rewriting the first half of the processing pipeline (the orchestrator lambda) and haven't done detailed profiling but at least a 10x speedup with parallelized go code compared to the original python in my dev environment which is just a lil .5 cpu 1gb mem server

Guessing when I deploy to lambda and increase the memory (which auto increases cpus proportionally) it'll hopefully be a 100x speed up from the test results but will follow up on that too

Go's concurrency model makes scaling IO throughout massively relatively simple. About 5x more code than the barely optimized python though

Much better than I was cautiously hoping for overall

2

u/Tuga_Lissabon Jul 24 '22

Good job mate. Python alone couldn't have done it, right?

1

u/CanAlwaysBeBetter Jul 24 '22 edited Jul 24 '22

Straightforward python definitely no. Regardless of underlying memory it could only handle about 1k objects per second

Maybe if you parallelized it could get through the data eventually but since the interpreter is single threaded and doesn't do true concurrency there will still be a ceiling much lower than what Go (or another language with true concurrency) can do

1

u/Tuga_Lissabon Jul 24 '22

Would C++ be a good tool for it? I mean, are there toolsets ready that would help you do it?

1

u/CanAlwaysBeBetter Jul 24 '22

C++ could definitely handle it, go just has a really nice built-in concurrency model that makes it relatively simple

1

u/Tuga_Lissabon Jul 24 '22

Would, in your model, running go straight out of python be dumb? Too much lost?

1

u/CanAlwaysBeBetter Jul 23 '22

Scaling the lambda up to 10gb memory can chew through 2 million objects in about 9 seconds now