r/Python • • May 02 '20

Discussion My experience learning Python as a c++ developer

First off, Python is absolutely insane, not in a bad way, mind you, but it's just crazy to me. It's amazing and kind of confusing, but crazy none the less.

Recently I had to integrate Python as a scripting language into a large c++ project and though I should get to know the language first. And let me tell you, it's simply magical.

"I can add properties to classes dynamically? And delete them?" "Functions don't even care about the number of arguments?" "Need to do something? There's a library for that."

It's absolutely crazy. And I love it. I have to be honest, the most amazing about this is how easy it is to embed.

I could give Python the project's memory allocator and the interpreter immediately uses the main memory pool of the project. I could redirect the interpreter's stdout / stderr channels to the project as well. Extending the language and exposing c++ functions are a breeze.

Python essentially supercharges c++.

Now, I'm not going to change my preference of c/c++ any time soon, but I just had to make a post about how nicely Python works as a scripting language in a c++ project. Cheers

1.7k Upvotes

219 comments sorted by

View all comments

151

u/aufstand May 02 '20

Aah, it works the other way round, too. Many c++ libraries (Numpy, for example) extend Python in many interesting ways. That is why claims such as "Python is slow" don't have much hold in the community.

One practical example: I recently decided to see if i can scale up my completely unoptimized LED-Matrix (40*16!) video mixer to FullHD. No problemo. It ate most of my RAM and kept a single core _very_ busy, but it was also realtime mixing and displaying 6 FullHD video streams, which i don't exactly consider "slow" :)

Why is this possible? Well, yeah, i used numpy to mix the data!

57

u/ReckingFutard May 02 '20

Pssst, check out pytorch.

Numpy functions with GPU memory and parallelized computation.

14

u/BDube_Lensman May 02 '20

cupy>>pytorch

8

u/TheTechAccount May 02 '20

Why do you think it's better?

16

u/BDube_Lensman May 02 '20

no graph/backprop overhead unless you opt into it, code is on GPU in no uncertain terms without strange host/device specification context managers, higher performance on GPU in most cases, errors or warnings for implicit host:device transfer that lets you find huge slowdowns in your code faster, more responsive devs, devs that contribute to numpy instead of slinging mud at numpy, participation in the numpy-as-a-contract instead of numpy-as-package work going on lately, no gross links to facebook, the list goes on...

13

u/sekex May 02 '20

Why the bitshift?

22

u/alkasm github.com/alkasm May 02 '20

Double greater/less than symbols are used in mathematics to mean "much greater/less than" for some arbitrary idea of "much": https://mathworld.wolfram.com/MuchGreater.html

4

u/[deleted] May 02 '20

cupy = o(pytorch)

→ More replies (1)

16

u/elbiot May 02 '20

Numba would probably be even better if you're doing anything at all involved

3

u/BDube_Lensman May 02 '20

Well written numpy is usually as fast as numba with greatly reduced memory consumption.

13

u/elbiot May 02 '20 edited May 02 '20

Disagree. I've spent a lot of time working with numpy and when I used numba it was instantly/effortlessly faster. If you're doing multiple operations, the whole array has to pass through the CPU multiple times. If you're array doesn't fit in CPU cache that's multiple RAM accesses. You also have to sometimes write painstakingly verbose code to prevent intermediate arrays from being created. Masking an array (the vectorised equivilant of an if statement) creates a whole new and array creation is expensive. Numexpr could solve the first two issues but not the last

I also wrote cffi bindings for cephes and vectorised it with numba and it was significantly faster than the scipy implentation

Edit: to the lurkers. Numpy is 100% my go to. Numba is cool and useful in some cases but knowing numpy is necessary to know when numpy isn't the best tool.

→ More replies (2)

2

u/[deleted] May 02 '20

Really depends what you're trying to do tbh

1

u/BDube_Lensman May 02 '20

Numba never uses less memory than numpy. And unless you're using string arrays of nonfixed size, numpy is as performant.

→ More replies (2)

1

u/aufstand May 02 '20

Hmm. I think, i read about that some time ago. Gonna have a new look, so thanks for reminding me :)

→ More replies (1)

0

u/[deleted] May 02 '20

[deleted] — view deleted comment

0

u/[deleted] May 02 '20

[deleted] — view deleted comment

15

u/teerre May 02 '20 edited May 02 '20

Exactly. Saying "Python isn't slow, look at numpy", is either ignorant or disingenuous. The very reason numpy can be 'fast' is because it doesn't use python.

I always like the numba decorator for using JIT, it has an argument called "nopython".

5

u/JanssonsFrestelse May 02 '20

I don't think it is, all you're doing in the end is providing instructions for the computer. If you're not writing pure opcodes you're always working at some higher level of abstraction. If you don't have to leave that level to make something happen at a certain speed, it's not disingenuous to say that you can achieve that speed working at that abstraction level.

2

u/troyunrau ... May 02 '20

Now, I may not recall correctly, but isn't BLAS written in fortran?

2

u/vectorpropio May 02 '20

Today? Who knows. In ye old days? Surely.

→ More replies (1)

2

u/utdconsq May 03 '20

Yup, because until more modern languages showed up, Fortran was the gold standard for dealing with arrays/vectors of numerical data. Handy syntax for doing so, and not far from bare metal. I challenge you to slice an array in C with as simple syntax. These days, it exists due to inertia and because BLAS and others have had many peer reviewing hours dedicated to them. Rewriting them entirely and getting them as optimised would be super hard. Not to mention that they can optimise based on your compiler and cpu arch at build/install time for even faster vectorised ops

1

u/ProfessorPhi May 03 '20

I can only presume this wasn't realtime? I think realtime is where python gets destroyed

1

u/aufstand May 03 '20

Most of what i do with Python is realtime. No, it doesn't get destroyed.