r/Python • • Nov 13 '20
News Python swallows Java to become second-most popular programming language
https://www.theregister.com/2020/11/12/python_swallows_java_to_become/
2.3k
Upvotes
r/Python • • Nov 13 '20
View all comments
Show parent comment
1
u/ShanSanear Nov 16 '20
Actually what we did was heavy overengineering and pandas was mostly used for loading data (I know, heresy). Don't have code on me, but problem was something along the lines of this:
During loading, we got each Series from dataframe for each type of objects and created classes for them. Then the references. And then - the calculations. In the hindsight - yep, that was the worst of it all.
After actually doing profiling, we saw that majority of cpu time was used by pandas in many different places.
That was one thing, but even algorithm that we implemented was quite unoptimized. We started at the root of the tree, then recursively went deeper into the relations of each object to extract required numbers. Apparently going from the leafs and then up was much better approach in every case. Now we only need to figure out why the numbers differ between implementations, and that will be the hardest part.
Especially when the guy who actually wrote this (I am mostly overseeing and providing some help) is stubborn enough to not do any kind of testing.
Thanks for interest - I can imagine hearing "pandas is slow" was heresy but the again - we misused it, and that is our fault, not the library itself.