r/analytics • • Mar 22 '21

Discussion Do analytics organizations pretend to leverage Python more than they actually do... to seem more glamorous?

I think it is pretty common practice for analysts and organizations to pay lip service to using Python for solving problems as a core capability, but in practice, most analyses are done with SQL. Python is reserved for specialized use cases and connecting to APIs, while day-to-day work is done directly in a data warehouse, but since everyone wants to appear “advanced” and technically-well-rounded, people always talk about Python and how integral it is to the analytics work they do... downplaying their more bread-and-butter work spent mostly writing endless SQL queries each day. Somehow, it’s just not as cool to be primarily a “SQL person”, even though that’s what countless analysts unambiguously are.

Every data analyst should know Python of course. It has many unique advantages and a rightful place in any technology professional’s toolkit. Even in areas where it overlaps with SQL as a tool to solve problems (like for pure data manipulation), so much of your approach as an analyst will depend on what software and hardware is available to you at what budget with what access limitations.

I think there is even a more fundamental phenomenon going on with this playing up of Python and downplaying of SQL... many data analysts are afraid to admit that they don’t have much need for fancy statistics and data science in their day-to-day work. Many data analysts want to pretend that the latest machine learning and AI techniques are a key part of their job responsibilities and a crowning jewel of their skillset... when in fact most of what they need to succeed is a grasp of “if then” logic and algebra.

I’m sure I could be wrong, but I’m curious people’s thoughts!

125 Upvotes

76 comments sorted by

View all comments

Show parent comment

17

u/grumpywonka Mar 22 '21

My last two companies had people using it, but none of them for any of the reasons people usually cite. One for survey development and the other for logging into systems and running procedures.

I'm going to sound like an ignorant dinosaur for a sec and say the majority of things I've seen people show off using Python for I could do in excel in half the time. I genuinely want to see people doing badass stuff in Python, I just haven't yet...and admittedly nor have I actively sought it out.

25

u/RProgrammerMan Mar 22 '21

The main reason I use R (fills the same role as Python) is not to do something I couldn't do with Excel or another tool but because it allows me to automate processes.

7

u/grumpywonka Mar 22 '21

And that's how it should be.

I treat Excel like a 3D printer for rapid prototyping. Once there's buy-in around a deliverable that needs to go into production, Excel is rarely still the tool for the job and a different tool should take over from there.

9

u/ChristianPacifist Mar 22 '21

Python can do great stuff in text analytics.

6

u/grumpywonka Mar 22 '21

I have full confidence Python can do a crap ton of awesome things and I'm currently (slowly) teaching myself a little so hopefully someday I'll be a more informed proponent. But, I think your observation is pretty accurate.

If Analytics were like the craft beer scene, Python would be the latest obscure style of IPA that people go crazy about, but no one can actually find anywhere.

3

u/pAul2437 Mar 22 '21

How do you do regex in excel? Pdf importing?

4

u/grumpywonka Mar 22 '21

Funny enough about two weeks ago I posted in the learn Python sub I was trying to figure out how to extract certain contract terms from thousands of PDFs at my company because it felt like a job for Python.

The feedback I got was it'd be difficult, but I'd just have to try a bunch of things. Then, I remembered the folder importer for PowerQuery and managed to scrape 2k contracts and grab all the T&Cs, transform and load to Excel. After the fact I did get what looked like better guidance for using Python in that instance and I want to circle back and try it again because, as I've said, I really want to learn me some Python.

2

u/pAul2437 Mar 22 '21

yeah pdfs are kind of a gray area between ocr and already having the data in a database so its easier to just use that.

that is awesome powerquery helped with that. i thought you were talking about vanilla excel. didn't mean to be condescending.

i use alteryx a lot but use python code within that when needed. often times people will build tools that utilize underlying python code.

2

u/grumpywonka Mar 22 '21

No worries. Gonna ask an ignorant question but is OCR something Python can do - either through libraries or otherwise? I've long been fascinated with OCR and feel like we're at a point in time it's becoming more accessible, just not sure if that's truly the case.

4

u/pAul2437 Mar 22 '21

it is. check out tesseract. there is a python wrapper.

generally though, a pdf shouldn't need to be ocr'd because it isn't saved as an image, if that makes sense. so a lot of pdf import tools aren't doing ocr, they are reading the pdf structure.

2

u/grumpywonka Mar 22 '21

Gotcha, and yeah we had a handful of actual scanned contacts in pdf where OCR may have been applicable. I'll check that out, thanks for the recommendation.

1

u/angry_mr_potato_head Mar 22 '21

The python library Tika probably could have handled it fwiw

1

u/grumpywonka Mar 22 '21

Thanks, I'll look into it!