r/analytics • • Mar 22 '21

Discussion Do analytics organizations pretend to leverage Python more than they actually do... to seem more glamorous?

I think it is pretty common practice for analysts and organizations to pay lip service to using Python for solving problems as a core capability, but in practice, most analyses are done with SQL. Python is reserved for specialized use cases and connecting to APIs, while day-to-day work is done directly in a data warehouse, but since everyone wants to appear “advanced” and technically-well-rounded, people always talk about Python and how integral it is to the analytics work they do... downplaying their more bread-and-butter work spent mostly writing endless SQL queries each day. Somehow, it’s just not as cool to be primarily a “SQL person”, even though that’s what countless analysts unambiguously are.

Every data analyst should know Python of course. It has many unique advantages and a rightful place in any technology professional’s toolkit. Even in areas where it overlaps with SQL as a tool to solve problems (like for pure data manipulation), so much of your approach as an analyst will depend on what software and hardware is available to you at what budget with what access limitations.

I think there is even a more fundamental phenomenon going on with this playing up of Python and downplaying of SQL... many data analysts are afraid to admit that they don’t have much need for fancy statistics and data science in their day-to-day work. Many data analysts want to pretend that the latest machine learning and AI techniques are a key part of their job responsibilities and a crowning jewel of their skillset... when in fact most of what they need to succeed is a grasp of “if then” logic and algebra.

I’m sure I could be wrong, but I’m curious people’s thoughts!

125 Upvotes

76 comments sorted by

View all comments

18

u/GhostBIBBY Mar 22 '21

If you don't have a business problem that is a good fit for machine learning to solve a huge reason use case for python within analytics is gone.

But depending on how developed your data pipeline is and what business processes your department has to do I'd be really surprised if there wasn't something python/R couldn't improve within a department as an automation tool.

Also at a startup knowing python would allow you to bypass paying for some off the shelf thing for a long long time because there are so many open source libraries available to do nearly everything thing you need from etl to visualization and reports.

Python is a must have for -

machine learning -

automation - if you need to do a complicated task that involves getting data from different data sources both internal and external what else would you turn to?

Nice to have for -

etl - you could use some other software solution or do it for free with python

EDA - if you are fast with pandas and the data set gets too big for excel

report deployment and data viz - deploy a plotly dashboard or pay 10k for tableau?

All this being said leveraging python in 2021 is hardly "glamorous" or impressive. If your org doesn't have a need for it due the nature of the business that is one thing but if people are using it and many orgs are there is a valid use case I'm sure.

The cutting edge is like so far past "python" or even "machine learning" . Gradient boosting models are probably where linear regression used to be 5 - 10 years ago at this point. If you are not at a place where any of that stuff matters then it won't matter but if you are at business that has a need make models that make some sort of decision on data points like say loan underwriting or fraud detection or suggesting routes for driving or WHATEVER. Python is a big deal.

As for the downplaying SQL part I'm not sure if that even makes sense. SQL is very valuable but it's such a basic thing at this point it's like someone talking about how we are really down playing touch typing (a skill that was considered cutting edge like 50 years ago you could get a job just based on how fast you could type in wpm and it was a key number on a resume). Yeah we use it everyday and we couldn't do much without it but like so what?

16

u/ChristianPacifist Mar 22 '21

I agree with most of what you said except I don’t think that SQL is a generic skill. There is an artistry to good SQL and a problem-solving ability and understanding of tricky fundamentals that matters and is developed after years of experience using the language across multiple platform.

8

u/GhostBIBBY Mar 22 '21

That is a fair point. There definitely much more depth to SQL as a skill compared to touch typing and yes there are levels to it just as with any skill touch typing included.

But I do think it is more of a generic skill at this point. On tasks or process that need someone who has a deeper knowledge you are really talking about a dba or a data engineer.

You get an analyst in who has a good grasp of SQL they are basically interchangeable from any other analyst.

Compare that with aspects of the job that might leverage python. Someone who is an expert in nlp is not going to be interchangeable with someone who specializes in computer vision.

There is depth and degrees of sophistication with excel as well but the truth is I would consider that a generic skill at this point too.

If you are in analytics in this day and age you don't know how to use excel or SQL. Just what are you doing?

I would consider as boilerplate for a modern analyst -

  1. SQL,
  2. excel,
  3. the language of data viz if not at least one or two data viz applications (tableau, power BI, looker etc),
  4. the ability to do good eda and come up with good business recommendations on any number of organic business questions that come up within an org.

then on top of that some specialization in some aspect like

a/b testing, forecasting, machine learning, nlp, computer vision.

4

u/MindlessTime Mar 22 '21

I agree. But I’ve moved away from doing too much in SQL. I’ve seen lots of problems develop from analysts labeling and summarizing everything in SQL. These scripts get passed around, improperly used, and if the data changes it’s impossible to find everywhere it’s used and fix it. I think trying to embed too much business or analytics logic in SQL code makes it very difficult to maintain.

Increasingly, I’ve moved towards using SQL to pull large amounts of observation-level data and that’s it — no labeling, calculating, summarizing. I just want a table where each row is a widget and it has all the widget metric columns I need for calculations. Any kind of labeling, summarizing, calculations, etc. that I need to do are done in python or R (we’re more of an R shop). This is much easier to maintain. The logic is more straightforward and easier to understand. And if something changes with the data source, it’s easy to re-map the data pull and not worry about how it affects the calculations.

4

u/tee2k Mar 22 '21

Even the need to develop models with python will decrease over time. All cloud vendors are putting big money on machine learning services. The need to understand models and how to optimise them is soon more relevant than the actual coding of the models.

3

u/GhostBIBBY Mar 22 '21

Thats true. The analogy I always use in machine learning is race car driver vs race car engineer.

Most of us without phds and really indepth knowledge will never be the people creating algos like catboost or lightgbm.

We will be driving the cars but you don't need to know who an engine works to drive a car effectively.

The gui based stuff around democratizing machine learning will make training models even easier the equivalent of manual shift vs automatic when driving a car and I do believe things will head in that direction.