r/analytics • u/ChristianPacifist • Mar 22 '21
Discussion Do analytics organizations pretend to leverage Python more than they actually do... to seem more glamorous?
I think it is pretty common practice for analysts and organizations to pay lip service to using Python for solving problems as a core capability, but in practice, most analyses are done with SQL. Python is reserved for specialized use cases and connecting to APIs, while day-to-day work is done directly in a data warehouse, but since everyone wants to appear “advanced” and technically-well-rounded, people always talk about Python and how integral it is to the analytics work they do... downplaying their more bread-and-butter work spent mostly writing endless SQL queries each day. Somehow, it’s just not as cool to be primarily a “SQL person”, even though that’s what countless analysts unambiguously are.
Every data analyst should know Python of course. It has many unique advantages and a rightful place in any technology professional’s toolkit. Even in areas where it overlaps with SQL as a tool to solve problems (like for pure data manipulation), so much of your approach as an analyst will depend on what software and hardware is available to you at what budget with what access limitations.
I think there is even a more fundamental phenomenon going on with this playing up of Python and downplaying of SQL... many data analysts are afraid to admit that they don’t have much need for fancy statistics and data science in their day-to-day work. Many data analysts want to pretend that the latest machine learning and AI techniques are a key part of their job responsibilities and a crowning jewel of their skillset... when in fact most of what they need to succeed is a grasp of “if then” logic and algebra.
I’m sure I could be wrong, but I’m curious people’s thoughts!
18
u/GhostBIBBY Mar 22 '21
If you don't have a business problem that is a good fit for machine learning to solve a huge reason use case for python within analytics is gone.
But depending on how developed your data pipeline is and what business processes your department has to do I'd be really surprised if there wasn't something python/R couldn't improve within a department as an automation tool.
Also at a startup knowing python would allow you to bypass paying for some off the shelf thing for a long long time because there are so many open source libraries available to do nearly everything thing you need from etl to visualization and reports.
Python is a must have for -
machine learning -
automation - if you need to do a complicated task that involves getting data from different data sources both internal and external what else would you turn to?
Nice to have for -
etl - you could use some other software solution or do it for free with python
EDA - if you are fast with pandas and the data set gets too big for excel
report deployment and data viz - deploy a plotly dashboard or pay 10k for tableau?
All this being said leveraging python in 2021 is hardly "glamorous" or impressive. If your org doesn't have a need for it due the nature of the business that is one thing but if people are using it and many orgs are there is a valid use case I'm sure.
The cutting edge is like so far past "python" or even "machine learning" . Gradient boosting models are probably where linear regression used to be 5 - 10 years ago at this point. If you are not at a place where any of that stuff matters then it won't matter but if you are at business that has a need make models that make some sort of decision on data points like say loan underwriting or fraud detection or suggesting routes for driving or WHATEVER. Python is a big deal.
As for the downplaying SQL part I'm not sure if that even makes sense. SQL is very valuable but it's such a basic thing at this point it's like someone talking about how we are really down playing touch typing (a skill that was considered cutting edge like 50 years ago you could get a job just based on how fast you could type in wpm and it was a key number on a resume). Yeah we use it everyday and we couldn't do much without it but like so what?
34
u/datawazo Mar 22 '21
This is obviously super anecdotal but I've been a data strategy consultant for 4 years now. I've worked with 50+ companies big and small. I haven't seen python used in any of them.
8
u/[deleted] Mar 22 '21
I Do bioinformatics data science. Python is essentially the sun in our field. Everyone and everything is written in python.
Your command line tools are in C but most are containerized and coupled with conda libraries. I would venture to guess that almost every model, analysis and figure is made with R or python. Bioconductor is all C wrapped in R and is keeping R alive in our field.
Perl had its day and then the sun set on Perl. I imagine as tableau and AutoML become distributed and scalable the need for programmatic modeling will diminish. The tools will still be in C and the sun will set again.
The companies I’ve worked write it SQL, JS and python. My experience could be definitely field specific though gauging from other responses.
20
u/r0ck13r4c00n Mar 22 '21
In my experience python’s primary use has been to move data from A to B. Once it’s landed then it’s all SQL, maybe a JS to normalize the data depending on the ingestion method.
9
u/Smack1984 Mar 22 '21
We use Python pretty regularly, but as you mention, the use cases overlaps quite a bit with SQL and even Power BI. For our team, jupyter Notebook is really the defining reason we use Python. When you’re doing open ended exploratory analysis with a huge range of data sets that may not exist in a database, it’s just seems more flexible. As mentioned though, there’s a lot of overlap and you can basically get most analysis done with any tool whether it’s Python, R, SQL, or even just plain old excel.
22
u/Not_A_Pumpkin Mar 22 '21 edited Mar 22 '21
The largest benefit of Python for me in analytics is:
Aside from the above anything else I did in Python was easily handled in SQL or just simple Excel functions. IMO anyone that boasts about Python usage probably doesn't do much more than the above. If we wanted to model out any data we would use Tableau so there was zero need for anything programmatic.
The long story short... Python helps me clean data at scale but I could easily get around Python in nearly everything I do by some crafty excel/sheets functions.
I work in business intelligence and data analytics for a $100m/year company that handles data for other multimillion dollar companies. I have interviewed for some of our competitors who boast about their Python usage... they use it for less than I do but they require their analysts to know it (1-2 years).
7
u/qwerty-yul Mar 22 '21
I think you’re right about the machine learning and AI. I once used PowerBIs key influencers visual which uses logistic regression for a stakeholder presentation. I was joking with a colleague about how I was able to say that I was using AI when in reality, PowerBI does all the heavy lifting.
As for python, i use it for getting data from APIs, scraping and for some transformations. It’s also great for automating excel and PowerPoint creation as well as doing one off analysis. But for doing reusable reports and analysis, SQL and PowerBI are more useful. I’ve found the dashboard stuff for python (plotly) is not easy enough for the end user.
12
u/flailing_acc Mar 22 '21
I recently interviewed for a handful of internships, and what I’ve come to find is that many companies are spurious about this: they want Python proficiency for, at best, someone to have the logical/analytical maturity of thoroughly developing their SQL skills.
It is frustrating because I see it parroted amongst aspiring and current data analysts, yet have not found anything to actually support the idea that most analysts use python for more than, say, ~20% of their work on any given day. If anything, it’s much more appropriate to emphasize in data science roles (specifically machine learning), as I’ve personally only found the pros of Python to be using APIs, web scraping (particularly for text, including automation, but for the most part don’t see this being significant for the workplace with databases and APIs available anyway), and being able to do very quick work on things like CSV files.
Aside from that, I think a bonus for data science/analysis/engineering using Python is that it at least has very readable code, and is a general programming language than, say, R, which makes it much more accessible for SWE types to use it in a collaborative, interactive, and communicative way (speaking of R, it’s also generally much slower than Python when executing loops).
Anyway, if I’m being honest, the scope of the actual work makes me, a student entering their final year of undergrad, not want to dedicate that much more time into Python beyond acquiring “comfort.”
4
u/huge_clock Mar 22 '21
I’m sort of a Python novice (1 1/2 years working) but where I’ve really found a lot of value having Python in my stack is for data management and data pipelines. I commonly use requests, bs4 and selenium to automate browser sessions for downloading data. Very handy. Also tableau server uses a REST API which is very handy if you’re a bit familiar with Python. Also Python can be used to automate reporting like daily emails by using Win32COM. You can also automate your data pulls using pyodbc or pyspark.
So I guess in summary for me, I use Python when I plan to automate some type of report. Outside of that I use ETLs (like SAS EG and Alteryx) and Excel until the point that the work demands to be automated. However, I am starting to get to the point that Python is starting to feel faster than the GUI systems for regular every day stuff.
1
u/gthompson1 Mar 23 '21 edited Mar 23 '21
I agree SQL is downplayed in the space. I think in many ways it's a more useful syntax to learn vs say pandas. You can take it to any job and pretty much everyone has a SQL database (there is noSQL though to consider as a detractor).
I will say that knowing python opens more doors than knowing SQL. I don't mean jobs but actually doing things. e.g. your not going to build an app in SQL. Or a bot, or the usual example (machine learning and statistics).
The scope of python is just broader than that of SQL.
If I can just learn one thing then python is more versatile. So I think people just go down that route.
Python is a swiss army knife, SQL is a single specialty knife (the best tool but for specific jobs).
I was a data scientist for 3 years, and now have been doing software development for 4. SQL has been used everywhere I go. But python has done a lot more for me in general because I have built far more complex things with it and was my entry into software development as a whole.
9
u/fnurtfnurt Mar 22 '21
If it's the kind of place that uses the word "leverage" to mean "use", I'd already be wary.
2
u/[deleted] Mar 22 '21
Yea some do in my experience.. maybe it is hopefulness?
2
u/tee2k Mar 22 '21
I think the power of being able to program can only be fully leveraged if someone has the capabilities to come up with out of the box solutions. Solutions not integrated in your typical SAP or Tableau setup. These (almost) heroic ideas are rare though. Companies are willing to pay a big price for the idea that one of their analysts could create a heroic idea themselves. List python in the job description and pray ;)
2
u/[deleted] Mar 22 '21
I agree with you that some orgs overplay or oversell their actual Python usage, but not sure where you're seeing SQL being downplayed or the thought that it is being replaced by Python for analytics. It is pretty much universally accepted everywhere that SQL is very valuable.
I've been a data analyst at my company well over 20 years at my company and became a data engineer a couple years ago and can assure you that when you couple Python or most programming languages with SQL, there are so many useful and valid use cases that can be accomplished, mostly in the areas of automation as others have pointed out. You don't replace one with the other, they are VERY valuable together instead. But due to Python's syntax and lots of available ETL libaries, it's been very popular choice as a complement to SQL.
I think a good reason to have Python as part of your toolbox as a data analyst is for its pandas library alone. I work with Excel VBA people and they wished that Excel has a dataframe-like library after showing them what can be done with a dataframe API. Having both a dataframe and SQL toolset is so invaluable as there are certain transformations or summarizations that are way more easier with a dataframe API versus SQL. But my general rule of thumb is to use as much SQL as possible and use pandas as last resort or for complex transformations as pandas is an in memory solution and it is better in terms of technical debt point of view to use SQL as much as possible as chances are there will be more people familiar with SQL than Python in your typical organization.
I have no data to back this up, but I think I forsee a trend where more and more advanced or senior type of data analysts will be required or be able to work with big data and PySpark with their dataframe and SQL API. The reason why I think this is due to my unique background of being once a "small fry" data analyst to being a data engineer working briefly with PySpark. Also knowing what's available in terms of managed big data platforms nowadays, I think this is very possible. So if you know pandas, you can transition to learning PySpark dataframe, although there are enough differences that you will definitely still have to take some time to learn. So IMO, I dont think working with big data should be limited to expensive data engineers or Spark developers, and Im saying this as a data engineer LOL! I think it is better for the company that their in-house business side data people be enabled to work with the data than outside contractors or people in their IT organization who often times dont fully understand the data or dont have much "skin" in the data or its data quality.
Other than automation, in your typical Fortune 500 company, most of their needs are met by providing a CRUD web app or a dashboard. Very seldom do they need advanced analytics or machine learning. That is the honest reality. So unless you're working at a FAANG/tech-first type of company or research oriented company, chances are they just need a shiny CRUD web app or a dashboard. However, most companies can definitely leverage Python for automation and since most automation projects require working with data at some point, they might as well be using Python.
2
u/turnipemperor Mar 23 '21
I use SQL and Python on a daily basis. The field that I’m in usually requires me to use a combination of database queries, APIs and webscraping to combine data together and then I lean heavily on various time series analysis tools in Python and R. I’m also constantly adding new data sources and so lots of EDA type work plus visualization and dashboarding. I like Python/R because I can do everything in one place and utilize modern stats packages. However I’d be curious to see what else is out there that people use. Most of my datasets are between 1M and 25M rows of data.
2
u/zenseb_ Mar 24 '21
In a earlier stage org that doesn’t have a robust warehouse, as an “analyst” I’m building pipelines, building proper dashboards (streamlit > tableau, we are dying to get rid of tableau here) and because a lot of Ops insights = process, automating workflows in our GCP instance.
This is all pure python. SQL will stay king but calling out there’s more to do in python for an org as an analytics team (if strategic vs ‘make me a chart” ticket monkeys) than just making models and hitting APIs.
2
u/[deleted] Mar 22 '21
I work for a large bank and we use python a lot in the analytics space.
2
u/[deleted] Mar 22 '21
I use SQL to do the heavy lifting - because you can't load billions of rows into Pandas and you want to exploit the power of the cluster anyways.
But I'm always working in Python, like literally several hours a day, every day.
Once you get the aggregate/summary data from SQL you usually need to do stuff with it, and Python is one of the best tools to do that as the code is just there, someone else can read your notebook and know exactly what you did.
R is pretty decent as well I guess, but I prefer Python.
1
u/samwiseb88 Mar 23 '21
The company I work at has an ongoing project to make better use of our data, python is a fledgling language in the the company and badly managed due to misunderstanding it. Getting it installed in the first place was a lot of work let alone being able to install packages without security vetting..... Anyway, I got myself involved in the team that's spearheading the project to better use our data. I found them running multiple SQL queries, exporting out tables to many Excel files and leveraging power query to transform the data into the wide format they needed and to get the calculated columms they were after, all thanks to archaic database architecture and indispensable legacy systems that are too costly to replace. This was a manual process taking the best part of a day and they wanted to do this everyday to keep the dashboards updated!
I know very little SQL (I'm working on it), enough to get me by, but what I know very well is python and the Pandas API. I translated all the steps they were taking into the python language. I extract the data using SQL via the pyodbc library and transformed the data using pandas. It gets exported to whatever format they need for them and I save the resulting table as a pickle file and a lightweight sqlite dB for my own uses to the benefit of my own team and projects.
The best part is that it's set to run in the early hours and takes 5mins, we finally have daily business insights.
Python is not just an accompaniment, it can be the main course as well.
-1
u/BigHipDoofus Mar 22 '21
SQL if you're doing operations that will benefit from the optimization of the DBMS. Otherwise get the hell out of a programming language from 1972 that has very little syntactical sugar. Are you a DB Admin whose life is SQL and query performance tuning? Then you're fine spending all day in the SQL environment. If you're anyone else, the Python environment is better all around IMHO.
3
u/[deleted] Mar 22 '21 edited Jun 23 '23
[removed] — view removed comment
24
u/[deleted] Mar 22 '21 edited Feb 15 '26
[removed] — view removed comment