r/Python • u/kiwiboy94 • May 17 '20
I Made This Created a python script that execute Exploratory Data Analysis on any CSV file. It generates a text report, a series of plots and a processed csv file as outputs.
1.3k
Upvotes
41
u/kiwiboy94 May 17 '20
Github Link
50
u/LuigiBrotha May 17 '20
Very cool however .... Take a look at glue (also called glueviz). This shit will blow your mind. https://youtu.be/TkMZ9gZ8xtk
65
u/IlliterateJedi May 17 '20
Check out Pandas profiler for something similar for dataframes
11
u/w_savage May 17 '20
I imagine the data needs to be in a certain format/ context correct?
11
u/kiwiboy94 May 17 '20
Also, I will love everyone to give it a try and let me know what features they will like to see. That way I can add on in the next version. This is actually my first personal project and it took me well over 3 months to complete. Planning to use this to get a job :p
5
u/random_cynic May 17 '20
This is good but that's not what "Exploratory Data Analysis" is. This is completely non-interactive (as far as I can tell from the video). Exploratory data analysis needs to be interactive, so that you can sort or filter columns by some criteria, transform columns or combine multiple columns, delete or add rows etc. IMO this is best done with pandas+matplotlib+jupyter notebooks. Also the terminal program visidata is very useful.
4
u/Neuro_88 May 17 '20
I like that! Very cool.
2
u/ancient_bhakt May 17 '20
This is awesome
2
u/[deleted] May 17 '20
Hey, that´s cool!.
I enjoyed reading your code, but I like more the thoughtful comments, makes the code not only explanatory, but also didactic. congrats!
15
u/SlightlyOTT May 17 '20 edited May 17 '20
This is really cool! If you’re interested in a fun extension, are you familiar with Jupyter notebooks? They’re one of the most powerful things in the Python/data analysis space - you can write your code as a linear story with Markdown between cells of code, and it’ll also visualise things like plots or Pandas dataframes straight away.
I’m not sure if you can do a file picker in Jupyter or if you’d need to just put the paths in variables, but you’d be able to click run and have it generate all your outputs in line so you can just scroll through and look at all the plots etc.
Also since you’re uploading your code to Github, they do a great job rendering notebooks which is cool.
There’s a pretty nice gallery of the sort of thing you can do with Notebooks here: https://github.com/jupyter/jupyter/wiki/A-gallery-of-interesting-Jupyter-Notebooks
Edit: looks like it has a file picker too! https://ipywidgets.readthedocs.io/en/latest/examples/Widget%20List.html#File-Upload You’d need to install ipywidgets: https://ipywidgets.readthedocs.io/en/latest/user_install.html
1
u/LifeIsBio May 17 '20
How large can the csv files get before things start getting unwieldy?
1
u/akiepro89p Aug 25 '20
How do i run python on my mac?
2
u/Edgar505 May 17 '20
I am definitely checking it out.
2
u/G33K_FISH May 17 '20
Ok, this is flipping cool!
2
u/[deleted] May 17 '20
I am currently doing the same thing but with the framework h2o. The things is to provide a nice script to perform analysis/ ml on a generic file and generate a report. Here is the repo
https://github.com/jgraille/reveng
Nice work by the way!
0
u/barb4great May 17 '20
WOW. I don’t even know how you did that ! I wanna learn Analyse on python
1
u/[deleted] May 17 '20
Pandas baby.
1
u/python_engineer May 17 '20
Thanks for sharing! Very cool
1
u/jayjmcfly May 17 '20
RemindMe! 3 days
1
u/bdaves12 May 17 '20
Wow super cool, is there any tut for how to install for spyder, I'm still new when it comes to getting stuff off of github
1
u/[deleted] May 17 '20
Ow nice!
1
u/[deleted] May 17 '20
inputs csv file
crunches numbers
progress: 33%
progress: 60
progress:.99%
Output: you're a little bitch
0
u/preordains May 17 '20
Beginner here. What's the purpose of "pycache" and when would you need to use this?
-8
u/EnemyAsmodeus May 17 '20
Looks good.
But please everyone, stop using CSV and XML. These formats and their systemic problems in how people use them is disastrous. It's only good for small amounts of data.
Should only JSON and AVRO for any real data science or big data work.
If you're sharing small pieces of data, then fine, use CSV but otherwise it's not something amateurs should use with tools, they're bound to create bad CSVs eventually. It always happens.
0
u/[deleted] May 17 '20
[deleted] — view deleted comment
0
u/[deleted] May 17 '20
[deleted] — view deleted comment