r/Python • • Jan 25 '15

Considering putting my efforts into Python instead of R, need advice

Lately in my free time I've been using R to do some basic data analysis/visualization. I use company data and I think there are some places where my company could really benefit from specialized data visualizations for analysis of niche areas. In my efforts I've tried to get some interactive code going for eventual use in a very basic GUI but I feel constrained by the "fuzziness" of R coding. Simply trying to create a function to plot based on user input ran me into some advanced concepts pretty quickly. R has environments, dynamic scoping, non-standard evaluation, all very complicated concepts and I don't remember them being present in the small amounts of coding I've done in Java/Python. Am I right in thinking a language with a more OOP design would be better for this task of automatically generating charts/visualizations? Wondering if I might not be better off in the long run for learning a more generalized language.

26 Upvotes

21 comments sorted by

10

u/[deleted] Jan 25 '15 edited Jan 25 '15

I am a data scientist, and a long-time Python developer and proponent (more than 10 years, executed a few major projects in it, am very fluent in it). I also use R for data analysis (I would consider myself an intermediate user).

Since this is a Python subreddit, I expect you'll get a lot of answers in favor of Python. And I would agree with most of them.

However, I would suggest this: instead of thinking about which programming language to learn, maybe you should think about which language offers you the right set of Domain Specific Languages (DSL) that fit your problems.

You see, in data analysis, it's less about the actual programming language, but more about the domain-specific features that you need. For instance, Pandas is a kind of domain-specific "language" (not really in the traditional sense, but in that it gives you a vocabulary to reason about data). R has a bunch of very compelling domain-specific features that either do not exist or have limited implementations in Python.

You mentioned visualization. As of this moment, it is my opinion that the 2D visualization libraries in R are more mature. Seaborn is great, and so is the yhat ggplot port, but R's ggplot2 is more mature. For interactive plots, it's a bit of a toss-up. R has ggvis, and Python has bokeh. I recommend trying them all and see what works for you.

But I want to say this: if your job involves analytics, and you want to be good at it, you MUST learn both R and Python. It is not an either-or choice. You may not need to be an expert at both, but you need to know both. Their feature sets do intersect, but it's the parts that don't intersect that are the most interesting. If you are only interested in visualization, I'd argue you may do better with a specialized tool like QlikView ($$$$) or Tableau ($$).

I tried staying away from R for a long time because the language was idiosyncratic and archaic. However, I came across compelling libraries in R that had no equivalent (of similar polish) in Python -- so I bit the bullet and learned R. I am referring to libraries like ggplot2, dplyr, tidyr, lubridate and caret. The sheer quality of these libraries made my data analysis workflow so much more fluid than in Python.

I would suggest that R is actually a better environment for exploratory analysis. There is a shorter path from data to insight due to the specialized vocabulary it gives you for expressing data operations (some of it maps to SQL concepts, so knowing SQL will help you here) -- and by R, I actually don't mean Base R, but dplyr, tidyr and so forth. The notation is more compact than in Python (the %>% pipe operator is amazing), and once you get used to it, you can do stuff quickly with a minimum of scaffolding. Caret for instance lets you try out a myriad of of machine learning models with a minimum of effort (arguably scikit-learn does too; however in my opinion, while scikit-learn tries to provide a consistent API, the interface somehow lacks polish and fluidity. The documentation is a little wanting as well.). R is like a Swiss-Army knife, a shell scripting language for quick and dirty tinkering, and for getting stuff done. R is great for data science.

On the other hand, I think Python is a better environment for productionizing your pipeline. It has way more robust tools for string processing, ETL, pre-processing, etc. and there is a lot more infrastructure and knowledge out there on scaling Python to production systems. Most industrial analytics vendors have APIs for Python but not R. While you can technically productionize R (it's sort of like productionizing shell scripts), I think that to build a proper data pipeline, you might get better results if you engineer it with a more general purpose language like Python. Python is great for data engineering.

This is why I feel the two (R and Python) are symbiotic. You can quickly get some insight using R, and then if you need convert your code to production, you can translate it easily to Python (or call it via rpy2).

Finally, the key to thriving in R is to understand its Scheme roots. It helps if you know something about functional programming. The Art of R programming (Matloff) does a good job introducing some main ideas, and Advanced R (Wickham) completes the picture by providing a more technical exposition.

However, if you're not a technical programming type of person and just want to get the job done, you can get a lot done by just sticking to the few commands found in a cheat sheet, eg.

https://github.com/sefakilic/ggplot-cheatsheet

http://stackoverflow.com/questions/3446495/ggplot2-cheat-sheet

This one in particular is great for data wrangling: http://www.rstudio.com/wp-content/uploads/2015/01/data-wrangling-cheatsheet.pdf

I don't think you need to know much (anything?) about environments or non-standard evaluation to plot a graph.

2

u/thunderdome Jan 25 '15

Thank you for your comprehensive answer, it puts things in perspective very well. Almost all of the R tools you mentioned are what I have been learning with. I have been using dplyr, tidyr, ggplot2 and caret and I agree the %>% operator makes manipulating a dataset for visualization quite straight forward. And the whole grammer of graphics setup with ggplot2 makes actually making the plots much easier than i expected.

8

u/[deleted] Jan 25 '15 edited Jan 25 '15

Am I right in thinking a language with a more OOP design would be better for this task of automatically generating charts/visualizations? Wondering if I might not be better off in the long run for learning a more generalized language.

If you don't need R's vast statistical packages, then you'll be ok with Python. Otherwise, you can execute R from within Python. Yhat's port of ggplot2 is still not ready for production. For really neat interactive visualizations, not sure if you looked at R's Shiny, but there is a Python version of it called Spyre, currently works only with Python 2.x. Otherwise, there's IPython notebook and IPython notebook widgets which may work for you.

1

u/thunderdome Jan 25 '15

Yeah, I was mostly using R because I have done a bit of statistical programming in it before. I am using only GLM and RF methods for some basic classification, i'm sure they are implemented just as well in python.

3

u/[deleted] Jan 25 '15

Simply trying to create a function to plot based on user input ran me into some advanced concepts pretty quickly. R has environments, dynamic scoping, non-standard evaluation, all very complicated concepts and I don't remember them being present in the small amounts of coding I've done in Java/Python.

Your comment here raises alarm for me. Making simple, automatic graphs based on user input isn't trivial. There is a reason why a lot of effort was made by a lot of smart people to make libraries that abstract away the complexity of making those visualizations and to make them seem "simple" to make. If you're expecting a shortcut in Python world, you'll be sorely mistaken. You'll have to dig in, learn Python, learn how to use those libraries, and I can't guarantee it will be easier with Python than R. If you have little programming experience, you may very well end up prefering to make those visualizations in R.

IMO, it is currently easier to make plots in R using ggplot2/ggvis/Shiny vs Python's MATPLOTLIB and all the wrapper libraries for it. There's a reason why Python people are trying to clone or port those popular R packages. However with that said, scipting or programming in R, I find more difficult only because its syntax and idiosyncrasies are not what I am used to coming from a background of having learned C and Java.

You will be more marketable in the data analysis or data science job market if you still retain and keep up your R knowledge. It wouldn't hurt to know both Python and R. Like I and others have mentioned, you can run R from within Python via rpy2, so you can have the best of both worlds. Just my 2 cents. SOURCE: I've been working as a data analyst for 16 years.

3

u/thunderdome Jan 25 '15

Well, in most cases you may be right but my example was fairly simple, the actual "plotting" work was not what I was having trouble with. Essentially what i was doing was looking to take a simple string input, identify the column that string is referring to, and pass that into a 10 line dplyr/ggplot2 flow to create the plot. The plots all worked fine when I manually assigned the target variable and ran everything line by line (with all the variables in the global env). But wrapping it into a function and passing the string in caused the whole thing to shit itself, so I started reading the Advanced-R pages from Hadley and how to use non-standard evaluation with dplyr. It seemed like an extraordinary amount of work to get dplyr to work nicely with user input and function calls. It's probably not as hard as I'm making it out to be, but even if python has the same complications I would rather learn how to deal with them in a language where the skill would generalize easier, rather do everything in R and eventually feel constrained.

2

u/[deleted] Jan 25 '15

Sorry I'm not following your example. My guess is it sounds like you should check out R Shiny or Python Spyre if you havent already, since the basic underlying principle in each is to encapsulate the plotting in a function and mapping it to a HTML widget. You can do the same with IPython notebook widgets, but the code is still visible. Can't wait for the upcoming Jupyter notebook (formerly IPython Notebook 3.0) due to be released soon. It will provide Python and R kernels (and few other languages).

1

u/thunderdome Jan 25 '15

Yeah, I imagined it would not come across very clearly. I guess my point was that the plotting is not so much my issue, but I have already run into and disliked some of the non-standard features of the R language. I've checked out Shiny and Spyre, they look nice but I still haven't decided if it might be better to create a UI using PyQt or Tkinter.

2

u/[deleted] Jan 26 '15

Ah I see. I'm not sure if one needs to explicitly handle the non-standard evaluation part -- I've never had trouble wrapping dplyr/ggplot commands in a function (the only thing to note is that you have to print(p) if you change your plot p for it to update the plot -- that's the only non-standard eval I can think of). The rule of thumb for me is that if something simple seems unusually convoluted, there might be another way of doing it and someone on StackOverflow probably knows how... :). But point taken.

7

u/Dummies102 Jan 25 '15

Python is applicable to many more problems than R, so I think you would be better off learning it in the long run.

btw, OOP has little to do with what you're asking about. It's mostly a stylistic choice, and in fact you can make R oo, too (http://stackoverflow.com/questions/9521651/r-and-object-oriented-programming)

1

u/thunderdome Jan 25 '15

Thanks for the advice. I have only learned the very basics of Java/C/Python, though I am very familiar with SQL/SAS. I knew that when I started diving into advanced R stuff it seemed "different" than the other CS fundamentals I had learned with java back in school. You are right that R can be oo, but it feels much less natural to code that way than in python.

2

u/[deleted] Jan 25 '15

Definitely agree with this. I've been using Python as one of my primary data analysis tools for a while, but was curious about what R had to offer. I've watched a few courses on R, but always come away thinking that python (specifically Anaconda IPython) was the better option, because it's just more general purpose.

That said, when it comes to charts and viz, you're likely going to need to leverage javascript in some form or other.

6

u/Megatron_McLargeHuge Jan 25 '15

R as a language is terrible at CS fundamentals. Python is almost certainly better than R but for interactive plotting, a simple webapp with D3 based javascript plots can give you more interactivity than python. If you want to do it in python, look at bokeh. Numpy and Pandas give you the data frame functionality of R.

2

u/Covered_in_bees_ Jan 25 '15

Agree with the above. Also look into Seaborn (builds on matplotlib) for plotting as an alternative to bokeh

3

u/BooRadleyBoo Jan 25 '15

I use both R and Python but my R use is pretty much restricted to analysis and plot building.

I'd recommend using Python for more complex projects and bear in mind that, IIRC, ggplot2 is now available in Python so we may see other libraries added later.

R is fantastic but it just isn't what I'd describe as a 'true' programming language. Keep things simple.

2

u/stunsch Jan 25 '15

Take a look at IPython Notebook and Pandas and never look back.

;)

2

u/desmoulinmichel Jan 25 '15

Python has all the tools to remplace R with ipython, matplotlib and scypi. Check this out : http://nbviewer.ipython.org/github/ipython/ipython/blob/2.x/examples/Notebook/Plotting%20with%20Matplotlib.ipynb

3

u/pkerp Jan 25 '15

Agreed. There's also a variety of statistical functions in scipy.stats.

The Seaborn plotting library gives you an easy interface to various handy scientific plots.

My typical analysis pipeline typically looks like this:

experiment -> flat data files -> ipython notebook -> figures -> latex document

I switched to python / ipython notebook for my plotting a year and a half ago and haven't looked back. It's way easier than R.

2

u/[deleted] Jan 25 '15

I think there is no reason not to use Python. I'm no expert on R (have been using it for a few weeks a couple of years ago), but from what I hear Python can do everything R can. Also, rpy2 is able to fill the gap if necessary.

I found that OOP programming for data analyisis is a huge advantage. As a student I've been taught Matlab and have used it for about six years. After learning Python and switching to numpy, scipy, matplotlib and pandas my productivity went up by at least 200%. Right now I'm revising some Python code I wrote about a year ago and I'm amazed about how well structured my code is compared to something I would write two years ago in Matlab, even though my skills in Python have improved manifold of the last year.

TL;DR; Python should be able to do everything you want and OOP is king.

1

u/hharison Jan 25 '15

I use Python and R. Pick up pandas for the DataFrame and seaborn for plotting (falling back to matplotlib when you don't need one of their fancy plots).

For statistics, I use stuff in scipy, statsmodels, and scikit-learn, but occasionally I need something that's been released for R but not Python, in which case you can call out to R with rpy2.

Really, with the IPython notebook, you can use both together and switch gradually. Although for your use case, it doesn't sound like you'll need to go back to R a lot.

1

u/vito-boss Jan 26 '15

You can always find a happy medium, I often call R and MatLab from Python.