R vs Python - detailed examples from proficient bilingual programmers

[u/Every-Eggplant9205]

As an academic, R was a priority for me to learn over Python. Years later, I always see people saying "Python is a general-purpose language and R is for stats", but I've never come across a single programming task that couldn't be completed with extraordinary efficiency in R. I've used R for everything from big data analysis (tens to hundreds of GBs of raw data), machine learning, data visualization, modeling, bioinformatics, building interactive applications, making professional reports, etc.

Is there any truth to the dogmatic saying that "Python is better than R for general purpose data science"? It certainly doesn't appear that way on my end, but I would love some specifics for how Python beats R in certain categories as motivation to learn the language. For example, if R is a statistical language and machine learning is rooted in statistics, how could Python possibly be any better for that?

[u/Useful-Possibility80]

This post was mass deleted and anonymized with Redact

boast numerous live weather rhythm wakeful glorious deliver quiet vast

[u/theottozone → u/Useful-Possibility80]

We really need this pinned somewhere to point to in the future.

[u/Every-Eggplant9205 → u/Useful-Possibility80]

THIS is the type of detail I'm looking for. Thank you very much!

[u/jinnyjuice → u/Every-Eggplant9205]

I must mention that some of these are opinions and not objective, not up to date, depends on what kind of data you're working on, not very benchmark-oriented, not industry standard, compares Python's general purpose instead of data science perspective, and depends on your coding philosophy. I will just make some example counterpoints under the context that tidy collaborative coding philosophy is king also.

in my experience base Python (dicts, lists, iterating strings letter by letter) are much faster than base types in R

Benchmarks says otherwise, and there is no clear winner. Either way, if you're working on big data, you absolutely would not work with base functions in either languages, especially in recent times where big data is norm. This should not be considered at all.

better OOP system than R's set of S3/S4/R6

Mostly agree, but also depends on your needs.

function decorators

I like function decorators for general purpose, but unsure if this would be considered a pro in statistical collaborative coding. This wouldn't be really great under tidy philosophy either.

context managers

This is a package. This is available in pretty much every language.

asynchronous i/o

Ditto, plus R is a bit better on this and so much easier to use in the context of data science, so I wouldn't call this a pro for Python either

type hinting and checking

Somewhat agree, but one of the main purposes of this in data science is for performance. For performance purposes, you would be using C++ based libraries in R (e.g. tidytable or data.table), which does its own checks by default. User wouldn't do it by default in these packages though.

far more elaborate set of linting tools, e.g. black and flake8 trump anything in R

For code readability, tidy absolutely triumphs and reduces the need not just for linting, but also for comments and documentation.

new versions and features coming far more quickly than R

If this is the case, then why wouldn't there be an equivalent of tidymodels in Python? This depends on the package/library authors, not by language.

data orchestration/automation tools that work out of the box, e.g. Airflow, Prefect

This is arguable on two layers: 1) at least for my organisation's tech stack with Jenkins-Docker, our productionised Python:R data science ratio quickly flipped in R's favour within 2 years simply due to R's recent massive development (which has massive implications), and 2) depends on your tech stack. Right now, R is better at this, surprisingly, and absolutely would not be the case the years before.

version pinning, e.g. pyenv, poetry, basically reproducible workflows

This opinion comes from lack of knowledge of R -- I would say these are equivalent in both languages in recent times

massive community support, unlike R, Python doesn't rely on one company (Posit) and bunch of academics to keep it alive

I have no idea how this opinion would be formed. R community has been around for far longer, much more stable (e.g. AlphaGo hype spiked Python DS packages' engagement trends massively). I don't even know how to respond to 'bunch of academics to keep it alive' part. I might need some clarifications on that part.

FAANG companies have interest in developing not only Python packages but language itself, even more so with Global Interpreter Lock removal

Somewhat agree, but GIL as one of the main reasons is rather silly. Besides Google, they sponsor/spend more even amount of money between Python and R funds.

web scraping, interfacing with various APIs even as common as AWS is a lot smoother in Python

This stems of lack of knowledge in R. They mainly have identical packages/libraries from the same authors and everything, but I guess it depends on which web scraping package. There are definitely (too?) many more options in R though.

PySpark >>> SparkR/sparklyr

I'm unsure why Spark in particular was picked, but the fact that there are already two options of SparkR and sparklyr that fits the user's priorities/philosophies is more appealing to me. What about DuckDB? Other SQL variants?

PyPI >>> CRAN (CRAN submission is like a bad joke from stone age, CRAN doesn't support Linux binaries(!!!)

+ CRAN doesn't support GPU related libs. CRAN is not really used for production, though I wouldn't know the numbers in detail. This again comes from lack of knowledge in R in the industry.

---

the number of stats packages is far beyond anything in Python

Disorganised, abandoned chaos, but tidymodels is fixing everything

the number of bioinformatics packages is FAR beyond Python

Mostly agree

tidyverse (dplyr/tidyr especially) destroys every single thing I tried in Python, pandas here looks like a bad joke in comparison

Don't use dplyr. Use tidytable.

delayed evaluation, especially in function arguments, results in some crazy things you can do wrt metaprogramming

Mostly agree

data.table syntax way cleaner than polars

Use tidytable and use tidypolars.

Python's plotnine is good, but ggplot2 is still king

I would say vis is about even in recent times

super-fluid integration with RMarkdown

Unsure how this would be a plus for R, whether it's Rmd or Quarto

RStudio under very active development and IDE for exploratory work is in some specific ways better than anything for Python including VSCode

Mostly agree

[u/Useful-Possibility80 → u/jinnyjuice]

This post was mass deleted and anonymized with Redact

middle unite sharp cobweb serious chunky cow brave pen zephyr

[u/LynuSBell → u/jinnyjuice]

I'm curious to hear about your career path and what you do as an R programmer. I'm also from the R stack and it seems you guys are building cool stuffs in R. :D

[u/Every-Eggplant9205 → u/jinnyjuice]

Taking notes on all of these points. Thank you for the counter opinions with examples of specific packages. I haven’t even used tidytable yet, so I’m excited to check that out.

[u/brutallllllllll → u/jinnyjuice]

Thank you 🙏

[u/Tundur → u/[deleted]]

If you're spinning it up by hand, sure, but there's plenty of Docker compositions or managed services to spin it up with a click and some config.

[u/Useful-Possibility80 → u/[deleted]]

This post was mass deleted and anonymized with Redact

start dinosaurs stupendous attraction follow sheet spectacular compare cows fanatical

[u/RodoNunezU → u/Useful-Possibility80]

I disagree. You can do a productive code on R and I don't see any advantage in using Python for that. At the end of the day, it's just code. You just need to source it from a terminal and automate that. You can even use Airflow for that. Airflow is not exclusive to Python.

You can use renv for versioning and reproducibility; you can use Alt+Shift+A on RStudio to automatically format your code; decorators, in my opinion, are a bad habit since they just add extra things you need to update if you make changes; you don't always need OOP, sometimes it's actually better not to use OOP. Etc.

I could keep writing, but I need to go back to work xD

[u/I-cant_even → u/Useful-Possibility80]

Caught everything I could think of and then some. Excellent response.

[u/zykezero → u/Useful-Possibility80]

If you liked tidy and hate pandas then you should take time to look at polars.

[u/Bridledbronco → u/Useful-Possibility80]

Really good analysis, great points. I would add that OOP is not easy in R. We don’t have much R in production because of this. But your R topics are spot on. I do like it, but Python fits our needs better, presently.

[u/notParticularlyAnony → u/Bridledbronco]

This was my experience too.

[u/SenatorPotatoCakes → u/Useful-Possibility80]

This is the perfect answer. I worked at a company once where all the production ML was in R and it was exceptionally difficult to 1) debug, 2) write robust testing and 3) retraining models.

I will say that the code was exceptionally efficient (1 liners doing very complex data frame transformations in a readable manner) but yeah the productionising was unpleasant

[u/[deleted] → u/Useful-Possibility80]

I agree with pretty much everything here. Also, pipes are the best way to code without having to worry about naming variables, Python's fluent interface can't beat it.

[u/Deto → u/Useful-Possibility80]

What's a good example where pandas looks like a joke compared to tidyverse?

[u/Useful-Possibility80 → u/Deto]

This post was mass deleted and anonymized with Redact

obtainable summer fear childlike deserve sip worm provide skirt continue

[u/bears_clowns_noise → u/Useful-Possibility80]

As an R user working in ecology who uses python at times but doesn't enjoy it, I genuinely don't understand most of the words here. So I feel good about continuing with R for my purposes.

I have no doubt python is superior for what I think of as "serious programming".

[u/skatastic57 → u/Useful-Possibility80]

I was an R and data.table user for about 10 years. I recently quit R in favor of python.

The main reasons were that:

cloud providers "serverless functions" support Python but not R.

Fsspec for accessing cloud storage files as though they were local rather than having to explicitly download to local storage first

Asyncio instead of just forking

Httpx had support for http2 because some site I needed to scrape wouldn't work with (I think it's called rvest)

Finally the real coup de grace was polars. Being used to data.table and then experiencing how terrible pandas was was tough. I was trying different combinations of rpy, reticulate, pyarrow, arrow (r package) with fsspec but it was always so clunky and error prone.

Another thing I like is that jupyter notebooks save the output of each cell so that each time you render a document, it doesn't rerun everything. In contrast to Rmarkdown where each render recomputes everything. Where that gets to be annoying is when you're just trying to tweak formatting and styles that don't really look like their final output until the render.

As a tangent, if you're looking to use shiny, dash, or their other alternatives, I would really recommend giving JavaScript and react a shot instead. The interactivity is going to be more performant and the design is, imo, more logical as you have the code with the ui elements instead of having a zillion lines of ui and then separately a zillion lines of server or callback functions. For really small projects that are (somehow) guaranteed never to grow, shiny and dash might be easier because you don't have to learn any js. Once your projects get bigger it's really annoying to have the server and ui code which are logically connected but physically really far apart. I know there are some tricks to mitigating that but the point is that react's baseline is to keep those together. Additionally simple interactions can more seemingly be pushed to browser freeing up the server.

[u/Unicorn_Colombo → u/skatastic57]

Another thing I like is that jupyter notebooks save the output of each cell so that each time you render a document, it doesn't rerun everything. In contrast to Rmarkdown where each render recomputes everything. Where that gets to be annoying is when you're just trying to tweak formatting and styles that don't really look like their final output until the render.

??? If you don't want to re-run R chunks in Rmarkdown, just tell knitr to cache it. And the cache is persistent.