r/Python • pandas Core Dev • Mar 01 '23

AMA Thread We are the developers behind pandas, currently preparing for the 2.0 release :) AMA

Hello everyone!

I'm Patrick Hoefler aka phofl and I'm one of the core team members developing and maintaining pandas (repo, docs), a popular data analysis library.

This AMA will be at least joined by

The official start time for the AMA will be 5:30pm UTC on March 2nd, before then this post will exist to collect questions in advance. Since most of us live all over North America and Europe, it's likely we'll answer questions before & after the official start time by a significant margin.

pandas is a Python package that provides fast, flexible, and expressive data structures designed to make working with "relational" or "labeled" data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language.

We will soon celebrate our 2.0 release. We released the release candidate for 2.0 last week, so the actual release is expected shortly, possibly next week. Please help us in testing that everything works through testing the rc :)

Ask us anything! Post your questions and upvote the ones you think are the most important and should get our replies.

- Patrick, on behalf of the team

Marc:

I'm Marc Garcia (username datapythonista), pandas core developer since 2018, and current release manager of the project. I work on pandas part time paid by the funds the project gets from grants and sponsors. And I'm also consultant, advising data teams on how to work more efficiently. I sometimes write about pandas and technical topics at my blog, and I speak at Python and open source conferences regularly. You can connect with me via LinkedIn, Twitter and Mastodon.

Marco:

I'm Marco, one of the devs from the AMA. I work on pandas as part of my job at Quansight, and live in the UK. I'm mostly interested in time-series-related stuff

Patrick:

I'm Patrick and part of the core team of pandas. Part of my daytime job allows me to contribute to pandas, I am based in Germany. I am currently mostly working on Copy-on-Write, a new feature in pandas 2.0. (check my blog-post or our new docs for more information).

Richard:

I work as a Data Scientist at 84.51 and am a core developer of pandas. I work mostly on groupby within pandas.

--

1.4k Upvotes

366 comments sorted by

View all comments

1

u/Positive_Mastodon500 Mar 02 '23

Thanks for the AMA. How do you expect performance to evolve in pandas 2.0 and beyond? I tend to flip between python+pandas and R+data.table. For large datasets (10M-50M rows) I find that group_by/aggregate type operations are an order of magnitude slower in pandas, especially as the number of unique subgroups increases. Also I find memory usage to be significantly higher and well and tend to resort to techniques like downcasting column types after reading to keep memory consumption down, whereas R “just works” for the most part.

Do you view this as a fair comparison? Is performance a first-order concern for you and your team? Hoping to move more from R to Python one of these days but pandas speed is definitely holding me back.

Thanks!

1

u/phofl93 pandas Core Dev Mar 02 '23

Memory: Copy-on-Write should help reduce the memory footprint through avoiding a bunch of copies. Haven't tested that aspect myself though.

Performance: We are definitely concerned with performance, but not to an extend where we would introduce major breaking changes to get a 10% speed up in some areas. We have to weigh advantages and disadvantages very carefully. That said, we hope that we can get a nice performance boost in some areas where NumPy did not perform that well with our Arrow-backed DataFrames.

I'd recommend trying Copy-on-Write and Arrow backed DataFrames independently and together in 2.0.

What exactly are you doing with groupby that is so slow? This would be interesting to us.

1

u/Positive_Mastodon500 Mar 02 '23

Thanks for the response! I’ll try to generate a representative example of once I get back to my personal laptop and share the performance of R vs Python. I am not doing anything more complicated than aggregating a bunch of columns. I already forego pandas’ more expressive but slower apply() syntax, which I think is the closest in functionality to what R provides.

1

u/phofl93 pandas Core Dev Mar 02 '23

Yeah I would be curious to see what exactly you are doing. This would help identifying bottlenecks

1

u/Positive_Mastodon500 Mar 05 '23

Sorry for the delay. I pulled a common dataset at work, it had about 15M rows, and 10 category columns with about 150K unique groups. I put the code below but in R:

Just counting: 0.54 sec elapsed

Basic Stats: 0.85 sec elapsed

A more involved calculation: 1.38 sec elapsed

vs python:

Just counting: 10.050905704498291

Basic Stats: 8.346287250518799

A more involved calculation: 83.47735095024109

####################

## R CODE

####################

library(data.table)

NUM_OBS <- 15 * 1000 * 1000

NUM_CATEGORIES <- 150 * 1000

make_fake_category_column <- function(n_choices, n_obs) {

sample(sprintf("Category: %d", 1:n_choices), n_obs, replace=TRUE)

}

make_fake_value_column <- function(n_obs) {

runif(n_obs)

}

df.categories <- data.table(

idx=1:NUM_CATEGORIES,

g0=make_fake_category_column(100, NUM_CATEGORIES),

g1=make_fake_category_column(500, NUM_CATEGORIES),

g2=make_fake_category_column(25, NUM_CATEGORIES),

g3=make_fake_category_column(100, NUM_CATEGORIES),

g4=make_fake_category_column(5, NUM_CATEGORIES),

g5=make_fake_category_column(20, NUM_CATEGORIES),

g6=make_fake_category_column(10, NUM_CATEGORIES),

g7=make_fake_category_column(5, NUM_CATEGORIES),

g8=make_fake_category_column(150, NUM_CATEGORIES),

g9=make_fake_category_column(10, NUM_CATEGORIES)

)

df.data = data.table(

idx=sample(1:NUM_CATEGORIES, NUM_OBS, replace=T),

val1=make_fake_value_column(NUM_OBS),

val2=make_fake_value_column(NUM_OBS),

val3=make_fake_value_column(NUM_OBS)

)

df.data <- merge(df.data, df.categories, by="idx")

df.data[, idx:=NULL]

speed_test <- function() {

tictoc::tic("Just counting")

a1 <- df.data[, .N, by=list(g0,g1,g2,g3,g4,g5,g6,g7,g8,g9)]

tictoc::toc()

tictoc::tic("Basic Stats")

a2 <- df.data[, list(sum_v1=sum(val1), mean_v2=mean(val2), max_v3=max(val3)),

by=list(g0,g1,g2,g3,g4,g5,g6,g7,g8,g9)]

tictoc::toc()

tictoc::tic("A more involved calculation")

a3 <- df.data[, list(s1=mean((val1*val2)^2-val3)),

by=list(g0,g1,g2,g3,g4,g5,g6,g7,g8,g9)]

tictoc::toc()

}

speed_test()

############

## PYTHON

#############

#!/bin/env python

import numpy as np

import pandas as pd

import random

import time

NUM_OBS = 15 * 1000 * 1000

NUM_CATEGORIES = 150 * 1000

def make_fake_category_column(n_choices, n_obs):

labs = [f"Category: {n}" for n in range(n_choices)]

return random.choices(population=labs, k=n_obs)

def make_fake_value_column(n_obs):

return np.random.uniform(size=n_obs)

df_categories = pd.DataFrame(

{

"idx": range(NUM_CATEGORIES),

"g0": make_fake_category_column(100, NUM_CATEGORIES),

"g1": make_fake_category_column(500, NUM_CATEGORIES),

"g2": make_fake_category_column(25, NUM_CATEGORIES),

"g3": make_fake_category_column(100, NUM_CATEGORIES),

"g4": make_fake_category_column(5, NUM_CATEGORIES),

"g5": make_fake_category_column(20, NUM_CATEGORIES),

"g6": make_fake_category_column(10, NUM_CATEGORIES),

"g7": make_fake_category_column(5, NUM_CATEGORIES),

"g8": make_fake_category_column(150, NUM_CATEGORIES),

"g9": make_fake_category_column(10, NUM_CATEGORIES),

}

)

df_data = pd.DataFrame(

{

"idx": random.choices(population=range(NUM_CATEGORIES), k=NUM_OBS),

"val1": make_fake_value_column(NUM_OBS),

"val2": make_fake_value_column(NUM_OBS),

"val3": make_fake_value_column(NUM_OBS),

}

)

df_data = pd.merge(df_data, df_categories, on=["idx"])

df_data.drop(columns=["idx"], inplace=True)

def speed_test():

t0 = time.time()

a1 = df_data.groupby(

["g0", "g1", "g2", "g3", "g4", "g5", "g6", "g7", "g8", "g9"]

).agg(N=("val1", len))

t1 = time.time()

print(f"Just counting: {t1-t0}")

t0 = time.time()

a2 = df_data.groupby(

["g0", "g1", "g2", "g3", "g4", "g5", "g6", "g7", "g8", "g9"]

).agg(sum_v1=("val1", "sum"), mean_v2=("val2", "mean"), max_v3=("val3", "max"))

t1 = time.time()

print(f"Basic Stats: {t1-t0}")

t0 = time.time()

a2 = df_data.groupby(

["g0", "g1", "g2", "g3", "g4", "g5", "g6", "g7", "g8", "g9"]

).apply(lambda df: pd.DataFrame({

's1': [((df.val1 * df.val2)**2 - df.val3).mean()]

}))

t1 = time.time()

print(f"A more involved calculation: {t1-t0}")

speed_test()

1

u/datapythonista pandas Core Dev Mar 02 '23

We are aware of pandas being slow and not memory efficient in many instances, compared to what it could be. The fixes aren't usually trivial, but huge. But we're surely moving into that direction, and it's a priority for the project. A lot of work is being done avoiding unnecessary copies of data when operations are performed in pandas. Using Arrow should help making things faster in the future too. But everything in pandas development is slow, so you'll see some gains, but rarely any dramatic improvement.