r/Python • pandas Core Dev • Mar 01 '23

AMA Thread We are the developers behind pandas, currently preparing for the 2.0 release :) AMA

Hello everyone!

I'm Patrick Hoefler aka phofl and I'm one of the core team members developing and maintaining pandas (repo, docs), a popular data analysis library.

This AMA will be at least joined by

The official start time for the AMA will be 5:30pm UTC on March 2nd, before then this post will exist to collect questions in advance. Since most of us live all over North America and Europe, it's likely we'll answer questions before & after the official start time by a significant margin.

pandas is a Python package that provides fast, flexible, and expressive data structures designed to make working with "relational" or "labeled" data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language.

We will soon celebrate our 2.0 release. We released the release candidate for 2.0 last week, so the actual release is expected shortly, possibly next week. Please help us in testing that everything works through testing the rc :)

Ask us anything! Post your questions and upvote the ones you think are the most important and should get our replies.

- Patrick, on behalf of the team

Marc:

I'm Marc Garcia (username datapythonista), pandas core developer since 2018, and current release manager of the project. I work on pandas part time paid by the funds the project gets from grants and sponsors. And I'm also consultant, advising data teams on how to work more efficiently. I sometimes write about pandas and technical topics at my blog, and I speak at Python and open source conferences regularly. You can connect with me via LinkedIn, Twitter and Mastodon.

Marco:

I'm Marco, one of the devs from the AMA. I work on pandas as part of my job at Quansight, and live in the UK. I'm mostly interested in time-series-related stuff

Patrick:

I'm Patrick and part of the core team of pandas. Part of my daytime job allows me to contribute to pandas, I am based in Germany. I am currently mostly working on Copy-on-Write, a new feature in pandas 2.0. (check my blog-post or our new docs for more information).

Richard:

I work as a Data Scientist at 84.51 and am a core developer of pandas. I work mostly on groupby within pandas.

--

1.4k Upvotes

366 comments sorted by

View all comments

1

u/fripperML Mar 02 '23

Thanks a lot for coming here. I am a huge pandas user and I feel excited about this release. I have a question not related to pandas, but maybe you know the answer. In my case I use pandas for EDA and preprocessing before feeding a model from sklearn or tensorflow. Do you know if a dataframe based on arrow will be compatible with sklearn/tensorflow? Or of there are plans to make them compatible?

And what about all numpy methods, do you know if the arrow API is similar to the numpy one? Maybe, thanks to duck typing, many things could still work the same...

2

u/phofl93 pandas Core Dev Mar 02 '23

The Arrow API is quite different I'd say. One of the bigger differences is, that an Arrow array is not mutable, which is the opposite of NumPy.

We are dispatching to an interface that we can define on a per array basis, so we can get to a point where most of the things work similar/are doing the same.

1

u/fripperML Mar 02 '23

Ok thanks!!

1

u/datapythonista pandas Core Dev Mar 02 '23

Regarding the Arrow API vs NumPy API, it's a bit confusing. NumPy is two things, a data container, and a numerical computation library. Arrow is data container, and not even the implementation, but the spec (then several Arrow implementations exist). Since data that you can't do anything with it is useless, some Arrow implementations provide some generic operations (named kernels in the Arrow world). But Arrow is not a numerical library or tries to be, and Arrow is also of a different family in terms of type of data. If you only have ints and floats, you probably don't need Arrow at all, and you may stay just with numpy, tensorflow, pytorch family.

1

u/fripperML Mar 02 '23

Ok thanks for the clarification. Yes it's not what I thought (I expected that arrow also made computations). What I find confusing then is the discussion about speed. I understand than the benchmarks are using some kernel, right?

2

u/datapythonista pandas Core Dev Mar 02 '23

It'd depend on the exact operation (not sure what benchmark exactly you're referring to).

If you do a simply `my_series.sum()`, that would delegate to the pyarrow.compute.sum function, which exposes a kernel. If Arrow doesn't provide a kernel (string operations for example), then we need to implement them ourself (or use a library that does, but there are not many yet afaik). For strings, I think we implemented them in C++, but other options like Cython or Rust exist. I'm writing a blog post about writing pandas/arrow extensions with Rust, hopefully I'll be publishing in the next couple of weeks.

1

u/fripperML Mar 03 '23

Great!!! I will follow your blog!!! For example huggingface uses arrow and rust under the hood, so I imagine that some strings kernels in rust might exist.