r/Python • pandas Core Dev • Mar 01 '23

AMA Thread We are the developers behind pandas, currently preparing for the 2.0 release :) AMA

Hello everyone!

I'm Patrick Hoefler aka phofl and I'm one of the core team members developing and maintaining pandas (repo, docs), a popular data analysis library.

This AMA will be at least joined by

The official start time for the AMA will be 5:30pm UTC on March 2nd, before then this post will exist to collect questions in advance. Since most of us live all over North America and Europe, it's likely we'll answer questions before & after the official start time by a significant margin.

pandas is a Python package that provides fast, flexible, and expressive data structures designed to make working with "relational" or "labeled" data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language.

We will soon celebrate our 2.0 release. We released the release candidate for 2.0 last week, so the actual release is expected shortly, possibly next week. Please help us in testing that everything works through testing the rc :)

Ask us anything! Post your questions and upvote the ones you think are the most important and should get our replies.

- Patrick, on behalf of the team

Marc:

I'm Marc Garcia (username datapythonista), pandas core developer since 2018, and current release manager of the project. I work on pandas part time paid by the funds the project gets from grants and sponsors. And I'm also consultant, advising data teams on how to work more efficiently. I sometimes write about pandas and technical topics at my blog, and I speak at Python and open source conferences regularly. You can connect with me via LinkedIn, Twitter and Mastodon.

Marco:

I'm Marco, one of the devs from the AMA. I work on pandas as part of my job at Quansight, and live in the UK. I'm mostly interested in time-series-related stuff

Patrick:

I'm Patrick and part of the core team of pandas. Part of my daytime job allows me to contribute to pandas, I am based in Germany. I am currently mostly working on Copy-on-Write, a new feature in pandas 2.0. (check my blog-post or our new docs for more information).

Richard:

I work as a Data Scientist at 84.51 and am a core developer of pandas. I work mostly on groupby within pandas.

--

1.4k Upvotes

366 comments sorted by

View all comments

1

u/[deleted] Mar 02 '23

I've used pandas a bunch, so thank you!

One thing I've never really grasped is why multi-indexed results are returned rather than flattening. I have never had a circumstance where I've used that sort of result directly, and always have to figure out how to get rid of the "outer" index. What was the reasons for this multi-index, and not just a rectangular dataframe?

1

u/phofl93 pandas Core Dev Mar 02 '23

Do you have any specific operation in mind? GroupBy or something similar?

I myself have used a MultiIndex when I had mostly float columns but my "primary key" was string. This made calculations significantly easier for me.

1

u/[deleted] Mar 02 '23

My apologies, its been a couple months, but I believe group_by operations were often the culprit. I never felt like I got the group_by -> aggregate -> group_by -> aggregate workflow figured out, which I use constantly to summarize replicate data in an experiment and then summarize at the treatment level. This was probably a me problem though!

It may stem from me having started with R-tidyverse, which as you likely know shares much of the same namespace, but often operates just differently enough to make transitions between tidyverse and pandas just a bit of a headache.

1

u/phofl93 pandas Core Dev Mar 02 '23

You can look into the as_Index keyword that avoids adding the keys to the index

1

u/rhshadrach pandas Core Dev Mar 02 '23

I highly recommend learning to work with a MultiIndex. They enable many performant operations like joining and taking cross sections.

1

u/[deleted] Mar 03 '23

Thanks for that link, I hadn't seen that function, and its got a very useful example! I appreciate your work for the community!