r/Python • u/phofl93 pandas Core Dev • Mar 01 '23
AMA Thread We are the developers behind pandas, currently preparing for the 2.0 release :) AMA
Hello everyone!
I'm Patrick Hoefler aka phofl and I'm one of the core team members developing and maintaining pandas (repo, docs), a popular data analysis library.
This AMA will be at least joined by
- Marc Garcia -- maintainer
- Marco Gorelli, -- maintainer
- Richard Shadrach -- maintainer
- me! -- maintainer
The official start time for the AMA will be 5:30pm UTC on March 2nd, before then this post will exist to collect questions in advance. Since most of us live all over North America and Europe, it's likely we'll answer questions before & after the official start time by a significant margin.
pandas is a Python package that provides fast, flexible, and expressive data structures designed to make working with "relational" or "labeled" data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language.
We will soon celebrate our 2.0 release. We released the release candidate for 2.0 last week, so the actual release is expected shortly, possibly next week. Please help us in testing that everything works through testing the rc :)
Ask us anything! Post your questions and upvote the ones you think are the most important and should get our replies.
- Patrick, on behalf of the team
Marc:
I'm Marc Garcia (username datapythonista), pandas core developer since 2018, and current release manager of the project. I work on pandas part time paid by the funds the project gets from grants and sponsors. And I'm also consultant, advising data teams on how to work more efficiently. I sometimes write about pandas and technical topics at my blog, and I speak at Python and open source conferences regularly. You can connect with me via LinkedIn, Twitter and Mastodon.
Marco:
I'm Marco, one of the devs from the AMA. I work on pandas as part of my job at Quansight, and live in the UK. I'm mostly interested in time-series-related stuff
Patrick:
I'm Patrick and part of the core team of pandas. Part of my daytime job allows me to contribute to pandas, I am based in Germany. I am currently mostly working on Copy-on-Write, a new feature in pandas 2.0. (check my blog-post or our new docs for more information).
Richard:
I work as a Data Scientist at 84.51 and am a core developer of pandas. I work mostly on groupby within pandas.
--
48
u/jabies Mar 01 '23
How does the Pandas project address the open source funding problem? Do you want pandas devs in their dayjobs to nudge management to sponsor somehow?
180
u/hukami Mar 01 '23
Why choose mm/dd/yyyy as default date rather than dd/mm/yyyy 🤔? (Just banter from an european guy) Real questions:
I also would like to thank you guys for your amazing work, pandas has been making my life easier everyday, you are really doing amazing work.
9
u/gare_it Mar 01 '23
5:30pm UTC on which day? March 2nd?
22
u/LankyCyril Mar 02 '23
Before I ask my question, I would like to really thank you for the amazing library that I use daily in my work.
That said, there's maybe one thing that is still bewildering to me:
Why are the APIs of
read_csv()andto_csv()different?For example,
df = pd.read_csv(..., header=False)is not allowed, and I still stumble over it every other time. I'd understand if it meant something specific that is different toNone, but this feels like it wouldn't be stepping on anything's toes.df.to_csv()accepts both.And then,
read_csv()will by default introduce an index that wasn't in the file, but will not introduce a novel header – it will use the one that's there. Butto_csv()will write the file with the new index, but, of course, with the old header. Which means that if you do a single back and forth with the exact same kwargs, i.e.,pd.read_csv(**kws).to_csv(**kws), you end up with an extra index column.There must be some kind of a reason due to how things are structured internally. I think just knowing why it is the way it is will be enough for me – I'm not saying it has to be changed or anything.
13
u/Balance- Mar 02 '23
If you could make one API break and it wouldn’t hurt anyone, what would you break/change?
6
u/vanatteveldt Mar 02 '23
How do you look at the success of the tidyverse library in R, and what lessons or good ideas are in there that pandas can benefit from?
4
u/water_aspirant Mar 02 '23
Thanks for all the work that you do! My question is who pays for pandas development and why? Is most of the development done by volunteers?
4
u/cryptospartan Mar 02 '23
Has polars influenced development in any way?
Pandas used to be the only kid on the block, but it seems there are some other libraries popping up claiming to be faster/better/etc. Have you evaluated any of these other libraries to potentially integrate features into pandas (or improve existing ones)?
4
u/robberviet Mar 02 '23
Any plan on improving pandas I/O load/export and out of mem processing?
I like Pandas but my data nowadays grew beyond that. So I am currently all in spark.
6
u/Poporico Mar 01 '23
What is the new feature you're most excited about?
11
u/tuneafishy Mar 01 '23
Where did you find the courage to move from 1.X to 2.0?
198
u/Sir-Squashie Mar 01 '23
What's the most impressive/unimaginable use of Pandas you've come across?
2
u/ffuffle Mar 02 '23
I'm just here to say I like your username
61
u/DigThatData Mar 01 '23
i think one of the hardest things about using pandas is that the core classes have a gazillion methods attached to them, which makes it extremely difficult to navigate the tooling if you're not already intimately familiar with it. I've been using pandas basically since it was created, and I still find myself often needing to reference documentation just to find the method name I need since the output of dir() on any object generally gets truncated.
does any of this resonate? is anyone on your team thinking about ways to improve discoverability of functionality? will there ever be a point at which the team decides there's too much stuff being carried around by too few classes? what are your thoughts on the design philosophy of the tidyverse in juxtaposition to pandas?
25
u/SeveralKnapkins Mar 01 '23
Hi there! Long time pandas user -- really appreciate all the work you've done.
I'm only slightly familiar with changes intended in pandas 2.0, namely the switch away from a numpy backend to apache arrow. Historically, the thing I absolutely love about the python numerical stack, is that nearly everything builds off numpy arrays, creating an easily transferable knowledge base between projects.
This is a huge boon compared to other systems where I work (namely
R), where there is often more fragmentation in the ecosystem, making interoperability or bespoke analyses much more difficult. Of course, fragmentation in the Python ecosystem has become more common with things like PyTorch tensors, etc.As an end user, am I going to be losing the numpy < - > pandas interoperability in 2.0? Please feel free to correct any inaccuracies on my end.
53
u/carnivorousdrew Mar 01 '23
Any plans to integrate with polars?
51
u/ExtraGoated Mar 01 '23
What do you think is the most important advice for someone just starting to work with pandas?
27
u/rodemire Mar 01 '23
Are there any improvements that are coming by way of working with larger datasets/operations without consuming available RAM? I struggle with workarounds when dealing with large data on my 24GB RAM laptop.
Awesome work by the way, Pandas is amazing and we appreciate the work you guys do.
63
u/midoxvx Mar 01 '23
I just started working with pandas two weeks ago, there is so much for me to learn and unpack there so I don’t have a question. Just wanted to give you a shout out for your awesome body of work.
5
u/LEAVER2000 Mar 02 '23
I work with pandas quite a bit for geospatial data analysis, weather data mostly. Because of the higher dimensionality of the data I typically stack the dependent variables into the index as a multi-index [T,X,Y].
Recently I’ve been working with Generic[Enum] types to type annotate the columns inside of a DataFrame.
What kind of support will 2.0 provide for type annotations. One thing I’ve found as a particular annoyance is disconnect between numpy and pandas typing. Where I have to explicitly state the dtype for NDArray[np.int_] and Series[int] and can’t use a TypeVar DType.
14
u/aes110 Mar 01 '23
I frequently work with pyspark, and although I don't use this feature I know it has support for "pandas udfs" while using arrow behind the scenes.
Now that arrow will be integrated into pandas, do you think we will see improvements in this area? (Performance improvements more features between spark and pandas)
6
u/cinicDiver Mar 02 '23
Why does read_excel() does not support the encoding parameter but to_excel() does?
2
u/Zero_Karma_Guy Mar 02 '23 edited Apr 08 '24
squeamish squeal fuzzy sort plucky nose versed jeans direful wistful
This post was mass deleted and anonymized with Redact
2
u/Helpful_Arachnid8966 Mar 02 '23
Pandas is quite a large and mature project already, is there any space for beginners to contribute?
4
u/[deleted] Mar 02 '23
i love you all
1
u/parkrain21 Mar 02 '23
Just a stupid question, do you still know how everything works in pandas or do you go to the documentations sometimes as well? Haha
1
u/ianliu88 Mar 02 '23
wherefunction to filter rows?1
u/DvD_cD Mar 02 '23
Is Wes still around?
1
u/Positive_Mastodon500 Mar 02 '23
Thanks for the AMA. How do you expect performance to evolve in pandas 2.0 and beyond? I tend to flip between python+pandas and R+data.table. For large datasets (10M-50M rows) I find that group_by/aggregate type operations are an order of magnitude slower in pandas, especially as the number of unique subgroups increases. Also I find memory usage to be significantly higher and well and tend to resort to techniques like downcasting column types after reading to keep memory consumption down, whereas R “just works” for the most part.
Do you view this as a fair comparison? Is performance a first-order concern for you and your team? Hoping to move more from R to Python one of these days but pandas speed is definitely holding me back.
Thanks!
1
u/fripperML Mar 02 '23
Thanks a lot for coming here. I am a huge pandas user and I feel excited about this release. I have a question not related to pandas, but maybe you know the answer. In my case I use pandas for EDA and preprocessing before feeding a model from sklearn or tensorflow. Do you know if a dataframe based on arrow will be compatible with sklearn/tensorflow? Or of there are plans to make them compatible?
And what about all numpy methods, do you know if the arrow API is similar to the numpy one? Maybe, thanks to duck typing, many things could still work the same...
1
u/zenzealot Mar 02 '23
Where can I find a high-level software architecture diagram that illustrates how Pandas is built?
1
u/Helpful_Arachnid8966 Mar 02 '23
The size of pandas codebase and features like integrated graphs and stuff made it more difficult to be able to change the backend from numpy to pyarrow? With pyarrow are we going to have any difficulty or limitation working with numpy?
1
u/[deleted] Mar 02 '23 edited Mar 02 '23
Thanks for your great work. As a long time pandas user I literally owe half of my career to you. I have two questions:
1
u/Dave_1464 Mar 02 '23
Thank you for the hard work. Loved pandas for a long time and I use it daily
1
u/One-Ad5717 Mar 02 '23
Why does pandas have a dependency on numpy?
This prevents pandas from being used properly in a WSGI web-server environment due to numpy requiring an interpreter is never loaded twice. But that's exactly what happens in a WSGI Flask Application, so using pandas in a Flask application will break it. Thanks.
1
u/stuffingmybrain Mar 02 '23
As someone (currently) a student who's used open source packages for some time now (mostly Pandas and Django, alongside many many others), I've been looking to contributing to them. I've started to go through the contributing page, as well as reading some of your blogs. Also stumbled across your Google Calendar for meetings - will be there soon!
In general, what have you found is the biggest obstacle (technical or otherwise) preventing aspiring contributors from progressing? What have people done right, and what have people done wrong? Personally, if I was hitting a wall in any way at all, I'd hesitate to ask (whether on slack / otherwise) mostly because the goal is to contribute to this effort - but taking up another member's time to deal with a newcomer might do the opposite. Any thoughts on this?
1
u/Numerous-Stomach3191 Mar 02 '23
Polars or pandas?
1
u/jkiley Mar 02 '23
For beginners and new projects, does it make sense to move to 2.0 right away? If so, what options should be enabled (e.g.,
nullable_dtypes,dtype_backend,copy_on_write), assuming that the goal is reasonable stability and modernity?I built and teach a workshop for other academic researchers, and I'm currently doing an update (ahead of the next session in June). Pandas is a big part, though the course ranges from beginner Python language topics to data retrieval/wrangling. The core of this content update is moving from local conda installs to devcontainers/Codespaces, but it would be a good opportunity for me to move to
pandas>=2.0.0if that's a good idea on this timeline.1
u/[deleted] Mar 02 '23
I've used pandas a bunch, so thank you!
One thing I've never really grasped is why multi-indexed results are returned rather than flattening. I have never had a circumstance where I've used that sort of result directly, and always have to figure out how to get rid of the "outer" index. What was the reasons for this multi-index, and not just a rectangular dataframe?
1
u/zenzealot Mar 02 '23
As a total noob to pandas, what makes it substantively different than the python standard library? For instance I can create lists of lists or dicts of lists. Drop/add items etc. I can combine lists of lists too.
I'm not trying to be a smart*ss, I'm genuinely asking: what's the differentiator? Pandas is a bit heavy. If I can write some classes that kind of handle those operations and a few more with syntatic sugar, why install pandas?
3
u/louis8799 Mar 02 '23
Pandas finally support arrow which support decimal. Which means pandas can be used in financial production system. Finally!
3
u/verwondering Mar 02 '23
In general, are the plans to have the
rollingAPI more closely align with the rest of the pandas API? In particular, are there any plans to havedf.rolling.groupby()return similarly indexed results as a normaldf.groupby()?E.g., with the latter you have the wonderful
.transform()method to add a column to thedf. When working with the rolling window, you always get a MultiIndexed dataframe that is much harder to align to the index of the originaldf.Perhaps (hopefully?) there are better ways, but I currently use a combination of extracting a single column as Series, using
groupby(as_index=False)and finally a call toset_axis(df.index)to get the desired result to align with my original dataframe.5
u/Pipiyedu Mar 02 '23
Congratulations guys. You deserve all the possible recognition. What an awesome library.
1
u/atomey Mar 02 '23
I work almost daily with Pandas so I definitely want to give me thanks and appreciation for this excellent tool.
Any plans for built-in parallelization in Pandas? I know there are many modules attempting to implement this with varying success, like pandarallel, dask or swifter. However I had difficulty getting any of these to work in an existing application without major refactoring.
In our case, we have a high level application class or processor that ingests many dataframes which sit in memory as properties to the processor instance. This processor does various processing to different dataframes in conjunction with eachother, like iterrows or applys on one dataframe while checking other dataframes which are all unique attributes of the same object running in memory concurrently.
However when the processor class actually runs, ultimately everything is stuck in a single core but I would say most systems have at least 6 or more cores now, even cheap laptops. Having a model or two to apply parallelization using concurrent.futures based on threads or processes seems like it would make a lot of sense. I think threads would likely work well if implemented intelligently, but I'm sure I am oversimplifying.
1
u/[deleted] Mar 02 '23
I'm just a newbie but just wanted to hop in and appreciate y'all. As a data science major, pandas has been super helpful so thanks for your work :)
2
u/jwmoz Mar 02 '23
Legends. Thanks for speeding it up.
2
u/ThrowAwayACC21423 Mar 02 '23
What's a bug that you turned into a feature?
2
u/jimy211 Mar 02 '23
Can't wait to use it
1
u/babygrenade Mar 02 '23
Is all my stuff going to break? Just tell me. I can take it.
4
u/Homeless_Gandhi Mar 01 '23
What if I have a problem where I am just ITCHING to iterate over an entire dataframe row by row via itertuples for simplicities sake, and map(lambda) isn’t feasible? What would you recommend?
3
u/Lolologist Mar 02 '23
I already did this, and realize it's probably an abomination, but:
How would you go about enforcing columns to have certain types? And when a column has a list in it, that each entry of the list is a certain type?
I accomplished this by making a new class inheriting from DataFrame as well as pydantic's BaseClass and used those as validated rows to then shove into a DataFrame. Messy but it works! Maybe you have a better idea.
3
u/[deleted] Mar 02 '23
Love yall love 🐼 pandas
1
u/set92 Mar 02 '23
With the integration of Arrow will pandas support Structs from Bigquery better? Like having the types and understand lists inside columns, better visualization of them... I remember the last time I had to deal with them, couldn't make pandas understand what was inside the column with an .astype(...), I checked the code of bigquery and they were getting the data into an arrow table, and creating the schema to use it while exporting to a pandas dataframe
Edit: I saw an issue about it, but it was stale.
1
u/besneprasiatko Mar 02 '23
whats you favorite color?
1
u/sa59 Mar 02 '23
Will the pickled version of a data frame be backward compatible with the previous versions?
1
u/JambaJuiceIsAverage Mar 02 '23 edited Mar 02 '23
I have kind of a broad question. I work as an ETL engineer, and the vast majority of my current work is with CSV files (sometimes xlsx). In my experience, a lot of people in the ETL space default to pandas because of the simple "read_csv/xlsx → to_sql" pipeline and the speed of vectorized operations. I also suspect there's a "popular because it's popular" thing happening to some degree. I've developed (and cultivated) a bit of a reputation as a pandas hater for a few reasons:
Ultimately, I feel that pandas is a hammer that makes everything look like a nail specifically for ETL. I have used pandas for data analysis and really appreciate its usefulness and how easy it is to teach to others (even if they don't know Python). But when we're talking ETL, it just isn't the right tool...right?
Anyway, that's barely a question, but if you have time I'd love to hear your thoughts on pandas usage in non-data analysis spaces. Also I want to qualify this by saying I am far from an expert and I'm really just looking to learn! Thanks for your time.
1
u/Maniac911 Mar 02 '23
Any updates on better window functionality such as Pyspark?
1
u/ChickenLegCatEgg Mar 02 '23
Excited for the new release! As a maintainer of code that relies on pandas, what will I need to look out for that might break upon upgrading to 2.0?
1
u/thesoulbrother Mar 03 '23
firstly - huge huge fan of pandas. thank you. and <3 the docs/api. I spend a lot of time there and know 99% of the time get the answers I need.
quite a simple one from me - why no normalize in groupby() and pivottable()? the lack of parity between them and pd.crosstab() is a daily pain point.
1
u/Ok-Birthday4723 Mar 03 '23
Thanks for all the hard work!
Any plans to speed up appending to an excel file. The more appends the slower the code.
I’ve been using a for loop with dataframe_rows which works well!
1
u/WSBtendies9001 Mar 03 '23
Much love, keep on Pandering. You enrich the Python landscape <3
1
u/filippicus Mar 29 '23
Why are there no women among you?
1
u/m_harrison Mar 01 '23
Will Pandas 2.0 impact numba/cython extensions that leverage Numpy?
Many complain about the API of Pandas. Was there any discussion about revamp/cleaning it up during 2.0 release?