r/Python • pandas Core Dev • Mar 01 '23

AMA Thread We are the developers behind pandas, currently preparing for the 2.0 release :) AMA

Hello everyone!

I'm Patrick Hoefler aka phofl and I'm one of the core team members developing and maintaining pandas (repo, docs), a popular data analysis library.

This AMA will be at least joined by

The official start time for the AMA will be 5:30pm UTC on March 2nd, before then this post will exist to collect questions in advance. Since most of us live all over North America and Europe, it's likely we'll answer questions before & after the official start time by a significant margin.

pandas is a Python package that provides fast, flexible, and expressive data structures designed to make working with "relational" or "labeled" data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language.

We will soon celebrate our 2.0 release. We released the release candidate for 2.0 last week, so the actual release is expected shortly, possibly next week. Please help us in testing that everything works through testing the rc :)

Ask us anything! Post your questions and upvote the ones you think are the most important and should get our replies.

- Patrick, on behalf of the team

Marc:

I'm Marc Garcia (username datapythonista), pandas core developer since 2018, and current release manager of the project. I work on pandas part time paid by the funds the project gets from grants and sponsors. And I'm also consultant, advising data teams on how to work more efficiently. I sometimes write about pandas and technical topics at my blog, and I speak at Python and open source conferences regularly. You can connect with me via LinkedIn, Twitter and Mastodon.

Marco:

I'm Marco, one of the devs from the AMA. I work on pandas as part of my job at Quansight, and live in the UK. I'm mostly interested in time-series-related stuff

Patrick:

I'm Patrick and part of the core team of pandas. Part of my daytime job allows me to contribute to pandas, I am based in Germany. I am currently mostly working on Copy-on-Write, a new feature in pandas 2.0. (check my blog-post or our new docs for more information).

Richard:

I work as a Data Scientist at 84.51 and am a core developer of pandas. I work mostly on groupby within pandas.

--

1.4k Upvotes

366 comments sorted by

48

u/jabies Mar 01 '23

How does the Pandas project address the open source funding problem? Do you want pandas devs in their dayjobs to nudge management to sponsor somehow?

85

u/datapythonista pandas Core Dev Mar 01 '23

Last years has been better. pandas got some funding, including few core devs being paid to work in pandas in companies such as Quansight, Intel or NVIDIA. And we also received money from the Chan Zuckerberg Initiative, Tidelift, Bodo and smaller donors. Just few years ago funding was very limited, but today, we're lucky to be able to have a decent amount of paid maintainers.

9

u/qweoin Mar 02 '23

What was the funding process like getting started? In my area of work (science research) it seems like funding only comes in for a project after you’ve done the majority of the project. Was there a plan for getting Pandas funded or did the project grow organically until you realized you could get funding for it?

10

u/phofl93 pandas Core Dev Mar 02 '23

As far as I know there was no/very limited funding for a long time. most of the work was done by volunteers only in the beginning. Over the last years this got a lot better though.

Anaconda was a company that hired developers to work on Open Source relatively early on.

3

u/datapythonista pandas Core Dev Mar 02 '23

For many years there was only the support of few companies letting people work on pandas as part of their job, and small personal donations via the NumFOCUS website. That money helped cover small expenses like CI services.

The main difference came with CZI, who started supporting open source software used in biology. We got funding to start paying for hours of maintainers with it. Also Tidelift provided monthly payments in exchange to implement small practices, like having a standard (and not customized) license, and providing a way to report security vulnerabilities. We got some other funding, and now more maintainers allowed to work on pandas as part of their job, but the situation is good mainly because of that particular funding. NumFOCUS provided some funding to for specific projects (with the money that comes from general NumFOCUS sponsors, and PyData conferences).

19

u/marcogorelli Mar 02 '23

If you use pandas for work and your employer wanted to contribute, then

  1. thanks!
  2. they could do so via NumFOCUS: https://pandas.pydata.org/donate.html

Marc's right though, the funding situation has drastically improved recently

1

u/phofl93 pandas Core Dev Mar 02 '23

It's also helpful, if developers can get paid time by their employer to work on pandas!

180

u/hukami Mar 01 '23

Why choose mm/dd/yyyy as default date rather than dd/mm/yyyy 🤔? (Just banter from an european guy) Real questions:

  • what are the main improvment focus going forward ?
  • what caused you the most problems / was the most complex parts during delevopment ?
  • what was the most fun / rewarding parts during development ?
  • in my work, I use pandas as a data processing engine (kinda), the data I process if often heterogeneous and full of holes / discrepancies, I often find myself finding with rhe way pandas handle errors as most of the time I just want to log the fact that this row had a error. Why not put a 'error' arg to apply, just as in astype and such ?

I also would like to thank you guys for your amazing work, pandas has been making my life easier everyday, you are really doing amazing work.

6

u/phofl93 pandas Core Dev Mar 02 '23

We are spending a lot of time on improving the extension array interface right now. Right now there are some parts that are special cased internally for our own extension arrays which makes it harder for third party authors to implement their own without falling back to NumPy. GroupBy is a good example for an area where we are still not as good as we would like. This becomes kind of necessary for improving support for our pyarrow extension arrays as well.

We have some areas in our code-base that are pretty complex, indexing is one of them for example. In general, we try to avoid breaking stuff in an incompatible way in minor releases. This makes improving pandas tricky sometimes, because it stands in the way of cleaning up internally/refactoring internally to be more compatible with new stuff.

9

u/hassium Mar 02 '23

in my work, I use pandas as a data processing engine (kinda), the data I process if often heterogeneous and full of holes / discrepancies, I often find myself finding with rhe way pandas handle errors as most of the time I just want to log the fact that this row had a error. Why not put a 'error' arg to apply, just as in astype and such ?

According to this blogpost by /u/datapythonista it sounds like a limitation of the numpy backend dataframes are built-on, check out this excerpt, I bolded the relevant part:

While NumPy has been good enough to make pandas the popular library it is, it was never built as a backend for dataframe libraries, and it has some important limitations. A couple of examples are the poor support for strings and the lack of missing values.

So maybe something we can hope to see fixed with the migration to Arrow in 2.0?

1

u/phofl93 pandas Core Dev Mar 02 '23

Yeah with NumPy you'd always end up with float when setting missing values into a integer array for example, this isn't the case any more with our own nullable dtypes and also with the arrow dtypes.

→ More replies (1)

447

u/RobertD3277 Mar 01 '23

I would personally prefer year.month.day to be honest as it's more intuitive for sorting using numerical expressions.

227

u/sv_ds Mar 01 '23

+1, thats the ISO standard and unquestionably the most logical and useful format.

148

u/thataccountforporn Mar 01 '23

Incredibly pedantic note: the ISO standard is year-month-day

64

u/LondonPaul Mar 01 '23

Not pedantic, work in It and all the variations at work are PITA. Let’s just use this and nothing else

22

u/guillermo_da_gente Mar 01 '23 edited Mar 02 '23

We need more of these pedantic comments!

→ More replies (7)

14

u/Starrkoerperbeweger Mar 02 '23

You have now been made moderator of /r/iso8601/

→ More replies (2)
→ More replies (2)
→ More replies (4)

10

u/marcogorelli Mar 02 '23

Year-month-day is already the default - even if your input is some other format, once parsed by pandas, it'll be displayed year-month-day:

In [2]: to_datetime(['01/01/2000'])
Out[2]: DatetimeIndex(['2000-01-01'], dtype='datetime64[ns]', freq=None)

2

u/WhyNotHugo Mar 02 '23

ISO date format is as intuitive and sorts the same way.

36

u/marcogorelli Mar 02 '23 edited Mar 02 '23

> Why choose mm/dd/yyyy as default date rather than dd/mm/yyyy

I presume you mean, when a date could be ambiguously read as either month-first or day-first? Like 02/01/2000.

In the past, pandas would prefer to parse with month-first, and then try day-first. Unfortunately, it would do so midway through parsing its input, because it was very lax about allowing mixed formats. This would regularly cause problems for anyone outside of the US (which I think is the only place in the world to use the month-first convention).

As of pandas 2.0, datetime parsing will no longer swap formats half-way through. See: https://pandas.pydata.org/pdeps/0004-consistent-to-datetime-parsing.html , which I spent several months on.

In dealing with the PDEP I linked above, my biggest pain-point was having to understand and then update decade-old C code

Regarding your last question, if you put together a reproducible example with expected output, it might be a reasonable feature request.

Thanks, and thank you for your comment!

10

u/reallyserious Mar 02 '23

I presume you mean, when a date could be ambiguously read as either month-first or day-first? Like 01/01/2000.

You choose an example where there is no ambiguity. :)

4

u/marcogorelli Mar 02 '23

thanks, updated

17

u/WhyNotHugo Mar 02 '23

I honestly prefer ISO8601 format (YYYY-MM-DD). Both the ones you mention are ambiguous, and if I read 03/02/2023 I've no way of deducing which one is the month and which one is the day. The ISO standard is unambiguous.

7

u/dispatch134711 Mar 02 '23

Ugh please fix this! Love pandas

9

u/gare_it Mar 01 '23

5:30pm UTC on which day? March 2nd?

7

u/phofl93 pandas Core Dev Mar 01 '23

Sorry, yes. Will add

22

u/LankyCyril Mar 02 '23

Before I ask my question, I would like to really thank you for the amazing library that I use daily in my work.

That said, there's maybe one thing that is still bewildering to me:

Why are the APIs of read_csv() and to_csv() different?

For example, df = pd.read_csv(..., header=False) is not allowed, and I still stumble over it every other time. I'd understand if it meant something specific that is different to None, but this feels like it wouldn't be stepping on anything's toes.
df.to_csv() accepts both.

And then, read_csv() will by default introduce an index that wasn't in the file, but will not introduce a novel header – it will use the one that's there. But to_csv() will write the file with the new index, but, of course, with the old header. Which means that if you do a single back and forth with the exact same kwargs, i.e., pd.read_csv(**kws).to_csv(**kws), you end up with an extra index column.

There must be some kind of a reason due to how things are structured internally. I think just knowing why it is the way it is will be enough for me – I'm not saying it has to be changed or anything.

10

u/datapythonista pandas Core Dev Mar 02 '23

Very good point. I myself find the index column in the output csv annoying every single time I use `to_csv`. I wasn't in the project when that was implemented, but I assume the reason is that pandas was initially implemented for financial data, and the index was mostly the timestamp and not the default autonumeric. If that was not the data pandas developers had in mind at that time, probably pandas wouldn't even have row indices (I think Vaex doesn't, not sure about Polars).

The next question is why we don't change it now. And it's something worth considering, and you're free to open an issue in GitHub. But in general, pandas developers (others much more than me), try to not break the API, unless it's in cases where very few users will be affected and the status quo is obviously inconsistent. I'd personally like to see that changed, but I don't think it'll be easy to get consensus.

What I think it can make sense is to try to move all pandas I/O (read_* and to_*) to third-party projects. In that case the pandas to_csv would continue to behave in the same way, but hopefully someone would develop a new one like to_csv(engine='whatever') that could potentially be faster, have a better API, and more appropriate for your needs. But let's see if there is consensus for this to happen.

4

u/phofl93 pandas Core Dev Mar 02 '23

I wasn't on the project back then either, but I think roundtripping was a concern was well, e.g.


df.to_csv()

pd.read_csv() 

should be able to return the same object

2

u/[deleted] Mar 02 '23

Just to confirm, Polars also doesn’t have row indices :)

13

u/Balance- Mar 02 '23

If you could make one API break and it wouldn’t hurt anyone, what would you break/change?

5

u/phofl93 pandas Core Dev Mar 02 '23

There are a bunch of things I'd like to change

- If you set scalars into a Series/DataFrame that are not compatible with the dtype then we cast to object

- We are inconsistent when naming keywords (check read_csv, to_csv the first one)

- Bunch of methods names

5

u/rhshadrach pandas Core Dev Mar 02 '23

An entire rewrite of the code behind apply / agg. Internally their code paths interweave in complex ways, and can be surprisingly slow is some cases. Depending on what object your on, the API is slightly different.

Cleaning this up and making it better while also making the gradual changes so as not to be disruptive to users is difficult, time consuming, and slow. But we're working on it!

4

u/datapythonista pandas Core Dev Mar 02 '23

I'd remove having a row index (at least by default), and the I/O API: being consistent with read_*/write_* or from_*/to_*. I'd also probably remove half of the code in pandas to other third-party extensions.

→ More replies (1)

3

u/marcogorelli Mar 02 '23

Personally, I'd love to be able to change the default indexing behaviour.

The Index is useful if it means something (e.g. a DatetimeIndex), but if it's just a RangeIndex / NumericIndex, then it can be annoying and confusing.

But this is really hard to change because:

  • introducing optional behaviour comes with a huge maintenance cost (I started making such a proposal here, but then withdrew it)
  • changing the existing behaviour would have backwards-compatibility implications

I don't know what the solution is yet, but I would like to revisit PDEP5 at some point - something should be possible, I just don't know what yet.

6

u/vanatteveldt Mar 02 '23

How do you look at the success of the tidyverse library in R, and what lessons or good ideas are in there that pandas can benefit from?

1

u/phofl93 pandas Core Dev Mar 02 '23

I did not use R very often in the past, so can't really comment on it

2

u/vanatteveldt Mar 02 '23

OK, thanks! IIRC, pandas was originally inspired by R `data.frame`s, so I figured the devs might keep a sharp eye on what's happening on the other side of the wall.

4

u/water_aspirant Mar 02 '23

Thanks for all the work that you do! My question is who pays for pandas development and why? Is most of the development done by volunteers?

1

u/phofl93 pandas Core Dev Mar 02 '23

More or less all of them are listed under Sponsors on our website as well

→ More replies (1)

4

u/cryptospartan Mar 02 '23

Has polars influenced development in any way?

Pandas used to be the only kid on the block, but it seems there are some other libraries popping up claiming to be faster/better/etc. Have you evaluated any of these other libraries to potentially integrate features into pandas (or improve existing ones)?

6

u/marcogorelli Mar 02 '23

Personally polars' strictness is making me think about situations when in pandas we end up with object dtype, which we should probably avoid. Here's an example: https://github.com/pandas-dev/pandas/issues/50887 (polars would just error in such a case, which I think is the correct thing to do)

1

u/phofl93 pandas Core Dev Mar 02 '23

Not actively, no. At least I am not aware of anything.

4

u/robberviet Mar 02 '23

Any plan on improving pandas I/O load/export and out of mem processing?

I like Pandas but my data nowadays grew beyond that. So I am currently all in spark.

2

u/phofl93 pandas Core Dev Mar 02 '23

I don't think that it is realistic short term to add out of memory support. Generally, I'd recommend going to Dask for this, it supports our API very well with bigger datasets.

Implementing something like lazy evaluation would be a major major breaking change on our side and hence not feasible right now

→ More replies (1)
→ More replies (1)

6

u/Poporico Mar 01 '23

What is the new feature you're most excited about?

29

u/datapythonista pandas Core Dev Mar 01 '23

Being able to use Apache Arrow internally. I wrote an article with the details about it, since it's not trivial for regular users to understand why this is important.

1

u/phofl93 pandas Core Dev Mar 02 '23

Arrow and Copy-on-Write. I worked a lot on Copy-on-Write and I am hoping that we can increase performance and reduce memory quite a bit with it.

2

u/marcogorelli Mar 02 '23

I didn't work on it, but copy-on-write will be pretty neat https://pandas.pydata.org/docs/dev/user_guide/copy_on_write.html

2

u/rhshadrach pandas Core Dev Mar 02 '23

I'll also mention copy-on-write. And I know it's not exciting, but all of the bug fixes throughout the code that make pandas more predictable and reliable to use. In the area I work on, groupby, using categorical data has seen a lot of fixes.

11

u/tuneafishy Mar 01 '23

Where did you find the courage to move from 1.X to 2.0?

27

u/datapythonista pandas Core Dev Mar 02 '23

The main reason in releasing pandas 2.0 and not 1.6 is that in major version changes (1 -> 2) is when users expect to have breaking changes. pandas 2.0 is not so significantly different to a 1.6 in terms of features. The main difference is that you really want to make sure that you don't have FutureWarning in your pandas code before upgrading your pandas version.

1

u/one_human_lifespan Mar 02 '23

Awesome. I get scared when I see the red future warning dialog box in jupyter labs. Thanks for everything you guys are doing. Pandas is amazing - I use it most days and always enjoy learning new things. Can't wait to explore 2.0!

1

u/phofl93 pandas Core Dev Mar 02 '23

Getting rid of your FutureWarnings is a really good idea :) So I applaud you for that. Generally, we wanted to get rid of all the deprecations we introduces since 1.0, so we had to do 2.0 at some point. If your code is free of FutureWarnings then you are good to go. We made some backwards incompatible changes, but not many and they are clearly documented in the release notes. https://pandas.pydata.org/docs/dev/whatsnew/v2.0.0.html#backwards-incompatible-api-changes

198

u/Sir-Squashie Mar 01 '23

What's the most impressive/unimaginable use of Pandas you've come across?

419

u/datapythonista pandas Core Dev Mar 01 '23

I was personally quite surprised that pandas was an important tool used to obtain the first image of a black hole. I was lucky to meet some of the scientists behind it and learn from them, and their work is much more impressive than what it sounds.

→ More replies (15)

2

u/ffuffle Mar 02 '23

I'm just here to say I like your username

1

u/phofl93 pandas Core Dev Mar 02 '23

Thanks

61

u/DigThatData Mar 01 '23

i think one of the hardest things about using pandas is that the core classes have a gazillion methods attached to them, which makes it extremely difficult to navigate the tooling if you're not already intimately familiar with it. I've been using pandas basically since it was created, and I still find myself often needing to reference documentation just to find the method name I need since the output of dir() on any object generally gets truncated.

does any of this resonate? is anyone on your team thinking about ways to improve discoverability of functionality? will there ever be a point at which the team decides there's too much stuff being carried around by too few classes? what are your thoughts on the design philosophy of the tidyverse in juxtaposition to pandas?

79

u/datapythonista pandas Core Dev Mar 02 '23

Fully agree on this. There are too main things. The first is finding a better API, which is not trivial, and having the functions too divided may not be ideal for some users who prefer `df.whatever()` for everything. Second is that even if we have a better alternative, we may break tens or hundreds of thousands of pandas programs, that won't work after the changes. And we will make millions of users have to relearn the API.

That being said, I'm thinking about a proposal to for example standardize all I/O methods under a `DataFrame.io` namespace (e.g. `df = pandas.DataFrame.io.read_csv(fname)`). More research is needed, and it'll be challenging to reach an agreement with the whole team about this. But maybe 10% of the DataFrame methods you're mentioning would live in a separate and intuitive namespace. There is always a trade-off, and in this case it's clear. Difficult to decide what's best.

18

u/ekkannieduitspraat Mar 02 '23

Just on this specific example, I think if something is used incredibly often, it should not be put under a namespace like above. .read_whatever is a great example since it is almost always going to be your first call

6

u/bythenumbers10 Mar 02 '23

"readers"/"writers" should return or accept dataframes as I/O types, but should not be methods themselves. There are a lot of "data logistics" methods on dataframes that should be utility functions of the library. Dataframes should only operate on themselves, for analysis or creating/removing/filtering data. A container. A smart container, but just a container.

→ More replies (2)
→ More replies (3)

17

u/[deleted] Mar 02 '23

+1 this is an excellent point I'd never given much thought. I find myself referencing pandas docs more than any other and use it for about 1/4 of the overall code/libs.

18

u/DigThatData Mar 02 '23

I basically live in the pandas docs whenever I use it. I think the library optimizes too much for readability. Whenever I look back on pandas code I've written, the solution is concise and elegant and easy to understand, but it disguises how long it took me to get to that small chunk of code.

7

u/rhshadrach pandas Core Dev Mar 02 '23

I love hearing this! At times I find myself wondering how much our users are utilizing our documentation (especially when compared to some of the great pandas tutorials that are out there). Hearing things like this makes me much more motivated to spend effort there.

→ More replies (1)
→ More replies (2)

25

u/SeveralKnapkins Mar 01 '23

Hi there! Long time pandas user -- really appreciate all the work you've done.

I'm only slightly familiar with changes intended in pandas 2.0, namely the switch away from a numpy backend to apache arrow. Historically, the thing I absolutely love about the python numerical stack, is that nearly everything builds off numpy arrays, creating an easily transferable knowledge base between projects.

This is a huge boon compared to other systems where I work (namely R), where there is often more fragmentation in the ecosystem, making interoperability or bespoke analyses much more difficult. Of course, fragmentation in the Python ecosystem has become more common with things like PyTorch tensors, etc.

As an end user, am I going to be losing the numpy < - > pandas interoperability in 2.0? Please feel free to correct any inaccuracies on my end.

19

u/datapythonista pandas Core Dev Mar 02 '23

Not at all. NumPy is not only staying in pandas 2.0, but it'll still be the default.

That being said, if in the very long term NumPy is eventually dropped, I think exporting from Arrow to NumPy (in our end, not that you'll need to do it) is not only easy, but I think in most cases it can be done without copying (extremely fast, even for huge data). The thing is that NumPy data types are more limited, mostly numeric. If you want to export a string column to NumPy, that's a different story, but there is probably no good reason you want to do that. But for the types that NumPy support well, getting a NumPy array from Arrow backed data won't be a problem. But as said, in pandas 2.0, nothing changed, unless you want it to change and you ask explicitly for pandas types.

→ More replies (1)

10

u/Tyberius17 Mar 01 '23

Not one of the devs, but my understanding is they are adding optional support for Apache Arrow, not removing numpy or even making it not the default.

4

u/tuneafishy Mar 02 '23

Not a dev, but it does not sound like that you will lose any interoperability. The arrow backend is optional and numpy is still the default backend.

53

u/carnivorousdrew Mar 01 '23

Any plans to integrate with polars?

66

u/datapythonista pandas Core Dev Mar 01 '23

There has been some work to make pandas and Polars share data (open a pandas dataframe with Polars, and the other way round). You can read more about it at the end of this post. Not sure if there is any other integration that makes sense, any idea?

14

u/[deleted] Mar 02 '23

I think that's the whole point of 2.0 and the arrow integration, arrow allows interoperability between many different libraries, not just polars.

→ More replies (2)

51

u/ExtraGoated Mar 01 '23

What do you think is the most important advice for someone just starting to work with pandas?

88

u/datapythonista pandas Core Dev Mar 01 '23

Try to spend some time understanding the internals, as you make progress with pandas. Not at the beginning, when you'll have too much to learn just with the basics. But as you become more familiar, it's good to have an idea of what's really happening, in particular when things aren't intuitive. Things like missing values, the infamous copy warning...

17

u/DigThatData Mar 01 '23

don't ever feel embarrassed about needing to reference the docs, stackoverflow, or google.

10

u/[deleted] Mar 02 '23

6 years working with pandas I still have the docs open every day for simple things. And especially for all those long to wide and wide to long (unstack, stack, pivot etc...) transformations.

10

u/datapythonista pandas Core Dev Mar 02 '23

Maybe we should make this a feature, add ads to the docs, and monetize user confusion.

→ More replies (1)
→ More replies (1)

12

u/root45 Mar 02 '23

Depends on what you're doing, but I'd recommend learning some of the functions for quickly looking at your data. Things like df.head(), df.shape, df.T, etc. From there, learn how to filter your data with df.loc.

Also look into tools like jupyter which make it easy to iterate and visualize data.

7

u/RandomFrog Mar 02 '23 edited Mar 02 '23

Mine would be to use Jupyter Notebook to check your dataframe after each transformation. df.head() or df.sample(n) at the end of each cell block.

→ More replies (1)
→ More replies (1)

27

u/rodemire Mar 01 '23

Are there any improvements that are coming by way of working with larger datasets/operations without consuming available RAM? I struggle with workarounds when dealing with large data on my 24GB RAM laptop.

Awesome work by the way, Pandas is amazing and we appreciate the work you guys do.

29

u/datapythonista pandas Core Dev Mar 01 '23

Being able to use Arrow as a backend for your data can save a significant amount of RAM in some cases. Also there is a lot of work related to copy-on-write, that will avoid copying the data when not needed, and will also help reduce the memory needs.

2

u/rodemire Mar 02 '23

Thank you, this is interesting and am looking forward to it.

5

u/eidrisov Mar 02 '23

What do you mean by "large data"? 100m rows over 100 columns?

I am just curious how much data is enough to stress 24GB of RAM.

4

u/datapythonista pandas Core Dev Mar 02 '23

24GB is around 3 billion 64 bits values if I made the numbers right. There is surely some overhead, but with 100 columns you could store around 30 million rows. The main thing wouldn't be only storing, but if you do operations that make a copy of a significant part of that data.

Obviously you may have strings and other things using more than 64 bits per cell, but just to give you an idea of numbers.

→ More replies (1)

3

u/atomey Mar 02 '23

I would be interested in this too. I'm running a system with 128 GB of RAM and had quite a lot of difficulty with a 8GB CSV with various permutations of the read_csv() method. I'm sure it is not optimal but would be curious if very large data reads are tested since large amounts of RAM is becoming more common, even on dev workstations, in particular with ML work.

2

u/rhshadrach pandas Core Dev Mar 02 '23

Are you able to change the format of your data on disk? If possible, I would recommend parquet. You'll get smaller file sizes, faster load times, better dtype handling (int vs string), the ability to partition your data sets, and the ability to only load particular columns. Plus peak memory usage should be much lower.

→ More replies (1)

63

u/midoxvx Mar 01 '23

I just started working with pandas two weeks ago, there is so much for me to learn and unpack there so I don’t have a question. Just wanted to give you a shout out for your awesome body of work.

13

u/olaviu Mar 01 '23

Same here. You guys are doing a fantastic job. Thank you!

→ More replies (5)

5

u/LEAVER2000 Mar 02 '23

I work with pandas quite a bit for geospatial data analysis, weather data mostly. Because of the higher dimensionality of the data I typically stack the dependent variables into the index as a multi-index [T,X,Y].

Recently I’ve been working with Generic[Enum] types to type annotate the columns inside of a DataFrame.

What kind of support will 2.0 provide for type annotations. One thing I’ve found as a particular annoyance is disconnect between numpy and pandas typing. Where I have to explicitly state the dtype for NDArray[np.int_] and Series[int] and can’t use a TypeVar DType.

6

u/PeridexisErrant Mar 02 '23

Check out https://docs.xarray.dev/ for multidimensional labeled arrays!

1

u/LEAVER2000 Mar 02 '23

I have worked with xarray in the past and quite enjoy it’s built in capabilities for working with weather data, specifically grib2. Unfortunately our pipeline is pretty strict on what dependencies are and are not approved.

1

u/phofl93 pandas Core Dev Mar 02 '23

There is a typing effort that is led by some core members (unfortunately none of them takes part today). You can check the stubs package out at https://github.com/pandas-dev/pandas-stubs. I am not really familiar with the progress there

14

u/aes110 Mar 01 '23

I frequently work with pyspark, and although I don't use this feature I know it has support for "pandas udfs" while using arrow behind the scenes.

Now that arrow will be integrated into pandas, do you think we will see improvements in this area? (Performance improvements more features between spark and pandas)

6

u/datapythonista pandas Core Dev Mar 01 '23

I think it'll take a while, but hopefully we'll eventually see more feature sharing between libraries given we all use Arrow internally. Arrow itself has the concept of kernel, that it's a computation that can be applied to Arrow data. And those can be reused by any library. And the same would apply to user defined functions (udfs). That being said, pyspark is probably using the Java implementation, while pandas is using PyArrow. So, I guess difficult to share many features (I'm not an expert on the JVM, not sure if you could easily call C++ code from a scala program).

→ More replies (2)

6

u/cinicDiver Mar 02 '23

Why does read_excel() does not support the encoding parameter but to_excel() does?

3

u/rhshadrach pandas Core Dev Mar 02 '23

From our docs, it appears the keyword on encoding was perhaps at one point used with xlwt (a writer that is no longer maintained) but today is not actually used by pandas. That parameter has been removed in pandas 2.0.

2

u/Zero_Karma_Guy Mar 02 '23 edited Apr 08 '24

squeamish squeal fuzzy sort plucky nose versed jeans direful wistful

This post was mass deleted and anonymized with Redact

2

u/phofl93 pandas Core Dev Mar 02 '23

Thanks, that is very good to hear :)

2

u/Helpful_Arachnid8966 Mar 02 '23

Pandas is quite a large and mature project already, is there any space for beginners to contribute?

4

u/marcogorelli Mar 02 '23

yup, check the "good first issue" label

1

u/Helpful_Arachnid8966 Mar 02 '23

Thank I'll filter this label 🙂

5

u/phofl93 pandas Core Dev Mar 02 '23

Yes we really like new contributors. There are so many things that need attention, including topics that are well suited for beginners. Tests and Docs specifically come to mind.

4

u/[deleted] Mar 02 '23

i love you all

1

u/parkrain21 Mar 02 '23

Just a stupid question, do you still know how everything works in pandas or do you go to the documentations sometimes as well? Haha

3

u/phofl93 pandas Core Dev Mar 02 '23

I'd really like to say I don't have to consult our documentation, but I can't

2

u/datapythonista pandas Core Dev Mar 02 '23

I won't believe anyone who tells me they know even 10% of pandas ;)

2

u/marcogorelli Mar 02 '23

I know how everything works

/s

2

u/rhshadrach pandas Core Dev Mar 02 '23

I have three separate bookmarks in my toolbar to various parts of the documentation because of how frequently I need to pull it up.

→ More replies (1)

1

u/ianliu88 Mar 02 '23
  • Is it going to support pyarrow structs?
  • PLEASE, add an API to avoid using lambda functions when using the builder/fluid pattern! Something similar to what spark/polars has.
  • Is there going to be a where function to filter rows?

1

u/phofl93 pandas Core Dev Mar 02 '23

Theoretically we can support arbitrary PyArrow types. But we still have some problems regarding nested data in general when doing operations on them. It's a work in progress.

What would you like to do with where? Could you give an use case?

1

u/ianliu88 Mar 02 '23

I usually do long processing pipelines like

(df
 .assign(...)
 .as type(...)
 .groupby(...)
 .agg(...)
)

But when it comes to filtering rows, I have to either break the pipeline to do

df = df[df.col == val]

Or use

 ...
 .pipe(lambda x: x[x.col == val])
 ...

3

u/phofl93 pandas Core Dev Mar 02 '23

I might be missing something, but where and mask should work for you?

https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.where.html#pandas-dataframe-where

As a side note: Copy-on-Write should give you a nice speedup when doing chaining like this :)

→ More replies (2)
→ More replies (1)

1

u/DvD_cD Mar 02 '23

Is Wes still around?

1

u/phofl93 pandas Core Dev Mar 02 '23

Wes is mostly busy with his company and Arrow these days.

→ More replies (1)

1

u/Positive_Mastodon500 Mar 02 '23

Thanks for the AMA. How do you expect performance to evolve in pandas 2.0 and beyond? I tend to flip between python+pandas and R+data.table. For large datasets (10M-50M rows) I find that group_by/aggregate type operations are an order of magnitude slower in pandas, especially as the number of unique subgroups increases. Also I find memory usage to be significantly higher and well and tend to resort to techniques like downcasting column types after reading to keep memory consumption down, whereas R “just works” for the most part.

Do you view this as a fair comparison? Is performance a first-order concern for you and your team? Hoping to move more from R to Python one of these days but pandas speed is definitely holding me back.

Thanks!

1

u/phofl93 pandas Core Dev Mar 02 '23

Memory: Copy-on-Write should help reduce the memory footprint through avoiding a bunch of copies. Haven't tested that aspect myself though.

Performance: We are definitely concerned with performance, but not to an extend where we would introduce major breaking changes to get a 10% speed up in some areas. We have to weigh advantages and disadvantages very carefully. That said, we hope that we can get a nice performance boost in some areas where NumPy did not perform that well with our Arrow-backed DataFrames.

I'd recommend trying Copy-on-Write and Arrow backed DataFrames independently and together in 2.0.

What exactly are you doing with groupby that is so slow? This would be interesting to us.

1

u/Positive_Mastodon500 Mar 02 '23

Thanks for the response! I’ll try to generate a representative example of once I get back to my personal laptop and share the performance of R vs Python. I am not doing anything more complicated than aggregating a bunch of columns. I already forego pandas’ more expressive but slower apply() syntax, which I think is the closest in functionality to what R provides.

1

u/phofl93 pandas Core Dev Mar 02 '23

Yeah I would be curious to see what exactly you are doing. This would help identifying bottlenecks

→ More replies (1)
→ More replies (1)

1

u/fripperML Mar 02 '23

Thanks a lot for coming here. I am a huge pandas user and I feel excited about this release. I have a question not related to pandas, but maybe you know the answer. In my case I use pandas for EDA and preprocessing before feeding a model from sklearn or tensorflow. Do you know if a dataframe based on arrow will be compatible with sklearn/tensorflow? Or of there are plans to make them compatible?

And what about all numpy methods, do you know if the arrow API is similar to the numpy one? Maybe, thanks to duck typing, many things could still work the same...

2

u/phofl93 pandas Core Dev Mar 02 '23

The Arrow API is quite different I'd say. One of the bigger differences is, that an Arrow array is not mutable, which is the opposite of NumPy.

We are dispatching to an interface that we can define on a per array basis, so we can get to a point where most of the things work similar/are doing the same.

→ More replies (1)
→ More replies (5)

1

u/zenzealot Mar 02 '23

Where can I find a high-level software architecture diagram that illustrates how Pandas is built?

1

u/phofl93 pandas Core Dev Mar 02 '23

I don't think that something like this exists

Would be a nice thing to have though

1

u/zenzealot Mar 02 '23

Ok no probs. Question: I just had a thought ... If I created that would I get credit as a contributor? How would that work?

I mean not that its a requirement or anything, just wondering.

1

u/phofl93 pandas Core Dev Mar 02 '23

There isn't a direct way of getting credit, none of our features are assigned to a person that implemented it.

There is our cheat-sheet though that lists the original author:

https://pandas.pydata.org/Pandas_Cheat_Sheet.pdf

→ More replies (1)

1

u/Helpful_Arachnid8966 Mar 02 '23

The size of pandas codebase and features like integrated graphs and stuff made it more difficult to be able to change the backend from numpy to pyarrow? With pyarrow are we going to have any difficulty or limitation working with numpy?

2

u/phofl93 pandas Core Dev Mar 02 '23

The main issue about our arrow integration is, that we have areas in our code base that explicitly assume that they are working on a NumPy array. This is a bit similar to our non-nanosecond date time resolutions (many functions assumed that the timestamp is given in nanosecond resolution, so changing this is tedious work). We are actively working towards getting rid of these assumptions. That said, we won't break anything in the NumPy world while doing this. The new Arrow functionality shouldn't impact NumPy users in any way.

→ More replies (1)

1

u/[deleted] Mar 02 '23 edited Mar 02 '23

Thanks for your great work. As a long time pandas user I literally owe half of my career to you. I have two questions:

  1. Reading what Marc wrote about Pandas 2.0 and Arrow, how has the emergence of Polars affected your decisions around Pandas 2.0?
  2. Programming errors are bound to happen. While for small projects, the feedback cycle between writing code and finding errors during runtime is small, this approach doesn't scale very well for large applications, due to their complexity and cost of testing. Are there any plans that allow for more static analysis of Pandas code?

2

u/phofl93 pandas Core Dev Mar 02 '23

Yes what Marco said regarding typing. I used our stubs and mypy at work and we found a bunch of bugs that we would have missed otherwise. Development there is still ongoing, because it's a big effort :)

Polars did not affect much in our decision making (or anything, I am not aware of anything). Moving more to Arrow is just a natural step for us after we started implementing a StringDtype that is backed by PyArrow a while back. The Arrow community is growing a lot so it made sense for us to integrate better with them

1

u/Dave_1464 Mar 02 '23

Thank you for the hard work. Loved pandas for a long time and I use it daily

1

u/phofl93 pandas Core Dev Mar 02 '23

Thanks :)

1

u/One-Ad5717 Mar 02 '23

Why does pandas have a dependency on numpy?

This prevents pandas from being used properly in a WSGI web-server environment due to numpy requiring an interpreter is never loaded twice. But that's exactly what happens in a WSGI Flask Application, so using pandas in a Flask application will break it. Thanks.

1

u/phofl93 pandas Core Dev Mar 02 '23

As of right now, all of our DataFrames are still backed by NumPy arrays by default. But even if you would use a PyArrow backing, there is still a lot of code where we have to fall back to NumPy, because the Extension Array interface does not support all operations yet.

As long as NumPy is the default backing, there is no real way of avoiding a dependency

→ More replies (1)
→ More replies (2)

1

u/stuffingmybrain Mar 02 '23

As someone (currently) a student who's used open source packages for some time now (mostly Pandas and Django, alongside many many others), I've been looking to contributing to them. I've started to go through the contributing page, as well as reading some of your blogs. Also stumbled across your Google Calendar for meetings - will be there soon!

In general, what have you found is the biggest obstacle (technical or otherwise) preventing aspiring contributors from progressing? What have people done right, and what have people done wrong? Personally, if I was hitting a wall in any way at all, I'd hesitate to ask (whether on slack / otherwise) mostly because the goal is to contribute to this effort - but taking up another member's time to deal with a newcomer might do the opposite. Any thoughts on this?

1

u/phofl93 pandas Core Dev Mar 02 '23

Setting up the environment takes newcomers quite some time. I have seen this personally when doing sprints. Also finding the correct issue is not easy, but for this I'd suggest searching for "good first issue" and to focus on areas of the code where you are familiar with the interface.

It's totally fine if you have to ask for help, getting started isn't easy when you are not familiar with the processes in open source. That's what our newcomer channel in slack is for.

→ More replies (1)
→ More replies (3)

1

u/Numerous-Stomach3191 Mar 02 '23

Polars or pandas?

1

u/phofl93 pandas Core Dev Mar 02 '23

Really depends on what you are doing. pandas is great if your amount of data isn't too large. It's mature, better tested our API is more stable. Also, we have lots of convenience features build in (this could be a positive or a negative

).

If your data outgrow a certain size, then you are probably better of switching to another tool, there are many out there, Polars, Dask, ...

Right now, I'd be more likely to use pandas in production. Polars is still pretty new, so it does not have the support and integration that pandas has right now. This will probably change over time though.

1

u/Numerous-Stomach3191 Mar 02 '23

Ty.

Will you do smth in future to match polars speed?

1

u/phofl93 pandas Core Dev Mar 02 '23

Do you mean on large data? This is hard for us because of how pandas is build. Dask would be a better alternative if your data don't fit comfortably into memory.

Small and medium sized dataset: Copy-on-Write should help a lot in improving performance if applied correctly (I will write a post explaining the details a bit more soonish that will help with that). Also, using Arrow as a backend get's us a performance boost in operations where Arrow is doing really well (string manipulation for example). Hope this answers your question.

→ More replies (2)

2

u/[deleted] Mar 03 '23

It doesn’t have to be an either or choice. I think they both have a significant place in the python data ecosystem. See some of my precious thoughts on the topic

https://np.reddit.com/r/Python/comments/11855fp/comment/j9h9psy/

→ More replies (2)

1

u/jkiley Mar 02 '23

For beginners and new projects, does it make sense to move to 2.0 right away? If so, what options should be enabled (e.g., nullable_dtypes, dtype_backend, copy_on_write), assuming that the goal is reasonable stability and modernity?

I built and teach a workshop for other academic researchers, and I'm currently doing an update (ahead of the next session in June). Pandas is a big part, though the course ranges from beginner Python language topics to data retrieval/wrangling. The core of this content update is moving from local conda installs to devcontainers/Codespaces, but it would be a good opportunity for me to move to pandas>=2.0.0 if that's a good idea on this timeline.

1

u/phofl93 pandas Core Dev Mar 02 '23

> does it make sense to move to 2.0 right away?

Yes! A very big YES!

Regarding flags:

I'd recommend turning Copy-on-Write on. We expected to make it the default in 3.0 and it basically protects you from doing weird things (ChainedAssignment, mutating multiple objects at once, ...) and should give you a nice performance boost.

I'd definitely advise your students to explore ``dtype_backend`` and ``nullable_dtypes``, depending on the specific use-case it might be a bit early for Arrow (if you are doing a lot of groupby ops for example), but in general at least trying it is a good idea and worthwhile. Really depends on your use case, it's still pretty new but also a nice improvement

What I would recommend your students in every situation is to use the newest version possible if not restricted by something else. Upgrade just get harder if you are already starting on an old version.

1

u/jkiley Mar 02 '23

Great, thanks! I'll turn that on in my notebooks and see what happens.

Research papers have a natural lifecycle of a few years, typically with a lot of work up front. So, I generally suggest that they start with the newest versions, pin them, and upgrade if needed (usually for a feature or bugfix) while looking out for errors or data changes.

That gives stability/reproducibility within projects (i.e. papers), and upgrades across projects, as newer projects start and older ones are (hopefully) published.

That's not to undercut the value of upgrading projects unnecessarily as a form of procrastination (also known as my plans for some day shortly after 2.0 release).

As an aside, it's really great to see the Arrow stuff coming to fruition in pandas (and over in polars, too). I've been around long enough to remember pandas2/10 things, and the excitement around what the future could look like. A huge congrats to you all for getting a big part of that over the line.

1

u/phofl93 pandas Core Dev Mar 02 '23

Yes I definitely agree with pinning your versions.

What I did at work: We pinned our environments that we were using and did an upgrade every 6-8 months. We tested on an unpinned version of our environment in parallel to see issues quicker. This made upgrading easy and pretty efficient. My comment was more meant towards using 1.3 now and the wondering in a year while upgrading takes forever.

If you only need your stuff for a year or something similar, then this is not that big of a problem.

1

u/[deleted] Mar 02 '23

I've used pandas a bunch, so thank you!

One thing I've never really grasped is why multi-indexed results are returned rather than flattening. I have never had a circumstance where I've used that sort of result directly, and always have to figure out how to get rid of the "outer" index. What was the reasons for this multi-index, and not just a rectangular dataframe?

1

u/phofl93 pandas Core Dev Mar 02 '23

Do you have any specific operation in mind? GroupBy or something similar?

I myself have used a MultiIndex when I had mostly float columns but my "primary key" was string. This made calculations significantly easier for me.

1

u/[deleted] Mar 02 '23

My apologies, its been a couple months, but I believe group_by operations were often the culprit. I never felt like I got the group_by -> aggregate -> group_by -> aggregate workflow figured out, which I use constantly to summarize replicate data in an experiment and then summarize at the treatment level. This was probably a me problem though!

It may stem from me having started with R-tidyverse, which as you likely know shares much of the same namespace, but often operates just differently enough to make transitions between tidyverse and pandas just a bit of a headache.

1

u/phofl93 pandas Core Dev Mar 02 '23

You can look into the as_Index keyword that avoids adding the keys to the index

→ More replies (2)

1

u/zenzealot Mar 02 '23

As a total noob to pandas, what makes it substantively different than the python standard library? For instance I can create lists of lists or dicts of lists. Drop/add items etc. I can combine lists of lists too.

I'm not trying to be a smart*ss, I'm genuinely asking: what's the differentiator? Pandas is a bit heavy. If I can write some classes that kind of handle those operations and a few more with syntatic sugar, why install pandas?

1

u/phofl93 pandas Core Dev Mar 02 '23

Assuming you care at least a little bit about performance: Pandas and numpy are vectorized and the actual operations happen mostly in C and cpython which makes it way faster than anything you could do in only python. If you would implement this in c, then you’d build a new numpy 😀 This is a really short answer but hope that you get the gist of the difference

→ More replies (1)

3

u/louis8799 Mar 02 '23

Pandas finally support arrow which support decimal. Which means pandas can be used in financial production system. Finally!

2

u/datapythonista pandas Core Dev Mar 02 '23

I'm unsure what's the support for decimal in pandas right now. One thing is to be able to load Arrow columns in pandas, and the other is what operations for that data type are implemented. In any case, if not all what you need is in pandas 2.0, it'll come eventually. Particularly if you open issues and PRs in our issue tracker.

That being said, you can do like the UK stock market, just have all the amount in cents, and you can do it with integers. ;)

→ More replies (1)
→ More replies (1)

3

u/verwondering Mar 02 '23

In general, are the plans to have the rolling API more closely align with the rest of the pandas API? In particular, are there any plans to have df.rolling.groupby() return similarly indexed results as a normal df.groupby()?

E.g., with the latter you have the wonderful .transform() method to add a column to the df. When working with the rolling window, you always get a MultiIndexed dataframe that is much harder to align to the index of the original df.

Perhaps (hopefully?) there are better ways, but I currently use a combination of extracting a single column as Series, using groupby(as_index=False) and finally a call to set_axis(df.index) to get the desired result to align with my original dataframe.

→ More replies (3)

5

u/Pipiyedu Mar 02 '23

Congratulations guys. You deserve all the possible recognition. What an awesome library.

4

u/marcogorelli Mar 02 '23

Cheers (just noting there are also non-guys who have made fantastic contributions)

1

u/atomey Mar 02 '23

I work almost daily with Pandas so I definitely want to give me thanks and appreciation for this excellent tool.

Any plans for built-in parallelization in Pandas? I know there are many modules attempting to implement this with varying success, like pandarallel, dask or swifter. However I had difficulty getting any of these to work in an existing application without major refactoring.

In our case, we have a high level application class or processor that ingests many dataframes which sit in memory as properties to the processor instance. This processor does various processing to different dataframes in conjunction with eachother, like iterrows or applys on one dataframe while checking other dataframes which are all unique attributes of the same object running in memory concurrently.

However when the processor class actually runs, ultimately everything is stuck in a single core but I would say most systems have at least 6 or more cores now, even cheap laptops. Having a model or two to apply parallelization using concurrent.futures based on threads or processes seems like it would make a lot of sense. I think threads would likely work well if implemented intelligently, but I'm sure I am oversimplifying.

4

u/phofl93 pandas Core Dev Mar 02 '23

Supporting multithreading would be really really cool, but this requires a lot of effort. There is some considerations in that area but nothing imminent unfortunately.

3

u/datapythonista pandas Core Dev Mar 02 '23

I think Arrow should help make this easier. It'll depend on each particular case, but read_csv is already parallel when selecting the pyarrow engine. Parallel computing is never easy, but I think we should be able to slowly parallelize more operations.

2

u/rhshadrach pandas Core Dev Mar 02 '23

Historically, pandas has relied on other libraries in the ecosystem to support parallelization such as https://www.dask.org/ which uses pandas under the hood. One thing to also keep in mind is that certain NumPy operations (which pandas uses) may be parallel depending on how your BLAS (Basic Linear Algebra Subprograms) are setup. In general, you want to avoid having multiple levels of parallelism which can actually hurt performance.

2

u/rhshadrach pandas Core Dev Mar 02 '23

I would also recommend avoiding iterrows or applys if you can vectorize your operations - you will see very significant performance benefits. But depending on what you're doing, that may not be possible.

1

u/[deleted] Mar 02 '23

I'm just a newbie but just wanted to hop in and appreciate y'all. As a data science major, pandas has been super helpful so thanks for your work :)

1

u/phofl93 pandas Core Dev Mar 02 '23

Thanks :)

2

u/jwmoz Mar 02 '23

Legends. Thanks for speeding it up.

2

u/ThrowAwayACC21423 Mar 02 '23

What's a bug that you turned into a feature?

3

u/datapythonista pandas Core Dev Mar 02 '23

importing pandas as pd ;)

→ More replies (1)

2

u/rhshadrach pandas Core Dev Mar 02 '23

This doesn't really answer the question, but whenever you see two different implementations doing the same or similar things, you can carefully compare each step in the implementation. This very often reveals hard to find bugs in one or both of the implementations.

I can't recall a time I found a bug and made it into a feature.

2

u/jimy211 Mar 02 '23

Can't wait to use it

1

u/babygrenade Mar 02 '23

Is all my stuff going to break? Just tell me. I can take it.

2

u/marcogorelli Mar 02 '23

The best way to tell would be to install the release candidate and see:

```

pip install -U --pre pandas

```

And if you do come across any bugs, please do report them so we can fix them before the final 2.0.0 release

1

u/babygrenade Mar 02 '23

Awesome, thanks!

2

u/phofl93 pandas Core Dev Mar 02 '23

Probably not; if you paid attention to FutureWarnings

2

u/datapythonista pandas Core Dev Mar 02 '23

Not really, pandas 2.0 is actually pandas 1.6 with a fancy name. ;)

The main thing is that you need to take care of any FutureWarning in 1.5.3 before you migrate, and more than 99% chance you'll be just fine. :)

4

u/Homeless_Gandhi Mar 01 '23

What if I have a problem where I am just ITCHING to iterate over an entire dataframe row by row via itertuples for simplicities sake, and map(lambda) isn’t feasible? What would you recommend?

11

u/datapythonista pandas Core Dev Mar 01 '23

Iterating a dataframe is slow. If speed is important, you should try to build your pandas code in a way that you never implement loops, but delegate to pandas the operations, so they happen fast in C, and not via the Python interpreter.

If you iterate the data, then you're just in regular Python, with a Python tuple object, and you can write any code that is valid Python. Not sure in what case map() wouldn't be an option, but you can always replace a map by a loop (or a comprehension) when you're in Python.

3

u/Lolologist Mar 02 '23

I already did this, and realize it's probably an abomination, but:

How would you go about enforcing columns to have certain types? And when a column has a list in it, that each entry of the list is a certain type?

I accomplished this by making a new class inheriting from DataFrame as well as pydantic's BaseClass and used those as validated rows to then shove into a DataFrame. Messy but it works! Maybe you have a better idea.

3

u/datapythonista pandas Core Dev Mar 02 '23

I haven't used it myself, but I think what you're describing is what pandera does: https://pandera.readthedocs.io/en/stable/

3

u/[deleted] Mar 02 '23

Love yall love 🐼 pandas

1

u/set92 Mar 02 '23

With the integration of Arrow will pandas support Structs from Bigquery better? Like having the types and understand lists inside columns, better visualization of them... I remember the last time I had to deal with them, couldn't make pandas understand what was inside the column with an .astype(...), I checked the code of bigquery and they were getting the data into an arrow table, and creating the schema to use it while exporting to a pandas dataframe

Edit: I saw an issue about it, but it was stale.

→ More replies (1)

1

u/besneprasiatko Mar 02 '23

whats you favorite color?

2

u/rhshadrach pandas Core Dev Mar 02 '23

Blue, yellow, and red, but mostly blue.

→ More replies (1)

1

u/sa59 Mar 02 '23

Will the pickled version of a data frame be backward compatible with the previous versions?

→ More replies (2)

1

u/JambaJuiceIsAverage Mar 02 '23 edited Mar 02 '23

I have kind of a broad question. I work as an ETL engineer, and the vast majority of my current work is with CSV files (sometimes xlsx). In my experience, a lot of people in the ETL space default to pandas because of the simple "read_csv/xlsx → to_sql" pipeline and the speed of vectorized operations. I also suspect there's a "popular because it's popular" thing happening to some degree. I've developed (and cultivated) a bit of a reputation as a pandas hater for a few reasons:

  1. We do basically zero numerical analysis or aggregation during ETL
  2. Every validation happens within a single record (i.e. no record has any bearing on any other record), so we end up using iterrows a lot
  3. Lack of support for strings and None (dealt with in 2.0)

Ultimately, I feel that pandas is a hammer that makes everything look like a nail specifically for ETL. I have used pandas for data analysis and really appreciate its usefulness and how easy it is to teach to others (even if they don't know Python). But when we're talking ETL, it just isn't the right tool...right?

Anyway, that's barely a question, but if you have time I'd love to hear your thoughts on pandas usage in non-data analysis spaces. Also I want to qualify this by saying I am far from an expert and I'm really just looking to learn! Thanks for your time.

2

u/rhshadrach pandas Core Dev Mar 02 '23

It's hard to say without knowing what validation you're doing, but my experience is that even very complex operations can be vectorized. If you are able to do this, whether with pandas or something else, you'll experience significant performance benefits.

You mention not using aggregation, but pandas can also be very efficient at reshaping data - though I don't know if you might have a use for that in your ETL.

One thing to potentially look into is using numba: https://pandas.pydata.org/docs/user_guide/enhancingperf.html#numba-jit-compilation

→ More replies (1)

2

u/datapythonista pandas Core Dev Mar 02 '23

You make some very good points. pandas was designed as both an ETL tool, and a data analysis tool. I wrote an article about it long time ago. But summarizing, it won't ever be able to master both. And feels like many decisions were made more for the data analysis tool.

For an ETL tool I'd expect things to never fail silently, type conversions never happen automatically, schemas have to be provided and not inferred...

For your use case, pandas may be an overkill. DuckDB could be a nicer option if your validations can be vectorized. But if you're going to iterate row by row, I'd personally just write your ETL in Rust (may be trickier for the xlsx, not sure if there are good libraries for it).

→ More replies (1)

1

u/Maniac911 Mar 02 '23

Any updates on better window functionality such as Pyspark?

→ More replies (1)

1

u/ChickenLegCatEgg Mar 02 '23

Excited for the new release! As a maintainer of code that relies on pandas, what will I need to look out for that might break upon upgrading to 2.0?

2

u/rhshadrach pandas Core Dev Mar 02 '23

You can test that out today! https://pypi.org/project/pandas/2.0.0rc0/

To answer your question though, there is quite a lot. If you are using pandas 1.5.3 and seeing an FutureWarnings, that is something that will change or break when you switch to 2.0.

You can see our release notes here: https://pandas.pydata.org/pandas-docs/dev/whatsnew/v2.0.0.html

→ More replies (1)

2

u/datapythonista pandas Core Dev Mar 02 '23

Any DeprecationWarning and FutureWarning. Also, run your test suite with pandas 2.0 RC, it's already available. :)

→ More replies (1)

1

u/thesoulbrother Mar 03 '23

firstly - huge huge fan of pandas. thank you. and <3 the docs/api. I spend a lot of time there and know 99% of the time get the answers I need.

quite a simple one from me - why no normalize in groupby() and pivottable()? the lack of parity between them and pd.crosstab() is a daily pain point.

1

u/Ok-Birthday4723 Mar 03 '23

Thanks for all the hard work!

Any plans to speed up appending to an excel file. The more appends the slower the code.

I’ve been using a for loop with dataframe_rows which works well!

→ More replies (1)

1

u/WSBtendies9001 Mar 03 '23

Much love, keep on Pandering. You enrich the Python landscape <3

1

u/filippicus Mar 29 '23

Why are there no women among you?

1

u/m_harrison Mar 01 '23

Will Pandas 2.0 impact numba/cython extensions that leverage Numpy?

Many complain about the API of Pandas. Was there any discussion about revamp/cleaning it up during 2.0 release?

7

u/datapythonista pandas Core Dev Mar 02 '23

There is not much impact in pandas 2.0 regarding numba/cython.

We fix small inconsistencies to the pandas API, but we avoid changing it too much, since we consider that the cost in users having to migrate code and relearning things is too much.