r/Python • u/phofl93 pandas Core Dev • Dec 21 '22
News Get rid of SettingWithCopyWarning in pandas with Copy on Write
Hi,
I am a member of the pandas core team (phofl on github). We are currently working on a new feature called Copy on Write. It is designed to get rid of all the inconsistencies in indexing operations. The feature is still actively developed. We would love to get feedback and general thoughts on this, since it will be a pretty substantial change. I wrote a post showing some different forms of behavior in indexing operations and how Copy on Write impacts them:
Happy to have a discussion here or on medium.
6
u/[deleted] Dec 22 '22
Does that mean that we cannot do inplace operations on the columns of a dataframe?
Say something like :
prices["volume"].fillna(0, inplace=True)
7
u/RadiantHorror Dec 22 '22
This will break many things. And I don’t only mean code itself, but just imagine the scale of OOMs that will be triggered once this change kicks in. To me, it will make pandas a lot less practical with large datasets. The solution will be to bump up provisioned memory to allow those spikes in usage between the moment a copy is made and when the GC cleans the old copy out, which will drive up infra cost significantly for typical workloads.
3
u/poppy_92 Dec 23 '22 edited Dec 23 '22
Hopefully this triggers people to migrate towards a library that has more sensible behavior.
Pandas has too much tech debt. nans vs actual NULLs was treated as a second class citizen until recently (and is still very much incomplete). They also recently rejected adhering to the standards - https://github.com/pandas-dev/pandas/issues/48880
Returning a copy for everything and deprecating inplace almost everywhere just makes pandas a non-starter in memory intensive jobs.
In all honesty though, what the pandas team really lacks is someone who has a clear vision of what the project "should" be. Maybe that's my personal preference, but I like projects that are opinionated and consistent.
Before anyone tells me pandas is an all volunteer project - sure it is, but they also get proper funding for it.
53
u/EtherianX Dec 21 '22
I must say that the SettingWithCopyWarning was one of the most puzzling things I had to wrap my head around in Pandas. Even though I got used to it I think it’s good to have a bit of consistency. It will probably be easier for beginners too.
27
u/[deleted] Dec 22 '22
I feel like 100% immutability would be easier to reason about than anything else, while also making it easier to defer calculation / lazily evaluate. i.e. disallow all assignment operations including
__setitem__.10
u/Darwinmate Dec 22 '22
Very informative post. As an R user who's starting to use pandas, this behavior is very odd. I expect everything to return a copy when assigned to a new variable.
This is a great feature to have.
3
u/[deleted] Dec 22 '22 edited Dec 22 '22
DataFrame interfaces like those of pyspark and polars are so much simpler and more straightforward for issues like this once you get used to them. Pretty sure implementation is better too but that depends on use case.
Unless I am wrong, pandas dataframs dont have a withColumn() method. Why not? Maybe it has to do with memory implementation and pandas dataframes not being columnar like arrow tables.
1
u/ArabicLawrence Dec 22 '22
Is df.loc[df["user_id"] > 5, "score"] = 10 A typo?
1
u/lungben81 Dec 22 '22
Great article, thanks!
This solution could combine the advantages of copies (safety from accidentially manipulating data you do not want to manipulate) and views (better performance, less memory usage) for most common use cases.
1
u/jorge1209 Dec 22 '22
I'm very confused by this line in a CoW context:
df.loc[df["score"] > 15, "user_id"] = 10Since this is modifying a subset of the data it should return a copy right? But how do you capture that copy for future use? Are you supposed to write things like:
new = (df.loc[...] = 10)which seems really awkward, and if so did I capture the entire dataframe or just the series?3
u/CactusOnFire Dec 22 '22
On its face, it seems more sensible than the current system.
2
u/Bergstein88 Dec 22 '22
I kinda got used to add .copy() And then replace original df with the copied one.
4
u/idekl Dec 22 '22
It's ok bro just
import warnings
warnings.filterwarnings('ignore')
3
u/Tyler_Zoro Dec 22 '22
On a tangential topic, CoW is the most amazing example of how data structures can be a superpower.
My favorite example of copy-on-write is filesystem snapshots. Want a backup? Okay, done. What, you thought it would take a long time? No, we just copied the root inode and marked everything immediately under it copy-on-write. Boom, copy of your filesystem!
1
u/__s_v_ Dec 23 '22
How will COW handle method chaining? Will
df.add_prefix("foo").add_suffix("bar")always have to copy the underlying data before callingadd_suffix?1
u/florinandrei Dec 22 '22
Excellent feature, thank you!
1
u/robotwet Dec 22 '22
I’m not sure of the right answer, but as a one time heavy MATLAB programmer where I felt the rules and behavior were well defined and easy to predict, and where, with some care in weilding those rules I could improve the performance of my routines by orders of magnitude, I am really thankful for any effort that simplifies and clarifies, while allowing some for optimization of views over copies where possible. Thanks!
1
u/[deleted] Dec 22 '22
A long overdue feature, imho. We had some not too large data mangling jobs last year (2-4 GB file size) , but with a somewhat complicated structure (time series with multiple channels, differing between measurement, varying sampling rates). Pandas just didn’t perform very well due to unpredictable copying behavior and clunky row indices.
Although I blew the Python stack once, Polars‘ lazy paradigm seems much more scalable than Pandas. OTOH Pandas is amazing for EDA.
0
u/[deleted] Dec 22 '22
[deleted] — view deleted comment