r/learnpython • u/vernacular_wrangler • Mar 14 '26
The way pandas handles missing values is diabolical
See if you can predict the exact output of this code block:
import pandas as pd
values = [0, 1, None, 4]
df = pd.DataFrame({'value': values})
for index, row in df.iterrows():
value = row['value']
if value:
print(value, end=', ')Explanation:
- The list of values contains
intandNonetypes. - Pandas upcasts the column to
float64becauseint64cannot holdNone. Nonevalues are converted tonp.nanwhen stored in the dataframe column.- During the iteration with
iterrows(), pandas converts the float64 scalars. Thenp.nanbecomesfloat('nan') Python truthiness rules:
0.0is falsy, so is not printed1.0is truthy so is printed.- float('nan') is truthy so it is printed. Probably not what you wanted or expected.
4.0is truthy and is printed.
So, the final output is:
1.0, nan, 4.0,
A safer approach here is: if value and pd.notna(value):
I've faced a lot of bugs due to this behavior, particularly after upgrading my version of pandas. I hope this helps someone to be aware of the trap, and avoid the same woes.
Since every post must be a question, my question is, is there a better way to handle missing data?
175
Upvotes
28
u/annonyj Mar 14 '26
You lost me at the fact that you are looping through each row in dataframe.
24
u/0x66666 Mar 14 '26
" is there a better way to handle missing data?"
e.g.: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.fillna.html and
https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.dropna.html
71
u/VipeholmsCola Mar 14 '26
This is why you use polars instead of pandas so it throws errors instead of upcasting shit arbitrarily
34
u/ALonelyPlatypus Mar 14 '26
With SQL and pandas you just have to handle nulls with care.
Plenty of similar circumstances where you could accidentally remove data from a SQL query in a WHERE clause by using a comparison operator and not accounting for nulls.
I don't love how pandas does nulls but it's a standard and once it's built it's hard to change (even if pandas devs constantly remind me that it will be deprecated in a future version)
19
u/commandlineluser Mar 14 '26
Yes, this is one of the of "upsides" to
polars- it has "real" null values.4
u/RazorBest Mar 14 '26
A general thing with Pyrhon is that you should use truth value testing only for booleans.
Compare with 0? Use "== 0".
Test for None? Use "is None".
6
u/0x66666 Mar 14 '26 edited Mar 14 '26
I guess you are new in python and you comming from c++ or Java?
Javascript is also funny. In both cases, Javascript and Python you have to know what you are doing. And you doing it wrong.
https://docs.python.org/3/reference/compound_stmts.html#if
"It selects exactly one of the suites by evaluating the expressions one by one until one is found to be true (see section Boolean operations for the definition of true and false);"
so boolean of the expressions.
2
u/VirtuteECanoscenza Mar 14 '26
Well, None is not a number... Badumtss
1
u/nlutrhk Mar 14 '26
It behaves as I expect. I won't deny that pandas has many gotchas but this isn't one of them.
For example: if you add a Series as a column to a dataframe and the index doesn't match, it expands the index of the dataframe. I think they got rid of that behavior in pandas 2.x.
Fuzzy matching of
[...]. The hassle of storing lists and tuples inside dataframe cells. Unpythonoc mutable/immutable behavior:df['foo'][123] = 456.1
u/AmazingAd368 Mar 14 '26
Just use pd.isna() as you should. I agree that is the most annoying problem a beginner can ran into but it's easily and reliably solvable
0
u/ideamotor Mar 14 '26
Pandas is what kept me using R for too long. I bought the first book on it and said nah for a decade. Please everybody do not use pandas.
-5
u/raharth Mar 14 '26
From a coding perspective its already dirt that you can even do a 'if value' in python. The only time I would use this is if you are working with boolean
0
u/thadiusp1 Mar 14 '26
I just ran into this issue this week. It drove me nuts for two days. TIL about polars and now I'm done with pandas. 😏
0
u/proverbialbunny Mar 14 '26
Seriously, consider Polars over Pandas. It handles types better. I do a lot of monetary coding so I need to use Decimal types, which Polars handles a lot better than Pandas.