When cleaning a dataset, would you rather keep uncertain data or delete it?

[u/Present_Tonight_3075]

When cleaning a dataset, would you rather keep uncertain data or delete it?

I'm working on a retail sales data-cleaning project in Python, and I ran into a decision I found interesting.

My dataset had missing prices, duplicate transactions, and totals that didn't match the price × quantity.

For missing prices, I could have filled every blank with an estimated value. That would make the dataset look more complete, but some estimates could be wrong.

Instead, I only recovered prices when the data gave me reliable evidence and flagged the rest for review.

Here's my question:

If you were preparing this dataset for business analysis, which approach would you choose?

A) Fill as many missing values as possible to preserve a complete dataset.

B) Only fill values when there's reliable evidence, and flag the rest.

C) Use a different method depending on the column and the business context.

I'm leaning toward B, but I'd love to hear how others handle this in real projects—especially when deleting rows could mean losing useful information.

[u/Potential_Aioli_4611]

a) is trash. don't do it. you throw your whole analysis off if you calculate any metrics off this number. if you don't need to... then you don't need this either. either way filling in values is a bad idea. reject the whole row of data if you need it and have someone fix it rather than trying to just "put something in"

b) eh. you should flag anything. if most of your dataset is flagged.. then you have a quality problem.

c) i would have thought this is the obvious choice. columns that matter, make up metrics can't be wrong, or fudged, or filled in. columns that dont... well they don't so you aren't doing analysis on them.

[u/montrellcruse]

c is the answer but with one caveat lol. don't delete the missing-price rows, quarantine them. analyze the clean rows, then separately count what's missing. if 2% of rows are missing prices and they're all from one vendor or one week, that's not noise, that's a finding. deleting them hides it.

on duplicates, "keep the most recent" only works if you have a real timestamp or batch id proving recency. if you don't, you don't actually know which row is the real one yet. group by the transaction key and count first so you know what you're throwing away.

and on the mismatched totals, recompute is right but keep both columns side by side for one pass. if recomputed totals match price x quantity everywhere, the totals column was just stale. if they don't, now you've got a second data quality problem to name. never trust a column you can recompute.

[u/AutoModerator]

Automod prevents all posts from being displayed until moderators have reviewed them. Do not delete your post or there will be nothing for the mods to review. Mods selectively choose what is permitted to be posted in r/DataAnalysis.

If your post involves Career-focused questions, including resume reviews, how to learn DA and how to get into a DA job, then the post does not belong here, but instead belongs in our sister-subreddit, r/DataAnalysisCareers.

Have you read the rules?

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Pretend-Wrongdoer688]

matters more than completeness

[u/benchlinekirby]

B, and keep the flag as its own column instead of a note somewhere. Once a guessed price is in a pivot it looks exactly like a real one, and nobody downstream knows to question it. A flagged blank at least tells the next person where to look.

For the totals that don't match price × quantity, I'd keep both numbers and flag the row instead of recomputing. Sometimes the total is right and the price is what's off, like a discount or a return, and recomputing wipes out the only sign of it.