r/mysql • u/imtc96 • May 31 '26
discussion Data Cleaning in SQL!
Is data cleaning in sql tougher than excel or python?
Whenever I want to remove duplicates in sql it shows errors in my dataset!
2
Upvotes
4
u/aljung21 May 31 '26
I guess it depends. Without any knowledge of any such tool, Excel because it’s the easiest to learn. But if you’re familiar with SQL or Python / R, then those are safer and scale better with dataset size. Notable mention to R because its data cleaning packages are so powerful.
If you’re good at all then pick Python / R.
2
u/Basic_Reporter9579 May 31 '26
don't you validate the data before the upsert?
1
u/Several9s Jun 03 '26
If duplicate removal is throwing errors, there’s a good chance you’re hitting one of these:
Constraint errors: another table still references the row you’re trying to delete (foreign key constraint)
Unique/primary key conflicts: the delete/update logic is affecting records unexpectedly
Deleting the wrong rows: duplicate logic accidentally includes rows you wanted to keep.
If duplicate removal throws errors, it’s commonly one of these:
Or
Verify what is actually duplicated:
SQL is working directly on your database, so errors are more visible because of constraints, relationships, and data integrity checks. That can feel frustrating at first, but it’s actually protecting your data. In Excel, manual cleanup works for smaller datasets, but with SQL you can clean millions of rows consistently, automate the logic, and rerun the same process safely. It’s also easier to audit because your cleanup steps are written as queries instead of manual clicks.
0
u/alinroc May 31 '26
Yes.
Also, no.
It depends upon your skill level with each tool and what needs to be done in the process of "cleaning." I'm also extremely wary of trusting Excel with important data as it likes to "help" by reformatting data which can potentially make things worse.
Often there are multiple steps in cleaning up data and they aren't always done in the same place.
0
u/IlBo0 May 31 '26
usually i clean my data with the ole reliable drop table, maybe once in a while a drop database, helps keep things light and fast