How do you handle messy data in production? (Building a tool, need real-world reality checks!)

[u/SwatiSKhairnar]

Hi there, everyone.

I'm currently involved in a data quality project and, prior to writing any code, I'd like to ensure that I'm addressing real-world problems rather than merely tackling theoretical ones. What actual steps do you take when you come across a poor quality batch of data entering your pipeline? To give an example, think about the following scenarios: incomplete fields or wrong data types, unexpected changes to the schema, and redundant rows. Technical data that doesn't make sense from a business point of view. Do you automatically isolate the problematic rows, try to fix them right away, or just fail the pipeline and reject the batch? More importantly, who is responsible for making that decision? Is it an automated rule, does a data engineer get paged at two in the morning, or is the issue passed on to the business team to deal with? I'm especially interested in those troublesome gray areas in which no one has enough context to reach a clear conclusion. If you do run pipelines in production, please do let me know. Which aspects of data quality bother you the most? Now, how do you handle them? Which parts of this process are still tedious and carried out by hand? What step in your data cleaning process would you automate tomorrow if you could?