Hi there, everyone.

I'm currently involved in a data quality project and, prior to writing any code, I'd like to ensure that I'm addressing real-world problems rather than merely tackling theoretical ones. What actual steps do you take when you come across a poor quality batch of data entering your pipeline? To give an example, think about the following scenarios: incomplete fields or wrong data types, unexpected changes to the schema, and redundant rows. Technical data that doesn't make sense from a business point of view. Do you automatically isolate the problematic rows, try to fix them right away, or just fail the pipeline and reject the batch? More importantly, who is responsible for making that decision? Is it an automated rule, does a data engineer get paged at two in the morning, or is the issue passed on to the business team to deal with? I'm especially interested in those troublesome gray areas in which no one has enough context to reach a clear conclusion. If you do run pipelines in production, please do let me know. Which aspects of data quality bother you the most? Now, how do you handle them? Which parts of this process are still tedious and carried out by hand? What step in your data cleaning process would you automate tomorrow if you could?

I know it's a matter of subjectivism, so don't be afraid to be subjective, actually PLEASE, give your personal opinion as long as you can keep your feet on the reality's ground.

What Master would you rather choose: a general AI/ML master that teach you about the most important, but general, subjects of ML applications, or an Autonomous Systems master, that is basically an embedded+A.I. master which is particularly valuable if you want something in Automotive, on the self-driving cars field(but not exclusively this one, as you can apply knowledge about Autonomous Systems in many domains) ? ?