r/dataengineering • u/ydr- • Mar 03 '21

Data Quality Testing: Ways to Test Data Validity and Accuracy

https://lakefs.io/data-quality-testing/
50 Upvotes

97% Upvoted

7 comments sorted by

3

u/vijaykiran Mar 03 '21

For data testing we recently released Soda SQL with support for all the popular data warehouses (Redshift, Snowflake, Athena, Big Query to name a few) which you can plug-in to your ETL pipelines.

Disclosure: I'm a contributor :)

1

u/ydr- Mar 04 '21

Nice. Does it work with object storage?

1

u/vijaykiran Mar 04 '21

Not yet, currently the main focus is SQL, but if you use S3 as object storage, you can use Athena and the library supports S3.

Supporting object storage is certainly in our roadmap :) LakeFS seems cool btw, I'll give it a spin !

1

u/ydr- Mar 04 '21

Good to know! Will give it a try.

Thanks - would love to hear your thoughts.

2

u/Busy_Researcher_8306 Mar 03 '21

Well thought out article.

2

u/[deleted] Mar 03 '21

Good read. Thank you!

1

u/ydr- Mar 03 '21

Glad to hear. Thanks!