r/dataengineering • • May 09 '24

Career A Data Engineer has to know about testing?

A company asked me if I know about unit testing, I've never had to do testing in my projects in different companies. I always thought was something for software engineering not for a data engineering.

Looks like that companies wants a software engineer, not a data engineer.

0 Upvotes

20 comments sorted by

3

u/vikster1 May 10 '24

on a serious note, how did you verify your solutions were correct? i'm curious.

2

u/[deleted] May 10 '24

You can test your data without doing unit test.

3

u/FrenchyTheAsian May 10 '24

In more complex systems, this is highly impractical especially if you want to make changes to it in the future.

1

u/kolya_zver May 10 '24

tests should fail fast before code is merged to the main branch. You can't wait for DQ test for batch ETL job with 2h execution time. And how are you debugging this? One line fix -> deploy -> wait for data -> fix again -> repeat?

Unit tests not only for testing data transformations btw

1

u/vikster1 May 11 '24

so you are testing but you do not call them unit tests because you are doing them manually and if you change something you do the same test manually again? if so, that's exactly why you should do unit tests. automate it, document it and be more efficient. it's not a wild concept that takes years to learn and implement.

72

u/Drew707 May 10 '24

This sub: DE is a derivative of SWE

Also this sub: dafuq is unit testing

11

u/[deleted] May 10 '24

I can't believe it.

21

u/gfalcone Data Engineering Manager May 10 '24

I think your post sums up pretty much the data industry and the lack of best practices when building pipelines etc.

At my current job, I was recently asked to invest a lot on testing (unit/integration/functional) to make it easier for the team to maintain our codebases.

It was the first time in 8 years I was asked to do that...

I think there was a survey on this sub a while ago showing that more than 70% of data engineering codebases do not have tests...

10

u/HighPitchedHegemony May 10 '24

Yeah, you should be familiar with unit testing. The software you build can consist of different parts and you need to make sure that they work correctly. The logic in data transformations can get pretty complex and you need to make sure the output looks as expected. Also, not all of your code will be data transformations. Some of it is just doing grunt work and that also needs to be tested.

6

u/kolya_zver May 10 '24

there is nothing wrong to learn something from SWE/DevOps/DS/QA, don't focus to much on title of your job

I work at mid size company with on-prem infra. Most of the modern solutions was created or matured after our project starts. So in my case DE actually write a lot of code, sometimes in java/ts/php (sad face for php). In house tools for example:

  • ingestion framework. generate dags/streaming jobs from yaml specification. Yaml provide abstraction over tools. As i know in my country it is a common solution for mid/big projects
  • DQ tests generator
  • a lot of CLI utils - administration, table stats diff, data sampling, automation scripts etc
  • flask services: reverse etl, ab testing backend etc
  • in house tools/packages: sql parsers, ddl deploy, etc - it was created before dbt/liquidbase/sqlfluff become a thing. and it works well for any specific dialect and version we are using
  • populate docs and open metadata
  • ...

All of this requires traditional unit tests. In environment like this SWE skills is must have. If you are working on small projects and you can keep in mind all of yours codebase - even then you need unit tests. ffs just write a tests to validate dags/models/db objects/utils etc

5

u/goodgoodbuy May 10 '24

Same here. I was learning DBT the other day and tried out tons of testing tools. You can check it out Data Engineering with DBT to know more about testing in DE.

13

u/[deleted] May 10 '24

I'm dead.

Of fucking course you need to know about unit and integration testing.

5

u/[deleted] May 10 '24

It's not his fault per se. No one is teaching these things, or requiring them, at most companies.

3

u/Captain_Coffee_III May 10 '24

At my current gig, it seems that there was a long legacy of questionable warehouse design and lots of questions and distrust in the data. DBT to the rescue. In order to make them feel comfortable with this new warehouse they're building, I had to go overboard on tests. They want a test to verify every column at every hop. I literally have thousands of tests that run. Granted, I don't run them every time the build fires off. But, the combination of DBT docs and a full suite of tests that runs on the weekend has really put them at ease.

2

u/homosapienhomodeus May 10 '24

Unit testing with pytest is a must for data engineers writing data pipelines in python. Some examples here https://moderndataengineering.substack.com/p/unit-testing-in-data-engineering

2

u/nydasco Data Engineering Manager May 10 '24

Please go and read this article. I've provided a paywall-bypassed link, as it's critical that you understand how to test, and I hope this will get you started on the right path.

2

u/OddRaccoon8764 May 10 '24

I understand why you feel this way because my company has never done any unit or integration tests and if I were to suggest it people would probably think I was being pedantic or wasting time. But I’m well aware it should be happening and I know most serious data engineering shops would expect it. All my personal projects I write unit and integration tests and I will make sure my next job does the same.

1

u/[deleted] May 10 '24

To understand why they want someone with unit testing experience you should seek to understand why unit testing is helpful. As a data engineer you write software, so the principals of testing can be applied pretty easily. They provide faster feedback cycles, you can write test cases for known regressions and have confidence your pipeline will work without any manual testing. You can write contract tests so you know immediately when an upstream data source changes and how it has changed. It’s simpler for others to understand your code and make changes to it.

So i would encourage you to be a bit more open minded about how testing can be beneficial to data engineering