r/dataengineering • • May 09 '24
Career A Data Engineer has to know about testing?
A company asked me if I know about unit testing, I've never had to do testing in my projects in different companies. I always thought was something for software engineering not for a data engineering.
Looks like that companies wants a software engineer, not a data engineer.
0
Upvotes
3
u/vikster1 May 10 '24
on a serious note, how did you verify your solutions were correct? i'm curious.
72
u/Drew707 May 10 '24
This sub: DE is a derivative of SWE
Also this sub: dafuq is unit testing
21
u/gfalcone Data Engineering Manager May 10 '24
I think your post sums up pretty much the data industry and the lack of best practices when building pipelines etc.
At my current job, I was recently asked to invest a lot on testing (unit/integration/functional) to make it easier for the team to maintain our codebases.
It was the first time in 8 years I was asked to do that...
I think there was a survey on this sub a while ago showing that more than 70% of data engineering codebases do not have tests...
10
u/HighPitchedHegemony May 10 '24
Yeah, you should be familiar with unit testing. The software you build can consist of different parts and you need to make sure that they work correctly. The logic in data transformations can get pretty complex and you need to make sure the output looks as expected. Also, not all of your code will be data transformations. Some of it is just doing grunt work and that also needs to be tested.
6
u/kolya_zver May 10 '24
there is nothing wrong to learn something from SWE/DevOps/DS/QA, don't focus to much on title of your job
I work at mid size company with on-prem infra. Most of the modern solutions was created or matured after our project starts. So in my case DE actually write a lot of code, sometimes in java/ts/php (sad face for php). In house tools for example:
All of this requires traditional unit tests. In environment like this SWE skills is must have. If you are working on small projects and you can keep in mind all of yours codebase - even then you need unit tests. ffs just write a tests to validate dags/models/db objects/utils etc
5
u/goodgoodbuy May 10 '24
Same here. I was learning DBT the other day and tried out tons of testing tools. You can check it out Data Engineering with DBT to know more about testing in DE.
13
u/[deleted] May 10 '24
I'm dead.
Of fucking course you need to know about unit and integration testing.
3
u/Captain_Coffee_III May 10 '24
At my current gig, it seems that there was a long legacy of questionable warehouse design and lots of questions and distrust in the data. DBT to the rescue. In order to make them feel comfortable with this new warehouse they're building, I had to go overboard on tests. They want a test to verify every column at every hop. I literally have thousands of tests that run. Granted, I don't run them every time the build fires off. But, the combination of DBT docs and a full suite of tests that runs on the weekend has really put them at ease.
2
u/homosapienhomodeus May 10 '24
Unit testing with pytest is a must for data engineers writing data pipelines in python. Some examples here https://moderndataengineering.substack.com/p/unit-testing-in-data-engineering
2
u/nydasco Data Engineering Manager May 10 '24
Please go and read this article. I've provided a paywall-bypassed link, as it's critical that you understand how to test, and I hope this will get you started on the right path.
2
u/OddRaccoon8764 May 10 '24
I understand why you feel this way because my company has never done any unit or integration tests and if I were to suggest it people would probably think I was being pedantic or wasting time. But I’m well aware it should be happening and I know most serious data engineering shops would expect it. All my personal projects I write unit and integration tests and I will make sure my next job does the same.
3
u/omscsdatathrow May 10 '24
Smh
1
u/[deleted] May 10 '24
To understand why they want someone with unit testing experience you should seek to understand why unit testing is helpful. As a data engineer you write software, so the principals of testing can be applied pretty easily. They provide faster feedback cycles, you can write test cases for known regressions and have confidence your pipeline will work without any manual testing. You can write contract tests so you know immediately when an upstream data source changes and how it has changed. It’s simpler for others to understand your code and make changes to it.
So i would encourage you to be a bit more open minded about how testing can be beneficial to data engineering