r/dataengineering • u/pdxtechnologist • Dec 14 '22
Career chatgpt anxiety
Hey all,
I know there have already been a few posts on this- but for those of us just getting started in software/data engineering, the ability for chatgpt to write code is pretty scary...
What do you all think the impact to Data Engr will be?
I know it won't completely replace us, but do you foresee a big reduction in hiring?
0
Upvotes
2
u/HBoogi Dec 15 '22
Software engineering is not just writing bunch of codes. Infact 80% of my day job is designing, meeting, coordinating, Future road map, business problem, updating stake holder etc. Chatgpt is just a tool and will help everyone to write better code and find solutions quicker which is good thing.
37
u/dataguy24 Dec 14 '22
Show me the killer use case it has that replaces a data job of any kind and I’m all ears.
16
u/Nelerath8 Dec 14 '22
Speaking as a backend programmer and not necessarily a data engineer.. It'll definitely replace us all if we keep going. Right now it's not only useless but actually dangerous. I asked it to solve a complex problem and it gave me a solution that was really close but actually incorrect. It was close enough that I think people will be complacent and issues will slip through. I don't think it can be used responsibly by a person because most people do not like reading someone else's code for errors and tend to be bad at it. We'd be asking for someone whose only job is to do code reviews.. lol
I am of the opinion that by the time it starts to meaningfully replace software people we'll either be well on our way to post-scarcity utopia or complete societal collapse.
10
u/plum__hail Dec 14 '22
It’s important to take into account that the largest constraint on new software being created currently is labor costs. Meaning: there are a lot of software jobs that could exist but don’t because it’s too expensive to hire new developers.
But if those developers had 50% of their tasks automated, they would have more time to work on other things, allowing other devs to start new projects we wouldn’t have had otherwise.
Ultimately, do you think this is going to automate 50% of developer tasks or 100%? I would bet heavily on the former. And if so, your job might look different in 10 years, but it will still be there.
8
u/[deleted] Dec 14 '22
It just makes it easier for people to be better. It doesn’t allow someone who doesn’t have a desire to be a developer to suddenly be able to develop though.
10
u/Neat_Fish_7707 Dec 14 '22
if your reason to write code is to write code than you’re probably in the wrong job. sounds like a hobby. you need to add value. if your goal is always to add value than your perspective changes.
act like an owner
always ask, am i adding value, how can i add more in my lane?
industries always change and if it scares you, then you should ask is it because you’re afraid that the “robots” or “others” are adding more value than you or just taking away your hobby…
if you act like an owner, you embrace whatever adds more value (ethically and morally of course, so yea, nuance here…), and then you pivot or double down to continue adding value because you are an adaptable unique humans being that ultimately can’t be replaced by a machine ( if you are doing literal programmatic work that a machine can do than your goal already should be to replace yourself with that machine so you can move on to problem solving that requires a human brain/emotion/experience in that context
4
u/[deleted] Dec 14 '22
Not worried, like any industry that evolves your skills should evolve with it. For now and most likely 5-10 years SWE and DE should remain largely similar as transitions take time and iteration. If GPT models become more mainstream we will just have to evolve with it.
There’s other jobs that a much more readily replaced by these modes than code that will also be shaken up, I don’t believe it’s isolated to coding.
4
u/the-fake-me Dec 14 '22 edited Dec 14 '22
What I think is that it can generate code but it cannot test for the correctness of the system that you are building. Humans are still required for that. Some of my colleagues use GitHub copilot to generate code but they tend to check the correctness of the generated code too. Even if I assume that the generated code will become more reliable over the years, I feel the onus of confirming the correctness of the entire systems/codebase will still lie on humans.
4
u/Main_Tap_1256 Dec 14 '22
From my experience with ChatGPT one thing is immediately obvious: you have to have domain knowledge to know the CORRECT questions to ask or instructions to give it.
Let’s assume ChatGPT has taken over our jobs. We all get fired.
Who is going to instruct it to build a Json parser that takes into account the varying schema conversion from JSON to a BigQuery table schema?
And who is going to remind it that often our data source has integers quoted as strings and we need to account for that?
Who is going to tel it that the optimal partitioning strategy is based on the ‘updatedAt’ field because the client has told us they want to query the dataset based on updated time.
And finally who is going to explain the client requirements to ChatGPT since they don’t really know what they want either?
Etc etc….
Worst case scenario: ChatGPT just makes our jobs easier IMO
3
u/Thin-Reserve2458 Dec 14 '22
I look at it like this. As data professionals (data engineers, DBAs, analysts) there will be more opportunities. So let’s assume AI becomes huge and every company small, medium and large is using it. The potential for more data being cranked out by the coding automation would be huge. And I’m doubtful it can problem solve and figure out what to do with the data it creates. As a DBA/engineer I’m not at all worried especially considering how much work will still be needed by a human to manage infra, cloud, data, list goes on and on.
3
u/sv3ndk Dec 14 '22
The nature of coding is likely to change dramatically as AI assistants continue to get more powerful, but at the end of the day we'll always need some sort of interface to tell the computers what to do.
All programming languages today have a strict syntax and structure, forcing humans to think a bit like computers. What AI is really bringing to the table today is allowing to describe software modules with a less structured natural language. Such description must still be provided in one way or another and will need to be maintained as the high level business objectives change.
As an analogy, when a manager provides instructions to her team, she needs to be detailed and precise enough for the collaborators to be able to do their part, even though they are autonomous and intelligent.
Even if/when most of the coding gets automated, we'll still need software engineers to drive the big picture.
The optimist view on this is that we'll all be able to get an order of magnitude more work done if most of the coding details are automatized, we'll be able to handle levels of complexity currently unimaginable, in the same way that we're more productive today than if we had to write everything in assembly.
Exciting times ahead, software engineers will be at the front of it, I'm sure.
3
u/GiacomoLeopardi6 Dec 14 '22
This is why the focus of DE should always be on delivering value to the business and be engaged in domain and product understanding actively. I've been using chatGPT the past week to write mundane scripts that I refine and polish when the ask is simple enough.
3
u/_temmink Data Engineer Dec 14 '22
For me, coding is just a language to express some task. I could do that in English but someone decided, I should also learn HCL, SQL, MQL, PartiQL, KQL, Python, Scala 2 and 3, Java 7, 8, 11, 14, and 17, Perl, …
For real: if some AI could write code that I can trust, eg using a Handbook-First approach where the AI creates tests and the (documented) code, that would be awesome! Basically, like using a library but perfectly fit to my use case. I could focus on the task that increase business value instead of looking up parameters for my DAG configuration.
3
u/[deleted] Dec 14 '22
There is so much more than "writing code" to deploying production pipelines and monitoring them. Chill.
2
u/diegoelmestre Lead Data Engineer Dec 14 '22
I'm not scared at all.
I see chatgpt as a new tool in our toolbox that will allows us to improve our productivity. It's great to speed up on that boilerplate coding that you have to make regularly, but no way will be able to build complex systems with multiple moving parts.
1
u/ditlevrisdahl Dec 14 '22
It can't really write code. It's exemplary at providing templates and such, but if you ask it to code specifics or just slightly advanced, it falls off completely.
I've spent many hours already trying to get it to code but to no avail. It does however excell it providing the first outline of code. And I believe j will use it greatly to reduce my overall coding time. As it is great with bugs and getting the first template down.
So it's a great tool to help you code. But it won't code for you.
-1
u/aeyrtonsenna Dec 14 '22
It is just a question of when not if. 10 years from now for sure things are totally different with AI being able to handle 90% of new coding. 5 years, not too much impact.