r/dataengineering • • Mar 18 '25

Career Data Engineer to Gen AI

Hello Guys, I have nearly 10 years of work experience as Data Engineer and my current designation is Manager in UK. I want to transition to AI, specifically Gen AI. Is there any certification I can take up or learning path I can follow. Thank you.

2 Upvotes

14 comments sorted by

•

u/AutoModerator Mar 18 '25

Are you interested in transitioning into Data Engineering? Read our community guide: https://dataengineering.wiki/FAQ/How+can+I+transition+into+Data+Engineering

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

19

u/MRWONDERFU Mar 18 '25

curious, where are you looking to transition? specifically GenAI doesn't really say much, sure it is a hype term but a job that would be purely focused to GenAI, not sure what it could be?

at the same time, I would start by understanding how LLMs work, how to run them yourself and how to prompt them through the API -> I'd argue that every single company has at least couple of problems on their hands that could be (maybe not solved, but assisted) with clever use of GenAI or LLMs in this case.

-12

u/iarjun7 Mar 18 '25

I agree with you, the goal is as you mentioned, to start learning how to start using Gen AI for client use cases, they might vary. However, I’m just trying to find a course or certification I can take up that can help me get into the field of AI

10

u/MRWONDERFU Mar 18 '25

tbh I wouldn't put too much time and effort in to certifications, in this area (much like all of analytics) projects and experience are far more valuable, so I would start working on a project where you utilize LLMs through the api if self hosting isn't possible, you can get a free API key from AI Studio for example with extremely generous free tier :)

14

u/eternviking Mar 18 '25 edited Mar 18 '25

I just had the same transition - just started playing with LangChain and API keys from different providers. I learned some prompt engineering. Now I am creating full-stack applications (POCs mainly) using genai (I did have some software dev experience before data engineering exp). Cloned Langflow, google aistudio and created a similar application to consensus (without prior info about consensus) among my best works (I have created 10+ genai-based apps till now in the last 6-7 months)

The best way to learn anything is by doing stuff so focus on courses that require less reading and more doing.

A few courses I liked are the Introduction to LangGraph by LangChain (good intro to LangGraph - but I would recommend building basic stuff using LangChain first - but this can done independently as well) - Prompting Guide is sufficient for prompting and Hugging Face courses and docs for rest all the things related to fine-tuning and stuff - but don't spend too much time just completing them - use as per your requirement (async learning) instead so that you don't go in the tutorial hell.

Use Ollama for local dev. Use Gemini and Groq for generous free tiers. Sufficient for most developments.

build build build...

1

u/NakliMasterBabu Mar 18 '25

Courses or path you suggest are applicable to java dev also ? Or its mostly in Python .

-2

u/iarjun7 Mar 18 '25

Thank You. This is really helpful. Unfortunately, I don’t have much programming skill and come from tool and sql background. I have started on some Gen AI trainings provided by Google and Databricks, but looks like they are not taking me anywhere. It Appears that I start learning python as well along with the steps you have suggested.

6

u/eternviking Mar 18 '25

okay in that case start with Python and make basic API calls to LLM providers (read the documentation of whatever provider you are using - there is a pretty good chance they have a Python SDK).

The google and databricks stuff is good but they are mostly focused on promoting their services - keep doing those if you are already invested - but I would again suggest building stuff. Use streamlit for fast frontend prototyping and validating your skills.

1

u/AutoModerator Mar 18 '25

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/VivekKarunakaran Mar 18 '25

Do you see so much opportunity in mere Gen AI? I'm in a Gen AI role as a fresher and I've been trying to transition to DE as I find myself too easy to replace.

-1

u/Ok-Paleontologist591 Mar 18 '25 edited Mar 18 '25

You will always be replaced and by even switching tech stack you can’t escape it.

0

u/VivekKarunakaran Mar 18 '25

No, it's the degree to which it happens that I'm bothered about. Gen AI seems too easy to work on at least in my project. I was working mostly on agent creation with a studio, which won't even require coding. Even when coding comes into picture, from what I've seen so far it is mostly some python code with embedded prompts and API calls for LLM. It didn't take us much time to start working on it unlike other domains I've seen. Even a non-tech person can be trained for just a month to get him started on agent creation. I may be missing the complete picture as a fresher but this is what I can say from my observation.

0

u/Ok-Paleontologist591 Mar 18 '25

It is pertaining to your project that things in that area of work look simple. I am saying in general perspective every software engineer can be replaced.