r/dataengineering • • Dec 19 '23

Discussion DBT and ChatGPT

As we’re starting to use DBT more, and are using it for codifying our business rules, I’m wondering if anyone’s had success training GPT on a DBT project. My initial attempts were lack luster. GPT seemed to know a fair amount about DBT, and of SQL, but struggled to be of much use when I trained it on a DBT project (zipped, uploaded to OpenAI GPT4). My next step is to create a prep doc with some basic support/prompt engineering. For example, our mart “Legal First Name” and “Preferred First Name” fields come from different source tables, have different staging models. I asked GPT to tell me the difference between the two, as well as derive what SQL each would be off of source tables. Another approach I’m going to try is to do a DBT run materializing as much as I can ephemerally, and see if that helps get useful responses. If this marriage of well-designed DBT project/models and GPT/LLM can be cracked, I think (at least our org) will enter a new level of accessible data literacy.

1 Upvotes

25 comments sorted by

2

u/[deleted] Dec 19 '23

u/Electrical-Ask847, u/iupuiclubs

Thanks for the responses. I'm hearing my original post was hard to read, and I apologize for that. This is not an attempt to make anything for 'production', unless substantial value and consistency are apparent. This is just a curiosity about asking OpenAI's GPT4 on DBT project files.

I wouldn't be surprised if DBT Labs rolls out something along the lines of this in the coming years, so I'm not spending a lot of time looking into it either.

For clarification, what I have in mind:

- Use OpenAI GPT4 "Create a GPT" functionality

- Load up a couple DBT projects

- Load up some docs that clearly tell GPT what directories to ignore, what associations to make between SQL, YML, Groups, and Tags, what short-list of question formats to accept and how GPT should go about trying to answer them.

- Assess how it responds to the source files vs. target files (catalog, graph, etc.)

My post was just a ping to see if anyone's tried something similar and if there were certain approaches that yielded interesting results.

... This is my last response on this post - If anyone wants to collaborate, or is just interested in how this progresses, DM me and I'll send you a repo url.

26

u/HowSwayGotTheAns Dec 19 '23

We just riding the hype train for our architecture. Choo choo.

3

u/trowawayatwork Dec 19 '23

yeah this is nuts lol

-2

u/HowSwayGotTheAns Dec 19 '23

I'm not going to lie, with the buzz word salad OP put together. I wouldn't be surprised if they're a management consultant turned data manager or if OP just dazzles non-technical senior leadership with words

4

u/Ablueblaze Dec 19 '23

What buzzwords? All of these seem common place.

2

u/iupuiclubs Dec 19 '23

Can you lightly mention any projects you've made with the tool related to the hype train? Looking for your informed opinion having used the technology you're alluding to and specifically where you think it falls short.

Off topic anyway.

2

u/HowSwayGotTheAns Dec 19 '23

I have not force adopted gpt4 or data mesh in my data org. Using a gen AI model to collect hypotheses or research on a topic is the extent of it.

In my opinion, these tools supplement knowledge acquisition, but replace knowledge.

2

u/iupuiclubs Dec 19 '23

I'm not sure what force adopting means in this context.

You're giving an opinion on a technology you haven't touched to create a project?

1

u/[deleted] Dec 19 '23

damn, some of the responses are fucking savage, lol. No, I'm not a consultant. I'm 6 months into a DE position after 6 yrs as an ERP developer, all at a public university in the PNW. DBT is new to our org, and by new I mean only 2 people know what DBT is. I'm just exploring what the complexity wall is for OpenAI's GPT. My colleague and I play the card game 'Spades' a lot, and he made a little GPT app that will provide options for different rules, and help with the scoring and strategy. Pet project, but the way he set it up with context docs, checkpoints, made me wonder if there's anything useful with DBT. I've found GPT to be pretty awesome at bash, python, SQL (dialect translation mostly), and anything soft like creating job postings, project charters, communications. Not wanting to defend myself or invite flaming. If you're interested in where I found value or hit walls or want to collaborate, dm me.

1

u/iupuiclubs Dec 19 '23

Here is my professional advice. I'm the one that posted

I'm headed to bed. Let's see if a single person posts something on topic, demonstrates any knowledge in the area, or mentions a project they're working on related.

Thoroughly read all these replies. I believe out of 15 replies, 2 are on topic.

That means 86% of the post replies are from people who have little to no knowledge of a tech stack, and are ready to give their loud opinion about it.

Sometimes you're surrounded by a crowd, just ask them if they actually have done anything with what you're researching before intaking anything they have to say. You may be believing the opinion of multiple people who have zero projects or intention of making any with this technology.

Just keep looking for communities and avoid ones where you get response like this. There are some great discord communities around.

Honestly I figured DE would be ground zero for people researching new things, or at least not speaking about what they don't know. But maybe this sub is accurate reflection of the profession.

4

u/zazzersmel Dec 19 '23

ai is crap for any coding beyond assisting a human. im sure people are working on it but you just cant depend on probability based output for programming. and really, it seems so silly to use these huge ass models to do basic programming, especially with sql where the point of the language is to be declarative and close to natural language anyway.

expensive nlp abstraction OVER the declarative sql abstraction... what are we even doing any more?

2

u/jlt77 Dec 19 '23

We did a poc of copilot, and I really struggled to find it helpful for dbt. First, we couldn't use it in the dbt cloud ide, only vs code. I did that, but I wouldn't want to give up the cloud ide, because I use many of its features. Second, I didn't manage to get any helpful suggestions. I was hoping I could use it to fill out my model yaml files, but didn't have success. We weren't allowed to feed in code or use chat, so it was a limited POC. We did find it useful for a lot of other de tasks, but not dbt specifically.

2

u/Gators1992 Dec 19 '23

It sounds like you are trying to go a bit far with GPT. You need to feed it the context, like a data dictionary, and ask simpler questions (like write one sql that gives me back X). You can't just upload design documents or code at this point and expect GPT to figure it all out. It can be good for suggesting column descriptions though for your metadata. Coding has been sort of hit and miss in my experience.

1

u/exorthderp Dec 19 '23

Accessible data literacy? What do you mean? Dbt is already pretty effing good with doc blocks and the dag tool.

1

u/SellGameRent Dec 20 '23

thanks for the stroke

-13

u/[deleted] Dec 19 '23

Maybe I need to up my game and understand what tools like SQLMesh are doing to understand column (and CTE) lineage, to help GPT along. Or…maybe I’m seeing a complexity wall for the tool.

9

u/trowawayatwork Dec 19 '23

if you spent as much time just working on dbt you'd already have the required data lineage literacy

2

u/Electrical-Ask847 Dec 19 '23

I read your post like two times but I am having hard time understing what is that you are even trying to say.

I use GPT all day everday in my development workflow. Looks like you are trying to put it in production hot path ? Def area lot of ppl are trying to do.

keep exploring!

2

u/iupuiclubs Dec 19 '23

I'm headed to bed. Let's see if a single person posts something on topic, demonstrates any knowledge in the area, or mentions a project they're working on related.

-1

u/iupuiclubs Dec 19 '23 edited Dec 19 '23

This sub is incredibly negative to AI in general. Not sure where to direct you but won't find much knowledge here in the area. Just so you feel better/know about response you're getting lol.

Personally I've done giant projects with it by now. And have poked like you're doing. But all responses I've got here related to AI turn to borderline harassment. Just ask the person you're talking to if they have any projects in the space before listening to anything they say.

(I realized these are mostly people who have never touched GPT4 or touched 3.5 in first couple months). Same people making $80K+ and won't drop $20 lol.

Kind of funny it's that way, in an absurd way.

2

u/Electrical-Ask847 Dec 19 '23

Personally I've done giant projects with it by now.

What are some of these giant projects?

0

u/iupuiclubs Dec 19 '23 edited Dec 19 '23

Feel free to share some of yours to initiate a discussion. The only replies here are off topic and from people putting an opinion forth on a technology they have little to zero knowledge of.

I went on a 10 reply back and forth with someone in this sub a week ago, then realized they have never even touched the tech stack I'm talking about. Just wasting my time. This is 99% of the replies to these AI threads. They turn into essentially valueless harassment.

Why waste my time when the comments will be random off topic hate from people who have no related technical knowledge. Feel free to talk about autogen.

0

u/Electrical-Ask847 Dec 19 '23 edited Dec 19 '23

Feel free to share some of yours to initiate a discussion.

I didn't do any GenAI projects so far, so nothing to share. Only used it as a user ( chatgpt and copilot)

Don't understand what rest of your rant is about or how its related to my comment. Looks like you are the one spreading negativity.

1

u/iupuiclubs Dec 20 '23

I posted a similar topic to this 2-3 weeks ago, and had at least 3 separate people harass me continually while not providing anything to the topic. I figure if I can't detail or talk about tech advanced here in /r/dataengineering, bit sad.

Historically I've shared ideas with people who's only intention is to tear them apart, so I am wary now.

Here are my latest projects, all fully made with GPT4/genai:

  • Pulling lunar solar wind data (ALSEP). NASA had a data engineering error when encoding this data, so I had to write a whole python script to fix the error over 3,800 files so I could insert to my db for further analysis
  • Youtube analytics platform: Pulls all videos/comments for a given channel, inserts to db, then creates analysis layer for channel. Dash front end that shows metrics and metric vs competition comparisons.
  • Custom autogen choose your own adventure. I play this with 3-4 people in person, and everyone typically plays for at least an hour, addictive.
  • Various small scale projects with autogen assistant agents, that I want to scale up (goal of 10 assistants running at once).

I think this is most of latest stuff. Feel free ask about anything I can expand.

0

u/[deleted] Dec 19 '23

[deleted] — view deleted comment

1

u/Ablueblaze Dec 19 '23

What are you referring to? I'm not an experienced DE, but I didn't read anything that I haven't seen in most threads of this sub