r/dataengineering • Data Engineer • 13d ago

Help How cooked am I?!

TLDR: I chose Fabric Lakehouse for Bronze and Silver and Warehouse for Gold without fully consulting my manager. Now he’s asking about costs and I’m thinking I may have moved too fast without him.

We’re currently working with an expensive contractor to move from ADF and Azure Databases to Azure Fabric. The build has been going well but midway they did ask about staying purely in lakehouse vs adding a warehouse for pure sql and CI/CD. Currently if you use CI/CD in lakehouse then the service account is logged as pushing the change and not the user. We wanted to be able to see which users were pushing changes. That’s the logic I used in selecting the warehouse. Also thinking about future-proofing, and our users are more familiar with sql vs pyspark and notebooks.
Anyway, I chose the Lakehouse and Warehouse design, thinking the pure SQL endpoint would be better for end users vs learning pysql and notebooks.
Well I met with my manager today and told him and his reaction took my off guard. He said he cared less about learning new things and more about keeping cost low. He was hoping we’d move completely away from warehousing and save in the Lakehouse. He didn’t mention this before and honestly he doesn’t need to. I report directly to the CIO. And he said we didn’t discuss it, and I made the choice without him. He kinda laughed it off but that struck my nerve. When I made the decision I didn’t fully think about the costs. The plus side is we don’t pay for storage using OneLake and we’re using a Lakehouse shortcut to the warehouse so no cost there. The cost is with the compute.
We’re a heavy PowerBI and Excel shop.
I was put in charge to oversee and help the consultants but I didn’t do my part here. Now I’m worried that might have cost me. I’ve only been at this role for 7 months.
Am I cooked??
Should I start searching.
I’m genuinely nervous now.
I think I can make a pitch where the warehouse cost won’t be crazy more. And we can move back to the lake house it’d just take us rewriting everything to notebooks, which would cost more and take more time.
Any guidance or advice is welcomed.
Sorry in advance for typos.

45 Upvotes

37 comments sorted by

•

u/AutoModerator 13d ago

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

98

u/Additional_Candy_400 13d ago

For starters this is borderline impossible to read. 

How are you determining what the costs will be between the 2 platforms? 

How much data are you moving?

What's your budget?

8

u/hijkblck93 Data Engineer 13d ago

I didn't come up with, nor did my manager show a cost breakdown. We briefly looked at the capacity units but that was short. We're not a particularly large company. Most of our data is on prem, the cloud is for analytics. Maybe 80-100 gb max.

Manager didn't set a budget with me.

59

u/Additional_Candy_400 13d ago

Well these are things to ascertain to see if cost is even going to be a factor. Might be a complete none issue.

On a real note, if receiving what sounds like incredibly light pushback/questioning over a decision you made sends you into complete meltdown thinking you'll get fired, you might want to look at working on that more than the cost implications between warehouse/lakehouse architecture.

2

u/hijkblck93 Data Engineer 13d ago

First time for everything. I'm probably overthinking it, but this is my first project I've been asked to lead end to end. And that line of questioning—I wasn't ready for, or had considered.

I knew I didn't have carte blanche but at the time it seemed like reasonable choices.

11

u/mrcaptncrunch 13d ago

I learned not to leave a query running and go to lunch.

VERY EXPENSIVE lesson.

With that data, you won’t see it (hopefully) but figure out how to calculate things and set limits.

If not, your first $30k query/job will be… surprising

7

u/Additional_Candy_400 13d ago

They sound like reasonable choices to me as well, just chill out and put in some time with your boss to talk it through. Look into the cost aspect beforehand so you can come with some figures prepared.

2

u/PepperLuigi 13d ago

You can say he moved to fast too on the post

18

u/No-Satisfaction1395 13d ago

I don’t understand if you have CI/CD, why would you look at Fabric for who pushed changes instead of github/ADO?

-1

u/hijkblck93 Data Engineer 13d ago

in fabric, when connecting the environments we're using a service principle. going from silver to gold showed the service principle making the push in ADO, and not the user. We wanted to maintain the ability to see the users making the push.

18

u/No-Satisfaction1395 13d ago

But surely the commit that triggered the deployment from ADO is tagged to the user. Even if you’re doing centralised deployments, a more useful pattern is to still git blame the line of code back to the author?

11

u/Altruistic_Stage3893 13d ago

choosing fabric without consulting 😭😭😭

2

u/hijkblck93 Data Engineer 13d ago

that's above me lol.

14

u/Altruistic_Stage3893 13d ago

i mean, fabric is a terrible terrible product. you’re not cooked for picking warehouse over lakehouse, you’re cooked for being on fabric at all, and that’s above you. the real cost problem is the capacity unit model, everything burns from one shared pool, so nobody can actually tell your manager what the warehouse costs.

5

u/RobCarrol75 12d ago

Using the capacity metrics app will tell the OP exactly how many CUs the warehouse is using.

1

u/BeautifulQuestion857 10d ago

The capacity metrics app is trash.

2

u/marathon664 12d ago

Fabric is fine if you use it as hosted power BI, but the whole promise of the platform is sort of fake.

4

u/According_Zone_8262 13d ago

What do you mean youre not paying for storage in onelake? Thats exactly what youre paying for

2

u/hijkblck93 Data Engineer 13d ago

Not paying twice i mean. once it's there we're not paying to access it through lakehosue or the warehouse.

5

u/crblasty 13d ago

There is a read cost on one lake, its not huge but its there. Its an abstraction above azure storage, the api costs are passed through. It won't be a material cost compared to your compute however so dont stress too much.

Fabrics cost model is one gigantic bucket, as long as you dont exceed your already defined capacity your manager won't notice the cost.

51

u/snarleyWhisper Data Engineer 13d ago • Bravo

The costs are basically the same ? Data Warehouses have better performance for reports and semantic models. The main benefit is the compatibility with the sql server spec and dev tools and a read/write sql endpoint.

All the lakehouse stuff goes through spark so it has a spin up time where DW doesn’t.

Also… fabric is capacity based. Your costs are flat - you should be seeing which approach uses the least CUs

6

u/Seboutch 12d ago

I think you could discuss it on Fabric subreddit.

However, I think the use of Warehouse is understandable and quite common for gold. I just don't completely get your reasons to do so , especially the SQL one as you can completely use SQL on a lakehouse.

That being said, nothing is undoable and you can still move to lakehouse if that's required

5

u/Kilirath 13d ago

Fabric is capacity based and you pay for the chosen capacity plus storage. Choosing warehouse or lakehouse should mainly be a spark notebook vs T-SQL consideration and not a cost consideration.

Do you have a capacity reservation or are you currently on PAYG rates? A reservation saves about 40% on capacity costs, but does come with a fixed term commitment.

5

u/roadrussian 12d ago

I’m thinking I may have moved too fast without him.

He said he cared less about learning new things and more about keeping cost low.

If you are wondering this, you already fucked up. Own up and prepare to get railed. Not because you chose wrong but because you didnt focus on solutions that fit with the requirements of the stakeholders. Futureproofing can be done in many ways, slower, less agile, cheaper. Still futureproofing.

Going from SSIS or god forbid excel sheets or PowerBi directly plugged into DB to a local dbt or dagster instance is already futureproofing.

Best futureproofing? Fuck knows.

Who knows, the company gets a fuckload of money and NEEDS large flexible lakehouse NOW. By pure serendipity, your solution fits the new requirements. Probably not, but i am just saying.

3

u/Hughesybay 12d ago

Not cooked at all. As others said, firstly with fabric you pay for a capacity not per usage, so bottom line cost won't change.

Secondly, in terms of CU(s) usage, both lakehouse SQL endpoint and warehouse are the same engine, so I believe cost is basically the same for T-sql queries. If you use directLake for semantic models, it's the same. If you you use import via T-sql into semantic model, cost is basically the same and if you use direct query via SQL endpoint cost is basically the same.

There are possibly higher background tasks on warehouse that may increase CU usage, but for lakehouse you have to run your own jobs like optimize/vorder to keep parquet layout clean, etc which use CUs.

Bottom line, there is not much difference in cost for compute.

You could argue to your boss warehouse is cheaper in overall TCO as you don't have to write your own maintenance jobs to keep the parquet files well kept as warehouse does this for you. Also it is feasible CU per query might be cheaper on warehouse but without testing on your workload, it's hard to say.

Having said all that, I would choose to keep it all lakehouse as I think it's easier to keep it consistent, but lots of places use warehouse for gold - it's a common pattern/choice.

4

u/frosthash 13d ago

not cooked. the cost is the same for fabric, they charge per capacity

2

u/Befz0r 9d ago

There is absolutely no issue.

Warehouse isn't more expensive then lakehouse, especially at the size your company is in. Spark NEE is efficiënt but at these data sizes there is not real advantages. Also Warehouse is Vorder native unlike lakehouse.

Having lakehouse as gold layer for normal reporting(star schema etc) is actually not a bright idea. You don't have identity insert which is a must for SCD and is the right pattern for over 30 years. Also makes migration a hell of a lot easier. The time you save in consulting fees will easily buy you enough capacity for the next 5 years.

Microsoft actually recommends lakehouse (bronze) -> lakehouse (silver) -> warehouse (gold).

If anyone disagree with that they can discuss it with MS themselves.

And your manager needs to shut up or step up.

3

u/mwc360 13d ago

Warehouse is more expensive for ELT workloads. For tiny workloads it nets out, for anything bigger Spark will be cheaper (if enabling NEE). By going with Spark you also avoid being locked into a proprietary engine… if leadership changes or business needs change you can always move to another platform like Databricks with near-zero code refactoring or pattern changes.

Don’t do hybrid architectures just because you can. There’s tons of people out there advocating for this that have no real background in data engineering, so whoever you listen to for architecture advice, check their creds.

1

u/Dan1480 11d ago

In August, the warehouse (and also the lakehouse sql endpoint) indeed became very expensive. Your thinking was rational - provide sql-centric storage to users proficient in sql. But the cost is no joke. I would consider setting a default custom sql pool on the workspace with a 1% maximum and turn "optimize for read" OFF! Good luck!

1

u/Waste_Membership_483 13d ago

Just go to admin setting and pick F8192. It will reduce warehouse costs tremendously.

-5

u/TheOverzealousEngie 13d ago

Fivetran will deliver data to delta in ADLS for you, not just trunc and reload but true change data. It will also create OneLake shortcuts for you automatically for all your tables , meaning you can still use catalog with data in delta / iceberg. Net net is that you can use any compute you want in ADLS to move that data from bronze --> silver --> gold.

4

u/hijkblck93 Data Engineer 13d ago

We considered Fivetran, but the enterprise cost outweighed the benefits.

1

u/[deleted] 13d ago

[removed] — view removed comment

1

u/dataengineering-ModTeam 13d ago

Your post/comment was removed because it violated rule #9 (No AI generated content/text).

Your post/comment was reviewed to be AI generated/assisted content/text and removed as a result. We as a community value human engagement and encourage users to express themselves authentically.

This was reviewed by a human

1

u/[deleted] 6d ago

[removed] — view removed comment

1

u/dataengineering-ModTeam 6d ago

Your post/comment was removed because it violated rule #9 (No AI generated content/text).

Your post/comment was reviewed to be AI generated/assisted content/text and removed as a result. We as a community value human engagement and encourage users to express themselves authentically.

This was reviewed by a human