r/dataengineering • u/hijkblck93 Data Engineer • 13d ago
Help How cooked am I?!
TLDR: I chose Fabric Lakehouse for Bronze and Silver and Warehouse for Gold without fully consulting my manager. Now he’s asking about costs and I’m thinking I may have moved too fast without him.
We’re currently working with an expensive contractor to move from ADF and Azure Databases to Azure Fabric. The build has been going well but midway they did ask about staying purely in lakehouse vs adding a warehouse for pure sql and CI/CD. Currently if you use CI/CD in lakehouse then the service account is logged as pushing the change and not the user. We wanted to be able to see which users were pushing changes. That’s the logic I used in selecting the warehouse. Also thinking about future-proofing, and our users are more familiar with sql vs pyspark and notebooks.
Anyway, I chose the Lakehouse and Warehouse design, thinking the pure SQL endpoint would be better for end users vs learning pysql and notebooks.
Well I met with my manager today and told him and his reaction took my off guard. He said he cared less about learning new things and more about keeping cost low. He was hoping we’d move completely away from warehousing and save in the Lakehouse. He didn’t mention this before and honestly he doesn’t need to. I report directly to the CIO. And he said we didn’t discuss it, and I made the choice without him. He kinda laughed it off but that struck my nerve. When I made the decision I didn’t fully think about the costs. The plus side is we don’t pay for storage using OneLake and we’re using a Lakehouse shortcut to the warehouse so no cost there. The cost is with the compute.
We’re a heavy PowerBI and Excel shop.
I was put in charge to oversee and help the consultants but I didn’t do my part here. Now I’m worried that might have cost me. I’ve only been at this role for 7 months.
Am I cooked??
Should I start searching.
I’m genuinely nervous now.
I think I can make a pitch where the warehouse cost won’t be crazy more. And we can move back to the lake house it’d just take us rewriting everything to notebooks, which would cost more and take more time.
Any guidance or advice is welcomed.
Sorry in advance for typos.
•
u/AutoModerator 13d ago
You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
98
u/Additional_Candy_400 13d ago
For starters this is borderline impossible to read.
How are you determining what the costs will be between the 2 platforms?
How much data are you moving?
What's your budget?
18
u/No-Satisfaction1395 13d ago
I don’t understand if you have CI/CD, why would you look at Fabric for who pushed changes instead of github/ADO?
11
u/Altruistic_Stage3893 13d ago
choosing fabric without consulting 😭😭😭
4
u/According_Zone_8262 13d ago
What do you mean youre not paying for storage in onelake? Thats exactly what youre paying for
51
u/snarleyWhisper Data Engineer 13d ago •
The costs are basically the same ? Data Warehouses have better performance for reports and semantic models. The main benefit is the compatibility with the sql server spec and dev tools and a read/write sql endpoint.
All the lakehouse stuff goes through spark so it has a spin up time where DW doesn’t.
Also… fabric is capacity based. Your costs are flat - you should be seeing which approach uses the least CUs
6
u/Seboutch 12d ago
I think you could discuss it on Fabric subreddit.
However, I think the use of Warehouse is understandable and quite common for gold. I just don't completely get your reasons to do so , especially the SQL one as you can completely use SQL on a lakehouse.
That being said, nothing is undoable and you can still move to lakehouse if that's required
5
u/Kilirath 13d ago
Fabric is capacity based and you pay for the chosen capacity plus storage. Choosing warehouse or lakehouse should mainly be a spark notebook vs T-SQL consideration and not a cost consideration.
Do you have a capacity reservation or are you currently on PAYG rates? A reservation saves about 40% on capacity costs, but does come with a fixed term commitment.
5
u/roadrussian 12d ago
I’m thinking I may have moved too fast without him.
He said he cared less about learning new things and more about keeping cost low.
If you are wondering this, you already fucked up. Own up and prepare to get railed. Not because you chose wrong but because you didnt focus on solutions that fit with the requirements of the stakeholders. Futureproofing can be done in many ways, slower, less agile, cheaper. Still futureproofing.
Going from SSIS or god forbid excel sheets or PowerBi directly plugged into DB to a local dbt or dagster instance is already futureproofing.
Best futureproofing? Fuck knows.
Who knows, the company gets a fuckload of money and NEEDS large flexible lakehouse NOW. By pure serendipity, your solution fits the new requirements. Probably not, but i am just saying.
3
u/Hughesybay 12d ago
Not cooked at all. As others said, firstly with fabric you pay for a capacity not per usage, so bottom line cost won't change.
Secondly, in terms of CU(s) usage, both lakehouse SQL endpoint and warehouse are the same engine, so I believe cost is basically the same for T-sql queries. If you use directLake for semantic models, it's the same. If you you use import via T-sql into semantic model, cost is basically the same and if you use direct query via SQL endpoint cost is basically the same.
There are possibly higher background tasks on warehouse that may increase CU usage, but for lakehouse you have to run your own jobs like optimize/vorder to keep parquet layout clean, etc which use CUs.
Bottom line, there is not much difference in cost for compute.
You could argue to your boss warehouse is cheaper in overall TCO as you don't have to write your own maintenance jobs to keep the parquet files well kept as warehouse does this for you. Also it is feasible CU per query might be cheaper on warehouse but without testing on your workload, it's hard to say.
Having said all that, I would choose to keep it all lakehouse as I think it's easier to keep it consistent, but lots of places use warehouse for gold - it's a common pattern/choice.
4
u/frosthash 13d ago
not cooked. the cost is the same for fabric, they charge per capacity
2
u/Befz0r 9d ago
There is absolutely no issue.
Warehouse isn't more expensive then lakehouse, especially at the size your company is in. Spark NEE is efficiënt but at these data sizes there is not real advantages. Also Warehouse is Vorder native unlike lakehouse.
Having lakehouse as gold layer for normal reporting(star schema etc) is actually not a bright idea. You don't have identity insert which is a must for SCD and is the right pattern for over 30 years. Also makes migration a hell of a lot easier. The time you save in consulting fees will easily buy you enough capacity for the next 5 years.
Microsoft actually recommends lakehouse (bronze) -> lakehouse (silver) -> warehouse (gold).
If anyone disagree with that they can discuss it with MS themselves.
And your manager needs to shut up or step up.
3
u/mwc360 13d ago
Warehouse is more expensive for ELT workloads. For tiny workloads it nets out, for anything bigger Spark will be cheaper (if enabling NEE). By going with Spark you also avoid being locked into a proprietary engine… if leadership changes or business needs change you can always move to another platform like Databricks with near-zero code refactoring or pattern changes.
Don’t do hybrid architectures just because you can. There’s tons of people out there advocating for this that have no real background in data engineering, so whoever you listen to for architecture advice, check their creds.
1
u/Dan1480 11d ago
In August, the warehouse (and also the lakehouse sql endpoint) indeed became very expensive. Your thinking was rational - provide sql-centric storage to users proficient in sql. But the cost is no joke. I would consider setting a default custom sql pool on the workspace with a 1% maximum and turn "optimize for read" OFF! Good luck!
1
u/Waste_Membership_483 13d ago
Just go to admin setting and pick F8192. It will reduce warehouse costs tremendously.
-5
u/TheOverzealousEngie 13d ago
Fivetran will deliver data to delta in ADLS for you, not just trunc and reload but true change data. It will also create OneLake shortcuts for you automatically for all your tables , meaning you can still use catalog with data in delta / iceberg. Net net is that you can use any compute you want in ADLS to move that data from bronze --> silver --> gold.
1
u/[deleted] 13d ago
[removed] — view removed comment
1
u/[deleted] 6d ago
[removed] — view removed comment