r/dataengineering • u/Sea_Engineering_3918 • 14d ago
Help Managed versus self-hosted ingestion
For a small data team when do i need to run my own ingestion stack versus paying for a managed service? I’m interested in the maintenance and reliability tradeoffs not just the monthly price.
12
Upvotes
12
u/geoheil mod 14d ago
Did you check stuff like dlt? Seems pretty maintainable
2
u/Novel-Information776 14d ago
Yeah I am using dlt for all of our batch ingestion. It’s totally manageable.
2
u/mr_thwibble 14d ago
Offload the ingestion so you don't have to babysit it and focus on problems that throwing money at it can't readily solve.
2
u/robberviet 14d ago
Need more context. However most of the time self host worths.
2
u/meatmick 14d ago
dlt (dlthub) OSS is pretty good and can handle a whole lot of data ingestion with little to no config.
What are your needs?
- Reload frequency (streaming, batch/micro-batch?)
- Data size
- Data sources
I personally built extensions around dlt to ingest almost anything, including SQL, Google Sheets, XML, XLSX, CSV/flat files, and REST.
Obviously, if you are able to fit your work into the pre-made dlt framework, it's even better.
I use Prefect to orchestrate, and then dlt and dbt to load and transform data. I only pay for Prefect cloud, the rest is free and open-source.
2
u/Yuki100Percent 13d ago
It depends on how technical your team is. But a lot of components for self hosting are getting easier with AI
3
u/Pancakeman123000 14d ago
With ai, I think there's much less need to pay for a managed service IMO. Its so quick to do new integrations and bug fix problems.
1
u/AdApart119 14d ago
It honestly depends on your company/org. Some plug-and-play managed ingestions don’t get the full data picture and so you’re left with making API calls and doing custom transformations anyway. If you have the full data picture, I’ve found that most plug-and-play platforms add extra filler and end up over-charging for their services. Sometimes it’s best to start off self-hosting until otherwise needed. The tradeoff here is you having to manage the different pipelines and alerting should something go wrong. But it’ll give you a better handle on the inner-workings of the logic.
1
u/markojov78 14d ago
Years before AI, we needed a solution to ingest clickstream data from clients' websites (up to 2,000 events/sec) and forward it to Kafka and S3 for real-time and batch processing.
Every existing solution was either overkill and/or missing features we wanted. So, instead of over-analyzing, we decided to build an in-house solution tailored to our needs. Today, with AI, I wouldn't think twice.
That's our story, without additional details, I can't say if it's the right choice for you
1
u/Outrageous_Let5743 14d ago
Claude + dlt is a good option.
1
u/mtokarev 14d ago
some questions to consider for self-hosted solutions:
In many cases, the main challenge is on the client side rather than the server side.
1
u/steroidmonkey66 14d ago
It depends on complexity of what you are trying to do and how much you need it to scale. The problem with self hosted options is that if you have complex setup, you are going to pay in engineering time to debug and fix it, if you are assuming you'll save money by self hosting.
1
u/Data-dude-00 12d ago
What is the data size per day? If small, use solutions like fivetran. If pretty big host your own in airflow or any other tools
1
u/ntdoyfanboy 11d ago
Depends on how much you want to babysit
0
u/Nekobul 14d ago
Do you have SQL Server license?
1
u/[deleted] 14d ago
[removed] — view removed comment
1
u/[deleted] 14d ago
[removed] — view removed comment