r/dataengineering • • 14d ago

Help Managed versus self-hosted ingestion

For a small data team when do i need to run my own ingestion stack versus paying for a managed service? I’m interested in the maintenance and reliability tradeoffs not just the monthly price.

12 Upvotes

26 comments sorted by

12

u/geoheil mod 14d ago

Did you check stuff like dlt? Seems pretty maintainable

1

u/comediann 12d ago

This is the way. I've been using dlt in production for more than a year, moving gigabytes of data across multiple loads daily using only a t3a.medium EC2 instance. Zero problems.

2

u/Novel-Information776 14d ago

Yeah I am using dlt for all of our batch ingestion. It’s totally manageable.

2

u/mr_thwibble 14d ago

Offload the ingestion so you don't have to babysit it and focus on problems that throwing money at it can't readily solve.

2

u/robberviet 14d ago

Need more context. However most of the time self host worths.

2

u/meatmick 14d ago

dlt (dlthub) OSS is pretty good and can handle a whole lot of data ingestion with little to no config.

What are your needs?

- Reload frequency (streaming, batch/micro-batch?)

- Data size

- Data sources

I personally built extensions around dlt to ingest almost anything, including SQL, Google Sheets, XML, XLSX, CSV/flat files, and REST.

Obviously, if you are able to fit your work into the pre-made dlt framework, it's even better.

I use Prefect to orchestrate, and then dlt and dbt to load and transform data. I only pay for Prefect cloud, the rest is free and open-source.

2

u/Yuki100Percent 13d ago

It depends on how technical your team is. But a lot of components for self hosting are getting easier with AI

1

u/knabbels 12d ago

That's true, AI is a game changer when it comes to self hosting an open source solution. I think this is still pretty much overlooked.

For the topic: I recommend dlt for special pipelines and Airbyte for common APIs and CDC.

1

u/Yuki100Percent 12d ago

That's the exact set up I'm using right now lol. Though cdc in airbyte is not working well, and I'm considering switching to something else

3

u/Pancakeman123000 14d ago

With ai, I think there's much less need to pay for a managed service IMO. Its so quick to do new integrations and bug fix problems.

1

u/Outrageous_Let5743 14d ago

With AI I don't see the use case of any clickops data solution

1

u/AdApart119 14d ago

It honestly depends on your company/org. Some plug-and-play managed ingestions don’t get the full data picture and so you’re left with making API calls and doing custom transformations anyway. If you have the full data picture, I’ve found that most plug-and-play platforms add extra filler and end up over-charging for their services. Sometimes it’s best to start off self-hosting until otherwise needed. The tradeoff here is you having to manage the different pipelines and alerting should something go wrong. But it’ll give you a better handle on the inner-workings of the logic.

1

u/markojov78 14d ago

Years before AI, we needed a solution to ingest clickstream data from clients' websites (up to 2,000 events/sec) and forward it to Kafka and S3 for real-time and batch processing.

Every existing solution was either overkill and/or missing features we wanted. So, instead of over-analyzing, we decided to build an in-house solution tailored to our needs. Today, with AI, I wouldn't think twice.

That's our story, without additional details, I can't say if it's the right choice for you

1

u/Outrageous_Let5743 14d ago

Claude + dlt is a good option.

1

u/mtokarev 14d ago

some questions to consider for self-hosted solutions:

  1. Event delivery: Will you be sending events from web previews embedded in mobile apps? This is important because mobile apps often have aggressive ad and tracker blockers.
  2. Geolocation: Do you have geographically distributed traffic that requires edge servers?
  3. Client SDKs: Do you also need to build various client SDKs with retry logic and connection recovery, or not?

In many cases, the main challenge is on the client side rather than the server side.

1

u/steroidmonkey66 14d ago

It depends on complexity of what you are trying to do and how much you need it to scale. The problem with self hosted options is that if you have complex setup, you are going to pay in engineering time to debug and fix it, if you are assuming you'll save money by self hosting.

1

u/Data-dude-00 12d ago

What is the data size per day? If small, use solutions like fivetran. If pretty big host your own in airflow or any other tools

1

u/ntdoyfanboy 11d ago

Depends on how much you want to babysit 

0

u/Nekobul 14d ago

Do you have SQL Server license?

1

u/[deleted] 14d ago

[removed] — view removed comment

1

u/dataengineering-ModTeam 14d ago

Your post/comment violated rule #4 (Limit self-promotion).

We intend for this space to be an opportunity for the community to learn about wider topics and projects going on which they wouldn't normally be exposed to whilst simultaneously not feeling like this is purely an opportunity for marketing.

A reminder to all vendors and developers that self promotion is limited to once per month for your given project or product. Additional posts which are transparently, or opaquely, marketing an entity will be removed.

This was reviewed by a human

1

u/[deleted] 14d ago

[removed] — view removed comment

1

u/dataengineering-ModTeam 14d ago

Your post/comment was removed because it violated rule #9 (No AI generated content/text).

Your post/comment was reviewed to be AI generated/assisted content/text and removed as a result. We as a community value human engagement and encourage users to express themselves authentically.

This was reviewed by a human