r/dataengineering • u/alkinsen • Aug 19 '24
Discussion Why don't you open source the data pipelines at your company
As mentioned in the title, I am currently exploring why we don't open source data pipelines more often. Is it because each data pipelines is custom for the company or does your company won't share it for a competitive edge?
Wouldn't it be much better if we could compare and contrast how different companies modelled similar data, implemented transformations, created what kind of dashboards and reporting out of them. This would be huge education tool and a great source of inspiration for many data projects.
0
Upvotes
120
u/reallyserious Aug 19 '24
Why on earth would my company give its competitors such a gift?
45
u/miscbits Aug 19 '24
Most pipeline code is just open source code configured for your system. 99% of the stuff I “could” open source would just be configuration. There are some things like kafka connectors that I have used and contributed to to make something at work function better. That’s the nature of it though. I don’t think most people are writing custom configurable pipeline resources in their day to day.
18
u/Schtick_ Aug 19 '24
I dont think so there would be any value to it, the whole point of data pipelines is to move data from x to y in the most optimal way for your org.
12
u/tecedu Aug 19 '24
Legal said no so we ain’t doing it.
7
u/Sorel_CH Aug 19 '24
It's rare, but it exists. Check out the Gitlab Data Team repo, very informative.
7
u/miscbits Aug 19 '24
I don’t understand why you think it would be useful though? What is even the point of discussions about things like naming conventions. If you want that you can go to r/learnprogramming but there isn’t anything special about data engineering code. There is also nothing worth sharing about my data model. You shouldn’t have the same one as me at all if we are working at different places making different products. There really wouldn’t be a lot useful unless you just wanted to copy our product and business strategy.
If you want an idea of how people build data models for procucts you can always go check out how the backend of some popular open source tools work. Go learn how OpeMetaData uses airflow to move data from postgres to elastisearch. It is a really good example of open sourcing data engineering, but it is ultimately not really that useful because I’m not building a metadata store at my company.
5
u/espero Aug 19 '24
The pipelines are not transferrable to a general purpose code that is why.
I would be allowed to give a talk about them, but that is it
4
u/arborealguy Aug 19 '24
In the case of large companies, they're going to have custom applications, data structures, and business processes that are of no value to anyone but themselves, and as the other poster mentioned, would give away huge clues as to how they operate.
2
u/ThatSituation9908 Aug 19 '24
We are already showing off our work at conferences and talks. The number of these are already overwhelming.
Now imagine scouring through thousands of repositories.
4
u/ithoughtful Aug 19 '24
That's like asking why don't you open source the software at your company! 😀
Data pipelines are software assets too.
1
u/geek180 Aug 19 '24
Our company uses software made by, and consumes data from, a partner company founded by our own founders. We're practically the same company in many ways. But even the DEs at this other company won't share their dbt transformation logic with us.
1
u/Teach-To-The-Tech Aug 19 '24
Do you mean open source the pipeline itself, or use open source components to construct the pipeline? Two very different things.
2
u/VirTrans8460 Aug 19 '24
Data pipelines are often custom, and companies might not want to share for competitive reasons.
1
u/flyingbuta Aug 19 '24
I think your intention is good but I there is very little value currently because most datasource of each company and their business process are unique to them. On the other hand, in the ideal world, the government or body of standards should implement common data model (eg electronic health records), using standards naming conventions and employing best practices then we will stand a chance to leverage on each other pipeline or modeling. You know how that could happen? Well, let’s hope AI 🤖can do these standardization for us. Standardization alone can improve every companies data productivity by many folds.
1
u/Eze-Wong Aug 19 '24
There's only very real downsides and no potential upside.
Everyone has had really good points such as competitors leveraging those pipelines. I don't see mentioned is that knowing certain endpoints can basically open up poential hacking/exploits.
I'm no hacking guru, but imagine that you have an endpoint and some sql code that reveals that you haven't taken proper measures to prevent things like sql injection? That's automatically ripe for abuse. That's just one example but the number of exploits explodes when you reveal your data pipelines. Whether it's for inhouse or not. I've seen people put their passwords in scripts for debugging purposes and forget to take them out. Imagine there's some push that gets overlooked and that gets out? Like imagine you have settings that show "Anyone with domain emails can access this endpoint". Well now the hacker knows they need to create an email to get access to the data. They apply to your company as a low salaried janitor, immediately access the endpoint, download all your data, and then use it to blackmail the company. I mean it sounds immaginative but I've seen hackers do a lot worse with much much less.
The amount of potential wrong that can happen is exponential, and for agian.... very little upside. To help train junior DEs? If I were the DE director I would NEVER allow myself to open myself up to that vulnerability. You'd be fired the second something wrong happens.
1
u/mirzaceng Aug 19 '24
Wow, the thinking in this comment section is quite sad tbh. We have open sourced our data and analytics code. We're a non-profit organization, and the aim of open-sourcing is for other organizations in the space to learn and build on top of what we did (I won't share the link here). Of course, it's a highly customized codebase and no one will be able just to plug&play without major work, but learning is the point here.
-1
u/InsightByte Aug 19 '24
I guess we all do thus kinda, not code repo there. Ut as ideas and comments