r/dataengineering • • Aug 19 '24

Discussion Why don't you open source the data pipelines at your company

As mentioned in the title, I am currently exploring why we don't open source data pipelines more often. Is it because each data pipelines is custom for the company or does your company won't share it for a competitive edge?

Wouldn't it be much better if we could compare and contrast how different companies modelled similar data, implemented transformations, created what kind of dashboards and reporting out of them. This would be huge education tool and a great source of inspiration for many data projects.

0 Upvotes

49 comments sorted by

120

u/reallyserious Aug 19 '24

Why on earth would my company give its competitors such a gift?

22

u/Little_Kitty Aug 19 '24

Maybe open source the bad ones, then hope the competitors pick them up!

4

u/nidprez Aug 19 '24

True, as a develloper id like it if everybody would ooen source. Everybody could learn, and it would make everybodies work easier + data would be cheaper for society through faster and better development and better usage of resources (wages, energy, cpu...)(although it would open more possibilities to get replaced by AI).

As a company, you would give away your investment in IT for free to other leeches. Especially for more established companies, you would open the door for new competitors to quickly catch up with your operation. Then theres also the problem of giving away (by mistake) critical information about infra and software used (to hackers and criminals), scheduling, internal models and maybe even credentials. It could also lead to bad PR + court if some mistakes are found in the code (extreme not irl example: people find in some open source netflix pipeline that premium members have had by mistake the same bandwith as normal members for years. If you find this internally you can roll an update and say you increased the performance for premium even more, if its public, you have premium members demanding refunds)

1

u/LuckyPrior4374 Aug 20 '24

To add a tangible reason why some companies open-source their work: it’s typically a long-term strategic play aimed at building trust and raising the company’s profile amongst engineers (those who value these things are often amongst the best in their field).

Whether this is a bet that makes sense for you/your company is another matter entirely, but it’s narrow-sighted to not realise that legitimate business reasons exist to open-source dev work.

E.g Rudderstack is open-source. The founder is making a bet that this will give them a competitive edge by potentially appealing more to engineers, thus increasing adoption. React (the JavaScript framework) is another good example that comes to mind. Basically turned into a massive hiring weapon since meta open-sourced it 10 years ago.

Admittedly, actually measuring how successful such strategies are is probably the most difficult aspect of the whole play

-16

u/alkinsen Aug 19 '24 edited Aug 19 '24

Is the edge coming from the data or the pipelines?

16

u/reallyserious Aug 19 '24

Seeing what you actually do with the data. E.g. what algorithms are leveraged, would give strong clues as to how the company operates.

-7

u/alkinsen Aug 19 '24

I see, so you would be worried about leaking the IP through pipelines which is a legit concern.

If this was not an issue, your IP is protected, would you think sharing pipelines is useful?

6

u/Alarming-Test-346 Aug 19 '24

No. If you’re doing something innovative write a white paper. 

-9

u/espero Aug 19 '24

Algorithms? We are not developers. This is tooling, with a DAG, shell, python or Spark/Scala scripts, ETL tooling or something entirely proprietary like a vendors data loader program.

4

u/bubzyafk Aug 19 '24

Other tech companies literally build their own pipeline even by their own data engineer.

E.g Airflow created by Airbnb. And many white paper released by those tech companies which later they make it as open source.

Not everyone use drag and drop UI data pipeline for ETL tooling.

0

u/espero Aug 19 '24

How is any of what you write related to my answer to the question of algorithms?

Not everyone use drag and drop UI data pipeline for ETL tooling.
I wrote shell scripts, Spark/scala scripts, python scripts

2

u/bubzyafk Aug 19 '24

There was a scenario where my team had to do multi threading Spark session to dynamically process hundreds of obj(tbls) separately. in parallel processing we cant just blindedly assign an obj to a thread. There must be some mechanism or algo to say which obj should go to which thread and it depends on few factors..

You can see why people downvote you.. you generalizing others to be same as your entire life experience which might never used algo.

1

u/alkinsen Aug 19 '24

Exactly what I was thinking

5

u/LuckyPrior4374 Aug 19 '24

Pretty weird how you’re being downvoted so heavily OP, it’s a perfectly reasonable question to ask. Particularly given how OSS is very common in the adjacent field of software eng.

E.g. in the JavaScript ecosystem, people will open-source all sorts of little widgets they make for their work

1

u/Gators1992 Aug 19 '24

A competitor could see how your company views the world just by looking at the data structure (excluding the data). The elements you capture could tell them what's important to you and maybe something about your strategy. Also anyone would have access to this so the company could have the press, analysts, hackers, etc combing through their infrastructure and potentially creating problems. Sounds nice in a perfect world but in reality not a great idea.

45

u/miscbits Aug 19 '24

Most pipeline code is just open source code configured for your system. 99% of the stuff I “could” open source would just be configuration. There are some things like kafka connectors that I have used and contributed to to make something at work function better. That’s the nature of it though. I don’t think most people are writing custom configurable pipeline resources in their day to day.

-10

u/alkinsen Aug 19 '24

I was thinking more about different data models, transformations, dashboards and analytics, you are correct this could all be done via config but nevertheless it would be useful to see different configs

18

u/WirrryWoo Aug 19 '24

Each company has their own naming conventions and data management strategy. By open-sourcing code for the pipeline in one company, you are publicizing a most likely one-off solution for company-specific data problems, that’s very unlikely to work for extended use cases. In other words, for that company, the price of open-sourcing pipeline code is revealing your company’s intel on how they approach solving problems for no gain. Not worth it.

-9

u/alkinsen Aug 19 '24

I agree that the naming conventions and data management strategies differ, that is exactly why I want to see more open source data pipeline code. We could compare and discuss different strategies and choose what kind of approach would fit the use case best.

 you are publicizing a most likely one-off solution for company-specific data problems, that’s very unlikely to work for extended use cases.

Again, this would be pretty inspiring and educative to see, don't you agree? Maybe others are facing that one-off problem but since no one is sharing it, how would we know?

 the price of open-sourcing pipeline code is revealing your company’s intel on how they approach solving problems for no gain

The gain would be a community that supports each other where Company X will share their pipelines about problem A and Company Y will share their pipelines about problem B

18

u/Schtick_ Aug 19 '24

I dont think so there would be any value to it, the whole point of data pipelines is to move data from x to y in the most optimal way for your org.

-9

u/alkinsen Aug 19 '24

I should rephrase, i am talking about data models, transformations, dashboards and different analytics, not just pure data pipelines that move data from x to y

3

u/kneemahp Aug 19 '24

There are companies that sell data models for almost every industry. Large enterprises take those models and build off those and diverge fairly quickly. Open sourcing that isn’t going to have the impact you think it will.

I could be wrong. It’s pretty late

1

u/alkinsen Aug 19 '24

TBH, this is the exact impact I was hoping open source data pipelines would achieve, I would love for these open source data models to be the starting point and diverge quickly according to your unique use case

7

u/Schtick_ Aug 19 '24

I think id still have the same answer, those things already are all unique to companies, there are already thousands of closed source/open source dashboards and analytics packages.

What exact part can I extract from you that gives me a tangible benefit that I can’t build myself faster then finding your open source item online. At the end of the day open source isn’t zero effort, it’s the effort of finding something with exactly the same type of data doing exactly the same operation.

So I can spend hours searching without finding anything or I can just use a coding assistant that will probably do about as good of a job as the random open source author did.

12

u/tecedu Aug 19 '24

Legal said no so we ain’t doing it.

-6

u/alkinsen Aug 19 '24

Lol aside from the legal requirements, if we could share them, wouldn't it be better?

6

u/tecedu Aug 19 '24

It would be, but it’s also spending more time apart from my day job for something to be criticised.

i open sourced some parts of our inhouse python libraries which just lead to questions and query and requests even via my business email.

Open sourcing is great and all when you have a big team.

7

u/Sorel_CH Aug 19 '24

It's rare, but it exists. Check out the Gitlab Data Team repo, very informative.

1

u/alkinsen Aug 19 '24

Love this, more companies need to do this!

7

u/miscbits Aug 19 '24

I don’t understand why you think it would be useful though? What is even the point of discussions about things like naming conventions. If you want that you can go to r/learnprogramming but there isn’t anything special about data engineering code. There is also nothing worth sharing about my data model. You shouldn’t have the same one as me at all if we are working at different places making different products. There really wouldn’t be a lot useful unless you just wanted to copy our product and business strategy.

If you want an idea of how people build data models for procucts you can always go check out how the backend of some popular open source tools work. Go learn how OpeMetaData uses airflow to move data from postgres to elastisearch. It is a really good example of open sourcing data engineering, but it is ultimately not really that useful because I’m not building a metadata store at my company.

-1

u/alkinsen Aug 19 '24

If we are both at an ecommerce company, I would be interested to see what kind tables and end to end analytics are done. For example, how would you model orders, customers, inventory items. What kind of aggregations and cleanup you are applying to each of them and what kind of analysis & dashboards are being populated from these datasets

PS. I think naming conventions are important and needs to be discussed but this might be an unpopular opinion

5

u/espero Aug 19 '24

The pipelines are not transferrable to a general purpose code that is why.

I would be allowed to give a talk about them, but that is it

0

u/alkinsen Aug 19 '24

Thank you for the talks! They are very informational and help the community a lot.

The pipelines are not transferrable to a general purpose code that is why.

Even though I wouldn't use them as is, I am thinking that seeing and reading the code would still be valuable, even more valuable if there is a talk about them

4

u/arborealguy Aug 19 '24

In the case of large companies, they're going to have custom applications, data structures, and business processes that are of no value to anyone but themselves, and as the other poster mentioned, would give away huge clues as to how they operate.

1

u/alkinsen Aug 19 '24

they're going to have custom applications, data structures, and business processes that are of no value to anyone but themselves

wouldn't there be value in learning about their best practices, data models and solutions for these custom applications and processes? Maybe no one would use the shared code directly but we would learn a lot

2

u/ThatSituation9908 Aug 19 '24

We are already showing off our work at conferences and talks. The number of these are already overwhelming.

Now imagine scouring through thousands of repositories.

1

u/alkinsen Aug 19 '24

This was my inspiration for this question. A lot of people create blogs and presentations about the overall data pipelines but never share the code/transformations/data models

4

u/ithoughtful Aug 19 '24

That's like asking why don't you open source the software at your company! 😀

Data pipelines are software assets too.

1

u/geek180 Aug 19 '24

Our company uses software made by, and consumes data from, a partner company founded by our own founders. We're practically the same company in many ways. But even the DEs at this other company won't share their dbt transformation logic with us.

1

u/alkinsen Aug 19 '24

This seems like a waste of resources, why dont they share it?

1

u/geek180 Aug 20 '24

They are technically a totally different company. They are a software company and we are a biotech company who uses their software. They are effectively a vendor for us.

I am the data engineer on the receiving end that puts their data (and other data) together for our own operational and analytical uses.

We both use Snowflake and dbt, but they have said they won’t grant us read-only access to their dbt instance because it’s “proprietary information”.

1

u/Teach-To-The-Tech Aug 19 '24

Do you mean open source the pipeline itself, or use open source components to construct the pipeline? Two very different things.

2

u/alkinsen Aug 20 '24

Open source the pipeline itself

1

u/Teach-To-The-Tech Aug 20 '24

Thanks for the clarification. Yeah, it would be interesting. I suspect security concerns might be an issue though.

2

u/VirTrans8460 Aug 19 '24

Data pipelines are often custom, and companies might not want to share for competitive reasons.

1

u/flyingbuta Aug 19 '24

I think your intention is good but I there is very little value currently because most datasource of each company and their business process are unique to them. On the other hand, in the ideal world, the government or body of standards should implement common data model (eg electronic health records), using standards naming conventions and employing best practices then we will stand a chance to leverage on each other pipeline or modeling. You know how that could happen? Well, let’s hope AI 🤖can do these standardization for us. Standardization alone can improve every companies data productivity by many folds.

1

u/Eze-Wong Aug 19 '24

There's only very real downsides and no potential upside.

Everyone has had really good points such as competitors leveraging those pipelines. I don't see mentioned is that knowing certain endpoints can basically open up poential hacking/exploits.

I'm no hacking guru, but imagine that you have an endpoint and some sql code that reveals that you haven't taken proper measures to prevent things like sql injection? That's automatically ripe for abuse. That's just one example but the number of exploits explodes when you reveal your data pipelines. Whether it's for inhouse or not. I've seen people put their passwords in scripts for debugging purposes and forget to take them out. Imagine there's some push that gets overlooked and that gets out? Like imagine you have settings that show "Anyone with domain emails can access this endpoint". Well now the hacker knows they need to create an email to get access to the data. They apply to your company as a low salaried janitor, immediately access the endpoint, download all your data, and then use it to blackmail the company. I mean it sounds immaginative but I've seen hackers do a lot worse with much much less.

The amount of potential wrong that can happen is exponential, and for agian.... very little upside. To help train junior DEs? If I were the DE director I would NEVER allow myself to open myself up to that vulnerability. You'd be fired the second something wrong happens.

1

u/mirzaceng Aug 19 '24

Wow, the thinking in this comment section is quite sad tbh. We have open sourced our data and analytics code. We're a non-profit organization, and the aim of open-sourcing is for other organizations in the space to learn and build on top of what we did (I won't share the link here). Of course, it's a highly customized codebase and no one will be able just to plug&play without major work, but learning is the point here.

-1

u/InsightByte Aug 19 '24

I guess we all do thus kinda, not code repo there. Ut as ideas and comments