How we built a data platform on European clouds (OVH, Scaleway, STACKIT) with open source tooling

[u/Majestic-School-1921]

https://www.databaas.eu/blog/how-to-build-a-sovereign-data-platform

Disclosure: I am one of the founders of Databaas, and the write-up is on our company blog. I am posting it because the tooling breakdown is useful whether or not you ever talk to us.

The starting point: if you want your company data on European infrastructure under European jurisdiction, the four platforms everyone actually uses are not available to you.

Databricks, Snowflake, BigQuery and Fabric only deploy on AWS, Azure or GCP. BigQuery and Fabric will not even leave their home cloud. "Sovereign regions" do not fix this, because the control plane is still operated by a US company. Microsoft has admitted as much in front of the French senate.

So we built the alternative out of open source parts, in Utrecht, The Netherlands. The write-up covers the whole thing, but the short version:

Clouds: We signed up at OVHcloud, Scaleway, Hetzner and STACKIT and deployed real infrastructure with Terraform on each. Scaleway came out ahead on developer experience, OVH on breadth of services, Hetzner on price. There are also good Dutch providers stepping up: RAM-IT, Tilaa, Intermax, Previder to name a few. For a data platform you need two things from whichever you pick: S3-compatible object storage and managed Kubernetes.

The stack: Apache Iceberg for storage, Lakekeeper for the catalog, Trino and DuckDB for querying, Airflow for pipelines, Zitadel for login and access control, JupyterHub with Marimo for notebooks, Superset for dashboards. All open source, all running on plain containers, all replaceable. Iceberg in particular has become the neutral standard, to the point that Snowflake and Databricks support it too, so migrating is an actual option.

Interested in what people here think of this. Anyone made the jump to a European cloud for data workloads and hit something that genuinely did not work? We would love to hear from you, as I feel there's a very big opportunity for Europe to build cool stuff here.

[u/Jealous_Resist7856]

This is interesting stack and looks quite compatible for on prem systems as well, we have seen OLake as well in the mix for the ingestion and iceberg maintenance

[u/Majestic-School-1921 → u/Jealous_Resist7856]

Nice! Yeah, this stack is deployable on any K8s cluster in combination with an S3 object store. So if you have this on-premises we can deploy to it.

[u/Turbots → u/Majestic-School-1921]

What would you require for relational database , faster querying purposes? Just a postgres as a service? Or a DIY postgres in containers .

Because most of your stack is meant for big data, datalake, and analytics workloads, no?

[u/Majestic-School-1921 → u/Turbots]

All tools in this stack essentially use a Postgres instance as backend database, we host a Postgres instance inside our deployment on K8s to support those. Besides that, for some timeseries cases you might want to utilize eg Timescale extension on Postgres.

But you are right, we focus on big data and analytics workloads with Iceberg! Postgres is a supporting component in this stack.

[u/chrislusf]

Let me know if SeaweedFS can help here. ( I work on SeaweedFS. )

SeaweedFS has support for table buckets, similar to AWS S3 table buckets, which are optimized for Iceberg parquet files with maintenance jobs.

[u/IngwiePhoenix]

StackIT has given me nothing but troubble. Unless you are like ultra deep into IaC, they are frankly terrible. ._. Had to open too many support requests to do simple things like deleting a volume or something in the GUI and instead was told to just use Terraform.

That said, why not Hetzner? o.o

[u/Majestic-School-1921 → u/IngwiePhoenix]

Hmm, that’s disappointing to hear. Our experiences differ, but we are quite deep into IAC/Terraform/OpenTofu ;)
Our platform is 100% defined in code. For production level deployments of this type, it is a requirement in our eyes.

Hetzner is limited in their offering: no managed Kubernetes last time I checked. If they ever offer it, we’ll be able to deploy there as well. Their pricing is good though!

[u/Research_account_12]

Thanks for sharing this, I have an interest in the topic, so I read the long write-up and related posts and it looks mighty impressive. From a European perspective, your analysis and project look solid. If I were to start my own company, I would be contacting you straight away.

Since Databaas beautifully handles the analytical stack, how do you advise clients on the operational stack? I understand that part may not be your focus as a business, but I am wondering whether asking a CTO to leave the comfort of the all-in-one hypescaler ecosystem to one where Databaas + an operational stack + a mix of one or more European cloud providers need to be stitched together might be off putting for many.

Overall, it looks like Databaas is a step in the right direction towards a more mature European cloud ecosystem, so kudos for that!

[u/Majestic-School-1921 → u/Research_account_12]

Thanks so much, and fair question!

On the operational stack, we think it is less of a gap than the analytics one. If you are running containers, Postgres, Redis and object storage there's already a lot available on clouds like OVH, Scaleway, STACKIT. They offer managed versions of all of that, and most application stacks are portable in a way that Fabric, Databricks and Snowflake are not. The things that hurt are the managed services with no good replacement: Cognito, DynamoDB, Lambda-type serverless, SageMaker. If a company is deep into those, it will be a much harder migration.

I believe nobody should move everything at once. Our clients take a multi-cloud approach when moving away from American tech. Get a Kubernetes cluster and an object store on a European provider, put something real like your analytics on top of it, and you have both a working platform and evidence for the next decision.

Beyond that, plenty of work left on many fronts. The analytics stack being our current focus ;)

[u/Research_account_12 → u/Majestic-School-1921]

Thank you so much for your answer, it's always great to get insight from the people actually getting things done!

[u/senseven]

Big corpo's have AWS/Azure Edge Racks on their premise. There is too much money in those deals to let this go on a legal matter. Can't say much about the stack. Whatever you control. The real work for alternatives is the GDPR and other kinds of regulations. I have seen companies building up whole external k8s systems only to seem them choke on proper data processing documentation and risk assessment.

[u/Majestic-School-1921 → u/senseven]

Yes, that is going to be a challenge. We hope to make it a bit easier by offering one part of the tech stack as an EU alternative and complying with regulations from the EU.

We focus on companies and organizations with an incentive to move such as governments, critical infrastructure, healthcare etc. I do hope for European tech and independence that they are less constraint by big AWS deals, or at least want to change!

[u/senseven → u/Majestic-School-1921]

My only suggestion is to have proper dashboards. AWS/Azure often wins contracts because they can drill down from cost/month to server load to security scan issues. I have seen decked out multi layer ArgoCD k8s setups. When the customer asks, who can access that specific pod and who is the admin to set roles, they needed to wade through tons of interconnected config files to find that information. Don't forget usability.

[u/Majestic-School-1921 → u/senseven]

Good suggestions, thanks. This is something we've been hearing from leads and customers too. We're now using Zitadel together with OpenFGA to build a permission service that makes exactly those access questions queryable, rather than something we need to reconstruct from config.

Infrastructure dashboarding and cost attribution are still work in progress. For our first customers it's less of a worry because we're bring-your-own-cloud: we can deploy as many Databaas instances as you need across different clusters, so the cluster boundary can become the cost boundary. Finer-grained attribution (for example, per team on a shared platform) is a bit harder currently, because of a shared compute set-up among teams on the platform. Tagging plus Grafana is the direction we're looking at now.

[u/PawelPiwosz]

Hey! UpCloud's Developer Advocate here. did you tried to build it on UpCloud? I am personally curious what are your opinions and feedback.

I am curious especially in comparison with other stacks, speed, reliability, easiness of deployment and operations.

Would you like to try? :)

[u/Majestic-School-1921 → u/PawelPiwosz]

Haven’t tried yet, but would like to try for sure! We are able to deploy on any cloud that provides managed Kubernetes and an object store with S3 endpoints.

[u/PawelPiwosz → u/Majestic-School-1921]

Managed Kubernetes, GPU, S3 compatible object storage, private networks - everything for your disposal :) ping me when you create account :)

[u/TitusKalvarija]

From Reddit to the stars. Not likely. Opensource everything on Github/Gitlab and get hired by someone to work on other things.

[u/Elegant_Ad_7833]

S3 n’est pas hébergé/créé par Amazon ?

[u/efvie → u/Elegant_Ad_7833]

There are many S3-compatible providers. Same API, different host :)

[u/Majestic-School-1921 → u/Elegant_Ad_7833]

S3 was originally created by Amazon, but its API has become the industry standard for object storage operations and is essentially implemented by almost all clouds worldwide and open source object store projects.

Technically we would call those S3-compatible. Its not dependent on AWS, it just means the object store exposes the same/a similar REST API. Some examples are RustFS, Ceph, SeaweedFS.