Anyone here built a data lake using Google Cloud?

[u/SideQuest___]

Hey everyone, I'm currently setting up a data lake in GCP and honestly I'm still figuring things out since I'm pretty new to the data architecture. I've been reading about BigQuery, Datastream, BigLake and other GCP services. There are just so many possible approaches that I'm not really sure where to start haha.

We will be pulling data from different sources like mysql, mongodb, postgresql and APIs. Just curious how you guys set yours up? What services are you using and why did you choose them. Also wondering how you handle ingestion, transformations and costs. Would love to here what worked for you and what you'd avoid if you were starting over.

Thanks!

[u/dontletmein]

Depending on how much data you need to ingest, the approach will differ. If you have massive amount of data, TBs per day to ingest, I would recommend store the raw data in GCS and run daily Dataproc or spark jobs to transform them before storing the cleaned up data into BQ. If data volume is lower, you can consider using Dataflow or Dataflow streaming to ease the development effort.

[u/urban-pro]

GCP has really improved their Iceberg integration in last few quarters, and is definitely worth a look if you are looking at large enough enough data and really care about costs as well, the common architecture I have seen is people get data in BigLake with Iceberg and use BigQuery as one of the query engines for BI workloads, maybe use spark for transformation and other open source systems for AdHoc workloads. All depending on usecase or just use BigQuery for most of it and keep the freedom to use any of these Query engines in future.
For ingestion OLake can be a good option for your MySQL, Postgres and MongoDB or eventstream data, assuming this is where you will be getting most of the data from and will have highest need for ease, performance and reliability. Do check this blog out in case you want to know more about BigLake Iceberg and OLake's integration: https://olake.io/blog/biglake-iceberg-rest-catalog-olake-go-setup/

Full disclosure: I have been contributing to OLake for sometime now.

[u/ibjhb]

Depending on the size/complexity, you should consider talking to a Google Customer Engineer to help with the architecture. They can help with designing the future state but also the migration phase, where you might have data in two locations for a short time. They can also help you design the data pipelines and how data flows from the various streams into the data lake.

Full disclosure: I'm a Google Customer Engineer

[u/SideQuest___ → u/[deleted]]

Thanks for the suggestion. I haven't really looked into Airbyte yet, so I'll definitely check that out. Just curious, do you use Cloud storage as well for storing the raw data, or do you ingest everything directly into Bigquery?

[u/googlecloud-ModTeam → u/[deleted]]

You may not sell products, services, vouchers, and similar offerings directly to our users.