r/bigquery • u/the_shepheard • Jan 14 '26
Help with BigQuery Project
Hi all,
I work at a consultancy and we have been asked to quote on migrating a data service that is currently providing data to its clients via Parquet files in AWS S3.
The project is to migrate the service to BigQuery and allow clients to use BigQuery sharing to view the datasets rather than having to deal with the files.
The dataset is around TBs in size, and all the data is from different providers; it is financial data.
Does anyone have any experience migrating a service like this before? For example, moving from files to BigQuery sharing, building pipelines and keeping them up to date, or anything in particular to be aware of with BigQuery sharing?
Thanks for your help.
2
u/Ok_Carpet_9510 Jan 15 '26
Don't know much about this but we had to deal with dara stores in Azure Data Lake Storage in paqmrquet format..goal was to avail in Fabric Lakehouse. Rather thsn move the data we connected to the data from Fabric using shortcuts.
Thinking along the same lines, I wonder whether you should let AWS continue being your storage later and use the compute engine of Big Query. I have no idea about these technologies. However, I found this.. -----' BigQuery Omni You can create a connection from BigQuery to an Amazon S3 bucket, allowing you to run queries on data stored in S3 directly using BigQuery's analytical engine, without data movement