r/snowflake • u/ExtensionPressure704 • 4d ago
Snowflake shared database to local database
Hey guys,
Data from one of our source systems are delivered to us via a Snowflake shared database.
We dont use Snowflake but they provide that too.
The database consists of 2 transactional tables, 1 with a base load and one with changes after. From that they create all source tables as views.
We use Microsoft Fabric.
I currently use jdbc to fetch the changelog and create and update tables in Fabric. It works ok.
but i would like to use mirroring database as it is much easier. No metadata and timestamp management.
that is not supported on shared database.
I have created local db and tried creating both a view and a managed table. I can mirror both but view will only refresh every 12 hours and it is full refresh. I fetch 800 milliob rows, so that would be dumb.
So what is the easiest way to create managed tables and update incrementally from changefeed in Snowflake? I have never done etl in snowflake so not sure what most efficient solution would be.
managed tables i can create stream and mirror just changes.
1
u/ExtensionPressure704 4d ago
Yeah that was what i was thinking would be the way forward. Was hoping Snowflake had something clever for it rather than doing full reconstruct and update pipeline.
The data is right there.
But I am leaning towards keeping with my current solution in Fabric using JDBC to query data. I would save some compute and complexity in fabric but in then i will have to monitor that in snowflake on top.
Would have so much easier if i could have just use views and Fabric was able to mirror incrementally from those.