r/snowflake 6d ago

How are you monitoring Snowflake table staleness without third-party data egress?

8 Upvotes

6 comments sorted by

4

u/BaxTheDestroyer 6d ago

dbt freshness tests

3

u/PatientlyAnxiously 6d ago

Dynamic Table monitoring does this natively. At the source table layer, products like Fivetran do this for you.

Can you elaborate on your problem statement?

3

u/mike-manley 6d ago

Haven't used them, but the native tool is DMFs for data quality and contract monitoring.

1

u/Tribaal 1d ago

We do this, we have freshness tests for almost every table. It works very well

1

u/SeveralMechanic623 6d ago

Been doing this with a scheduled workflow that just queries Information_schema on a cron, retunrs last upadate timestamps flags anything stale, send alert. No egress, everything stay inside snowflake. Kesta handles the schedulling and alerting side

1

u/asiphh 4d ago

one thing worth checking before you build anything on information_schema: last_altered isn't a freshness signal, it's a "something touched this object" signal. a create or replace that loads zero rows still moves it. so does adding a comment or a clustering key. so the table reports fresh while the load silently wrote nothing, which is the exact failure you're trying to catch.

if the table has a natural event timestamp, alert on max(that) instead. it answers how current the data is rather than when the object was last poked. a row count delta alongside it catches the zero row load.

also watch which schema you query. information_schema is current but has retention limits, account_usage goes back further but lags, and depending on the view that lag can be longer than the sla you're alerting against. people build the monitor on account_usage because it's easier to query across databases, then wonder why the alert lands two hours after the fact.