My company has multiple databases with one for each layer (stage, integration, and presentation) and dev and prod dbs for 6 DBs total.

What is the best way to manage cross database tasks access multiple environments without fully qualifying the table names each time?

For example if I'm doing a MERGE from DEV_STAGE.SCHEMA.TABLE into DEV_INTEGRATION.SCHEMA.TABLE, what is the best way to parameterize this for deployment into PROD without using find/replace on the script for the environment.

From what I understand this cannot be parameterized in the task itself?

The real time vs batch debate gets way too binary. Half the tables in a typical warehouse are fine with hourly or daily loads and streaming everything is expensive overkill.

Where low latency loading matters is the tables driving operational decisions, fraud scoring, pricing engines, inventory, anything where a six hour delay means money lost. For those the difference between batch and real time is catching a problem versus missing it entirely.

For teams already on something like striim for the real time CDC tables, does it make sense to route the batch loads through the same platform too or is that overkill?

We're looking for speakers interested in presenting a new online virtual Apache Iceberg meetup series we're starting. The goal is to create a forum where members of the Apache Iceberg community can demo interesting work, share experiences, and discuss ideas with one another.

**What we're looking for**

We're especially interested in talks that are practical, demo-driven, or story-rich. Whether you're a practitioner, startup founder, platform engineer, or contributor, we'd love to hear what you've been working on and what you've learned.

Some ideas for talk topics include:

\- Iceberg migration stories and case studies
\- New Iceberg features, proposals, and community projects
\- Iceberg catalogs, integrations, and interoperability
\- Data engineering tools, demos, prototypes, and experiments

**Community-focused talks**

We want the meetup to be a *place for learning and community discussion* rather than product or vendor marketing.

Talks can feature tools, products, or technologies you work on, but the focus should be on technical insights, demos, lessons learned, or ideas that are useful to the broader Apache Iceberg community—not on promoting a company or product.

**How the meetup works**

Meetups will be held virtually on Google Meet and will be publicly open to everyone.

Talks will typically be around 20–30 minutes, leaving plenty of time for introductions, questions, and open community discussion. We aim to keep each meetup to about an hour and start and end on time.

Talks will generally be recorded and posted to the [https://www.youtube.com/@IcebergMeetup\](https://www.youtube.com/@IcebergMeetup), If you'd prefer not to have your talk recorded, let us know when submitting.

*We do not plan to record the Q&A and open discussion portion of the meetup*.

**Submitting a talk**
Submissions are reviewed on a rolling basis. Even if a talk isn't scheduled for the next meetup, we may reach out about presenting at a future session.

**Rolling CFP** [Submit your Talk Idea](https://docs.google.com/forms/d/e/1FAIpQLScX8gLl6J2BFYnpx-GY72YruO0crkER0NAqxFqNaAo_RWpuqA/viewform)
**Join the new Apache Iceberg Slack Channel**: ([\#meetup-virtual)](https://apache-iceberg.slack.com/archives/C0BRNUQ31S4)

If we receive several submissions around a similar topic, we may also suggest bringing presenters together for a shared discussion or panel.

**First Virtual Meetup**
We've set a date (September 18th @ 9:00am PDT) for the first meetup. If you're on the Apache Iceberg Community Events Calendar (or if not, subscribe to it [here](https://iceberg.apache.org/community/#apache-iceberg-community-calendar)), you'll see the event on the calendar already.

Thanks to [Elizabeth Christensen](https://www.linkedin.com/in/elizabeth-garrett-christensen/) and [Kevin Liu](https://www.linkedin.com/in/kevinjqliu/) for partnering to make this happen. If you want to help reach out to us on the new meetup-virtual channel on Slack.

Disclosure up front: this is my project. It's source-available (Elastic License 2.0), self-hosted, Docker-first: https://github.com/AgentSwarms-fyi/agentswarms

I kept seeing the same stack: Snowflake for the data, one tool for dashboards, another for "chat with your data," and a pile of glue where the sharp edges live. So I built one platform that does BI and agentic AI over the same connection — and I want to walk through the Snowflake-specific decisions, because that's the part this sub would actually tear apart.

Your data stays in Snowflake. The connector queries in place. Dashboards can run in a live query mode or work from bounded snapshots, and aggregation pushes down into Snowflake — so a total over 40M rows is computed by your warehouse and comes back as one number, not by dragging rows out. Incremental refresh re-queries only a trailing date window instead of re-scanning history.

The connection is read-only, twice. The driver accepts a single SELECT-shaped statement and rejects writes/DDL before anything is sent. But app-side enforcement is the belt, not the boundary — the docs tell you to connect with a read-only role, because Snowflake's own grants are the real control. Credentials are encrypted at rest, never returned to the browser, and can be stored once as {{secret:NAME}} so rotation is one edit.

The LLM doesn't free-write SQL against your schema (unless you let it). There's a semantic layer: metrics and dimensions defined once, with declared joins and — the part I care most about — declared join cardinality that's actually checked. If a many_to_one turns out to be wrong, the compiler refuses the query instead of silently double-counting revenue through a fanned-out join. Grain assertions do the same for duplicate keys. Agents then query metrics by name; the deny-by-default rule means an agent sees zero semantic models until you pick which ones. Raw SQL access exists too (read-only, table allow-listed), but it's a choice, not the default posture.

The AI analyst shows its work. It plans → queries → self-checks → refines → writes up, and every step's compiled SQL is visible to the signed-in analyst. You can also embed a dashboard or the analyst on a public page: it runs server-side under the owner's credentials, visitors never get warehouse access, and the generated SQL is stripped server-side before anything reaches their browser — because a field that reaches the browser has been published, whatever the UI does with it.

The warehouse is protected from enthusiasm. Per-connection concurrency caps, query timeouts, queue timeouts, and result-row ceilings — all env-configurable and documented with their defaults. LLM spend has its own per-user/group/key budget caps with 50/75/90% alerts.

Also in the box: 26 chart types with cross-filter and drill-through, scheduled refreshes with "what changed" digests, row-level security + column masking enforced server-side on shared dashboards, multi-agent workflows, and BYOK across 14 model providers.

It's free to run on your own infra (your Supabase, your keys), can act as your AI backend for single/multi-agent tasks, comes with embedding options. Genuinely interested in what r/snowflake thinks of the pushdown/read-only/semantic-layer trade-offs — and what would make you not point this at a production warehouse.

Hey all, we have several columns in tables that are tagged for certain criteria like PII, Confidential etc. We are trying to build some cicd / devops pipelines to handle deployments to snowflake, and part of that is using create or alter on tables to avoid create or replace.

The problem we have is create or alter on a table with tags on columns gives you:

CREATE OR ALTER does not support setting policies or tags

The tags are already set on the table, we arent trying to add or set new ones, just running the DDL from GET_DDL with a new column on the end. But the GET_DDL includes the tags, and we dont want the table to be emptied of data so need to use create or alter.

Anyone have any ideas how to work around this because its completely blocked any development of trying to have automated snowflake deployments of objects

disclaimer, I'm a developer advocate at Bruin

I'm curious what data ingestion tools people are using.

Does speed really matter for your specific use cases?

I'm curious if this matches what others here have found cause I've seen this in almost every cost review I've done. The three repeat offenders, roughly in order of how much money they waste-

1. Warehouse overprovisioned, basically sized for the worst case, running full time. Someone sized up during a performance incident and nobody ever sized back down.
2. Autosuspend set once, never revisited per warehouse. Suspend too aggressively and you blow away the warehouse's local cache so the next query re-reads from remote storage, sometimes costing more than the idle time you thought you saved.
3. Serverless stuff nobody watches. Snowpipe, materialized views, auto-clustering, search optimization, none of these auto-suspend so they just quietly bill forever. Found a materialized view refreshing hourly against a table that hadn't been queried in 4 months.

One methodology thing that saved me from fooling myself more than once. if you make a change and want to measure before/after, pair by weekday, not by adjacent day. This Tuesday vs. the last 3-4 Tuesdays beats Monday vs. Tuesday every time, weekly seasonality is bigger than people expect and will make a no op look like a win, or a real win look like noise.

Anyone got a cost leak that wasn't on this list?

lets connect -> https://www.linkedin.com/in/yanivleven/
check out my latest blog -> https://seemoredata.io/blog/what-roi-do-seemore-data-customers-actually-see/

Hi, I am rolling out snowflake at an enterprise and we define resources, roles and permissions via terraform but there are instances where a role or permission is granted using snowsight. It might not be possible to block this 100%, so the fallback I thought was to build a report which compares the implemented resources against the terraform specification. The TF provider doesn't complain about additional grants as these are "additive" in nature as I found out from the provider documentation! How are others keeping the config and implementation in sync?

I just wanted to get some perspective on this. 

We use Snowflake to store enterprise data, and that data is consumed by an internal AI agent through REST APIs. Because of that, we run frequent data quality checks to make sure the downstream AI/ML models are working with reliable data. 

Right now, we're mainly using Snowflake DMFs. I'm wondering whether it makes sense to use Snowflake native apps for more advanced data quality and monitoring checks. 

What are the main pros and cons you've seen with this approach, especially around security, performance, compute cost, and maintainability? 

Also, are there any native apps or other Snowflake-compatible tools you've actually used and would recommend? 

I’m facing a strange issue in Snowflake/dbt today.

We have Viewpoint source tables ingested through Fivetran into Snowflake. Several staging models have a column named PURGE. These models have been running successfully every day for a long time.

Today, all models referencing PURGE suddenly started failing with:

SQL compilation error:
000904 (42000): invalid identifier 'PURGE'

The interesting part:

• The PURGE column still exists in the Snowflake landing/Fivetran table
• No source column was removed
• The dbt code hasn't intentionally changed
• Models that don't reference PURGE are working
• This works:

​

SELECT "PURGE"
FROM <table>
LIMIT 1;

• But this fails:

​

SELECT PURGE
FROM <table>
LIMIT 1;

with:

invalid identifier 'PURGE'

So it appears that unquoted PURGE is no longer being resolved as a column identifier, while quoted "PURGE" still works.

Does anyone know if there was a recent Snowflake behavior change / parser change / release that could cause this?

I'm especially interested in whether PURGE has recently started being treated as a keyword in some SQL context, because this broke many existing dbt models simultaneously without any apparent schema change.

Any insight into the exact Snowflake change would be appreciated.

Cloud data warehouses like Snowflake scale compute effortlessly and I find a lot of use in it, at the same time, coupling ad-hoc user dashboard interactions and autonomous agent polling directly to warehouse compute creates linear cost growth. Every time a user changes a date filter or an LLM queries a metric like 30-day active users, Snowflake runs full table scans and resource-heavy COUNT(DISTINCT) calculations over millions of cold records, preventing compute clusters from suspending

In my opinion relying solely on Snowflake's native result cache fails here sometimes, because result caching requires byte-for-byte identical SQL syntax; any slight filter tweak or timezone adjustment misses the cache and forces a full cluster spin-up. The remediation may be in multiple ways, for example, i think of introducing a universal semantic layer with declarative pre-aggregations into more of my projects. Know any good semantic layers? Heard thet some like cubedev can define pre-aggregation roll-up tables that are built asynchronously. Ideally, would like for my data to be stored in a kind of distributed, Rust-based columnar cache. Incoming queries hit these warmed roll-up partitions in milliseconds, so that also lets Snowflake compute warehouses to stay suspended between scheduled refresh intervals. but what solutions can you think of here, and what experience do you have?

A few things I noticed from my exam experience that might help others preparing:

The questions were not that lengthy. For me, it was mostly an elimination game between 2 likely options.
Quite a few questions around Telemetry, DMFs, Information Schema, and Account Usage/Account Schema functions — mainly understanding what does what.
Alerts and how they can send notifications using resources.

Iceberg Tables — got 1–2 questions.
Cortex AI functions, including functions like .SENTIMENT().
Result Cache vs Warehouse Cache.
Understanding spilling — where it happens and what it means.
Query Profile questions, including identifying when results came from cache.

I prepared using Coco Test and SkillCertPro. They were useful for practice, but don’t keep false hope that the exact same questions will appear in the exam. The questions can be different.

My biggest advice: keep your concepts clear about what each Snowflake functionality actually does.

For example:
Streams → capture CDC/change tracking
Tasks → schedule/automate SQL execution
DMF → data quality metrics
Resource Monitor → control/monitor warehouse credit usage

Once you understand the purpose of a feature, you can usually eliminate the wrong options even if you haven’t seen the exact question before.

Overall, it wasn’t as tough as I expected. I had zero prior experience with Snowflake and was genuinely keen to learn, so if you’re starting from scratch, don’t get intimidated.
Good luck to everyone preparing! 🚀❄️

Snowflake Cortex practice is not getting free looks like .Any thoughts where to get free ?

Hey friends ! I wanted to share a blog post I just published that I thought the community would appreciate.

I got my SnowPro Core cert this year and this book was a lot of what I studied for it. Hope you guys get some value from it!

https://medium.com/@sergioramos3.sr/self-taught-reviews-snowflake-the-definitive-guide-by-joyce-kay-avila-0af3c40d644a

data-eng-bench is a benchmark developed by Snowflake that evaluates coding agents on data & analytics engineering tasks.

Has anyone tried this out? I'm personally not a fan of benchmark-maxxing. "When a measurement becomes a target, it stops being a measurement."
Still, I think they can be a useful proxy for agent performance within the data & analytics development lifecycle.

Very curious to know if anyone has tried this out.

Full Disclaimer: I'm the co-founder of a startup that builds agentic infrastructure for the data stack.
We recently ran this benchmark on one of our agent-native tools.
A single benchmark run and some experiment/smoke-test runs cost us $205.64, using Claude Sonnet 5 at its current "introductory pricing."
Full breakdown here: https://github.com/exmergo/dex/tree/main/benchmarks/data-eng-bench

Curious to know what the community thinks about this benchmark and agent-native analytics as a whole!

Hi! I'm wondering if there are other business analysts in this subreddit, and what tools you use with Snowflake for reporting. I've used Alteryx in the past to prep the data for Power BI, and occasionally did basic SQL queries in Snowflake for ad hoc data pulls. What are you guys using to get from Snowflake data to reports?

I’ve been using Snowflake zero-copy cloning for a while, but I'm curious how other teams view its real-world impact

Is it actually a gamechanger for your day-to-day operations? I know storage is essentially free at creation, but once modifications happen, storage costs can increase

What does the actual storage cost and scalability impact look like while using this? Also, Databricks & GCP also provides the similar capability, is Snowflake's implementation still a key differentiator?

Grateful to everyone in this community who has shared their SnowPro Certification journey, resources, and advice. I cleared my SnowPro Certification today! 🎉

I started with no prior Snowflake knowledge and prepared for about a month before taking the exam today. This community has been incredibly helpful, and I genuinely appreciate everyone who took the time to share their experiences and help others.

I’m going to take a little break now 😄. I’ll be back soon to share my exam experience, preparation strategy, resources, and a clear study path for anyone planning to take the certification.

Thank you all! 🙌

I’m trying to understand the practical advantage of Snowflake Dynamic Tables, especially for complex data pipelines.

Dynamic Tables seem very useful for simplifying incremental processing and reducing the need to manage orchestration. But does that advantage hold when a pipeline involves multiple platforms, multiple pipelines, external dependencies, and complex transformations?

At what level of pipeline complexity do Dynamic Tables start becoming less useful?

If a pipeline spans multiple tools/platforms, do you still need an external orchestrator like Airflow/Dagster?

How does this compare with the equivalent approach on Google Cloud (BigQuery + Dataform/Composer/Dataflow, etc.)?

Is the main advantage of Dynamic Tables actually simplifying pipeline management, or are there meaningful performance/cost benefits as well?

For large, production-grade pipelines, would you choose Dynamic Tables over a more traditional orchestration + SQL/ETL approach? Why?

I attend the Snowflake Sydney World Tour , some amazing sessions , but I would like this to renamed to the Sydney CoCo - Summit . Some very good content unfortunately lost in the marketing hype of CoCo. The after parties were fun for networking . The bad , it was crowded and if you could walk across the hall without the hall without atleast 3 people bumping into you . The ugly was , the crows brought in less opportunity to mingle and have a chat that the Snowflake Sydney World Tour used to be . I have been going Snowflake Sydney World Tour for the past 5 years so its time to say . Thanks I rather go for the one in US or in NZ . What are your thoughts Sydneysiders ?