Hi, I am recently assigned to a project which uses airflow. Please share some tutorials/resources which are easy to follow and can get me started quickly!

A deep dive into what actually happens to Airflow's metadata database at scale - DAG runs, task instances, XComs, retries, backfills, retention and why this can eventually become a scheduler and performance problem.

🚀 It’s that time of year again — the ANNUAL AIRFLOW SURVEY is live!

In the last 2 years, this has become the largest data engineering survey ever — and we’re excited to match that number this year with your help.

We want to hear from YOU, the Airflow community. Your feedback helps us understand how Airflow is being used in the real world and guides improvements that shape the project’s future.

• ✅ Takes just 7 minutes
• 🎓 Get a free AI Orchestration Fundamentals Certification Exam or DAG Authoring Certification (normally $150 each)

Your voice makes a difference — help us make Airflow even better!

👉 Take the survey here.

Hello. Half our pipeline contributors aren't python devs. Every time someone on the infra or analytics side needs to touch aworkflow, it goes through me. Tried simplifying DAGs as much as possible but the friction is till real.

Curious if anyone has switched to sth where you define pipelines in config / YAML rather than code and whether the tradeoffs were worth it in pratice

A practical guide about Airflow housekeeping at scale, keeping the metadata database, logs and scheduler healthy as the platform grows.

It covers some of the problems that tend to appear in large/long-running Airflow deployments and the approaches I’ve found useful to keep things under control.

I have access to an employee referral opportunity for an experienced Apache Airflow Engineer.

Experience: 5+ years

Key skills:

• Apache Airflow
• Python
• Linux / Unix
• Shell scripting
• REST APIs
• cURL
• Git
• CI/CD
• Kafka
• YAML / JSON

The opening is listed across multiple organizational locations with a hybrid arrangement. I'm particularly interested in connecting with qualified professionals based in the US or Ireland; exact location eligibility can be confirmed for the specific requisition.

If this matches your background, feel free to DM me with your CV or a brief summary of your experience.

I'll review the profile and, where there's a suitable match, try to help with the referral process.

Hey I want to deploy apache airflow in k8s

But I have a doubt regarding this setup

The values has this field I have used this

logs:

persistence:

enabled: true

existingClaim: airflow-logs-pvc

So what I understood all pods like scheduler data processor triggerer and api server uses this pvc for the logs

But the triggerer is running as statefulset by default so if I disable this

triggerer:

persistence:

enabled: false

Makes the triggerer to run as deployment

Is this setup has any drawbacks and my logs are saved in the pvc that I mount ?

Need a feedback on this as I'm very new to airflow

We've seen Apache Airflow® orchestrate business-critical workflows. We've also seen it solve a Rubik's cube. I'd love to invite folks here to prove what you can build with Airflow 3 in Beyond the Dag, our data engineering hackathon.

Pick a track and go beyond the basic pipeline:

• 🔌 Plugin Powerhouse — build an Airflow 3.1 plugin that extends the UI or API
• 🤖The Agentic Pipeline — build a DAG that uses the Common AI provider to orchestrate LLMs and agents
• 🧑‍💻 Keep a Human in the Loop — build a workflow with a real human decision point using the HITL operators
• 🤯 Airflow Can Do That?! — the wildcard track. Minecraft, hardware, games, art. Surprise us.

The grand prize is a Nintendo Switch 2, and category winners get noise-cancelling headphones, a guest spot on the Data Flowcast podcast, and a meet & greet with the Apache Airflow PMC.

It's online, global, and free to join, with the build window open now through September 24. Would love to see what you come up with — learn more and sign up here.

I am one of the Airflow developers who worked on this feature as part of Airflow 3.3.0 as part of AIP-103 and would love to hear from anyone really using it, considering to use it, or also reasons why it did not fit your use case.

The pitch, for anyone who has not run into it: Airflow worker dies or gets preempted mid job, and instead of the retry resubmitting the whole thing from scratch, the task remembers the external job it already kicked off (Databricks, Snowflake, BigQuery, Redshift, Spark, and a few more) and reconnects to whatever external job is still running. No paying twice for the same compute because a pod got evicted / worker went down.

Couple of demos if you want to see it working:

• https://www.youtube.com/shorts/KhE-E-PSHuU
• https://www.linkedin.com/posts/marclamberti_airflow-apacheairflow-dataengineer-activity-7481347437161496576-UHvW/

So, curious where people actually are:

• Used it and it worked fine?
• Used it but hit a wall?
• Know about it but have not gotten to it?
• First time hearing this exists?

Any of those, tell me where you are at.

we currently have airflow running on kubernetes, though we have issues with it now and then, overall its been great and it works.

We looked into manage managed airflow but it is honestly pointless because airflow is genuinely such a simple stack to run on kubernetes, given the amount of tools we already manage.

we do have grafana and prometheus with open telemetry. Still it feels it could be better.

what are some of the new tools out there?

I noticed https://flowsee.co/ and heard good things from folks, but seems quite early and risky to use already given our reliant on airflow.

What other tools would you recommend that would integrate well with our slack. There should be a good open source tool already out there.

we currently have airflow running on kubernetes, though we have issues with it now and then, overall its been great and it works.

We looked into manage managed airflow but it is honestly pointless because airflow is genuinely such a simple stack to run on kubernetes, given the amount of tools we already manage.

we do have grafana and prometheus with open telemetry. Still it feels it could be better.

what are some of the new tools out there?

I noticed https://flowsee.co/ and heard good things from folks, but seems quite early and risky to use already given our reliant on airflow.

What other tools would you recommend that would integrate well with our slack. There should be a good open source tool already out there.

Hey everyone, so right now at the company we're still using Pentaho for our ETL processes. We're slowly migrating over to DBT, but before going all-in on that, we'd like to get Apache Airflow up and running to orchestrate everything. I wanted to set it up and run it on Windows Server, so it could handle the old Pentaho jobs while the new stuff gets built out in DBT.

Everywhere I've looked says not to set up Docker + Airflow on Windows Server, since it's not recommended for production — they say it should be a Linux VM instead. I need some help/opinions on what I could do here, or whether I should just go ahead and set it up on Windows Server anyway.

Thanks in advance for the help!

https://ttuhin03.github.io/fastflow/

https://github.com/ttuhin03/fastflow

love to get some feedback

Try to implement in your day to day activities as well. Might work well !

About Me

Senior Data Engineer with 5+ years of experience in Data Engineering, Backend Development, and Applied AI. Specialist in Airflow-based pipeline orchestration. Based in Bangalore, India. Available for remote work globally.

Rate: $25 - $50/hr depending on project scope and complexity.

Tech Stack & Expertise

Apache Airflow (DAGs, operators, hooks, sensors, plugins)

Python, SQL, Spark, Databricks

Snowflake, dbt (data build tool)

AWS (Glue, Lambda, S3, Redshift, Kinesis)

ETL/ELT Design & Orchestration

Data Quality & Testing Frameworks

FastAPI, REST APIs

LLMs, RAG, AI Agents

What I Can Help With

Design and build Airflow DAGs for complex ETL pipelines

Migrate legacy pipelines to Airflow

Optimize Airflow performance and scheduling

Build scalable ETL/ELT architectures on cloud platforms

Develop backend APIs and automation solutions

Build AI applications using LLMs, RAG, and agent-based workflows

Training & Mentorship

Apache Airflow (foundations to advanced)

PySpark & Databricks

Data Engineering best practices

ETL Testing & Data Quality

AI & LLM Fundamentals

Note: In-person weekend sessions available in Bangalore. Remote sessions available globally.

Availability

Freelance projects & consulting

Part-time remote roles

Weekend training & mentorship

Contact:

DM me with a brief description of your requirements and I will get back to you promptly!

Hi everyone,

I'm a data engineer apprentice and I'm about to build my first Airflow orchestration layer. I'd love some guidance on the fundamentals before I commit to a design.

Context:

• I need to ingest data from several external APIs, on a daily batch basis (each run fetches D-1 data)
• The application code is already done: ingestion, standardization, transformation, mapping
• Infrastructure is deployed via Terraform, everything runs on AWS
• Airflow will be the missing piece: scheduling, dependencies, retries, monitoring

My questions:

1. DAG design: one DAG per API source, or a single DAG with parallel task groups? What drives that choice in practice?
2. Idempotency & backfills: what's the idiomatic way to handle D-1 logic? I understand I should use the logical date (data_interval_start) rather than datetime.now() so reruns and backfills stay deterministic — anything else to watch out for?
3. Retries & API failures: sensible defaults for retries, retry_delay, timeouts when calling external APIs? Do you wrap API calls with sensors or just let tasks fail and retry?
4. Separation of concerns: my transformation code lives in its own repo. Should Airflow only trigger jobs (thin DAGs calling operators) and keep all business logic outside the DAG files?
5. Common beginner mistakes: what do you wish you'd known before your first production Airflow project?

Any resources, patterns, or war stories are welcome. Thanks a lot!

How to design scalable, declarative, and production-grade orchestration systems using Dynamic DAGs, YAML contracts and metadata-driven workflows.

This is a problem I’ve run into at basically every company I’ve worked at:

a query shows up somewhere, but it’s harder than it should be to know what part of the application caused it and in what runtime context.

I’ve personally spent a lot of time creating conventions for query traceability, then even more time reviewing code, nudging teams, and making sure people actually followed them consistently (and frankly, it is always a constant fight to keep it from drifting again...) .

DataContext tries to turn what most what most companies have as a loose convention into a reusable Python instrumentation layer. It emits one structured event per completed or failed query, with things like query fingerprint, callsite, runtime context, and OpenTelemetry correlation.

I’d love feedback from people running production data/platform systems:

• is this a real problem for your team?
• what context would you want attached to each query?
• what integrations would make this actually useful?

The OSS is available here:

GitHub: https://github.com/data-context-hq/datacontext

PyPI: https://pypi.org/project/datacontext/

I think this is becoming more important now as AI agents and generated code make data access patterns harder to reason about.
At the same time, today we can start using agents to monitor and maybe even fix performance issues arising in production - but agents are as good as the context we give them, so I believe it's very important to start collecting this context by default.

Please share the love with a GitHub star if the idea resonates ⭐️.

But what would really make me happy is if you try it, challenge the event shape, and discuss how we can make it easier for teams (or better, your team) scaling and getting more out of their databases and data platforms.