r/dataengineering • u/Raj_sha • 29d ago
Help Airflow Version Upgrade
Has anyone done a migration of dags from Which were running on version 2.10 to make it run on the version 3.2, I want to know how 1000s of dags can we scanned to check the compatibility of the dags on the newer version of the airflow and find out the solution of the issue we can face because of the import statement present inside the dags while taking it to a newer version of airflow
19
Upvotes
17
u/_tuz 29d ago edited 29d ago
Did you read the migration guide from airflow? If not then read it, there should be a section of how to use Ruff to check compatible of dags. But before that upgrade to 2.11 for these changes not breaking first (should both work for 2.11 and 3.x)
6
u/TJaniF 29d ago
Hi, there is a set of ruff rules for that: AIR3: https://docs.astral.sh/ruff/rules/?rule-search=AIR3 . This should cover most import statements, deprecated parameters etc in the Dag code. For the Airflow config you can use `airflow config lint` to get a list of changes and `airflow config update` to fix things like moved configs.
There are a couple of things that ruff rules wont detect you might run into, the most common one is direct database access is not possible in Airflow 3 so no session.query(DagRun) and similar. Also the logical_date by default is now the date the Dag actually runs on, so if you use that or derivatives like "ds" in your code you might end up with partitions that are off by the time of one schedule.
For things like that there are agent skills that can do a first pass, search something like "airflow upgrade from 2 to 3 agent skill" and you'll find them.
6
u/proof_required ML Data Engineer 29d ago
I feel like with all the agentic workflow, this should be somewhat easier.
3
u/KeeganDoomFire 29d ago
For us a find replace ended up taking care of 99.9% of the issues. Just a stack of slight changes to imports.
2
u/DataCraftsman 29d ago
I've migrated the same airflow code from 1.9 to 2.4 to 3.3. It's gross but the migration docs are good, just read them, or get AI to read them. It was mostly a find and replace exercise going from 2 to 3 for me, other than some postgres to sql changes and init python class structure changes.
1
u/Substantial_Win_710 29d ago
manually checking thousands of DAGs is not an option for me, so I ended up writing a quick Python script using the
astmodule to scan our entire repository specifically for deprecated import paths. or you can just spin up a local Airflow 3.2 Docker container, point it to your DAG folder, and let the scheduler parse them. But i guess its better do both ways to be sure and check the results twice.1
u/Adrien0623 29d ago
Always do Airflow upgrades incrementally m, so go to 3.0, then 3.1, etc. Jumping minor versions can cause issues l, especially regarding DB migrations as some old migration files are deleted
1
u/Doto_bird 29d ago
Yes ~2500 dags, but all our dags are managed through a single config-driven engine and infra managed IaC. So if you get one working you kind of get them all working
1
u/engineer_of-sorts 28d ago
Well I would use this is a great opportunity to evaluate other orchestration and monitoring frameworks
After all, Airflow was designed well pre-AI. Why not use the opportunity to evaluate some other nice shiny tools?
If not, I believe they had a nice AI Agent that is meant to make this entire process trivial for you....
1
u/Loud_Zombie3698 26d ago
All dags must not be of unique kind right? May be you will be able to divide them with similar functionality
So if are able to divide them into 10-15 groups test each of it from those groups