4.1.1.1 Upgrading Airflow

This section provides information about upgrading Airflow

To upgrade Airflow, follow these steps:

1. Take a backup of your Airflow instance. To do so:
   1. Back up the metadata database (using tools such as mysqldump or pg_dump), along with project DAGs, configuration files, and any custom plugins.

      Example: mysqldump -u root -p airflow_db > airflow_db_1.8.9.sql

   2. Note your current Airflow version to manage the upgrade path.
2. Deactivate Any Running Airflow Services. To do so, stop all running Airflow processes, including the web server, scheduler, and workers by running the following command:
   pkill -f airflow
3. Activate Your Airflow Virtual Environment (if using virtualenv). To do so, run the following command:

   source <YOUR_AIRFLOW_HOME>/venv/bin/activate

   Note:

   Skip this step if you are not using a virtualenv.
4. Upgrade Airflow and Providers
   1. To upgrade Airflow, install the desired Airflow version by running the following command:
      pip install --upgrade "apache-airflow==<NEW_VERSION>" \ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<NEW_VERSION>/<PYTHON_VERSION>.txt"

      Note:

      • Replace <NEW_VERSION> with the target Airflow version (for example, 2.9.3).
      • Replace <PYTHON_VERSION> with the required Python version (for example, 3.9 for Python 3.9). If you are unsure, run python --version and use cut -d. -f1,2 to extract the major and minor version.
   2. To Airflow Providers, run the following command:

      pip install --upgrade apache-airflow-providers-cncf-kubernetes \ apache-airflow-providers-docker

      Note:

      For more information about additional/updated providers relevant to your DAGs, see Airflow provider documentation.
5. Upgrade Custom or Linked Packages. To do so,upgrade custom dependencies such as OpenMetadata and ingestion tools by running the following commands:
   pip install --upgrade "openmetadata-managed-apis~=1.7.7"
   pip install --upgrade "openmetadata-ingestion[all]~=1.7.7"
   pip install --upgrade "python-daemon>=3.0.0"
6. Upgrade Airflow Metadata Database. To do so, run the database migration to update the schema for the new Airflow version:

   airflow db upgrade

   or

   airflow db migrate

7. Review and Update Airflow Configuration. To do so:
   1. Review your airflow.cfg and update it to reflect any new, modified, or deprecated configuration settings as described in the Airflow version upgrade notes.
   2. If you use SQLAlchemy connection URIs or other custom configurations, adjust them as required.
8. Start or Restart Airflow Services. To do so, start the Airflow web server, scheduler, and any workers:

   airflow webserver & airflow scheduler &

   or

   airflow standalone >> ./airflow.log 2>&1 & echo $! > airflow.pid

9. Validate the Airflow Upgrade. To do so:
   1. Verify the installed Airflow version:

      airflow version

   2. Open the Airflow web UI and confirm that all DAGs load correctly.
   3. Trigger a few test DAGs to ensure proper execution.
   4. Review Airflow logs for any warnings or errors.
10. Clean Up. To do so:
    1. Remove deprecated packages or old virtual environments that are no longer needed.
    2. Monitor the system and DAG execution over the next few cycles to detect any unexpected issues.