How to deploy Airflow to Heroku

Apache Airflow

This post goes over how to deploy Apache Airflow to Heroku. See the repository.

Stack

Env

Ensure the config vars from app.json are set in your Heroku app:

AIRFLOW__CELERY__WORKER_CONCURRENCY=2
AIRFLOW__CORE__FERNET_KEY=
AIRFLOW_HOME=/app
AIRFLOW__WEBSERVER__BASE_URL=
AIRFLOW__WEBSERVER__COOKIE_SAMESITE=Strict
AIRFLOW__WEBSERVER__COOKIE_SECURE=True
AIRFLOW__WEBSERVER__SECRET_KEY=
AIRFLOW__WEBSERVER__WORKERS=2

Install cryptography:

pip3 install cryptography

Generate a AIRFLOW__CORE__FERNET_KEY:

python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

Set AIRFLOW__CORE__FERNET_KEY in your Heroku app:

heroku config:set AIRFLOW__CORE__FERNET_KEY=$(python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())")

Generate a AIRFLOW__WEBSERVER__SECRET_KEY:

openssl rand -base64 32 # head -c 32 /dev/random | base64

Set AIRFLOW__WEBSERVER__SECRET_KEY in your Heroku app:

heroku config:set AIRFLOW__WEBSERVER__SECRET_KEY=$(openssl rand -base64 32)

Database

If you’re seeing the error in your Heroku logs:

ERROR: You need to initialize the database. Please run `airflow db init`.

Then verify your Airflow config looks good before initializing the database in your Heroku app:

heroku run bash
airflow info
airflow db init

User

Create a user in your Heroku app:

heroku run bash
airflow users create -e EMAIL -f FIRSTNAME -l LASTNAME [-p PASSWORD] -r ROLE [--use-random-password] -u USERNAME

Here’s an example of how to create an admin user:

airflow users create \
--username admin \
--firstname FIRST_NAME \
--lastname LAST_NAME \
--role Admin \
--email admin@example.org

Resources

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store