Skip to content
This repository was archived by the owner on Feb 28, 2025. It is now read-only.

Commit 64a8247

Browse files
avaidyanathaAbhi Vaidyanatha
avaidyanatha
and
Abhi Vaidyanatha
authored
Add Airflow demo script for one click deployment (airbytehq#4451)
* Add Airflow Demo Example * Finalize script instructions * Add down.sh and run gradle format * Note about down.sh * Address review comments Co-authored-by: Abhi Vaidyanatha <[email protected]>
1 parent 9708e4c commit 64a8247

12 files changed

+243
-0
lines changed

‎.gitignore

+4
Original file line numberDiff line numberDiff line change
@@ -32,3 +32,7 @@ crash.log
3232
*.tfstate
3333
*.tfstate.backup
3434
*.lock.hcl
35+
36+
# Airflow Demo
37+
resources/examples/airflow/logs/*
38+
!resources/examples/airflow/logs/.gitkeep

‎resources/examples/airflow/.env

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
AIRFLOW_UID=501
2+
AIRFLOW_GID=0

‎resources/examples/airflow/README.md

+17
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Airflow Airbyte Plugin Demo Example
2+
3+
## Running the Script
4+
Simply run `up.sh` to bring up the Airbyte and Airflow containers. You don't need to worry about cleaning up the containers, as we run a `docker-compose down -v` before attempting to recreate them.
5+
6+
After Airbyte starts running, head over to http://localhost:8000 and set up a connection. The script will then prompt you for an Airbyte connection id. You can find this in the URL after clicking on your connection.
7+
8+
![](./assets/airbyte_connection_id.png)
9+
10+
Simple enter that ID into your terminal and a connection will be set up in Airflow for you. Head over to http://localhost:8085 to access the Airflow UI. Enter `airflow` as your username and `airflow` as your password. You should see your DAG show up in the UI with no errors if your connection was configured correctly.
11+
12+
![](./assets/airflow_start_dag.png)
13+
14+
Trigger the DAG with the switch in the top right and you should be in business! If it doesn't automatically run, just hit the play button in the top right to kick off the sync.
15+
16+
## Cleaning Up
17+
Run `down.sh` to clean up the containers. Or run `docker-compose down -v` here and in the root directory, your call.

‎resources/examples/airflow/airflow.sh

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
#!/usr/bin/env bash
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing,
13+
# software distributed under the License is distributed on an
14+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
# KIND, either express or implied. See the License for the
16+
# specific language governing permissions and limitations
17+
# under the License.
18+
19+
#
20+
# Run airflow command in container
21+
#
22+
23+
PROJECT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
24+
25+
set -euo pipefail
26+
27+
export COMPOSE_FILE="${PROJECT_DIR}/docker-compose.yaml"
28+
exec docker-compose run --rm -e CONNECTION_CHECK_MAX_COUNT=0 airflow-worker "${@}"

‎resources/examples/airflow/assets/airbyte_connection_id.png

82.9 KB
Loading

‎resources/examples/airflow/assets/airflow_start_dag.png

21.7 KB
Loading

‎resources/examples/airflow/dags/dag_airbyte_example.py

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
from airflow import DAG
2+
from airflow.utils.dates import days_ago
3+
from airflow.providers.airbyte.operators.airbyte import AirbyteTriggerSyncOperator
4+
from airflow.models import Variable
5+
6+
airbyte_connection_id = Variable.get("AIRBYTE_CONNECTION_ID")
7+
8+
with DAG(dag_id='trigger_airbyte_job_example',
9+
default_args={'owner': 'airflow'},
10+
schedule_interval='@daily',
11+
start_date=days_ago(1)
12+
) as dag:
13+
14+
example_sync = AirbyteTriggerSyncOperator(
15+
task_id='airbyte_example',
16+
airbyte_conn_id='airbyte_example',
17+
connection_id=airbyte_connection_id,
18+
asynchronous=False,
19+
timeout=3600,
20+
wait_seconds=3
21+
)

‎resources/examples/airflow/docker-compose.yaml

+151
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
#
18+
19+
# Basic Airflow cluster configuration for CeleryExecutor with Redis and PostgreSQL.
20+
#
21+
# WARNING: This configuration is for local development. Do not use it in a production deployment.
22+
#
23+
# This configuration supports basic configuration using environment variables or an .env file
24+
# The following variables are supported:
25+
#
26+
# AIRFLOW_IMAGE_NAME - Docker image name used to run Airflow.
27+
# Default: apache/airflow:master-python3.8
28+
# AIRFLOW_UID - User ID in Airflow containers
29+
# Default: 50000
30+
# AIRFLOW_GID - Group ID in Airflow containers
31+
# Default: 50000
32+
# _AIRFLOW_WWW_USER_USERNAME - Username for the administrator account.
33+
# Default: airflow
34+
# _AIRFLOW_WWW_USER_PASSWORD - Password for the administrator account.
35+
# Default: airflow
36+
#
37+
# Feel free to modify this file to suit your needs.
38+
---
39+
version: "3"
40+
x-airflow-common: &airflow-common
41+
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.0}
42+
environment: &airflow-common-env
43+
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
44+
AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
45+
AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
46+
AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
47+
AIRFLOW__CORE__FERNET_KEY: ""
48+
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: "true"
49+
AIRFLOW__CORE__LOAD_EXAMPLES: "false"
50+
AIRFLOW__API__AUTH_BACKEND: "airflow.api.auth.backend.basic_auth"
51+
volumes:
52+
- ./dags:/opt/airflow/dags
53+
- ./logs:/opt/airflow/logs
54+
- ./plugins:/opt/airflow/plugins
55+
- ./requirements.txt:/opt/airflow/requirements.txt
56+
user: "${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}"
57+
depends_on:
58+
redis:
59+
condition: service_healthy
60+
postgres:
61+
condition: service_healthy
62+
63+
services:
64+
postgres:
65+
image: postgres:13
66+
environment:
67+
POSTGRES_USER: airflow
68+
POSTGRES_PASSWORD: airflow
69+
POSTGRES_DB: airflow
70+
volumes:
71+
- postgres-db-volume:/var/lib/postgresql/data
72+
healthcheck:
73+
test: ["CMD", "pg_isready", "-U", "airflow"]
74+
interval: 5s
75+
retries: 5
76+
restart: always
77+
78+
redis:
79+
image: redis:latest
80+
ports:
81+
- 6379:6379
82+
healthcheck:
83+
test: ["CMD", "redis-cli", "ping"]
84+
interval: 5s
85+
timeout: 30s
86+
retries: 50
87+
restart: always
88+
89+
airflow-webserver:
90+
<<: *airflow-common
91+
container_name: "airflow_webserver"
92+
command: 'bash -c "pip3 install -r requirements.txt && airflow webserver"'
93+
ports:
94+
- 8085:8080
95+
healthcheck:
96+
test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
97+
interval: 10s
98+
timeout: 10s
99+
retries: 5
100+
restart: always
101+
102+
airflow-scheduler:
103+
<<: *airflow-common
104+
command: 'bash -c "pip3 install -r requirements.txt && airflow scheduler"'
105+
healthcheck:
106+
test:
107+
[
108+
"CMD-SHELL",
109+
'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"',
110+
]
111+
interval: 10s
112+
timeout: 10s
113+
retries: 5
114+
restart: always
115+
116+
airflow-worker:
117+
<<: *airflow-common
118+
command: 'bash -c "pip3 install -r requirements.txt && airflow celery worker"'
119+
healthcheck:
120+
test:
121+
- "CMD-SHELL"
122+
- 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
123+
interval: 10s
124+
timeout: 10s
125+
retries: 5
126+
restart: always
127+
128+
airflow-init:
129+
<<: *airflow-common
130+
command: version
131+
environment:
132+
<<: *airflow-common-env
133+
_AIRFLOW_DB_UPGRADE: "true"
134+
_AIRFLOW_WWW_USER_CREATE: "true"
135+
_AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
136+
_AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
137+
138+
flower:
139+
<<: *airflow-common
140+
command: 'bash -c "pip3 install -r requirements.txt && airflow celery flower"'
141+
ports:
142+
- 5555:5555
143+
healthcheck:
144+
test: ["CMD", "curl", "--fail", "http://localhost:5555/"]
145+
interval: 10s
146+
timeout: 10s
147+
retries: 5
148+
restart: always
149+
150+
volumes:
151+
postgres-db-volume:

‎resources/examples/airflow/down.sh

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
#!/usr/bin/env bash
2+
cd ../../..
3+
docker-compose down -v
4+
cd resources/examples/airflow || exit
5+
docker-compose down -v

‎resources/examples/airflow/logs/.gitkeep

Whitespace-only changes.

‎resources/examples/airflow/requirements.txt

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
apache-airflow-providers-http
2+
apache-airflow-providers-airbyte

‎resources/examples/airflow/up.sh

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
#!/usr/bin/env bash
2+
cd ../../..
3+
docker-compose down -v
4+
docker-compose up -d
5+
cd resources/examples/airflow || exit
6+
docker-compose down -v
7+
docker-compose up -d
8+
echo "Access Airbyte at http://localhost:8000 and set up a connection."
9+
echo "Enter your Airbyte connection ID: "
10+
read connection_id
11+
docker exec -ti airflow_webserver airflow variables set 'AIRBYTE_CONNECTION_ID' "$connection_id"
12+
docker exec -ti airflow_webserver airflow connections add 'airbyte_example' --conn-uri 'airbyte://host.docker.internal:8000'
13+
echo "Access Airflow at http://localhost:8085 to kick off your Airbyte sync DAG."

0 commit comments

Comments
 (0)