Skip to content

gopida/e-verify-airflow-data-pipeline

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

14 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Airflow data pipeline to automate ETL

ETL-E-verified-Employer is linked as a submodule to this automation project. 

Installation (Raspberry pi setup)

1. Setup Airflow home directory
	export AIRFLOW_HOME = /path/to/my_airflow_directory

2. Install py package 
	pip install apache-airflow

3. Initialize SQLite DB
	airflow db init
	
4. Setup admin user
	airflow users create \
	  --username admin \
	  --firstname FIRST_NAME \
	  --lastname LAST_NAME \
	  --role Admin \
	  --email [email protected]
	  
5. Execute
	airflow webserver -p 8080 & airflow scheduler

About

Creating a pipeline to automate the ETL-E-Verified-Employer

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages