Skip to content

Latest commit

 

History

History
33 lines (21 loc) · 808 Bytes

README.md

File metadata and controls

33 lines (21 loc) · 808 Bytes

Airflow ETL Pipeline - Chemist

This is WIP

Progress

  • Seeding module
  • Spatial Information module: Base Spatial Information DWH
  • Transform module: DBT
  • Orchestration: Airflow
  • Container: Docker Compose

Seeding Module

The Seeding module is designed to publish a mock OLTP database, simulating business activities. Additionally, it generates flat files to emulate data provided by resellers.

Getting Started

  1. Create a virtual environment using your preferred environment manager.

  2. Install the required packages within the virtual environment:

pip install -r requirements.txt

  1. Create a .env file based on the provided example (.env.example).

  2. Seed the database and generate flatfiles by running:

python3 data-generate/initialise.py