Skip to content

Commit

Permalink
chore: folder skeleton for data preserved for future reference
Browse files Browse the repository at this point in the history
  • Loading branch information
mr.hasan committed Feb 23, 2025
1 parent e7e8573 commit 81564f4
Show file tree
Hide file tree
Showing 6 changed files with 4 additions and 3 deletions.
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -31,5 +31,6 @@ ENV/
config/config.yaml
logs/

/data/*
data/*
!data/**/.gitkeep
!/data/README.md
4 changes: 2 additions & 2 deletions data/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ This directory contains pre-collected data used for testing and development purp
## Structure

- `raw/`: Contains raw data files as collected from the source. There are two folders here
- `metafile/`: The schema for this `csv` file collection is filepath, start_date, end_date. Where the filepath
- `metafiles/`: The schema for this `csv` file collection is filepath, start_date, end_date. Where the filepath
is the location of the logs for the failed tasks. start_date and end_date are also the metainformation of the failed tasks.
- `logfile/`: The actual log files contents. As airflow stores logs in configured file location, we've collect log from those files.
- `logfiles/`: The actual log files contents. As airflow stores logs in configured file location, we've collect log from those files.
- `processed/`: Contains processed data files that have been cleaned or transformed for use in the project. Here resides the files after bieng
extracted and transformed for categorization.

Expand Down
Empty file added data/processed/.gitkeep
Empty file.
Empty file added data/raw/.gitkeep
Empty file.
Empty file added data/raw/logfiles/.gitkeep
Empty file.
Empty file added data/raw/metafiles/.gitkeep
Empty file.

0 comments on commit 81564f4

Please sign in to comment.