The data half of the final project. It runs end to end as it stands, against a public job board, on your team's own infrastructure. Nothing here is a stub waiting to be filled in.
That is deliberate. You learn more from a pipeline that works and has to be changed than from a set of empty functions: you can run it on day one, see real rows arrive, break something, and watch which test catches it. Your job is to make it yours, which is a different and more interesting problem than making it exist.
Team-a walkthrough. Examples in this folder use team-a's cloud resources (
sthyffpteama,team_a,rg-hyf-fp-team-a). If you are on team-b, team-c, or team-d, keep the same commands and replace those names with yours — see Setup for the mapping. Personal settings (LANDING_PREFIX,DBT_SCHEMA, your Databricks token) stay yours on every team.
flowchart LR
SRC["Source API or open data"]
subgraph az["Azure"]
ING["ACA job: ingestion container, image tagged by SHA"]
subgraph dbx["Databricks (Unity Catalog)"]
LAND[("landing zone, ADLS: /Volumes/catalog/landing/prod")]
WH["SQL warehouse, 2X-Small"]
MODELS["staging, then marts, plus dbt tests"]
ENRM["fct_postings_enriched: dbt Python model on serverless"]
PYM["LLM classifier: dbt Python model"]
OPSTBL[("ops.dbt_test_runs")]
end
AF["Airflow, one VM per team: ingest, dbt build, publish"]
end
subgraph be["Backend"]
PG[("Postgres: app schema theirs, analytics schema yours")]
end
SRC -->|"fetch"| ING
ING -->|"raw JSON"| LAND
LAND --> WH --> MODELS
MODELS --> ENRM
MODELS -.-> PYM
WH -.-> OPSTBL
ENRM --> PUB["publish: write, then swap"]
PUB -->|"outbound sync"| PG
PG -.->|"inbound sync"| MODELS
AF -.->|"trigger and wait"| ING
AF -.->|"dbt build"| WH
AF -.-> PUB
classDef opt stroke-dasharray:5 4
classDef orch fill:#dff3e0,stroke:#4a8055
classDef store fill:#dceaf7,stroke:#4a6080
class PYM,OPSTBL opt
class AF,PUB orch
class PG store
This is the Week 15 course architecture drawn for one team rather than all three. Dashed nodes are optional: the project is complete without them.
Everything around that pipeline, the parts you do not build yourself:
flowchart LR
REPO["team monorepo: data/src, data/dbt, data/airflow/dags"]
GHA["GitHub Actions and ACR, no stored secret"]
KV["Key Vault, one secret per team"]
ING2["ACA jobs"]
AF2["Airflow on your team VM"]
WH2["SQL warehouse"]
PG2[("Postgres, analytics schema")]
subgraph opsz["Operations"]
LOGS["Log Analytics: container job logs"]
SLACK["Slack: failure alerts"]
HEALTH["Streamlit health page: ACA app, public URL"]
end
REPO -->|"build"| GHA -->|"image tagged by commit"| ING2
REPO -->|"dags and dbt project"| AF2
KV -->|"the VM identity reads only its own secrets"| AF2
ING2 -->|"container stdout"| LOGS
AF2 -->|"on failure"| SLACK
WH2 -->|"rows and freshness"| HEALTH
PG2 -->|"published rows"| HEALTH
classDef opt stroke-dasharray:5 4
classDef code fill:#e8e2f7,stroke:#6b5b95
classDef sec fill:#fdebd2,stroke:#b07d3a
classDef orch fill:#dff3e0,stroke:#4a8055
classDef store fill:#dceaf7,stroke:#4a6080
class HEALTH opt
class REPO,GHA code
class KV sec
class AF2 orch
class PG2 store
The Databricks box is the part of your work nobody outside the data track ever sees. Staging, intermediate models and every model you build while figuring the data out live in your catalog. Only the marts cross into Postgres, and only through the sync task in Airflow.
Three things in that picture are provided for you rather than built by you. CI builds the image and pushes it to your registry with no credential stored anywhere, Key Vault holds the one secret your Airflow VM may read, and the Slack alert is already wired to every task.
Every team runs this shape: ingestion in a container, raw files in your team's landing zone, dbt building and testing models in your team's catalog, and Airflow publishing the finished mart into the database the backend reads.
Enrichment is a dbt model, not a separate job. The work that SQL expresses
badly, here deciding which discipline a job title belongs to, lives in
dbt/models/marts/fct_postings_enriched.py, a dbt Python model running on
Databricks serverless. It is a node in the graph like any .sql file, so
dbt build runs it in order after fct_postings, ref() works, and its
output is tested in _fct_postings_enriched.yml. There is no cluster to start
and no fourth task to trigger and wait for.
There is a second, optional Python model beside it. fct_title_discipline.py
does the same classification with an LLM instead of a dictionary, and ships
with enabled: false because it needs an API key first. See
optional/README.md for the key and the daily request
limit.
Your raw files live in your team's own storage account, in a container called
prod. That same container is registered in Unity Catalog as a volume, so
the file the container writes as
prod/raw/postings/ingest_date=2026-08-12/data.json is the file dbt reads
at /Volumes/team_a/landing/prod/postings/. One copy of the
bytes, two ways to reach it: Azure tooling on one side, SQL on the other.
The two tracks meet in the backend's database, which has one schema per side.
You write marts into analytics, which the backend reads. The backend writes
app, which you can read. Neither side can write to the other's, and that is
two Postgres roles rather than an agreement.
Everything below works. The "change this" column says what a team normally edits, not what is missing.
The Python is grouped by pipeline stage, so the folders match the boxes in the
diagram above. common/ is what more than one stage needs.
| Path | What it does | Change this? |
|---|---|---|
src/ingestion/models.py |
A Pydantic model for job postings | Yes: your source's shape |
src/ingestion/ingest.py |
Calls the API, validates, counts rejects | The parsing, if your source is nested |
src/ingestion/storage.py |
Lands raw JSON in your team's landing zone | Rarely |
src/ingestion/pipeline.py |
The ingestion job's entrypoint: settings, fetch, validate, land | Rarely |
src/publishing/sync.py |
Publishes the mart into the backend's database, atomically | Rarely |
src/common/warehouse.py |
Runs SQL against your warehouse over HTTP | No |
src/common/aca.py |
Starts a container job and waits for it | No |
dbt/models/ |
Staging reads the volume, the mart is the contract | Yes: your domain |
dbt/models/marts/fct_postings_enriched.py |
The enrichment model: classifies each posting in Python, on serverless | Yes: this is your domain logic |
dbt/models/marts/fct_title_discipline.py |
Optional: the same classification with an LLM, shipped disabled | Only if you turn it on |
dbt/tests/ |
Two custom tests, including a zero-row check | Add your own |
tests/ |
Unit tests, in folders mirroring src/. No credentials, under a second |
Add as you build |
airflow/dags/pipeline_dag.py |
The three tasks, wired in order | Only to add a step |
airflow/dags/alerts.py |
Posts to Slack when any task fails | No |
Dockerfile |
The image the ingestion job runs | Rarely |
optional/ |
A health dashboard and a dbt-results recorder. Neither required | As you like |
Most of this was done for you when your repository was created. What is already in place:
- your team's storage account, Databricks catalog, SQL warehouse and secret scope, and a Container Apps job to run your image
- your Airflow instance, already pulling this repository every minute, with the values it needs already set
- CI that builds the pipeline image and pushes it to your team's registry on
every merge to
main, with no credential stored anywhere - a Slack channel that receives an alert whenever a task fails
What you do once, on your own machine:
1. Copy .env and adjust for your team. data/.env.example is prefilled
with team-a as the worked example. If that is your team, you only need your
personal values (prefix, dev schema, Databricks token, Postgres password from
Key Vault). On another team, swap the four team identifiers:
| If you are on | Storage account | Catalog | Resource group | Registry |
|---|---|---|---|---|
| team-a | sthyffpteama |
team_a |
rg-hyf-fp-team-a |
acrhyffpa |
| team-b | sthyffpteamb |
team_b |
rg-hyf-fp-team-b |
acrhyffpb |
| team-c | sthyffpteamc |
team_c |
rg-hyf-fp-team-c |
acrhyffpc |
| team-d | sthyffpteamd |
team_d |
rg-hyf-fp-team-d |
acrhyffpd |
The full dev walkthrough is in docs/dev_flow.md.
2. Generate your Databricks token. The same one you made in Week 13, and
for the same reason: it authenticates dbt as you. Your name in the top bar,
Settings, Developer, Access tokens, Generate new token. Give it
a comment like final-project and copy the value once, because Databricks
will not show it again.
It is yours, not your team's. Paste it only into your local .env, never into
Slack, a pull request or an LLM prompt.
3. Create your local database. The backend's own script makes the
project_db database, the app and analytics schemas, and a login role for
each. Start the database first, from the repository root:
cd <repository root>
cp .env.example .env # the root one: Postgres container settings
docker compose up -d db
pip install "psycopg[binary]"
python scripts/db-setup.py --host localhost --port 5432 \
--admin-user admin --admin-password passwordCopy the analytics_user password it prints. It is shown once, and it is the
one your pipeline publishes with.
⚠️ If this fails withrole "admin" does not exist, something else on your machine already owns port 5432, usually a Postgres you installed yourself. Stop it, or give the container a different port, and use that everywhere.
4. Fill in .env.
cd data
cp .env.example .env # a different file: data/.env.example, the pipeline'sAdd your token, your LANDING_PREFIX and DBT_SCHEMA, and the Postgres
secrets from Key Vault (commands are in .env.example; team-a names are shown
— replace team-a with your team letter if needed). Step 3 is only for the
optional local Postgres container at the repository root.
5. Sign in to Azure. The pipeline authenticates as you locally, and as its managed identity in Azure. Same code, no secret either way.
az login6. Check you can reach your landing zone. Your teacher grants each team
member two roles, one per container: Storage Blob Data Contributor on dev,
so your own runs can write there, and Storage Blob Data Reader on prod,
so you can read what the scheduled pipeline wrote without being able to
overwrite it. Owner or Contributor on the resource group is not enough:
Azure separates managing a storage account from reading what is inside it, and
this trips up nearly everyone the first time.
# Team-a example — replace sthyffpteama with your storage account if needed.
az storage blob list --account-name sthyffpteama \
--container-name prod --auth-mode login -o tableAn AuthorizationPermissionMismatch here means the role is missing or has not
propagated yet. It can take a few minutes after it is granted.
7. Install and run.
uv sync --all-extras
uv run python -m src.ingestion.pipeline--all-extras, not one extra at a time: uv sync makes the environment match
exactly what you asked for, so a later uv sync --extra dev would uninstall
dbt again.
That fetches the default source and lands one file. Check it arrived, from the Databricks SQL editor:
SELECT count(*) FROM read_files('/Volumes/team_a/landing/prod/postings',
format => 'json');8. Point dbt at your landing zone, then build.
dbt reads LANDING_PATH from your .env, the same file your ingestion wrote
to, so the two cannot disagree. Step 7 printed the exact value to use as its
last line. Then:
cd dbt
uv run --env-file ../.env dbt show --inline "select session_user() as connected_as"
uv run --env-file ../.env dbt buildsession_user() must show your own email. If it shows somebody else's, you
pasted the wrong token.
--env-file matters: dbt/profiles.yml reads every value from the
environment, and uv run does not pick up .env on its own.
💡
dbt buildruns as you, through your own token. Airflow passes--target prodinstead, which uses the team's service principal. Same project, same models, two identities: yours can builddev_yournameand cannot touchanalytics, and the service principal is the only thing that publishes.
When staging reads your own file, you have an end to end path, and everything after that is shaping.
9. Run the tests. They need no credentials and no network, so they are the one thing you can run before anything else works.
uv run pytestThey run in under a second. They cover the parts that are painful to test any other way: what happens to a malformed record, whether the job poll loop notices a failed container, and whether the publish swaps its tables in an order that never leaves the backend looking at a missing one. Add to them as you go, and CI runs them on every pull request.
Everything except the two container jobs runs on your machine, against your
own copy of the data. Five settings in .env decide where that copy lives,
and they are the first thing to fill in:
| Setting | Yours | The scheduled run |
|---|---|---|
LANDING_CONTAINER |
dev |
prod |
LANDING_PREFIX |
your-name |
raw |
LANDING_PATH |
/Volumes/<catalog>/landing/dev/your-name/postings |
.../landing/prod/postings |
DBT_SCHEMA |
dev_yourname |
analytics |
BACKEND_PG_PUBLISH_SCHEMA |
analytics_dev |
analytics |
BACKEND_PG_USER |
analytics_dev_user |
analytics_user |
This is not a naming convention you have to remember. It is what your account
is allowed to do. You can write the dev container and only read prod.
You can create and own dev_ schemas and only read analytics. Point your
.env at the production names by mistake and the run stops with a permission
error, which is a much better afternoon than discovering at the demo that your
test data went out to the backend.
When you point the pipeline at a new API, the first question is what it actually returns, and you do not need a cloud account to answer it:
uv run python -m src.ingestion.pipeline --local # writes local-landing/It fetches and validates exactly as a real run does, then writes the file to
your own disk instead of the landing zone, in the same newline-delimited format
dbt would read. Open it, work out which fields matter, and write the renames in
stg_*.sql against something you have seen rather than something you assume.
--local takes an optional directory if you want it somewhere else, and it is
the one mode that needs no STORAGE_ACCOUNT.
This is a look, not a stage. The SQL warehouse cannot read your laptop, so there is no step where you land locally and then upload it. When the shape looks right, drop the flag and the same command writes the
devcontainer.
The loop, start to finish:
uv run pytest # no credentials needed at all
uv run python -m src.ingestion.pipeline # lands in dev/your-name
cd dbt && uv run --env-file ../.env dbt build && cd .. # builds dev_yourname,
# including the enrichment modeldbt build needs no --vars: it reads LANDING_PATH from the same .env
your ingestion wrote to, so the two cannot disagree. In VS Code, point the dbt
extension at data/dbt and it picks up the same profile.
cd data/airflow && astro dev startastro dev start prints the UI address, http://airflow.localhost:6563 on
current versions. Astro reads your data/.env, so a task running there uses your prefix, your
schema and your database. It overrides three values, because inside a
container localhost means the container itself: the database is reached as
db on 5432, with sslmode=prefer since the local Postgres has no
certificate.
Run the publish step against your own schema:
docker exec -it $(docker ps -qf name=scheduler) \
airflow tasks test final_project_pipeline publish_to_backend $(date +%F)It reads <catalog>.dev_yourname.fct_postings_enriched and writes
analytics_dev.fct_postings in the real backend database. Same table name as
production, one schema across, so promoting it later changes nothing the
backend selects. dbt_build runs the same way.
You share analytics_dev with your teammates, so the last publish wins. The
table carries a comment saying where the rows came from, which is how you tell
whose run you are looking at:
\d+ analytics_dev.fct_postings --> from team_a.dev_alex at 2026-08-13T11:21Z
Point BACKEND_PG_PUBLISH_SCHEMA at analytics by mistake and the run stops
with a permission error. analytics_dev_user cannot write production, which is
the point of it being a separate role. The ingest task does not: it starts a
Container Apps job, which needs the VM's identity, so run that one as the
script above.
The same DAG file runs in both places. It reads a secret from your
.envwhen there is one and from Key Vault when there is not, so nothing has to be commented out or stubbed to make local work.
Four tools, each with one job, all installed by uv sync --all-extras and all
run by CI on every pull request:
uv run ruff check . # mistakes: unused imports, bugs, naive datetimes
uv run black . # Python layout
uv run sqlfmt dbt/models dbt/tests # SQL layout
uv run ty check src tests # typesRun them before you push and the pull request goes green first time. Two of
them will teach you something the first week: ruff refuses a datetime with
no timezone, and ty refuses SQL built by pasting a name into a string, which
is why src/publishing/sync.py composes statements with psycopg.sql instead.
All three start from the repository root:
docker compose up -d db # from step 3, if it stopped
docker compose build pipeline # the image CI will build
cd data/airflow && astro dev startAstro needs no .env of its own. It reads data/.env, so the DAG runs
against the same source, prefix and schema your own commands use.
Build the image locally, but do not expect to run it locally. Inside the
container there is no az login and no Azure metadata service, so
DefaultAzureCredential has nothing to authenticate with and the run ends in a
wall of "credential unavailable" messages. That is correct behaviour: the image
gets its identity from Azure when the Container Apps job runs it. To exercise
the ingestion on your machine, use uv run python -m src.ingestion.pipeline, which
authenticates as you.
The template ships a job-postings example so the shape is concrete. Swapping in your team's source is five edits:
.env: pointSOURCE_API_URLandSOURCE_NAMEat your source.src/ingestion/models.py: change the model to match your records.dbt/models/: rename the models and columns to your domain.dbt/models/marts/_fct_postings.yml: rewrite the contract.dbt/models/marts/fct_postings_enriched.py: replace the classifier with whatever your product needs.
Do this in your first two days. Everything after that builds on the shape you choose here.
Verify your source before you commit to it: call it once, print a record, and confirm you can parse it. An idea you love with a source you cannot reach is worth less than a plain idea that works.
Raw files, not tables. A raw file is exactly what the source sent you, so when a column changes shape in three weeks you can re-read it and find out when.
Your team's storage account has two containers, each registered in Unity
Catalog as a volume. prod is the scheduled pipeline's, and the file it
writes as prod/raw/postings/ingest_date=2026-08-12/data.json is the file
dbt reads at /Volumes/<catalog>/landing/prod/postings/. One copy of the
bytes, two ways to reach it: Azure tooling on one side, SQL on the other.
dev is the other one, and it is where your own runs land. It appears next to
the first as /Volumes/<catalog>/landing/dev/. Two containers rather than two
folders because a permission can be given on a container and cannot be given
on a folder: this is the difference between separation you are asked to
observe and separation you cannot get around.
One file per source per day, so a re-run replaces its own file instead of
doubling your data. The date is a folder rather than part of the filename:
postings/ingest_date=2026-08-12/data.json. A folder named key=value is a
partition, a convention every engine that reads files understands, so dbt gets
an ingest_date column without anyone parsing a filename.
What that buys you is a bad day being one directory. When a source has an outage and sends nonsense for a day, the fix is to delete that folder and run the pipeline again for that date, and nothing else in the landing zone is touched:
az storage blob delete-batch --account-name sthyffpteama --source dev \
--pattern 'your-name/postings/ingest_date=2026-08-12/*' --auth-mode login
uv run python -m src.ingestion.pipeline --run-date 2026-08-12
⚠️ It does not speed updbt build. Skipping folders only helps a query that filters on the partition column, and staging deliberately reads every day so its de-duplication can pick the newest version of each record.
One sharp edge if you landed files before this layout existed: once a
ingest_date= folder appears beside them, read_files treats the folder as
the partition root and silently ignores files sitting at the old depth. No
error, they just stop appearing in your models. Move or delete them.
Change the layout if you like, and change LANDING_PATH to match.
Raw means raw. The job validates before it writes, but it lands what the source sent, not what validation produced. Parsing is a gate deciding whether the run is worth landing, not a transformation. Write the parsed objects instead and the "raw" file quietly carries your own type coercions, so re-reading it after a source change tells you about your bug rather than theirs. An empty batch raises: landing zero rows leaves yesterday's mart in place with every test still passing, and nobody finds out for a week.
src/common/warehouse.py sends statements over the Statement Execution API, which is
plain HTTPS. databricks-sql-connector would do the same over Thrift and a
much larger dependency tree.
It takes whichever credential it finds: your DATABRICKS_TOKEN on your
machine, so one value covers dbt and these steps alike, and the team's service
principal in Azure.
The one thing that surprises everybody about that second case: your team's
service principal authenticates at Microsoft Entra, not at the Databricks
workspace. The workspace's own /oidc/v1/token endpoint returns 401 for
principals created
this way, and the error does not hint that you are knocking on the wrong door.
Anything expressible as SQL belongs in a .sql model, where it is tested,
documented and rebuilt with everything else. For the rest there is the Python
model.
fct_postings_enriched.py reads each job title and decides which discipline the
posting belongs to. As SQL that is a hundred-line CASE nobody dares change; as
Python it is a dictionary with unit tests, and the day you replace it with a
real model or a call to another service, only that one file changes.
Ingestion stays a container rather than becoming a Python model too, and the
reason is worth knowing. Submitting a serverless job costs about a minute of
waiting per run, which is fine once a day at the end of the graph and painful
when you are iterating on a fetch. A container also runs anywhere, including
docker compose run, while a Python model only runs on Databricks.
So the rule is: SQL in a .sql model, Python that works on your tables in a
Python model, and a container when the work needs to reach outside the
warehouse. If your enrichment grows into something that calls another service on
every row, move it back into a container job and add a task for it.
Three tasks, in order: ingest, dbt_build, publish_to_backend. Enrichment
is not among them: it is a dbt Python model, so dbt_build already runs it in
the right order, and a step dbt owns cannot drift out of step with the models it
depends on.
The order is the point. Publishing a mart that failed its own tests is worse than publishing nothing, and the dependency is what stops it. Separate tasks also mean that when dbt fails you re-run dbt, not the fetch.
Every setting lives in the Airflow UI, under Admin -> Variables, and is read when the task runs. You are an admin on your team's instance, so changing where the pipeline points is a change you make yourself, in one place your whole team can see, with no deploy:
TEAM, AZURE_SUBSCRIPTION, AZURE_RESOURCE_GROUP, ACA_INGEST_JOB,
DATABRICKS_HOST, DATABRICKS_HTTP_PATH, BACKEND_PG_USER,
DATABRICKS_CATALOG, DBT_SCHEMA, AZURE_TENANT_ID, BACKEND_PG_HOST,
BACKEND_PG_DB.
They are set for you when your team is provisioned. Miss one and the task that needs it fails saying which one, rather than doing something surprising.
Secrets are not among them. Each one is fetched from Key Vault inside the task that needs it, using the machine's own identity, so nothing is stored on the VM and a typo fails with a 403 rather than reaching another team's data.
The DAG imports the same src package the containers run, so the publish logic
exists once. It is mounted inside the dags folder, which Airflow puts on the
Python path, so there is no PYTHONPATH to configure. A task failing with
ModuleNotFoundError: src means that mount is missing: a teacher question,
not something to work around in the DAG.
One pipeline lives in one file. pipeline_dag.py holds the whole pipeline
because it is one DAG of four tasks in a line, and splitting four tasks across
four files means opening four files to answer "what runs after dbt?".
When you add a second pipeline, add a second file rather than a second DAG in
this one. Name the file after its dag_id, so daily_report_dag.py defines
daily_report. The reason is not tidiness: Airflow parses each file on its
own, so a typo in your new DAG takes out only your new DAG. Put two in one file
and one mistake stops both.
Anything that is not a DAG belongs in src/, which the tasks import. Airflow
re-parses everything under dags/ every few seconds, so a helper module there
is read over and over for no reason. alerts.py is the deliberate exception:
it is DAG wiring, used by default_args, and it has to be importable by name
from the dags folder.
If you find yourself copying setting() and secret() into your second DAG,
move them somewhere shared instead. That is the moment they stop belonging to
one pipeline.
The other direction: your credential can read the backend's own tables, so a
model can join against how the application is actually being used.
src/publishing/sync.py has read_backend_table ready for it. Add a task once you have
agreed with the backend which table you are reading, and keep it before
dbt_build so the models can use what it lands.
Agreeing it matters more than it sounds. Their tables are theirs to change, and nothing warns you when they do: a column you depend on can disappear in a migration you never saw. Read as little as you need, and tell them what you read.
The two tracks meet in the backend's database, which has one schema per side:
analytics— you write, the backend reads. Your published marts.app— the backend writes, you read. Their operational tables.
There is a login role per schema, analytics_user and app_user, each owning
its own and holding read on the other. You connect as analytics_user. That is
what stops a stray publish from corrupting the application, and stops a backend
deploy from overwriting your marts.
scripts/db-setup.py at the repository root creates all of it, and prints your
password once. Run it against your local Postgres and you have the same shape
as the real database before the real one exists.
Anything personal is your problem to handle the moment you read it. Hash it or drop it in your staging model, so it never reaches a mart and never leaves the warehouse.
The publish must never leave a half-loaded table where the backend can see it, so loading and switching are separated:
- create
<table>__staging, empty - insert every row
- in one transaction:
drop table if exists <table>, then rename staging into its place
Note the if exists. The obvious version renames the current table out of the
way first, which cannot work the very first time you publish, because there is
nothing to rename: the sync then fails exactly once, on the run you most want
to succeed.
fct_postings is what dbt builds first, fct_postings_enriched is what the
Python model adds to it, and that is what gets published, under the name
analytics.fct_postings in the backend's database.
Adding a column is safe. Renaming or removing one breaks them, so agree it
first and change both sides at once.
Every column is documented in dbt/models/marts/_fct_postings.yml and
_fct_postings_enriched.yml. Those files are the contract: hand them to the
backend trainees on day one and they can write endpoints against columns that
have no data in them yet. See docs/mart_contract.md
for how to work on it together.
airflow/dags/alerts.py posts to your team's Slack channel when any task
fails. It is attached once in default_args, so every task inherits it,
including tasks you add later: alerting you have to remember is alerting you
will forget.
Airflow's own behaviour on failure is to colour a square red and wait for somebody to look. Nobody looks at 6am, which is when your pipeline runs.
No credentials live in this folder. dbt/profiles.yml is committed on purpose:
every value in it comes from env_var(...), so it holds nothing secret. Real
values live in .env, which is git-ignored, and in Key Vault.
You hold exactly one credential: your own Databricks token. Your team's service principal, the one that can write the schemas you cannot, stays in Key Vault where Airflow reads it, and is not yours to copy onto a laptop.
Never commit .env, and never paste a token or connection string into a chat
message or an LLM prompt.