Tally Split AI

Analyze receipts incl. handwritten annotations with AI analysis to auto-adjust totals and assign payers based on custom instructions.

Important

This is a personal app I built to solve a personal problem. It is open-sourced under the MIT license for learning purposes. On a best-effort basis, it aims to demonstrate sound architecture and engineering practices. It is not a product, not intended for distribution, and not meant to run out-of-the-box for others.

Use Case

It is challenging for me as a U.S. citizen to open a bank account in Germany due to FACTA. So without a joint bank account, how can we best split shared households expenses?

For receipts with shared and individual items, how can we indicate that some items are to be split between persons and others not?

As part of my pivot to AI Engineering and Architecture, I decided to build my own app instead of using existing commercial apps, which has added privacy benefit.

Example Analysis

The original total is 23.23 as per the OCR data. Handwritten annotations indicate that Julie paid, and MM owes. Two items, PUTENSTEAK (8.08) and SPEZI KROMBACHER (0.99), are circled, which defines the adjusted total as their sum: 9.07. The handwritten note 'Julie paid' clearly identifies the payer, and the annotations are unambiguous, leading to high confidence in both the adjusted total and payer determination.

• Overall: 90%
• splitAmount: 90%

Stack

Application

Component	Role
Nuxt	Full-stack web app (UI + API)
Postgres	Database for receipts, splits, and workflow state
SQLite	SQLite DB for @nuxt/content to serve the landing page.

Infrastructure

Component	Role
Vercel	App Platform that automatically deploys my application as serverless functions.
Trigger.dev	Platform for fully managed workflows (agent) and orchestration.
Supabase	Postgres development platform with built-in features and optimizations for serverless functions.
Turso	Hosted libSQL for @nuxt/content, not Postgres: Nuxt Content needs the pg driver, which conflicts with the postgres-js driver Supabase's pooler requires.
Azure Blob Storage	Receipt photo storage (direct client uploads via SAS)
Azure Document Intelligence	OCR for receipts (prebuilt-receipt model)
Vercel AI Gateway	Provider-agnostic LLM routing (ZDR, prompt caching, fallback)
GPT-4o	Handwritten annotation detection (via the Gateway)
GPT-4o-mini	Receipt normalization & split adjustment (via the Gateway)

Note

I chose Trigger.dev over Vercel Workflows for its developer experience and stronger feature set, for example, durable compute, remote-controlled execution.

Security

Highlights of the AuthN / AuthZ design (see docs/SECURITY.md for detail):

• Two principal types, never blended. Human users authenticate via session (GitHub OAuth); Trigger.dev tasks authenticate via HMAC. The auth pipeline commits to one path based on request headers — a failed task token does not fall back to session auth.
• Action-scoped HMAC tokens. Each task gets only the resource:permission pairs it needs (e.g. expense:write, upload:read). Tasks cannot mint their own tokens; only registered orchestrators can, and only for their declared children.
• Cryptographic scope binding. The token's scope is part of the signed HMAC payload, not a separate header — tampering invalidates the token. Verification uses crypto.timingSafeEqual and enforces a server-side expiry window.
• Household isolation at the query layer. Every resource read filters by the caller's household; resource-by-id reads go through a requireAuthorization guard that returns 404 (not 403) on mismatch to avoid leaking existence.
• Sessions hold identity + authZ scope only. Domain data lives in Pinia stores / DB queries, not in the session.
• User decides what personal information to send. The app can only send user initials to AI and agents with receipts. However, the user can choose to provide that information in custom instructions for their household to increase receipt analysis and payer assignment accuracy.

Important

If you find a security issue or bug, please feel free to open an issue or email me. Cheers.

Local Development

Note: offline development is not possible due to Azure, LLM, and Trigger.dev dependencies.

.env configuration

The .env.sample file describes which configuration is needed for the app to run. First, copy the file.

cp .env.sample .env 

Then open .env and fill in your own config values

Tip

You can use the scripts in this repo's Makefile to generate some of the secret values.

Finally, load the configuration.

. ./.env

Important

You need to load the env config before every command below - except for docker.

Postgres

docker compose -f docker-compose.dev.yaml up -d

Nuxt App

After loading configuration from your custom .env, start the app.

npm run dev

Open http://localhost:3000/

Trigger.dev

Connect to trigger.dev (orchestrator) to use local machine as a worker/agent.

npm run trigger:dev

Drizzle Studio (optional)

Drizzle provides a UI to explore database

npm run db:studio

Open https://local.drizzle.studio/

---

Docs

The docs are semi-scattered brain dumps.

Path	Target Audience
ARCHITECTURE.md	High-level system architecture
docs/	for Human understanding and reference
docs/SECURITY.md	Security model
trigger/README.md	Trigger.dev tasks — how to run and trigger them
shared/enums/README.md	Enum conventions
.claude/rules/	Instructions for agent (always loaded)
.claude/skills/	Loaded on Demand
CLAUDE.md	General agent instructions

Features

Frontend UI

• UI to upload files (drag-and-drop, queued, direct-to-Azure)
• UI to view/edit/delete receipts
• UI for splits (via receipt view page)
• Receipt inbox view (list + preview at /receipts/inbox) (temporary solution removed)
• Real-time workflow status updates via SSE
• SSO with GitHub

Backend Functionality

• Blob SAS tokens on-demand
• Document Intelligence API — OCR via Blob URL + SAS token
• Pino structured logging with domain-based child loggers
• Field-level change history for receipts and splits (human + agent mutations)

Annotation Functionality

• Deploy GPT-4o model for analyzing images for handwritten annotations
• Send API request with 1) OCR JSON and 2) Blob URL
• Process response, e.g. probability

Workflow Orchestration

• Trigger.dev orchestrator: OCR → annotations → normalize → split creation → adjust split
• Per-step progress tracking in workflow_runs table
• HMAC-authenticated HTTP API for tasks (no direct DB access)
• Action-scoped tokens — minimum permissions per child task