Tally Split AI

I built an AI app that turns a hand-annotated receipt into a fair split. Underneath the simple idea: a multi-layer LLM pipeline, computer vision for the handwriting, and a custom HMAC security model.

Problem

My partner and I split shared expenses, and existing apps all leave the thinking to me: they can divide a bill, but none I've found will split it automatically from your own preferences and a few handwritten notes.

I also wanted a real project to grow my AI engineering skills on. Tally Split AI is both: a fix for a daily annoyance, and a vehicle for going deep on agentic pipelines, secure authn/authz, and durable serverless workflows.

Solution

By default, the app behaves like most splitters: it reads the receipt total and divides it equally, 50/50. But the User can layer on their own curation and annotation rules — handwritten notes and household preferences that take the data-entry work off their plate. The User becomes a curator and approver rather than a typist.

Those rules land directly in the prompt, so the app is exactly as powerful as the User wants it to be, and as comfortable as they are sharing their data. In my own household, for instance, I've stored the last four digits of our cards so the AI can infer who paid.

Features

• Photo-to-split, no math in your head. Drag and drop a receipt; the app extracts line items, totals, vendor, and date, then splits based on your rules.
• Handwritten annotation detection. Circles, strikethroughs, initials, and handwritten totals are read as instructions.
• Agentic split and payer assignment. A reconciliation step combines OCR and annotations, with optional rules for identifying who paid.
• Human override everywhere. The user always gets a clear signal and the ability to correct anything.
• Household-scoped data. Resources roll up to a household; members share equal access to their own receipts and splits.

The Pipeline

A multi-layer pipeline split into single-purpose steps, each handed to the cheapest model that does the job well:

1. Upload — direct to Azure Blob Storage via short-lived scoped tokens, bypassing the app server.
2. OCR — Azure Document Intelligence's prebuilt-receipt model extracts structured line items, totals, vendor, and date.
3. Detect annotations — gpt-4o interprets handwriting against fixed conventions (initials, numbers, strikethroughs, circles).
4. Normalize — gpt-4o-mini cleans noisy OCR and consolidates franchised stores into colloquial names.
5. Split and assign payer — gpt-4o reconciles everything into a final split, with household rules to resolve ambiguity.

Full diagrams and prompts are on the project homepage.

Security Model

Custom security that's admittedly overengineered for a personal app — but the enterprise architect in me enjoys modeling it and trying to break my own designs. Even small AI applications deserve guardrails against unintended actions.

Authentication verifies two kinds of principal:

• Users sign in with GitHub SSO via nuxt-auth-utils; session cookies are httpOnly, encrypted, and MAC-signed.
• Tasks authenticate with a finely scoped HMAC-SHA256 token, verified by recomputing it from values in the database.

Authorization enforces scope on every endpoint:

• Household scope — reads and writes filter by householdId; mismatches return 404 Not Found, so one household can't confirm another's resources exist.
• Resource and action scope — each worker's token encodes the specific resource and resource:permission it may touch. A split:write task can't read receipts; a task scoped to one upload can't reach another.

N.B. The app is live but in closed private beta. Login works, but accounts outside the allow-list will see a not-authorized error.