Imagine you hire someone new and hand them the company card on day one. You also give them the manual. The manual says "refunds are in rupees." So a customer asks for ₹500 back, they type 500, and done.
Except the system actually counts in paise. So 500 means ₹5. The customer gets ₹5 back, nobody notices, and nothing ever shows an error.
That's basically what happens with AI agents today. They're starting to do real money stuff, like refunds, credits and plan changes, and they learn how from API docs. And they believe every word.
The thing is, docs are wrong more often than you'd think.
So for neatHack 2026 I built Notary. The idea is simple: the agent isn't allowed to touch money until it has proven the docs are telling the truth. (My first commit went in at 2:49 am, so yeah, it was that kind of hackathon.)
Here's a 15-second look at it before I explain anything:
Watch the red rows. Those are refunds that would've gone out wrong without Notary.
First thing I found: the docs were wrong
Before building anything fancy, I just pointed Notary at two real APIs: Crustdata (looks up company info) and Dodo Payments (where the money actually moves, in test mode only).
Crustdata's own docs were wrong in 3 places:
- Look up a company that doesn't exist, and the docs say you get a normal "200 OK" with an empty result. You actually get a 400 error. On two different endpoints.
- Send a field it doesn't know, and the docs say 400. You actually get 403.
None of this sounds dramatic, right? But picture an agent that reads "empty result means no match" and decides what to do with a customer based on that. It'll handle exactly these cases wrong, every time.
That one finding is honestly the whole reason Notary exists.
How it works, in plain words
The AI only suggests. Plain, boring code makes the actual decisions.
There are five steps. Here's each one, with a simple everyday version:
1. Read. Notary goes through the docs and turns every promise into something it can check. Like making a checklist out of a product manual: "it says the box has 3 cables, let's count."
2. Test. One AI model (I call it the Reflector) guesses what the API will do. Here's the important bit: the guess gets locked in before the real call happens. It's hashed, so it can't be changed later. Think of writing your match prediction on paper before the game starts, so you can't say "I knew it" afterwards.
3. Challenge. A second, stronger model (the Adversary) looks at the raw results and tries to prove the first one wrong. Basically, that one friend who pokes holes in everything you say. It can agree, say "true, but only in this narrow case," or reject it.
4. Certify. Whatever survives all that becomes a certificate: what was promised, what actually happened, the proof, and a fingerprint of all of it. Like a stamped receipt. And if the API changes later, the stamp gets cancelled.
5. Enforce. Every money action goes through a Guard. Refund, credit, plan change? It only runs if every certificate it depends on is still valid. If not, it waits for a real person, with all the evidence attached.
So the rule is literally the tagline: no certificate, no money moves.
What a run looks like
One recorded Dodo Payments run. You can see each step light up as it finishes.
Some numbers from the real runs:
- 3 promises in Crustdata's docs turned out false.
- 5 certificates issued, and the Adversary narrowed down every single one. No rubber stamps.
- All 19 live tests had their guess locked in before the call.
- 0 money calls that skip the Guard (checked across the whole codebase).
- About $0.03 of AI cost per run. Cheaper than a samosa.
And the one I like most: on Notary's test suite, which hides tricky API behaviour on purpose, an agent without Notary made 9 unsafe money moves. With Notary it made 0, and held all 9 refunds for a human to check.
What it doesn't prove (yet)
I'd rather tell you this myself:
- That 9 → 0 result is from the test suite, not a live queue. A real live test needs a person to set up a bunch of test checkouts first, and I haven't done that yet.
- The 3 wrong Crustdata promises aren't certified yet, so Notary just holds anything that depends on them. Which is kind of the point, honestly.
What I learned
The interesting part wasn't the AI. It was not letting the AI decide.
AI is great at saying "maybe it works like this." It's bad at being the final word on money. Put a few checks in between, like locked-in guesses, a second model whose job is to disagree, and plain rules that don't change their mind, and suddenly you can actually trust the thing.
Code, demo video and the full results are here: github.com/parthchilwerwar/notary.
Sources
- Notary on GitHub: the code, the demo video and the full README
- Notary evaluation: how the 9 → 0 test-suite result was measured
- Notary architecture: the Reflector, Adversary and Guard in more detail
- Incident records: the raw evidence for the 3 wrong Crustdata promises
- Crustdata API docs: the docs Notary checked
- Dodo Payments docs: where the refunds, credits and plan changes run (test mode)