Verification Systems someone else built or runs
An operator who audits. Not an auditor who sometimes operates.
A point-in-time verification of an AI system a vendor, an agency or your own team built. What it does with a hostile input. What it leaks. What it can do when it goes wrong. Whether its answers are grounded, what it costs under load, and whether its disclosures would survive a complaint. Written findings with a severity, a reproduction and a fix order for each. The same discipline runs inside every system I operate; this is the version you can buy on its own.
- Scope
- One system, one to three weeks, read-only access on your accounts.
- Frameworks
- OWASP Top 10 for LLM Applications 2025 · OWASP Top 10 for Agentic Applications 2026 · NIST AI RMF 1.0 (Jan 2023) and NIST AI 600-1 (Jul 2024)
- Independence
- Never a system I operate. If I run it, someone else verifies it, and I will name one.
- Ends with
- A findings report, the test set, a one-page summary for your insurer, counsel or customer, and a re-test if you want one.
01 What gets tested
Eight questions, each with a reproduction.
The codes are the entries of the OWASP Top 10 for LLM Applications (2025); “Agentic” refers to the OWASP Top 10 for Agentic Applications (2026). The frameworks supply the categories. The test cases are built from your system and your documents, and you keep them.
| Question | Framework entry | What is actually done |
|---|---|---|
| Prompt injection and jailbreaks | LLM01, LLM07 | Hostile input through every channel the system reads: callers, documents, web pages, tool results. Does it leak its instructions, change its behaviour, or act on an instruction it should not have taken? |
| Sensitive information disclosure | LLM02 | What a stranger can get it to say about other customers, your pricing, your staff, or the system itself. |
| Excessive agency | LLM06, Agentic | What it can do, not just say: which tools it can call, with what permissions, and whether a wrong turn can write, send, pay or delete. |
| Grounding and misinformation | LLM09, LLM08 | A golden-question set built from your documents. Does it answer from them, cite them, and say “I don’t know” when the document is silent? |
| Output handling | LLM05 | What happens downstream to what it writes: records, emails, texts, code. Is anything trusted that should not be? |
| Unbounded consumption | LLM10 | Cost and latency under load and under abuse. What one bad actor can run up on your account in an afternoon. |
| Supply chain and data | LLM03, LLM04 | Which vendors, models, plugins and data sources the system depends on, what each can see, and what you have agreed to in their terms. |
| Disclosure and consent | TCPA, CIPA, BPC §17941 | Does the bot say it is a bot? Is there consent evidence for recording, for outbound calls, for texts? Would it survive a complaint? |
02 What you get
A report you can act on, and a test set anyone can re-run.
The findings report
Each finding with a severity, the exact input that produced it, what happened, and what to change. Ordered by what to fix first, not by category.
The test set
Every case that was run, with the expected and observed result, in a plain file in your drive. Your vendor can re-run it; so can the next verifier.
The one-page summary
Written for the person who asked the question: your insurer, your counsel, a large customer’s procurement team. What was tested, what was found, what was fixed.
A re-test
After the fixes, the same set again, and a dated line in the summary saying what now passes. Optional, and priced in the proposal.
03 Who asks for this
Usually someone who was handed a system and a promise.
- An owner with a vendor’s word
An agency or a platform delivered an agent. It demos well. Nobody has tried to break it, and nobody knows what it can do with the CRM it is connected to.
- An MSP or accountant on behalf of a client
The client bought something. The MSP is being asked whether it is safe, and would rather answer with a report than an opinion. How that works.
- Counsel, an insurer, or procurement
A renewal, a claim form or a vendor questionnaire asks about the AI system, and the honest answer today is “we don’t know.”
- A team about to renew
The contract is up. Before signing for another year, they want the thing tested by someone who does not sell it.
04 What it is not
Four things this is not, so nobody buys the wrong one.
- Not a SOC 2 report or a certification
Those examine an organisation’s controls over a period. This examines one system at one point in time, and says so on every page.
- Not a legal opinion
The consent and disclosure findings are evidence for your counsel, with the statute named. Counsel decides what it means.
- Not a network penetration test
Your firewall, your endpoints and your email belong to your IT provider. This stays inside the AI system and what it is connected to.
- Not a verification of my own work
A system I operate is verified by someone else. Buying the operated subscription and the verification from the same person would make both worth less.
Have a system you were handed with a promise?
Thirty minutes, no prep. Bring the vendor’s claims; we work out what a verification would test.