# Use Case: Testing an AI Assistant Before Release: Injection, Leaks, Cost and a Regression Eval

Dokimore Support Team · October 28, 2026, updated 2026-11-01 · 8 min read

Format: Scenario walkthrough · Written for: QA leads and test managers, Developers who own their tests, Security and compliance reviewers · Where it applies: [Ireland](https://dokimore.com/for/ireland), [Germany](https://dokimore.com/for/germany), [France](https://dokimore.com/for/france), [Netherlands](https://dokimore.com/for/netherlands). The EU AI Act dates apply to an assistant offered to people in the EU; the four checks and the OWASP lists apply wherever the assistant runs.

> The feature works in the demo. The release question is what it does with a hostile prompt, a customer’s record, a runaway loop and the next model version.

## In short

- Four checks decide whether an AI assistant is releasable: what it does with a hostile prompt (injection), what it reveals (leaks of personal data and secrets), what it costs when something loops (a spend ceiling enforced before the call) and whether the next prompt or model change made it worse (a regression eval that gates the merge).
- The checks map to the OWASP Top 10 for LLM Applications, whose 2026 edition ranks prompt injection first, sensitive information disclosure second and excessive agency third; the agentic risks, tool misuse and goal hijacking, sit in OWASP’s separate Top 10 for Agentic Applications.
- Two AI Act dates matter in the scenario: Article 50 transparency, telling a person they are talking to a machine, applies from 2 August 2026 and was not deferred; the stand-alone high-risk obligations of Annex III were deferred to 2 December 2027 by the Digital Omnibus, Regulation (EU) 2026/1744. A support assistant is usually not high-risk, and the team writes down why.

This is a scenario walkthrough of a pre-release test plan for a feature built on a language model. It is not a customer story; the team is invented so the plan is concrete, and every outside fact is in the sources at the end.

## The scenario

A B2B product team adds a support assistant that answers questions about a customer’s account from the help centre and the customer’s own records, calling a third-party model through the provider’s API with a key the company holds. The demo impressed the board. Three weeks remain to the release, the assistant can read a customer’s invoices and open a ticket on their behalf, and the question the QA lead was handed is what has to be true before it ships.

## The landscape in numbers

Postman’s 2025 State of the API report, from more than 5,700 developers, architects and executives, found 89% using generative AI in their daily work and 24% designing APIs with AI agents in mind. Asked about security, 51% named unauthorised or excessive API calls from AI agents as their top concern, 49% worried about AI systems reaching data they should not see and 46% about AI systems leaking API credentials. Those three worries are the first three checks in this plan. The OWASP Top 10 for LLM Applications, in its 2026 edition built on reported incidents, ranks prompt injection first, sensitive information disclosure second and excessive agency third; OWASP now keeps a separate Top 10 for Agentic Applications for what a system with tools does, as opposed to what a model says.

## Check one: a hostile prompt

Prompt injection is the attack where text the model reads, in the user’s message or in a document it retrieves, instructs it to do something its operator did not intend. In the scenario the help centre articles are the retrieval source, and anyone with a support account can submit a ticket whose text the assistant later reads. The test is a set of probes: direct instructions in the user turn, instructions hidden in a ticket, control-token tricks, and attempts to make the assistant print its own system prompt. The pass condition is not "the model refused once"; it is a floor, the probability that every one of k repeated trials holds, reported with a confidence interval, because a model that refuses nine times in ten is a model that complies once a day at scale.

## Check two: what it reveals

The assistant can read invoices, so the response path is where personal data and secrets leave. The test runs detectors over both directions, the request and the response, for personal data, card and national identifiers with their checksums, API keys and tokens, and asserts that a prompt asking for another customer’s invoice gets a refusal and that a response never contains a credential. This is the check that an isolation bug in the retrieval layer, a query that forgot the tenant, turns into a leak, so it runs against the real retrieval layer with two test tenants, not against a mock.

## Check three: what it costs when something loops

An assistant that can open tickets can open a thousand. The test sets a spend ceiling and a per-tenant rate limit and proves that a call which would cross the ceiling is refused before it is made rather than reported after. In the scenario the ceiling is a day’s budget divided by tenants, and the proof is a loop that tries to exceed it and is stopped with a refusal the application handles gracefully. The cost ledger is priced from the provider’s own usage figures, so the number in the test is the number on the invoice.

## Check four: the regression eval

Every prompt edit and every model version is a release. The team builds a golden set of a few hundred cases from real, anonymised support questions with the expected answer or the expected refusal, scores each run against it, and makes the score a merge gate: a drop beyond an agreed threshold blocks the change. We published our own version of this discipline, 250 cases, five small local models, a paired statistical test between them; the method transfers, the models do not. The eval set is versioned with the prompt, and when a case is added to the prompt itself it is removed from the eval, because a case the model has seen measures memory, not behaviour.

## The law, in two dates

The EU AI Act applies generally from 2 August 2026, and Article 50 applies from that date without deferral: a person interacting with an AI system must be told so, unless it is obvious from the context. The assistant says what it is on the first turn, and that is a test case. The stand-alone high-risk obligations of Annex III, with their conformity assessment, logging and human-oversight duties, were deferred to 2 December 2027 by the Digital Omnibus, Regulation (EU) 2026/1744, published in the Official Journal on 24 July 2026. A support assistant that answers account questions is not usually in a high-risk category, and the team writes the classification and its reasoning down, because the document is what a customer’s procurement team will ask for.

## How Dokimore runs the four checks

Dokimore’s LLM testing module puts a gateway between the application and the provider whose key you supply. Injection probes and data-leak detectors run on every call in both directions; twelve multi-turn attack scenarios run in both polarities over repeated trials and report Pass^k floors with confidence intervals; a spend ceiling is enforced before the call and returns 402 when it would be exceeded, with per-tenant rate limits; and every call leaves an evidence envelope with the detector findings, the risk class, the cost and the latency. Findings use the vocabulary of the OWASP LLM and agentic lists. Agent mode lets the model call tools under your approval, with every guard decision listed.

The limits, stated: the detectors are rule-based and scan the first 192 KiB and the last 64 KiB of a payload; one of the twelve scenarios is published as a known limitation because its detector is keyword-anchored; and prompts go to the provider you chose, under your agreement with it, never to any other model. The regression eval is yours to build; the module gives you the gateway to run it through.

## What to do next

1. Write down whether the assistant falls in a high-risk category under Annex III and why; a customer’s procurement team will ask for that page.
2. Build the injection probe set, including instructions hidden in content the assistant retrieves, and report the floor over repeated trials with a confidence interval, not a single refusal.
3. Run the data-leak check against the real retrieval layer with two test tenants, never against a mock.
4. Set a spend ceiling and a per-tenant rate limit, and prove with a loop that a call which would cross the ceiling is refused before it is made.
5. Version a golden set with the prompt and make its score a merge gate; remove any case that is added to the prompt itself.
6. Make the first-turn disclosure a test case: a person must be told they are talking to an AI system, and Article 50 has applied since 2 August 2026.

## Questions people ask

### What should be tested before releasing an AI assistant?

Four things: how it behaves under prompt injection, including instructions hidden in content it retrieves; whether it reveals personal data or secrets in its responses; whether spending is stopped before a call that would cross a ceiling; and whether a prompt or model change made it worse, measured by a versioned regression eval that gates the merge. Each maps to an entry in the OWASP Top 10 for LLM Applications.

### Which OWASP list applies to an LLM feature?

The OWASP Top 10 for LLM Applications, whose 2026 edition ranks prompt injection first, sensitive information disclosure second and excessive agency third. If the feature can call tools or act on a user’s behalf, OWASP’s separate Top 10 for Agentic Applications covers what the system does, tool misuse and goal hijacking among them.

### Does the EU AI Act apply to a support chatbot from August 2026?

Article 50 does: from 2 August 2026 a person interacting with an AI system must be told so, unless it is obvious, and that date was not deferred. The high-risk obligations of Annex III were moved to 2 December 2027 by Regulation (EU) 2026/1744. A support assistant is usually not in a high-risk category; write the classification and the reasoning down.

### Why report Pass^k instead of accuracy?

Accuracy rises with retries and describes the best case. Pass^k is the probability that every one of k trials passes, which is the floor a customer gets, and under injected attacks the two diverge sharply. A floor with a confidence interval is a number a release decision can rest on; a single pass is not.

### Does Dokimore send our prompts to its own model?

No. The gateway runs its detectors over the request and the response and records the evidence envelope in your tenant. The prompt goes to the provider whose key you supplied, under your agreement with that provider, and to no other model.

## How we measured this

- The team, the assistant and its three-week deadline are a scenario, not a customer. Outside figures are quoted from the publisher’s own page on the date beside each reference.
- Dokimore’s behaviour is the product on the article date: detectors on both directions bounded to the first 192 KiB and last 64 KiB, twelve scenarios scored as Pass^k floors with one published known limitation, a spend ceiling enforced before the call with a 402 refusal, per-tenant rate limits, an evidence envelope per call, and prompts sent only to the provider you chose.
- Our own regression-eval figures are in the companion article on five small local models; they measure our workload, not yours.

## Sources

1. [Postman, 2025 State of the API report: 5,700+ respondents, 89% use generative AI, 24% design APIs for AI agents, 51% / 49% / 46% security concerns](https://www.postman.com/state-of-api/) (read 2026-10-28)
2. [OWASP, GenAI LLM Top 10 2026 (August 2026): the current edition, grounded in thousands of real-world incidents](https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/) (read 2026-10-28)
3. [Cloud Security Alliance, "The OWASP Top 10 for LLM Applications 2026: what changed and why it matters" (21 October 2026): LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM03 Excessive Agency](https://cloudsecurityalliance.org/articles/the-owasp-top-10-for-llm-applications-2026-what-changed-and-why-it-matters) (read 2026-10-28)
4. [Future of Privacy Forum, "The AI Act implementation timeline: what changes under the AI Omnibus": published in the Official Journal on 24 July 2026; Annex III obligations from 2 December 2027, Annex I from 2 August 2028, Article 50 generally applicable from 2 August 2026](https://fpf.org/blog/the-ai-act-implementation-timeline-what-changes-under-the-ai-omnibus/) (read 2026-10-28)
5. [OWASP GenAI Security Project, "OWASP Top 10 for Agentic Applications for 2026" (9 December 2025): the separate list for agents, from goal hijacking to tool misuse](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/) (read 2026-10-28)

Source: https://dokimore.com/blog/testing-an-ai-assistant-before-release
