Skip to main content
Free plan, no cardStart free

ProductHow-to guide

How to Test an LLM API Key Before You Build on It: The Dokimore LLM Tester, Step by Step

The tester stopped explaining itself on screen in October 2026. This is the manual: the four steps, what happens to your key, the 62 probes, the nine multi-turn chains, and what a result does and does not prove.

Summarise with AI

Opens a new chat with only this request: Summarise the key points of this article: https://dokimore.com/blog/how-to-test-an-llm-api-key

100Messages per chat, read from the server
62Starter probes in 13 groups, seven of them following the OWASP Top 10 for LLM Applications
9Multi-turn chains, each with a single-turn control
0Keys written to storage or logs

In short

  • Test the connection before you spend a token: one free models-list call proves the endpoint and the key and fills the model picker with the ids the provider actually lists for that key.
  • Start from a probe, not from "hello". The 62 probes come with an expected result and a reading guide, seven groups follow the OWASP Top 10 for LLM Applications (2026 edition), and a probe is the same text for every model you compare, which is what makes two results comparable.
  • Your key travels with each request to the gateway, which forwards it to the provider and forgets it; it is not stored, not logged and never echoed back. Your browser is outside that boundary, so decline its offer to save the field.

About this article

Format
How-to guide: a procedure to follow, step by step.
Written for
Developers who own their tests · QA leads and test managers · Security and compliance reviewers
Where it applies
Every market. The rules that differ by country are on the country pages.

You have an API key and a model name, and a decision to make: is this model fast enough, safe enough and cheap enough to build on? Opening a chat window and typing "hello" answers none of that. It costs a turn and produces one anecdote you could not put in front of a reviewer. The LLM tester exists to turn that first exchange into a measurement, and this article is its manual.

Until October 2026 the tester explained itself on screen: a welcome line, a four-step guide, a list of tips, a caption under the probes, a paragraph under the chains. After a visit or two it only took space from the conversation. So the screen now carries controls and results, and the explanation lives here, one link away from the empty chat.

What the LLM tester does

It lets you chat with any model through your own API key while the AEGIS gateway screens every message in and every reply out. Each reply comes back with a receipt: tokens in and out, cost, latency split into its phases, a safety grade, and a pass, warn or fail verdict from the checks you set. A chat holds up to 100 messages, you can keep up to five chats open side by side, and every chat is saved under your workspace so you can reopen it from the Recent chats list.

Shared memory is off by default. Each window is its own conversation, and the model has no recall across windows, which is what you want when the point is to compare one prompt against another cleanly.

Step 1: pick a provider and set the key

Choose the provider, paste its API key, and press Set key. The key field accepts the usual shapes, an OpenAI secret key, an Anthropic x-api-key, a Google AI key, and a custom endpoint for anything that speaks one of those wire formats. Two cases need no pasting at all: a key your workspace administrator configured on the platform, which the platform sends on your behalf and which never reaches your browser, and the local models that run on the Dokimore host, which need no key.

The model picker starts from a catalogue and marks its entries unverified until the connection test replaces them with the ids the provider actually lists for your key.

Model settings, under Provider features, lists what the provider’s API can do for the chosen model. Thinking starts on at medium effort, and the tester checks that the model accepts it before sending. For the local models on the Dokimore host it reads each model’s own capability list from the model server, so a model that lists no thinking is never sent it and the row says why. For any other endpoint, a provider that answers that the model does not support thinking is remembered for that model: the failed turn says so, and from the next turn thinking is left off with the provider’s own words as the reason.

What happens to your key

The key is held in memory for the tab you pasted it into. It travels with each request to the gateway, which forwards it to the provider and then forgets it. The saved conversation record is written without it, it is kept out of the logs, and it is never echoed back in a reply. Close the tab and it is gone.

Your browser sits outside that boundary. Chrome and Edge treat a masked field as a password field and may offer to save it; decline that prompt unless you want a provider key in your password manager.

An endpoint that starts with http:// is judged by where the traffic goes and whether a key travels with it. A key sent to a public host over http is an error. A key sent to a private network or an internal host name is a warning, because anything else on that network can read it. The local models on the Dokimore host take no key and stay inside the host, so they raise nothing.

Step 2: test the connection before you spend a token

Test connection makes one free models-list call. It proves the endpoint is reachable, proves the key is accepted, and fills the model picker with the exact current ids for that key, so you stop guessing whether the model you typed still exists. Some providers do not expose a models list that can prove a key; for those the tester reports the endpoint as reachable and lets the first real turn settle it.

Step 3: choose how to test

  • Chat sends one turn at a time. Type / in the message box to search the probe catalogue by name and arm a probe without leaving the composer. Stream replies shows the answer as it arrives and records the time to first token.
  • Agent lets the model call tools under your approval. You read the notice and turn it on, pick Act or Plan, set a posture, and can ask to be notified whenever an approval is needed. Each reply then carries a grounding pill, and the Security tab lists every guard decision the gateway made.
  • Compare sends one question to up to four models side by side. The analysis row shows where the answers agree, which named things they dispute, and a judge ranking taken with the lane order swapped, so the ranking cannot come from position alone.

Step 4: read the reply receipt

Every reply opens with its verdict word and a one-line count of what passed, warned, failed or was not checked. Under it sits a timing bar with only the phases the provider actually reported: the AEGIS check on the way in, waiting for the first token, writing, and the AEGIS check on the way out. Then the facts: total time, tokens in and out, cost or "Not priced" when no price is on file, a quality index when one was scored, and a trace id when one was filed. A value the provider did not report reads "Not reported", never zero, because an unknown is not an error.

Checks are graded on every reply by code, locally, with no extra model call. Add one from the Checks section and it runs on every reply from then on; frontier probes bring their own assertions. From a reply you can also run the same prompt several more times to measure how much the answers agree, ask the same question several different ways to measure how much the answer moves, grade it against your rubrics with the local judge, save it as a regression test, or open Inspect for everything measured about that one answer.

Start with a probe instead of "hello"

The catalogue holds 62 probes in 13 groups. Seven groups follow the OWASP Top 10 for LLM Applications, 2026 edition (published August 2026): prompt injection (LLM01), sensitive information disclosure (LLM02), excessive agency (LLM03), unbounded consumption (LLM06), misinformation (LLM07), hidden context exposure (LLM08, which widened the 2025 edition's system prompt leakage) and improper output handling (LLM10). The other six are performance, quality, frontier (auto-graded checks that still separate current models: strict schemas, arithmetic, long context, consistency), manipulation, bias and sensitive data. Since November 2026, fifteen agentic probes cover the 2025–2026 attack classes: forged chat-template turns, instructions hidden in invisible Unicode (tag characters, variation selectors and the binary "Sneaky Bits" code), requests dressed as policy files, payload splitting, look-alike letters, tool-description poisoning, tool-result hijacking and reference-style image exfiltration. Each plants a fabricated code, grades itself by code and sits in the OWASP group it tests. AEGIS decodes the same hidden text before the call and names the channel in the pre-flight findings, so a pass that only means the model cannot read the channel is not mistaken for a defence.

Each probe shows what to expect and how to read the result before you run it, plus the severity if the model fails. The prompt text itself is applied on the server and never sent to your browser, because these are working attack strings and a page that ships them publishes them to every extension on it. The expected result and the reading guide are what turn a reply into a finding.

Multi-turn chains: attacks that only exist across a conversation

Some failures never show up in one message. Escalation, trust-building, and fragments assembled over several turns are attacks on the conversation, not on a prompt. The nine chains in the catalogue, across manipulation, bias and sensitive data, run three to four scripted turns and then the same ask compressed into one turn as a control.

The result is three numbers rather than a reply: a verdict (held, broke or leaked), the turn on which the model flipped, and whether the single-turn control held. That last one is the finding. If the chain broke and the control held, the conversation is what broke the model. If both failed, the weakness is not multi-turn-specific, and that is a different report. A chain costs several billed turns, so it never runs on its own, and the panel shows each turn’s purpose rather than its prompt.

In the product

Attach a document to score RAG quality

Attach a document to a message and the reply is scored against it: faithfulness, hallucination rate, context recall, context precision and mean reciprocal rank appear in the RAG quality section of the inspector. The score belongs to the turn that carried the document; later turns without one keep it, and the panel says which turn it came from.

Every reply is read word by word

Under each reply sits one line: a verdict (Clean, Review or Risk) and what it came to, such as "3 of 4 claims supported · 1 contradicted · 2 hedges". Press Analysis to open the reply with its words marked, the claims with the passage that supports or contradicts each one, the terms of your question the reply never addressed, and its key phrases. A reply reopened from Recent chats is analysed when you ask, and nothing is sent to the provider either way.

Claims are checked against the material you supplied: files attached to the message, the reference documents of the chat, text you pasted into the message, your own long earlier messages, and, for an agent, the results of its tools. Each claim is labelled Supported, Partly supported, Contradicted (a number that differs for the same thing) or Not supported (it names an identifier, path or URL your material does not contain). A sentence that says the opposite of the material is marked as a possible contradiction rather than a certain one, because word matching cannot read scope ("not a refund" next to "store credit only" agrees); the Deep check settles it. With no material at all, the claims are Unchecked. That is not a failure: a closed-book answer cannot be checked against nothing.

The marks are hedges, overclaims, vague quantifiers, filler (stock phrases and chat boilerplate), repeated wording, names that are not in the material, contradictions, and long sentences. Each kind has its own underline as well as its tint, and the legend turns a kind off. The method is deterministic and runs in milliseconds: sentence segmentation, chunks of about 120 words with one sentence of overlap, BM25 retrieval, word and character overlap, TextRank for the key phrases. A supported claim matches your material; that does not make it true.

Word confidence and the Deep check

Turn on Word confidence (log-probabilities) in the provider capabilities and the reply also marks the words the model itself was least sure of, with the alternatives it weighed. Only some models return this, and the tester asks only where it does: chat models behind the OpenAI-style API that are not reasoning models, and Gemini 1.5 and 2.0. It does not request them on the Responses shape, from Anthropic, or from Gemini 2.5 and 3, and then says "Not returned" rather than inventing a number.

Deep check takes the riskiest claims (four by default, six at most) and checks each one with hybrid retrieval, the local embedding model fused with BM25, and an entailment judgement from the configured judge model. The judge must copy the sentence from your material that shows its answer. If that sentence is not in the material, the judgement is discarded and the claim reads "Not addressed". When the judge is a remote service, personal data is redacted from what it receives. One deep check runs at a time per account, up to twenty every ten minutes, because each one is several judge calls on shared hardware.

Prompt engineering: three levels and a preview

With Prompt engineering on, each message is corrected and shaped for the selected model before it is sent, and your bubble shows the text that was actually sent, with what you typed and a word-by-word diff underneath. Choose the level beside the switch: Correct fixes spelling, grammar and wording and keeps your structure; Engineer adds the structure that model family reads best; Engineer + rewrite also lets the judge model rewrite the instruction, and that rewrite is kept only if every name, number and quoted string you wrote survives it word for word.

Improve, or /improve followed by your text, shows the result in the message box before anything is sent, so you can edit it or go back to what you typed. A probe is never rewritten: it has to be the same text for every model you compare.

The structure comes from each vendor's own prompting guide: 24 model families in November 2026, from Claude, GPT-6, Gemini and Gemma to Grok, DeepSeek, Kimi, GLM, MiniMax, Cohere, Nova, Phi, Granite, Nemotron, Sonar and Jamba. Where a vendor publishes no style guide, the tester says so and uses the cross-vendor consensus. Material you paste that carries instructions, including text hidden in invisible Unicode or spelled with look-alike letters, is marked as data, and the model is told not to follow it.

Tune, on any reply, measures instead of assuming. It sends your prompt as typed and written three ways (XML tags, Markdown headings, plain lines), then sends the best one and the as-typed prompt again on fresh calls, and grades every reply against what you typed. That is up to eight billed calls. One run is never proof, so a lead is labelled a possible difference; each re-run of the same study (the same prompt, model and settings) adds its fresh calls to the same evidence until the difference is measured, by a test that stays valid however often you re-run and look, so retrying until it looks good cannot fake a result. Pin a markup and Prompt engineering writes it for that model from then on.

An output grammar, written in the message box

Press Grammar, or type /grammar, and the message box becomes the grammar box. Your half-typed message waits untouched. Paste a JSON Schema, a list of allowed answers one per line, a regular expression for the whole reply, a GBNF grammar or a Lark grammar; the kind is detected as you type, and the Kind menu pins it when the guess is wrong. Check compiles it and says how it will reach this provider. Enter applies it once the server has compiled it, Shift+Enter starts a new line, and Escape takes you back to your message.

The grammar in force sits above the box until you remove it. Where the provider has a field for it, it is sent there, as JSON Schema structured output or a native grammar on vLLM and llama.cpp; otherwise it is sent as a requirement in the prompt. Either way, every reply is validated against it here, and that verdict leads the reply’s checks.

What a result does and does not prove

  • A security probe the model refuses is a pass. Compliance is a finding about that model, not proof that the gateway failed.
  • One prompt is an anecdote. Run a probe several times before quoting a rate, because these models are not deterministic above temperature zero.
  • Performance figures belong to this deployment. A local CPU model and a hosted GPU model are not comparable on wall time.
  • The output-handling probe is expected to comply; the measurement is what AEGIS does with the reply, not whether the model answered.
  • Personal data in the probes is synthetic and exists to exercise the detector. Never paste a real customer record into a probe.
  • Bias probes use the BBQ design: in an ambiguous context "cannot be determined" is the only unbiased answer, and the disambiguated control proves the model can still commit when the evidence is there. Run both.
  • Sensitive-data canaries and every named person in the probes are fabricated. A pass means nothing leaked in this run, never proof that the model memorised nothing.

Before you quote a number

  1. Test the connection first. A key that fails there has cost you nothing.
  2. Start from a probe, not a free-typed prompt, so the result is comparable across models.
  3. Re-run before you quote a rate, and read the pass rate, not one reply.
  4. Run the single-turn control next to any chain result, and report which of the two broke.
  5. Save the prompt and its checks as a regression test, so the next model change is measured instead of argued.

Questions people ask

How do I test whether an LLM API key works?

Paste it into the tester and press Test connection. The tester makes one free models-list call, which proves the endpoint is reachable and the key is accepted and fills the model picker with the ids the provider lists for that key. Providers that expose no models list are reported as reachable, and the first real turn settles it.

Is my API key stored anywhere?

No. It is held in memory for the tab, sent with each request to the gateway, forwarded to the provider and forgotten. The saved conversation is written without it, it is kept out of the logs and never echoed back. Your browser may offer to save the masked field; decline that prompt.

What do the probes test?

Sixty-two starter probes in thirteen groups. Seven groups follow the OWASP Top 10 for LLM Applications, 2026 edition: prompt injection, sensitive information disclosure, excessive agency, unbounded consumption, misinformation, hidden context exposure and improper output handling. The others cover performance, quality, frontier capabilities, manipulation, bias and sensitive data. Fifteen of the probes test 2025–2026 agent and injection attacks with fabricated codes. Each probe states its expected result before you run it.

Can I compare two models on the same prompt?

Yes. Compare sends one question to up to four models side by side and shows where the answers agree, which named things they dispute, and a judge ranking taken twice with the lane order swapped, so position cannot decide it. A probe is never rewritten, so the text each model receives is identical.

How we measured this

  • The caps (100 messages per chat, 5 parallel windows, 4 compare lanes) are the values the tester reads from its own limits endpoint, and the probe and chain counts are the server catalogues as deployed on 2026-10-20. All of them change only on deploy; the tester always shows the live values.
  • Key handling describes the shipped code path: the key travels with each request to the gateway, which forwards it to the provider. The durable conversation record is written without it, and it is excluded from logs and from the reply. Your browser sits outside that boundary, which is why this guide tells you to decline its save prompt.
  • The "what a result does and does not prove" section reproduces the honesty notes the server serves with the probe catalogue, with their meaning unchanged. The OWASP list is cited by edition from OWASP’s own page on the date beside the reference. Corrected 2026-10-28: the log-probability sentence described provider APIs; it now describes what the tester requests, and the 2026 edition of the OWASP list is noted beside the 2025 names the catalogue still uses.

Sources

  1. OWASP Top 10 for LLM Applications, 2025 edition: the ten entries the probe groups follow read
  2. OWASP GenAI LLM Top 10 2026 (August 2026): the current edition, which the catalogue has not yet been renamed to read
  3. Parrish et al., "BBQ: A Hand-Built Bias Benchmark for Question Answering" (2022): ambiguous and disambiguated contexts read

See it on your own project

Every new account starts a 14-day Solo Pro trial, then 14 days on the Free plan. No card. Record one flow and measure the difference on your own numbers.

  • Free plan, no card
  • 14-day Solo Pro trial
  • Demo replies within one business day
  • Data hosted in India