Vendor evaluation checklist for test automation platforms: 30 questions, with our own answers
30 questions a procurement, security or QA lead asks before buying a testing platform, grouped by area, each with why it matters and our own answer. We answer yes to 20, partly to 5 and not yet to 5, and the five we cannot answer yes to are listed first so you do not have to hunt for them. Print it, send it to any vendor, and compare the columns.
No sign-up, no email gate. Send it to every vendor you are comparing.
How to use it
Print this page or save it as a PDF with the button above. The table prints without the site navigation.
Strike out the questions that do not apply to you. Add the ones specific to your regulator or your industry; the last section suggests a few.
Send the question and why-it-matters columns to every vendor you are considering and ask for a plain yes, partly or not yet, with evidence. A vendor who answers only with a brochure has answered.
Compare the columns side by side. A "not yet" with a date beats a "yes" with no evidence.
The five we cannot answer yes to today
A checklist that only shows a vendor's strengths is a brochure. These are our current gaps, stated the same way we would want a vendor to state theirs.
A lawful EU/UK transfer mechanism (executed Standard Contractual Clauses and a transfer impact assessment) is in progress. Until it exists, EU and UK customers who cannot accept a transfer to India should evaluate the on-premise deployment.
SCIM provisioning does not exist. Invitations are manual or through SSO sign-in on Enterprise.
A third-party penetration test of the platform has not been performed. Every release passes our own scanner and gates; that is not the same thing.
We hold neither a SOC 2 report nor an ISO 27001 certificate. The controls are documented and aligned; the badge is not earned.
There is one hosting region (Mumbai). Regional choice does not exist; on-premise is the alternative.
How we scored ourselves
Yes means the capability ships today and the evidence is on this site or available on request. Partly means it ships with a limit that matters, and the limit is named in the note. Not yet means it does not exist, and we say whether it is planned. We do not use "roadmap" as an answer to a yes-or-no question.
Every answer was checked against the shipped product on the date above. When something changes, the date changes with it.
Questions this list does not cover
Your regulator's specifics: PCI DSS scope if card data appears in test flows; HIPAA if test data could contain health records; sector rules for banking or public sector procurement.
Your own architecture: private-network targets, custom identity providers, air-gapped CI.
The people: who answers when a guide is wrong, and how fast. Our answer is the engineers who build each module, within one business day.
Data and hosting
Question
Why it matters
Our answer
Evidence or limit
Where is customer data stored, and in which legal jurisdiction?
Determines which privacy law applies and whether a transfer assessment is needed.
Yes
Mumbai, India. Stated in the privacy policy; DPDPA is the primary regime, GDPR applies to EU users.
Can the platform run inside our own network, on-premise or air-gapped?
The only complete answer to data residency and to targets that are not reachable from a cloud.
Yes
Enterprise plan, feature parity, including air-gapped.
Is there an executed lawful transfer mechanism for EU/UK personal data (SCCs, TIA)?
Without it, an EU or UK customer cannot lawfully send personal data to a non-adequate country.
Not yet
In progress. We will not name a mechanism we have not executed.
Is data encrypted in transit and at rest?
Baseline control on every questionnaire.
Partly
TLS in transit. Stored credentials and session state are encrypted at rest; the database volume relies on the hosting provider's managed storage.
Are backups taken, encrypted, and restore-tested?
A backup that has never been restored is a hypothesis.
Yes
Daily encrypted backups; a weekly automated restore drill checks the restored database against production.
AI and intelligence
Question
Why it matters
Our answer
Evidence or limit
Does all AI processing stay inside the platform, with no external AI provider?
Code, selectors and test data are confidential; a third-party model is a third-party disclosure.
Yes
Local open-source models inside the platform. Nothing is sent to OpenAI, Anthropic, Google or Meta.
Is customer data excluded from model training?
Training on customer data is a use most contracts forbid.
Yes
Inference only. No customer data trains or fine-tunes any model.
Do AI-assisted changes (heals, selections) require a human before they take effect?
A silent auto-repair can pass a broken product.
Yes
Every heal waits in a review queue; selection ships a written rationale and can be overridden.
Are the models and their licences disclosed?
Licence terms flow down to you.
Yes
Qwen (Apache-2.0) primary; Llama 3.2 as a local standby with the required notice on the site.
Identity and access
Question
Why it matters
Our answer
Evidence or limit
Phishing-resistant sign-in (passkeys) and 2FA?
Password-only access to a tool that holds your credentials vault is a single point of failure.
Yes
Passkeys (WebAuthn) and 2FA on every plan.
SSO with SAML or OIDC?
Joiner/leaver control lives in your identity provider.
Partly
Included on the Enterprise plan; not on Free, Solo Pro or Team.
SCIM provisioning and deprovisioning?
Deprovisioning by hand is how ex-employees keep access.
Not yet
Not available. Invitations are manual or via SSO sign-in.
Least-privilege, per-feature permissions, including a separate grant for active security scanning?
The ability to run an active scan is the most sensitive permission in the product.
Yes
Per-feature grants; active scanning is its own grant.
Can admins see active sessions and end them?
Needed for incident response and offboarding.
Yes
Sessions are visible and can be ended; a second-device sign-in ends the first.
Security of the product itself
Question
Why it matters
Our answer
Evidence or limit
Is target ownership verified before any active scan?
Scanning a system you do not own is illegal in most places and useless as evidence.
Yes
DNS TXT, a well-known token file or a meta tag, plus a signed admin override; recorded in a tamper-evident chain. Active phases are fail-closed until ownership is proven.
Is exploitation gated separately from confirmation, with an audit trail?
Confirming a finding and exploiting it are different risks.
Yes
Server-decided per authorisation record; every decision is an audit event.
Is there a vulnerability disclosure policy with a monitored address?
Researchers who cannot reach you go public instead.
Yes
/.well-known/security.txt (RFC 9116).
Has the platform had a third-party penetration test?
Self-testing has blind spots.
Not yet
Planned. Every release passes our own scanner and regression gates.
Are dependencies audited and an SBOM produced in the build?
Supply-chain exposure is now a regulatory topic (EU CRA).
Partly
Dependency audit and SBOM generation run in the pipeline; not every check is a blocking gate yet.
Compliance and evidence
Question
Why it matters
Our answer
Evidence or limit
SOC 2 or ISO 27001 certification?
Many procurement processes require the report.
Not yet
SOC 2-aligned controls are documented; a certification audit is on the roadmap. We do not print the badge.
Tamper-evident, exportable audit log of administrative and security actions?
Who did what, when, provable after the fact.
Yes
Integrity-attested entries; exportable.
Is evidence attached to every confirmed security finding?
A finding without the proving request is an opinion.
Yes
Request, response, proof of concept, cross-reference to the recorded step, remediation.
Is the product's own interface accessible, with a public statement?
The EAA and Section 508 apply to the tools you buy, not only the products you ship.
Partly
Partially conformant to WCAG 2.2 AA; the statement names the known gaps.
Can customers export and delete their data, with retention enforced?
Rights under DPDPA and GDPR must be exercisable, not promised.
Yes
Export and deletion on request; retention periods per data class are enforced in the database.
Operations
Question
Why it matters
Our answer
Evidence or limit
Is there a public status page?
When sign-in fails, your team needs somewhere that does not require signing in.
Yes
/status, standalone, no session required.
Is there a contractual SLA with a stated figure?
A commitment you can enforce, not a marketing number.
Partly
Uptime commitments are Enterprise contract terms; no public SLA figure is advertised.
Can we choose a hosting region?
Residency and latency.
Not yet
One region (Mumbai). On-premise is the alternative.
Commercial and exit
Question
Why it matters
Our answer
Evidence or limit
Can we export our test assets in open formats and leave?
Lock-in is a cost you pay at the worst moment.
Yes
Playwright, Cypress or Selenium code; k6, JMeter or Locust; open API collection formats.
Is pricing published, with quotas enforced and no overage charges?
Surprise invoices end tool relationships.
Yes
Published plans; hard caps enforced in the product; a refused run names the cap.
Is a DPA and a sub-processor list available on request?
Required for any processing of personal data on your behalf.
Yes
On request. Sub-processors today: the hosting provider (Mumbai) and the transactional email provider.
Questions people ask about this
Can we get this as a document?
Use the print button at the top of the page and save as PDF. The print version drops the site navigation and keeps the tables.
Will the answers change?
Yes, and the date at the top changes with them. If an answer here disagrees with what you see in the product, tell us through the contact form and we will correct the page, not the question.
Source: /documentation/vendor-evaluation-checklist, updated 2026-09-17. Answers are checked against the shipped product on that date.
Found something out of date? The people who build the module answer the contact form on the home page, and corrections to this guide are dated above. Tell us.
We use one optional analytics tool, run on our own servers, to see which pages help people. It stays off unless you turn it on. Storage needed to sign you in, keep the site secure and remember this choice is always on. Cookie Policy