How We Made a React SPA Readable by AI Answer Engines Without Adding a Dependency

Our public site is the same Vite and React application as the product. A crawler that does not execute JavaScript requests /home and receives a document whose body is one empty div and a script tag. Googlebot renders JavaScript, eventually and with a budget. Most AI crawlers, as far as their vendors document it, do not. So for the engines that now answer "which tool should I use for X", we did not exist. This article is the fix we shipped and the two things it deliberately does not do.

What we did not do

The textbook answer is a server-rendering framework. We rejected it for this site. The product runs on one eight-vCPU server that also runs CPU-only local inference for the platform's intelligence features; server rendering spends CPU on every page view, and our CPU is already spoken for. Rewriting the frontend of a large codebase for a dozen public pages is also the wrong trade. Our rule is reuse before write, and we already ship a browser.

Prerender at build with the browser we already have

Playwright is a core dependency of the product; the recorder runs on it. The prerender script starts a tiny static server over the built bundle, reads the list of public routes from sitemap.xml, loads each route in headless Chromium, waits for the page to settle, and writes the fully rendered HTML to dist/<route>/index.html. The web server serves that file when the exact path is requested, without a trailing-slash redirect, and the same HTML hydrates into the normal application for a person.

It runs inside the image build. A deploy cannot skip it, and the build fails if it fails.

Fail loud, not empty

A prerender that silently writes an empty shell is worse than no prerender, because it looks done. Each route is checked four ways before its file is written: the visible text must exceed a minimum length; the page must not be the sign-in screen (a public route that renders login means the route is missing from the public allowlist, and this check caught exactly that once); a meta description must exist; and routes that declare structured data must actually contain a JSON-LD block. Any failure fails the build and names the route.

The sitemap is the contract

Routes come from the sitemap, and the sitemap is drift-tested: a blog article or a documentation guide that exists in the data without a sitemap entry fails the test suite, and a guide marked "planned" that sneaks into the sitemap fails it too. Adding a page therefore means adding content, adding one sitemap line, and letting the build do the rest. There is no separate list to forget.

Noindex until the domain is real

Here is the part most guides skip. Our site URL is still a placeholder domain we do not own, because we have not launched publicly yet. A canonical tag or an Organization URL pointing at a domain someone else could register is not a cosmetic bug; it hands ranking signal to a stranger. So while the real URL is unset, every page emits noindex and nofollow, omits canonical and og:url, and drops the URL fields from its structured data. One configuration value flips all of it at once.

That is also the honest answer to "why does your site not rank": it is not reachable from the internet yet, and every page tells crawlers not to index it. Launch is a DNS record, a certificate, one variable, a rebuild, and a sitemap submission. Everything in this article was built so that launch is that short.

robots.txt and llms.txt

We name each crawler we know about and record the decision next to it. Search and answer bots, which fetch pages to cite them, are allowed. Training bots are allowed too, for now, on purpose: before launch, being discoverable matters more than being exclusive, and we can revisit that once we publish research worth protecting. We also note in the file that one commonly misread directive controls training use only, not answer features or ranking.

We ship an llms.txt with plain facts and links. As of our last check no engine has publicly committed to reading it. It costs one small file, so we keep it; we do not count on it.

What this does not solve

• Authority. A crawlable page with no links pointing at it is a crawlable page nobody ranks. Prerendering makes content visible; it does not make it referenced.
• Measurement. We added a script that reads the web server's access log and reports which crawlers fetched which paths, because a discovery channel you cannot observe is a guess. Search Console and Bing Webmaster Tools come after launch.
• Speed. Core Web Vitals depend on the bundle and the fonts, not on the prerender. The prerendered HTML paints fast; the interaction budget is a separate piece of work.
• Quality. This makes our content readable. Whether it is worth citing is a writing problem, and no build step fixes writing.

Steal this

• If you already depend on a browser automation library, you already have a prerenderer.
• Derive the route list from something that is already tested, and test that.
• Make every prerender check fail the build. A warning is a log line nobody reads.
• Never emit a canonical you would not want a stranger to own. Noindex is a safer default than a wrong claim.
• Write down, in the robots file itself, why each bot is allowed or not. The next person will not remember.