# Eight Days of Verified Crawler Logs on a New SaaS Site: OpenAI’s Bots Outread Googlebot

Dokimore Engineering Team · October 28, 2026, updated 2026-11-01 · 8 min read

Format: Original research · Written for: Developers who own their tests, Engineering managers · Where it applies: every market

> Five hundred and forty-four crawler requests, every source address checked against the vendor’s published ranges where the vendor publishes them, on a domain that had been live for less than three weeks. Here is who read what.

## In short

- On a domain a few weeks old, the training crawler read almost everything first: GPTBot fetched 124 distinct pages on the day it arrived and 129 over the eight days, against a sitemap of 126 to 132 URLs, while Googlebot read 72 in the whole period. OpenAI’s search crawler read another 65.
- The traffic that stands for a reader is single digits: five page fetches by ChatGPT-User and three by Claude-User in eight days. Training traffic is loud and reader traffic is quiet, which is the shape Cloudflare measures across its network.
- A crawler’s name is a claim. We count a request only after its source address is checked against the vendor’s published ranges; the day the ledger started, "Googlebot" was fetching /.env.old and /.git/config from addresses Google does not own. YandexBot and Claude-User are recorded as claims, because the script loads no published ranges for them.

## What was measured

Every ten minutes a script reads the last twenty minutes of the web server’s access log and appends one row per crawler request to a ledger that outlives the container the log lives in, because every deploy recreates that container. The rows below cover 20 October 2026 at 15:31 UTC to 28 October at 04:43 UTC: 544 requests. Googlebot fetched the sitemap for the first time on 22 October at 12:07 UTC, 47 seconds after a Search Console inspection request, so the period is the site’s first week of being indexed on purpose.

A row is written as verified only when the request’s source address falls inside the ranges the vendor publishes for that crawler: Google’s, Bing’s, OpenAI’s three lists for GPTBot, OAI-SearchBot and ChatGPT-User, Perplexity’s two and Apple’s. A crawler name from an address outside its vendor’s ranges is not that crawler and is not written. Crawlers whose ranges the script does not load, YandexBot and Claude-User among them, are recorded as claims and reported separately. No address and no query string is ever stored, and a path is stored only when it is in the sitemap or is one of the crawler files; anything else is written as off-sitemap, because a request path is chosen by whoever sends it.

## Who read what

Verified by address where the vendor publishes ranges, for the eight days; the two crawlers whose ranges the script does not load are recorded as claims.

*Crawler requests on dokimore.com, 20 to 28 October 2026, from the verified-crawler ledger*

| Crawler | Requests | Distinct sitemap pages | Sitemap fetches | Verified by address |
| --- | --- | --- | --- | --- |
| GPTBot | 152 | 129 | 5 | Yes |
| Googlebot | 102 | 72 | 2 | Yes |
| YandexBot | 98 | not stated | not stated | No, a claim |
| OAI-SearchBot | 80 | 65 | 0 | Yes |
| Bingbot | 57 | 31 | 3 | Yes |
| GoogleOther | 40 | 38 | not stated | Yes |
| ChatGPT-User | 5 | 5 | not stated | Yes |
| Applebot | 4 | 3 | not stated | Yes |
| Claude-User | 3 | not stated | not stated | No, a claim |
| PerplexityBot | 2 | 2 | not stated | Yes |
| Google-InspectionTool | 1 | not stated | not stated | Yes |

The shape says more than the totals. GPTBot arrived on 23 October at 01:53 UTC, fetched the sitemap less than three minutes later, and made 141 requests over 124 distinct pages that day. Since then it has come back for the sitemap at about 02:00 UTC every morning and been told each time, with a 304, that nothing changed. Googlebot read the sitemap and the hub pages first, then the pages the sitemap listed, a few dozen a day. OAI-SearchBot, which OpenAI documents as the crawler for its search features rather than for training, read 65 pages and never fetched the sitemap. Bingbot was the steadiest presence, 57 requests spread across the week.

Of the 544 requests, 350 answered 200 and 189 answered 304, the response that tells a crawler a page has not changed since its last visit; four were redirects and one a 403. A healthy crawl of a static site looks like that: a first full read, then conditional requests that cost almost nothing.

## The eight that stand for a reader

ChatGPT-User and Claude-User are not crawlers in the usual sense. OpenAI documents ChatGPT-User as the agent that visits a page when a person’s conversation needs it, and Anthropic describes Claude-User the same way. PerplexityBot is not in this group: Perplexity documents it as the crawler that surfaces and links websites in its search results, not used for training, and keeps a separate Perplexity-User agent for a person’s question, which did not appear in these eight days.

Eight fetches for a person, five of them address-verified, against 232 requests from OpenAI’s two crawlers alone, is the ratio a new site should expect: the models read everything to learn and fetch a specific page for a person rarely. Cloudflare’s network-wide figures put it starkly. Over the twelve months to July 2025, 80% of AI crawling was for training, 18% for search and 2% for user actions, and in July 2025 Anthropic crawled about 38,000 pages for every visit it referred, OpenAI about 1,100, Perplexity about 195 and Google about 5. Our eight fetches are the user-action slice of that chart, on one small site.

## What it changed in how we work

- We stopped reading the user-agent column as evidence. The presence board counts only address-verified requests, and the scanners sending Googlebot’s name to /.env.old are reported as what they are.
- We stopped treating training-crawler volume as a signal of interest. GPTBot reading 124 pages in a day is the default behaviour of a training crawler meeting a new sitemap, not a reader.
- We kept the crawlers in. Our robots policy allows the search and answer-engine crawlers and the user agents, because eight fetches for a person are eight readers we would not otherwise have had, and a page an answer engine cannot read cannot be cited by it.
- We watch the 304s. A rising share of conditional requests means the crawlers have what they need and are checking for change, which is the cheap state a small site wants to be in.

## What this is not

Eight days, one site, 544 rows. The numbers describe one new domain in its first week of being indexed and nothing else; a site with many inbound links, a news site or a store will see a different mix. Verification depends on the vendors’ range files staying published and on the edge handing the visitor’s own address to the server, both of which are checked and both of which can break; the ledger says "unchecked" for a vendor whose ranges failed to load rather than guessing. We will publish the thirty-day figures when there are thirty days.

## What to do next

1. Verify a crawler by its source address against the vendor’s published ranges before you count it; a user-agent string is a claim.
2. Separate training crawlers, search crawlers and fetches made for a person in your own ledger; they answer three different questions.
3. Decide per crawler, in the robots file itself, and write the reason beside it; blocking a search crawler removes you from answers, blocking a training crawler does not.
4. Watch the share of 304 responses: a rising share means the crawlers have what they need and are only checking for change.
5. Store no addresses and no query strings, and store a path only when it is in the sitemap; everything else is chosen by whoever sends it.

## Questions people ask

### How do you verify that a crawler is really Googlebot or GPTBot?

By the source address, never by the name. Google, Bing, OpenAI, Perplexity and Apple publish the address ranges their crawlers use; a request is counted for a crawler only when its address falls inside that vendor’s published ranges, which the script re-fetches and validates daily. A crawler name from any other address is not written to the ledger at all.

### Which AI crawlers visited a new site first?

On dokimore.com, Googlebot and Bingbot were seen on 20 October, the day the ledger started, OAI-SearchBot on 21 October, Claude-User on 22 October and GPTBot on 23 October, when it fetched the sitemap and read 124 pages in a day. Applebot, ChatGPT-User and PerplexityBot followed later in the week.

### What is the difference between GPTBot and OAI-SearchBot?

OpenAI documents GPTBot as the crawler whose content may be used to train its models, and OAI-SearchBot as the crawler for its search features, which is why each has its own address ranges and its own robots directive. On this site GPTBot made 152 requests over 129 pages and OAI-SearchBot 80 over 65, and only GPTBot fetched the sitemap.

### Is PerplexityBot a training crawler?

No. Perplexity documents PerplexityBot as the crawler that surfaces and links websites in its search results and says it is not used to crawl content for AI foundation models. Its separate Perplexity-User agent fetches a page when a person asks a question. Both have published address ranges.

### Should a new SaaS site block AI crawlers?

That is a policy choice, and ours is to allow them. The training crawlers cost us a few hundred requests to a static site; the user agents fetched pages for eight people in eight days who would not otherwise have found them, and a page an answer engine cannot read cannot be cited by it.

## How we measured this

- Counts are from our crawler ledger as read on 28 October 2026: 544 rows from 20 October 15:31 UTC to 28 October 04:43 UTC, summarised by the ledger script itself. The script reads the last twenty minutes of the access log every ten minutes and de-duplicates overlapping windows against every ledger row back to an hour before the window.
- A request is attributed to a crawler only when its source address is inside that vendor’s published ranges; the edge resolves the visitor’s own address and hands it to the server, which was checked on 20 October 2026. Rendering fetches (scripts, styles, images) are not counted. "Distinct pages" are distinct sitemap paths answered 200 or 304.
- The sitemap held 132 URLs when an outside review counted them on 25 October and 126 before the 28 October release; it changed with two deploys during the period, so "almost the whole sitemap" means 129 distinct pages against that range.
- Cloudflare’s figures are quoted from its own post as read on the date beside the reference; they describe its network in 2025, not this site.

## Sources

1. [Cloudflare, "The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals" (29 August 2025): 80% of AI crawling for training over twelve months; July 2025 crawl-to-refer ratios for Anthropic, OpenAI, Perplexity and Google](https://blog.cloudflare.com/crawlers-click-ai-bots-training/) (read 2026-10-28)
2. [OpenAI, "Overview of OpenAI crawlers": GPTBot, OAI-SearchBot and ChatGPT-User, each with published address ranges](https://developers.openai.com/api/docs/bots) (read 2026-10-27)
3. [Perplexity documentation, its crawlers: PerplexityBot for search results and not for training; Perplexity-User for user actions; published address ranges for both](https://docs.perplexity.ai/guides/bots) (read 2026-10-28)
4. [Anthropic, crawler documentation: ClaudeBot, Claude-User and Claude-SearchBot](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (read 2026-10-27)
5. [Google Search Central, "Verify requests from Google crawlers and fetchers": the published ranges for common crawlers, special-case crawlers and user-triggered fetchers](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot) (read 2026-10-28)

Source: https://dokimore.com/blog/eight-days-of-verified-crawler-logs-on-a-new-saas-site
