The proof, such as it is
Crawler report
Which AI crawlers fetched this site, from the server's own request log. Published as measured, zeros included.
Last 30 days
Fetches per day
2026-10-20 to 2026-11-18. Tallest bar: 7 fetches in a day. Outlined bars are days with unverified claims only.
All pages
By crawler
| Crawler | Verified, 7 days | Unverified, 7 days | Verified, 30 days |
|---|---|---|---|
| GPTBot OpenAI | 0 | 0 | 0 |
| ClaudeBot Anthropic | 4 | 0 | 4 |
| meta-externalagent Meta · cannot be verified: the vendor publishes no address list | 0 | 0 | 0 |
| Amazonbot Amazon | 0 | 0 | 0 |
| CCBot Common Crawl | 0 | 0 | 0 |
| MistralAI-Training Mistral · cannot be verified: the vendor publishes no address list | 0 | 0 | 0 |
| Bytespider ByteDance · cannot be verified: the vendor publishes no address list | 0 | 0 | 0 |
| Crawler | Verified, 7 days | Unverified, 7 days | Verified, 30 days |
|---|---|---|---|
| OAI-SearchBot OpenAI | 0 | 0 | 0 |
| Claude-SearchBot Anthropic | 0 | 0 | 0 |
| PerplexityBot Perplexity | 0 | 0 | 0 |
| meta-webindexer Meta · cannot be verified: the vendor publishes no address list | 0 | 0 | 0 |
| Amzn-SearchBot Amazon | 0 | 0 | 0 |
| Google-CloudVertexBot Google | 0 | 0 | 0 |
| MistralAI-Index Mistral | 0 | 0 | 0 |
| DuckAssistBot DuckDuckGo | 0 | 0 | 0 |
| Crawler | Verified, 7 days | Unverified, 7 days | Verified, 30 days |
|---|---|---|---|
| ChatGPT-User OpenAI | 0 | 0 | 0 |
| Claude-User Anthropic | 0 | 0 | 0 |
| Perplexity-User Perplexity | 0 | 0 | 0 |
| meta-externalfetcher Meta · cannot be verified: the vendor publishes no address list | 0 | 0 | 0 |
| Amzn-User Amazon | 0 | 0 | 0 |
| Google-Agent Google | 0 | 0 | 0 |
| Google-GeminiNotebook Google | 0 | 0 | 0 |
| Google-NotebookLM Google | 0 | 0 | 0 |
| MistralAI-User Mistral | 0 | 0 | 0 |
| Crawler | Verified, 7 days | Unverified, 7 days | Verified, 30 days |
|---|---|---|---|
| OAI-AdsBot OpenAI | 0 | 0 | 0 |
| Crawler | Verified, 7 days | Unverified, 7 days | Verified, 30 days |
|---|---|---|---|
| Googlebot Google | 7 | 0 | 7 |
| bingbot Microsoft | 0 | 0 | 0 |
| Applebot Apple | 2 | 0 | 2 |
llms.txt was fetched 0 times in the last 30 days by known crawlers (verified and unverified together). A study of 137,210 domains in 2026 found that 97% of llms.txt files were never requested, so this number is watched with interest and without hope.
Method
How this is counted
Every request for a page is looked at once. If its user agent names a crawler on the list below, the request's address is compared with the address ranges that the crawler's vendor publishes, or checked by reverse DNS with forward confirmation where the vendor documents that. A match counts as verified. No match counts as unverified, and is shown in its own column. Requests from ordinary browsers are not recorded here at all. No address is stored for anyone.
What a panel's own report counts
This page counts every fetch of the site. A panel's report is narrower. When a crawler fetches a page that shows panel texts (the billboard, a panel's own page, llms.txt, the JSON feed), the server notes which texts were on that response. A fetch counts for a panel only if the panel's text was on the page that was sent. So a panel starts at zero when its text goes on display, counts nothing while it is empty, and starts at zero again after a takedown. Each panel's report states the moment its count started.
What could not be verified
- meta-externalagent (Meta): Crawls for training Meta's models and for indexing. Meta publishes no way to verify it.
- meta-externalfetcher (Meta): Fetches single links at a user's request. Meta publishes no way to verify it.
- meta-webindexer (Meta): Indexes pages for Meta AI search. Meta publishes no way to verify it.
- MistralAI-Training (Mistral): Collects pages for training. Mistral publishes no IP list for it.
- Bytespider (ByteDance): Widely reported as ByteDance's training crawler. We found no vendor documentation, so it can never be verified here.
- Anthropic publishes one address list for ClaudeBot, Claude-SearchBot and Claude-User. A match proves the request came from Anthropic; which of the three it was is taken from the user agent.
- DuckAssistBot shares its address list with DuckDuckGo's search crawler, so the same applies.
Names that cannot appear in a log
- Google-Extended (Google): Controls whether Googlebot-crawled pages may be used for Gemini training and grounding. It has no user agent of its own.
- Applebot-Extended (Apple): Controls whether Applebot-crawled pages may be used to train Apple's models. It does not crawl.
These are switches in robots.txt, not crawlers. This site's robots.txt allows them. They are not listed in the tables because they never fetch anything.
| Crawler | Kind | What the vendor says it does |
|---|---|---|
| GPTBotOpenAI | Training | Collects pages for training OpenAI's models. Vendor page |
| OAI-SearchBotOpenAI | Search index | Builds the index behind ChatGPT search. Vendor page |
| ChatGPT-UserOpenAI | User-initiated | Fetches a page when a ChatGPT user's request needs it. robots.txt rules may not apply. Vendor page |
| OAI-AdsBotOpenAI | Other | Checks pages submitted as ChatGPT ads. Vendor page |
| ClaudeBotAnthropic | Training | Collects pages for training Anthropic's models. Anthropic publishes one IP list for all its bots. Vendor page |
| Claude-SearchBotAnthropic | Search index | Indexes pages for Claude's search results. Vendor page |
| Claude-UserAnthropic | User-initiated | Fetches a page when a Claude user asks a question that needs it. Vendor page |
| PerplexityBotPerplexity | Search index | Builds Perplexity's search index. Vendor page |
| Perplexity-UserPerplexity | User-initiated | Fetches a page for a Perplexity user's question. Generally ignores robots.txt. Vendor page |
| meta-externalagentMeta | Training | Crawls for training Meta's models and for indexing. Meta publishes no way to verify it. Vendor page |
| meta-externalfetcherMeta | User-initiated | Fetches single links at a user's request. Meta publishes no way to verify it. Vendor page |
| meta-webindexerMeta | Search index | Indexes pages for Meta AI search. Meta publishes no way to verify it. Vendor page |
| AmazonbotAmazon | Training | Crawls to improve Amazon products; may be used to train Amazon's models. Vendor page |
| Amzn-SearchBotAmazon | Search index | Indexes pages for search in Amazon products such as Alexa. Vendor page |
| Amzn-UserAmazon | User-initiated | Fetches a page live for a user's request. Vendor page |
| CCBotCommon Crawl | Training | Builds an open crawl archive that many AI models are trained on. Not a model vendor itself. Vendor page |
| Google-CloudVertexBotGoogle | Search index | Crawls pages that site owners asked Vertex AI Agents to index. Vendor page |
| Google-AgentGoogle | User-initiated | A Google agent browsing at a user's request. Vendor page |
| Google-GeminiNotebookGoogle | User-initiated | Fetches sources a user added to a Gemini notebook (formerly Google-NotebookLM). Vendor page |
| Google-NotebookLMGoogle | User-initiated | The earlier name of Google-GeminiNotebook. Vendor page |
| GooglebotGoogle | General search | Google's general search crawler. Its index also feeds AI Overviews and Gemini's search grounding. Vendor page |
| bingbotMicrosoft | General search | Bing's general search crawler. Its index also feeds Copilot. Microsoft documents no separate Copilot fetcher. Vendor page |
| ApplebotApple | General search | Apple's crawler for Siri, Spotlight and Safari. Vendor page |
| MistralAI-UserMistral | User-initiated | Fetches a page for a user's request in Mistral's assistant. Vendor page |
| MistralAI-IndexMistral | Search index | Builds Mistral's search index. Vendor page |
| MistralAI-TrainingMistral | Training | Collects pages for training. Mistral publishes no IP list for it. Vendor page |
| DuckAssistBotDuckDuckGo | Search index | Fetches pages for DuckDuckGo's AI-assisted answers. Vendor page |
| BytespiderByteDance | Training | Widely reported as ByteDance's training crawler. We found no vendor documentation, so it can never be verified here. No vendor page found. |
Search-grounded checks run this month: 0. They are capped at one per panel per day.