Notice to humansThis page is addressed to machines. You may look. Nothing is hidden from you. It is just not for you.
Bot Eyes Only

The proof, such as it is

Crawler report

Which AI crawlers fetched this site, from the server's own request log. Published as measured, zeros included.

Different verified AI crawlers, last 7 days
1
Verified AI crawler fetches, last 7 days (all time: 4)
4
Unverified claims to be a crawler, last 7 days
0

Last 30 days

Fetches per day

2026-10-20 to 2026-11-18. Tallest bar: 7 fetches in a day. Outlined bars are days with unverified claims only.

All pages

By crawler

Training crawlers
CrawlerVerified, 7 daysUnverified, 7 daysVerified, 30 days
GPTBot OpenAI000
ClaudeBot Anthropic404
meta-externalagent Meta · cannot be verified: the vendor publishes no address list000
Amazonbot Amazon000
CCBot Common Crawl000
MistralAI-Training Mistral · cannot be verified: the vendor publishes no address list000
Bytespider ByteDance · cannot be verified: the vendor publishes no address list000
Search index crawlers
CrawlerVerified, 7 daysUnverified, 7 daysVerified, 30 days
OAI-SearchBot OpenAI000
Claude-SearchBot Anthropic000
PerplexityBot Perplexity000
meta-webindexer Meta · cannot be verified: the vendor publishes no address list000
Amzn-SearchBot Amazon000
Google-CloudVertexBot Google000
MistralAI-Index Mistral000
DuckAssistBot DuckDuckGo000
User-initiated fetches
CrawlerVerified, 7 daysUnverified, 7 daysVerified, 30 days
ChatGPT-User OpenAI000
Claude-User Anthropic000
Perplexity-User Perplexity000
meta-externalfetcher Meta · cannot be verified: the vendor publishes no address list000
Amzn-User Amazon000
Google-Agent Google000
Google-GeminiNotebook Google000
Google-NotebookLM Google000
MistralAI-User Mistral000
Other
CrawlerVerified, 7 daysUnverified, 7 daysVerified, 30 days
OAI-AdsBot OpenAI000
General search crawlers whose index also feeds AI answers
CrawlerVerified, 7 daysUnverified, 7 daysVerified, 30 days
Googlebot Google707
bingbot Microsoft000
Applebot Apple202

llms.txt was fetched 0 times in the last 30 days by known crawlers (verified and unverified together). A study of 137,210 domains in 2026 found that 97% of llms.txt files were never requested, so this number is watched with interest and without hope.

Method

How this is counted

Every request for a page is looked at once. If its user agent names a crawler on the list below, the request's address is compared with the address ranges that the crawler's vendor publishes, or checked by reverse DNS with forward confirmation where the vendor documents that. A match counts as verified. No match counts as unverified, and is shown in its own column. Requests from ordinary browsers are not recorded here at all. No address is stored for anyone.

What a panel's own report counts

This page counts every fetch of the site. A panel's report is narrower. When a crawler fetches a page that shows panel texts (the billboard, a panel's own page, llms.txt, the JSON feed), the server notes which texts were on that response. A fetch counts for a panel only if the panel's text was on the page that was sent. So a panel starts at zero when its text goes on display, counts nothing while it is empty, and starts at zero again after a takedown. Each panel's report states the moment its count started.

What could not be verified

  • meta-externalagent (Meta): Crawls for training Meta's models and for indexing. Meta publishes no way to verify it.
  • meta-externalfetcher (Meta): Fetches single links at a user's request. Meta publishes no way to verify it.
  • meta-webindexer (Meta): Indexes pages for Meta AI search. Meta publishes no way to verify it.
  • MistralAI-Training (Mistral): Collects pages for training. Mistral publishes no IP list for it.
  • Bytespider (ByteDance): Widely reported as ByteDance's training crawler. We found no vendor documentation, so it can never be verified here.
  • Anthropic publishes one address list for ClaudeBot, Claude-SearchBot and Claude-User. A match proves the request came from Anthropic; which of the three it was is taken from the user agent.
  • DuckAssistBot shares its address list with DuckDuckGo's search crawler, so the same applies.

Names that cannot appear in a log

  • Google-Extended (Google): Controls whether Googlebot-crawled pages may be used for Gemini training and grounding. It has no user agent of its own.
  • Applebot-Extended (Apple): Controls whether Applebot-crawled pages may be used to train Apple's models. It does not crawl.

These are switches in robots.txt, not crawlers. This site's robots.txt allows them. They are not listed in the tables because they never fetch anything.

Crawlers on the list and where they are documented
CrawlerKindWhat the vendor says it does
GPTBotOpenAITrainingCollects pages for training OpenAI's models. Vendor page
OAI-SearchBotOpenAISearch indexBuilds the index behind ChatGPT search. Vendor page
ChatGPT-UserOpenAIUser-initiatedFetches a page when a ChatGPT user's request needs it. robots.txt rules may not apply. Vendor page
OAI-AdsBotOpenAIOtherChecks pages submitted as ChatGPT ads. Vendor page
ClaudeBotAnthropicTrainingCollects pages for training Anthropic's models. Anthropic publishes one IP list for all its bots. Vendor page
Claude-SearchBotAnthropicSearch indexIndexes pages for Claude's search results. Vendor page
Claude-UserAnthropicUser-initiatedFetches a page when a Claude user asks a question that needs it. Vendor page
PerplexityBotPerplexitySearch indexBuilds Perplexity's search index. Vendor page
Perplexity-UserPerplexityUser-initiatedFetches a page for a Perplexity user's question. Generally ignores robots.txt. Vendor page
meta-externalagentMetaTrainingCrawls for training Meta's models and for indexing. Meta publishes no way to verify it. Vendor page
meta-externalfetcherMetaUser-initiatedFetches single links at a user's request. Meta publishes no way to verify it. Vendor page
meta-webindexerMetaSearch indexIndexes pages for Meta AI search. Meta publishes no way to verify it. Vendor page
AmazonbotAmazonTrainingCrawls to improve Amazon products; may be used to train Amazon's models. Vendor page
Amzn-SearchBotAmazonSearch indexIndexes pages for search in Amazon products such as Alexa. Vendor page
Amzn-UserAmazonUser-initiatedFetches a page live for a user's request. Vendor page
CCBotCommon CrawlTrainingBuilds an open crawl archive that many AI models are trained on. Not a model vendor itself. Vendor page
Google-CloudVertexBotGoogleSearch indexCrawls pages that site owners asked Vertex AI Agents to index. Vendor page
Google-AgentGoogleUser-initiatedA Google agent browsing at a user's request. Vendor page
Google-GeminiNotebookGoogleUser-initiatedFetches sources a user added to a Gemini notebook (formerly Google-NotebookLM). Vendor page
Google-NotebookLMGoogleUser-initiatedThe earlier name of Google-GeminiNotebook. Vendor page
GooglebotGoogleGeneral searchGoogle's general search crawler. Its index also feeds AI Overviews and Gemini's search grounding. Vendor page
bingbotMicrosoftGeneral searchBing's general search crawler. Its index also feeds Copilot. Microsoft documents no separate Copilot fetcher. Vendor page
ApplebotAppleGeneral searchApple's crawler for Siri, Spotlight and Safari. Vendor page
MistralAI-UserMistralUser-initiatedFetches a page for a user's request in Mistral's assistant. Vendor page
MistralAI-IndexMistralSearch indexBuilds Mistral's search index. Vendor page
MistralAI-TrainingMistralTrainingCollects pages for training. Mistral publishes no IP list for it. Vendor page
DuckAssistBotDuckDuckGoSearch indexFetches pages for DuckDuckGo's AI-assisted answers. Vendor page
BytespiderByteDanceTrainingWidely reported as ByteDance's training crawler. We found no vendor documentation, so it can never be verified here. No vendor page found.

Search-grounded checks run this month: 0. They are capped at one per panel per day.