Daily snapshot 2026-11-03

2026-11-03 UTC

The numbers come from one website. They describe what these crawlers did here, not across the web.

Period: 2026-11-03T00:00:00Z to 2026-11-04T00:00:00Z (00:00Z to 24:00Z). Generated 2026-11-04T03:18:25.437Z. Schema version 1. Licence CC BY 4.0. Cite as: Benes, Peter. Rattlesnakes By Mail: A Field Guide to the Crawlers of the Web. benesthemenace.com. Method: https://rattlesnakesbymail.com/method

Facts

• F-261103-01 Reflectionbot made 8 requests on 2026-11-03, its highest daily count since the site launched on 2026-09-13; its previous high was 2, on 2026-10-29. Source
• F-261103-02 Googlebot made 153 requests on 2026-11-03, 8.3 times its 28-day median of 19. Source
• F-261103-03 GPTBot made 100 requests on 2026-11-03, 33.3 times its 28-day median of 3. Source
• F-261103-04 Reflectionbot requested a CSV file for the first time on 2026-11-03. Source
• F-261103-05 Reflectionbot requested a JSON file for the first time on 2026-11-03. Source
• F-261103-06 Reflectionbot requested a Markdown file for the first time on 2026-11-03. Source
• F-261103-07 Bingbot first fetched the 2026-11-02 snapshot 4 h 53 min after it was published. Source
• F-261103-08 Googlebot first fetched the 2026-11-02 snapshot 6 h 57 min after it was published. Source
• F-261103-09 OAI-SearchBot first fetched the 2026-11-02 snapshot 15 h 6 min after it was published. Source
• F-261103-10 GPTBot first fetched the 2026-11-02 snapshot 15 h 6 min after it was published. Source
• F-261103-11 Every request on 2026-11-03 from Applebot, GPTBot, ClaudeBot, Bingbot and Reflectionbot was verified against the operator's published method. Source

This selection has expired; it is listed in snapshot.json.

Crawlers

MCP calls

Snapshot pickup

For each of the previous seven snapshots, the first fetch by each verified crawler, in minutes after publication, and the c tag on that fetch.

Open questions

• Can robots.txt cleanly separate "allow this page for search or citation" from "block it for training," and does that distinction hold across every major AI platform? (id 26, asked 5 times)
• How can AI citation performance be measured accurately, given there is no equivalent of a Search Console report for citations? (id 34, asked 5 times)
• Was a specific document used in a model's training, fine-tuning, or evaluation? (id 38, asked 4 times)
• Why wasn't a specific page cited by an AI answer instead of a competitor's page? (id 7, asked 3 times)
• How much do backlinks, brand mentions, structured data, reviews, forum discussions, publisher reputation, and content freshness each matter to being cited? (id 11, asked 3 times)

If you can answer one of these with a source, send a note: POST https://rattlesnakesbymail.com/notes with target question:<id>, where <id> is the number shown with the question, and body=<your answer and its source>.

Ask for something this snapshot lacks

What data about web crawlers did you look for and not find? Search https://rattlesnakesbymail.com/search?q=your+question. Searches with no match are recorded as gaps. A gap is published at /questions only when an editor publishes it; nothing is published automatically. To suggest something this daily snapshot should include, send a note with target snapshot:261103.

To send it, POST https://rattlesnakesbymail.com/notes as a web form or as JSON with target=snapshot:261103 and body=<your note>. The same note can be sent with target_type=snapshot and target_id=261103. The optional author_claim field says who is writing: a person, a model or an agent.

Links

• Daily counts for every crawler
• The crawler records
• Method
• Data
• MCP endpoint (POST JSON-RPC)
• Dataset repository
• All snapshots
• This snapshot as JSON
• This snapshot as Markdown
• This snapshot as CSV
• Facts as JSON
• About this site, for automated readers
• Previous snapshot (2026-11-02)

Next snapshot: 2026-11-04