AI-crawler robots check
Crawler robots rules: inspect purpose before changing access
TL;DR
Review observed robots.txt patterns against your intended policy. Training, search and user-request retrieval are different purposes.
What the current check observes
The auditor reads /robots.txt, groups user-agent directives and detects simple root Disallow patterns with equal root Allow handling for a fixed agent list. It also observes a Sitemap directive. These policy observations are advisory.
Context and prerequisites
This is not a complete robots implementation, path-policy verdict or live crawler-access test. CDN, WAF, authentication and rate limiting may determine access separately. A legitimate licensing or privacy restriction is not a defect.
How to review the observation
OpenAI identifies GPTBot as a training crawler and OAI-SearchBot as a search crawler. ChatGPT-User handles user-triggered requests and has different robots behavior. Blocking GPTBot does not by itself prove ChatGPT cannot cite a page. Check the intended user-agent and path policy and actual response separately; preserve deliberate restrictions.
Illustrative example, not a universal prescription
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: /
This illustrative policy separates training and search intentions. It is not a universal recommendation to permit search or block training, and it does not prove a visit or citation.
Measurement and experimental legacy calculation
Public measurement separates Confirmed issue, Advisory, Unknown, Not applicable and Observed pass. An observed pass applies only to an explicit bounded rule; absence of an emitted failure is not a pass. Missing prerequisites do not earn repair credit.
Start 100; transport/access/non-200/non-404/non-410 retrieval returns 40; observed HTML response returns 0; missing/empty returns 100. Deliberate policy patterns and missing Sitemap directives do not deduct.
Legacy values are clamped to 0..100 and rounded. They are not readiness certification, risk indicators or predicted citation outcomes. Method changes do not rewrite historical records.
Full methodology and fixed weightsSources and further manual validation
The observations above describe the local implementation, not guarantees from these sources. Protocol and provider documents help interpret purpose and perform checks this audit does not implement.
What observations does this check return?
LLMScore audits eleven readiness categories. Free, no signup.
Run this check on your site →