Checks whether a URL's path may be crawled under robots.txt (RFC 9309)
Checks whether a URL's path may be crawled under robots.txt (RFC 9309): the verdict for a user agent (default: *, Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot), the rule that decided it and its line, crawl-delay, sitemaps and the matching group's lines. Longest rule wins, allow wins a tie, * and $ work. No robots.txt means everything is allowed; a server error is a 503, not charged.
Answeringour last check, 2026-10-18
1 of 1checks answered this week
38 msmedian answer time
$0.001listed price per call
$0.001price it asked us
Paid test badge: not yet. The checks above are free: we call the tool without paying and read the payment request it sends back. The Verified badge needs paid calls whose answers match the promised output, and nobody can buy a badge.
Endpoint
GET https://api.sitecheck-api.workers.dev/api/robots
| Category | Search and research |
|---|---|
| Provider host | api.sitecheck-api.workers.dev |
| Networks | eip155:5042, eip155:8453, solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp |
| Payment schemes | exact |
| Self-reported calls, 30 days | 2 from 2 payers (the provider's figure, not ours) |
Our checks, last 30 days
| Day | Result | HTTP | Asked | Time |
|---|---|---|---|---|
| 2026-10-18 | valid payment request | 402 | $0.001 | 38 ms |
Example input (from the provider)
{
"method": "GET",
"queryParams": {
"url": "https://www.google.com/search?q=x"
},
"type": "http"
}
Promised output schema (from the provider)
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"input": {
"additionalProperties": false,
"properties": {
"method": {
"enum": [
"GET"
],
"type": "string"
},
"queryParams": {
"properties": {
"url": {
"description": "A page URL on the site whose robots.txt is checked",
"type": "string"
},
"userAgent": {
"description": "One crawler name to check, e.g. Googlebot (optional; default lists the common crawlers)",
"type": "string"
}
},
"required": [
"url"
],
"type": "object"
},
"type": {
"const": "http",
"type": "string"
}
},
"required": [
"type",
"method"
],
"type": "object"
},
"output": {
"properties": {
"example": {
"type": "object"
},
"type": {
"type": "string"
}
},
"required": [
"type"
],
"type": "object"
}
},
"required": [
"input"
],
"type": "object"
}