Fetch a site's robots.txt and evaluate crawl permissions
Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule that matched, the user-agent group, the crawl-delay and any declared sitemaps. Implements the Robots Exclusion Protocol (RFC 9309) with longest-match-wins, Allow-over-Disallow tie-breaking and * / $ wildcards.
Answeringour last check, 2026-10-04
1 of 1checks answered this week
449 msmedian answer time
$0.006listed price per call
$0.006price it asked us
Paid test badge: not yet. The checks above are free: we call the tool without paying and read the payment request it sends back. The Verified badge needs paid calls whose answers match the promised output, and nobody can buy a badge.
Endpoint
POST https://web.openverbs.com/v1/robots
| Category | Search and research |
|---|---|
| Provider host | web.openverbs.com |
| Networks | eip155:8453 |
| Payment schemes | exact |
| Self-reported calls, 30 days | 0 from 0 payers (the provider's figure, not ours) |
Our checks, last 30 days
| Day | Result | HTTP | Asked | Time |
|---|---|---|---|---|
| 2026-10-04 | valid payment request | 402 | $0.006 | 449 ms |
Example input (from the provider)
{
"body": {
"paths": [
"example"
],
"url": "https://example.com",
"userAgent": "example"
},
"bodyType": "json",
"method": "POST",
"type": "http"
}
Promised output schema (from the provider)
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"input": {
"additionalProperties": false,
"properties": {
"body": {
"additionalProperties": false,
"properties": {
"paths": {
"description": "Optional extra paths (or same-origin URLs) to check in the same call, e.g. [\"/admin\", \"/search?q=x\"]. Up to 50.",
"items": {
"maxLength": 2048,
"minLength": 1,
"type": "string"
},
"maxItems": 50,
"type": "array"
},
"url": {
"description": "Public http(s) URL to check. Its origin's /robots.txt is fetched and its path is the first path evaluated.",
"format": "uri",
"maxLength": 2048,
"type": "string"
},
"userAgent": {
"description": "User-agent token to evaluate rules for, e.g. \"Googlebot\". Defaults to \"*\".",
"maxLength": 200,
"minLength": 1,
"type": "string"
}
},
"required": [
"url"
],
"type": "object"
},
"bodyType": {
"enum": [
"json",
"form-data",
"text"
],
"type": "string"
},
"method": {
"enum": [
"POST"
],
"type": "string"
},
"type": {
"const": "http",
"type": "string"
}
},
"required": [
"type",
"method",
"bodyType",
"body"
],
"type": "object"
}
},
"required": [
"input"
],
"type": "object"
}