Fetch and parse a public site's robots.txt into crawl directives and sitemap hin
Fetch and parse a public site's robots.txt into crawl directives and sitemap hints. $0.003 USDC per call on Base.
Answeringour last check, 2026-10-24
1 of 1checks answered this week
27 msmedian answer time
$0.003listed price per call
$0.003price it asked us
Paid test badge: not yet. The checks above are free: we call the tool without paying and read the payment request it sends back. The Verified badge needs paid calls whose answers match the promised output, and nobody can buy a badge.
Endpoint
POST https://api.avedaris.com/v1/paid/robots-inspect
| Category | Search and research |
|---|---|
| Provider host | api.avedaris.com |
| Networks | eip155:8453 |
| Payment schemes | exact |
| Self-reported calls, 30 days | 1 from 1 payers (the provider's figure, not ours) |
Our checks, last 30 days
| Day | Result | HTTP | Asked | Time |
|---|---|---|---|---|
| 2026-10-24 | valid payment request | 402 | $0.003 | 27 ms |
Example input (from the provider)
{
"body": {
"url": "https://example.com/"
},
"bodyType": "json",
"method": "POST",
"type": "http"
}
Promised output schema (from the provider)
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"input": {
"additionalProperties": false,
"properties": {
"body": {
"additionalProperties": false,
"properties": {
"url": {
"description": "Public http(s) URL on the site to check; the site's /robots.txt is fetched and parsed into crawler groups and sitemaps. 8 to 8,192 characters.",
"maxLength": 8192,
"minLength": 8,
"type": "string"
}
},
"required": [
"url"
],
"type": "object"
},
"bodyType": {
"enum": [
"json",
"form-data",
"text"
],
"type": "string"
},
"method": {
"enum": [
"POST"
],
"type": "string"
},
"type": {
"const": "http",
"type": "string"
}
},
"required": [
"type",
"method",
"bodyType",
"body"
],
"type": "object"
},
"output": {
"properties": {
"example": {
"properties": {
"result": {
"description": "Output of robots.inspect. The fields below are the documented core; some responses add further detail fields.",
"properties": {
"groups": {
"description": "Parsed rule groups, each with userAgents, allow and disallow path lists. Empty when no robots.txt.",
"type": "array"
},
"robotsUrl": {
"description": "The robots.txt URL that was fetched (site origin + /robots.txt).",
"type": "string"
},
"sitemaps": {
"description": "Sitemap URLs declared in robots.txt.",
"type": "array"
},
"status": {
"description": "HTTP status of the robots.txt fetch; 404 means the site has none.",
"type": "integer"
}
},
"required": [
"robotsUrl",
"status",
"groups",
"sitemaps"
],
"type": "object"
}
},
"required": [
"result"
],
"type": "object"
},
"type": {
"type": "string"
}
},
"required": [
"type"
],
"type": "object"
}
},
"required": [
"input"
],
"type": "object"
}