sitepeek (Tanod) robots.txt check
sitepeek (Tanod) robots.txt check: whether a crawler may fetch a URL under RFC 9309, with the matching rule, crawl delay and sitemaps. Read-only.
Answeringour last check, 2026-10-10
1 of 1checks answered this week
217 msmedian answer time
$0.001listed price per call
$0.001price it asked us
Paid test badge: not yet. The checks above are free: we call the tool without paying and read the payment request it sends back. The Verified badge needs paid calls whose answers match the promised output, and nobody can buy a badge.
Endpoint
POST https://tanod.dev/v1/robots
| Category | Search and research |
|---|---|
| Provider host | tanod.dev |
| Networks | eip155:137, eip155:8453 |
| Payment schemes | exact |
| Self-reported calls, 30 days | 2 from 2 payers (the provider's figure, not ours) |
Our checks, last 30 days
| Day | Result | HTTP | Asked | Time |
|---|---|---|---|---|
| 2026-10-10 | valid payment request | 402 | $0.001 | 217 ms |
Example input (from the provider)
{
"body": {
"url": "https://www.google.com/search",
"user_agent": "Googlebot"
},
"bodyType": "json",
"method": "POST",
"type": "http"
}
Promised output schema (from the provider)
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"input": {
"additionalProperties": false,
"properties": {
"body": {
"additionalProperties": false,
"properties": {
"url": {
"description": "Public http(s) URL to test; <origin>/robots.txt is fetched and the URL's path is matched against it.",
"maxLength": 2048,
"minLength": 1,
"type": "string"
},
"user_agent": {
"default": "*",
"description": "Product token (e.g. Googlebot) or a full User-Agent string; '*' (default) = any crawler.",
"maxLength": 200,
"minLength": 1,
"type": "string"
}
},
"required": [
"url"
],
"type": "object"
},
"bodyType": {
"enum": [
"json",
"form-data",
"text"
],
"type": "string"
},
"method": {
"enum": [
"POST"
],
"type": "string"
},
"type": {
"const": "http",
"type": "string"
}
},
"required": [
"type",
"method",
"bodyType",
"body"
],
"type": "object"
},
"output": {
"properties": {
"example": {
"properties": {
"allowed": {
"type": "boolean"
},
"crawl_delay": {
"type": [
"number",
"string",
"null"
]
},
"fetch": {
"enum": [
"ok",
"unavailable",
"unreachable"
],
"type": "string"
},
"group": {
"type": [
"string",
"null"
]
},
"http_status": {
"type": [
"integer",
"null"
]
},
"matched_rule": {
"properties": {
"line": {
"type": "integer"
},
"pattern": {
"type": "string"
},
"type": {
"enum": [
"allow",
"disallow"
],
"type": "string"
}
},
"type": [
"object",
"null"
]
},
"path": {
"type": "string"
},
"product_token": {
"type": "string"
},
"reason": {
"type": "string"
},
"robots_url": {
"type": [
"string",
"null"
]
},
"sitemaps": {
"items": {
"type": "string"
},
"type": "array"
},
"sitemaps_total": {
"type": "integer"
},
"stats": {
"type": [
"object",
"null"
]
},
"truncated": {
"type": "boolean"
},
"untrusted_content": {
"type": "boolean"
},
"url": {
"type": "string"
},
"user_agent": {
"type": "string"
}
},
"required": [
"url",
"robots_url",
"user_agent",
"product_token",
"path",
"http_status",
"fetch",
"allowed",
"matched_rule",
"group",
"crawl_delay",
"sitemaps",
"sitemaps_total",
"truncated",
"stats",
"reason",
"untrusted_content"
],
"type": "object"
},
"type": {
"type": "string"
}
},
"required": [
"type"
],
"type": "object"
}
},
"required": [
"input"
],
"type": "object"
}