Parse a site's robots.txt
Parse a site's robots.txt: whether a user-agent may crawl a path, which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot...) are blocked, and declared sitemaps. Check before your agent scrapes.
Answeringour last check, 2026-10-04
1 of 1checks answered this week
316 msmedian answer time
$0.003listed price per call
$0.003price it asked us
Paid test badge: not yet. The checks above are free: we call the tool without paying and read the payment request it sends back. The Verified badge needs paid calls whose answers match the promised output, and nobody can buy a badge.
Endpoint
POST https://api.vextorium.com/v1/robots-txt
| Category | Search and research |
|---|---|
| Provider host | api.vextorium.com |
| Networks | eip155:10, eip155:5042, eip155:8453, solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp |
| Payment schemes | exact |
| Self-reported calls, 30 days | 1 from 1 payers (the provider's figure, not ours) |
Our checks, last 30 days
| Day | Result | HTTP | Asked | Time |
|---|---|---|---|---|
| 2026-10-04 | valid payment request | 402 | $0.003 | 316 ms |
Example input (from the provider)
{
"body": {
"agent": "GPTBot",
"route": "/section/world",
"url": "https://www.nytimes.com"
},
"bodyType": "json",
"method": "POST",
"type": "http"
}
Promised output schema (from the provider)
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"input": {
"additionalProperties": false,
"properties": {
"body": {
"properties": {
"agent": {
"description": "User-agent to test (optional)",
"type": "string"
},
"route": {
"description": "Path to test (default /)",
"type": "string"
},
"url": {
"type": "string"
}
},
"required": [
"url"
]
},
"bodyType": {
"enum": [
"json",
"form-data",
"text"
],
"type": "string"
},
"method": {
"enum": [
"POST"
],
"type": "string"
},
"type": {
"const": "http",
"type": "string"
}
},
"required": [
"type",
"method",
"bodyType",
"body"
],
"type": "object"
},
"output": {
"properties": {
"example": {
"properties": {
"ai_bots": {
"properties": {
"Amazonbot": {
"type": [
"string",
"null"
]
},
"Applebot-Extended": {
"type": [
"string",
"null"
]
},
"Bytespider": {
"type": [
"string",
"null"
]
},
"CCBot": {
"type": [
"string",
"null"
]
},
"ChatGPT-User": {
"type": [
"string",
"null"
]
},
"Claude-SearchBot": {
"type": [
"string",
"null"
]
},
"Claude-User": {
"type": [
"string",
"null"
]
},
"ClaudeBot": {
"type": [
"string",
"null"
]
},
"Diffbot": {
"type": [
"string",
"null"
]
},
"DuckAssistBot": {
"type": [
"string",
"null"
]
},
"GPTBot": {
"type": [
"string",
"null"
]
},
"Google-Extended": {
"type": [
"string",
"null"
]
},
"MistralAI-User": {
"type": [
"string",
"null"
]
},
"OAI-SearchBot": {
"type": [
"string",
"null"
]
},
"Perplexity-User": {
"type": [
"string",
"null"
]
},
"PerplexityBot": {
"type": [
"string",
"null"
]
},
"YouBot": {
"type": [
"string",
"null"
]
},
"anthropic-ai": {
"type": [
"string",
"null"
]
},
"cohere-ai": {
"type": [
"string",
"null"
]
},
"meta-externalagent": {
"type": [
"string",
"null"
]
}
},
"type": [
"object",
"null"
]
},
"blocked_ai_bots": {
"type": [
"number",
"null"
]
},
"groups": {
"type": [
"number",
"null"
]
},
"query": {
"properties": {
"agent": {
"type": [
"string",
"null"
]
},
"allowed": {
"type": [
"boolean",
"null"
]
},
"applied_group": {
"items": {
"type": [
"null",
"string"
]
},
"type": [
"array",
"null"
]
},
"route": {
"type": [
"string",
"null"
]
}
},
"type": [
"object",
"null"
]
},
"robots_txt": {
"type": [
"boolean",
"null"
]
},
"site": {
"type": [
"string",
"null"
]
},
"sitemaps": {