Can I fetch this URL?
Can I fetch this URL? robots.txt + AI-crawler permission gate: Pre-fetch gate for agents and scrapers: is this exact URL allowed by robots.txt (RFC 9309: missing = allow, 5xx/unreachable = disallow) for generic agents and 16 named AI agents/crawlers (ChatGPT-User, Claude-User, PerplexityBot, GPTBot,
Answeringour last check, 2026-10-26
1 of 1checks answered this week
313 msmedian answer time
$0.01listed price per call
$0.01price it asked us
Paid test badge: not yet. The checks above are free: we call the tool without paying and read the payment request it sends back. The Verified badge needs paid calls whose answers match the promised output, and nobody can buy a badge.
Endpoint
GET https://shop.lordevo.com/api/quick/can-fetch
| Category | Search and research |
|---|---|
| Provider host | shop.lordevo.com |
| Networks | eip155:8453 |
| Payment schemes | exact |
| Self-reported calls, 30 days | 0 from 0 payers (the provider's figure, not ours) |
Our checks, last 30 days
| Day | Result | HTTP | Asked | Time |
|---|---|---|---|---|
| 2026-10-26 | valid payment request | 402 | $0.01 | 313 ms |
Example input (from the provider)
{
"method": "GET",
"queryParams": {
"url": "https://example.com"
},
"type": "http"
}
Promised output schema (from the provider)
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"input": {
"additionalProperties": false,
"properties": {
"method": {
"enum": [
"GET"
],
"type": "string"
},
"queryParams": {
"properties": {
"url": {
"description": "public http(s) URL or bare domain",
"type": "string"
}
},
"required": [
"url"
],
"type": "object"
},
"type": {
"const": "http",
"type": "string"
}
},
"required": [
"type",
"method"
],
"type": "object"
},
"output": {
"properties": {
"example": {
"properties": {
"agents": {
"additionalProperties": {
"type": "boolean"
},
"type": "object"
},
"allowed": {
"type": "boolean"
},
"content_signal": {
"type": [
"object",
"null"
]
},
"crawl_delay_seconds": {
"type": [
"number",
"null"
]
},
"noai": {
"type": "boolean"
},
"origin": {
"type": "string"
},
"page": {
"type": [
"object",
"null"
]
},
"path": {
"type": "string"
},
"robots_http_status": {
"type": [
"integer",
"null"
]
},
"robots_txt": {
"enum": [
"present",
"missing",
"unreachable"
],
"type": "string"
},
"sitemaps": {
"items": {
"type": "string"
},
"type": "array"
},
"summary": {
"type": "string"
},
"tool": {
"type": "string"
},
"url": {
"type": "string"
},
"verdict": {
"enum": [
"allowed",
"disallowed",
"allowed_but_no_ai_use",
"allowed_with_crawl_delay"
],
"type": "string"
}
},
"required": [
"tool",
"url",
"verdict",
"allowed",
"agents",
"robots_txt"
],
"type": "object"
},
"type": {
"type": "string"
}
},
"required": [
"type"
],
"type": "object"
}
},
"required": [
"input"
],
"type": "object"
}