Cheap robots.txt preflight before you call the Minifetch /extract and /run/seo-p
Cheap robots.txt preflight before you call the Minifetch /extract and /run/seo-page-audit endpoints: returns whether the Minifetch user agent may fetch a target URL (allow/deny) plus its crawl-delay. If blocked or setting a crawl delay read https://minifetch.com/tutorials/unblock-minifetch . Checks the URL as given — does not follow redirects. Supports GET & POST
Answeringour last check, 2026-09-24
1 of 1checks answered this week
920 msmedian answer time
$0.001listed price per call
$0.001price it asked us
Paid test badge: not yet. The checks above are free: we call the tool without paying and read the payment request it sends back. The Verified badge needs paid calls whose answers match the promised output, and nobody can buy a badge.
Endpoint
GET https://minifetch.com/api/v1/x402/preflight/url-check
| Category | Code and developer |
|---|---|
| Provider host | minifetch.com |
| Networks | eip155:8453, solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp |
| Payment schemes | exact |
| Self-reported calls, 30 days | 25 from 13 payers (the provider's figure, not ours) |
Our checks, last 30 days
| Day | Result | HTTP | Asked | Time |
|---|---|---|---|---|
| 2026-09-24 | valid payment request | 402 | $0.001 | 920 ms |
Example input (from the provider)
{
"method": "GET",
"queryParams": {
"url": "https://github.com"
},
"type": "http"
}
Promised output schema (from the provider)
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"input": {
"additionalProperties": false,
"properties": {
"method": {
"enum": [
"GET"
],
"type": "string"
},
"queryParams": {
"properties": {
"fresh": {
"description": "Optional. If true, bypasses the 24h robots.txt cache and re-fetches robots.txt for this check. Use after editing your robots.txt or crawl-delay (or adding/removing a Minifetch block) to confirm the change took effect.",
"type": "boolean"
},
"url": {
"description": "The target URL to check against robots.txt",
"type": "string"
}
},
"required": [
"url"
],
"type": "object"
},
"type": {
"const": "http",
"type": "string"
}
},
"required": [
"type",
"method"
],
"type": "object"
},
"output": {
"properties": {
"example": {
"properties": {
"results": {
"items": {
"properties": {
"data": {
"properties": {
"allowed": {
"description": "Whether the minifetch user agent may fetch this URL per robots.txt",
"type": "boolean"
},
"crawlDelay": {
"description": "Crawl-delay (seconds) for the minifetch UA; omitted when blocked or the domain is invalid",
"type": "number"
},
"message": {
"description": "e.g. 'Access allowed by robots.txt', 'URL is blocked by robots.txt', or 'robots.txt not found, defaulting to allowed'",
"type": "string"
},
"minifetchCache": {
"description": "robots.txt cache state (~24h TTL; pass fresh=true to bypass)",
"properties": {
"cachedAt": {
"type": "string"
},
"expiresAt": {
"type": "string"
},
"hit": {
"type": "boolean"
}
},
"type": "object"
},
"proxy": {
"description": "Fetch provenance \u2014 always minifetch UA here",
"properties": {
"proxyType": {
"enum": [
"minifetch"
],
"type": "string"
},
"userAgent": {
"type": "string"
}
},
"type": "object"
},
"url": {
"description": "The URL checked, exactly as given \u2014 redirects are not followed",
"type": "string"
}
},
"required": [
"url",
"allowed",
"message",
"proxy"
],
"type": "object"
}
},
"type": "object"
},
"type": "array"
},
"success": {
"type": "boolean"
},
"tip": {
"description": "Present only when allowed=false: how site owners can unblock the minifetch user agent",
"properties": {
"message": {
"type": "string"
},
"url": {
"type": "string"
}
},
"type": "object"
}
},
"type": "object"
},
"type": {
"type": "string"
}
},
"required": [
"type"
],
"type": "object"
}
},
"required": [
"input"
],
"type": "object"
}