Fetch and parse a domain's robots.txt file — returns all crawl rules by user-age
Fetch and parse a domain's robots.txt file — returns all crawl rules by user-agent, sitemap URLs, crawl delay settings, and checks whether a specific path is allowed or blocked for any bot — so agents can respect crawl policies and locate sitemaps before scraping.
Answeringour last check, 2026-09-24
1 of 1checks answered this week
941 msmedian answer time
$0.01listed price per call
$0.01price it asked us
Paid test badge: not yet. The checks above are free: we call the tool without paying and read the payment request it sends back. The Verified badge needs paid calls whose answers match the promised output, and nobody can buy a badge.
Endpoint
GET https://netintel.dev/robots-txt/analyze
| Category | Search and research |
|---|---|
| Provider host | netintel.dev |
| Networks | eip155:8453, solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp |
| Payment schemes | exact |
| Self-reported calls, 30 days | 1 from 1 payers (the provider's figure, not ours) |
Our checks, last 30 days
| Day | Result | HTTP | Asked | Time |
|---|---|---|---|---|
| 2026-09-24 | valid payment request | 402 | $0.01 | 941 ms |
Example input (from the provider)
{
"method": "GET",
"queryParams": {
"domain": "example.com"
},
"type": "http"
}
Promised output schema (from the provider)
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"input": {
"additionalProperties": false,
"properties": {
"method": {
"enum": [
"GET"
],
"type": "string"
},
"queryParams": {
"properties": {
"domain": {
"description": "Domain name to fetch robots.txt from (e.g. example.com)",
"type": "string"
},
"path": {
"description": "URL path to check permission for (e.g. /api/data)",
"type": "string"
},
"user_agent": {
"description": "Bot name to check rules for (default: *)",
"type": "string"
}
},
"required": [
"domain"
],
"type": "object"
},
"type": {
"const": "http",
"type": "string"
}
},
"required": [
"type",
"method"
],
"type": "object"
},
"output": {
"properties": {
"example": {
"properties": {
"domain": {
"type": "string"
},
"findings": {
"items": {
"type": "object"
},
"type": "array"
},
"found": {
"type": "boolean"
},
"grade": {
"type": "string"
},
"path_check": {
"type": "object"
},
"raw_content_preview": {
"type": "string"
},
"robots_url": {
"type": "string"
},
"rules": {
"items": {
"type": "object"
},
"type": "array"
},
"score": {
"type": "number"
},
"sitemaps": {
"items": {
"type": "string"
},
"type": "array"
},
"status_code": {
"type": "number"
}
},
"type": "object"
},
"type": {
"type": "string"
}
},
"required": [
"type"
],
"type": "object"
}
},
"required": [
"input"
],
"type": "object"
}