Discover a website's URLs in one call
Discover a website's URLs in one call: reads robots.txt, its declared sitemap(s) (sitemap indexes and gzipped sitemaps included, /sitemap.xml as the fallback) and the start page's internal links, then returns a same-host, normalized, deduplicated list (up to 500) with an optional substring filter. Hard budgets: at most 6 fetches, 15 seconds, 5 MB. Use it to pick which pages to crawl or extract next.
Answeringour last check, 2026-09-24
1 of 1checks answered this week
1062 msmedian answer time
$0.005listed price per call
$0.005price it asked us
Paid test badge: not yet. The checks above are free: we call the tool without paying and read the payment request it sends back. The Verified badge needs paid calls whose answers match the promised output, and nobody can buy a badge.
Endpoint
POST https://agent402.tools/api/site-map
| Category | Search and research |
|---|---|
| Provider host | agent402.tools |
| Networks | algorand:wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8=, eip155:10, eip155:1329, eip155:137, eip155:143, eip155:42161, eip155:42220, eip155:43114, eip155:4663, eip155:8453, solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp, stellar:pubnet |
| Payment schemes | exact, upto |
| Self-reported calls, 30 days | 4 from 1 payers (the provider's figure, not ours) |
Our checks, last 30 days
| Day | Result | HTTP | Asked | Time |
|---|---|---|---|---|
| 2026-09-24 | valid payment request | 402 | $0.005 | 1062 ms |
Example input (from the provider)
{
"body": {
"limit": 50,
"url": "https://www.iana.org"
},
"bodyType": "json",
"method": "POST",
"type": "http"
}
Promised output schema (from the provider)
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"properties": {
"input": {
"additionalProperties": false,
"properties": {
"body": {
"properties": {
"includeSubdomains": {
"description": "Also keep URLs on subdomains of the start site (default false; www and bare host always count as one site)",
"type": "boolean"
},
"limit": {
"description": "Max URLs to return, 1-500 (default 100)",
"type": "integer"
},
"search": {
"description": "Optional case-insensitive substring filter applied to the discovered URLs",
"type": "string"
},
"url": {
"description": "Start URL (the site's homepage or any page on it)",
"type": "string"
}
},
"required": [
"url"
]
},
"bodyType": {
"enum": [
"json",
"form-data",
"text"
],
"type": "string"
},
"method": {
"enum": [
"POST"
],
"type": "string"
},
"type": {
"const": "http",
"type": "string"
}
},
"required": [
"type",
"method",
"bodyType",
"body"
],
"type": "object"
},
"output": {
"properties": {
"example": {
"properties": {
"fetchedAt": {
"type": "string"
},
"fetches": {
"type": "integer"
},
"host": {
"type": "string"
},
"search": {},
"sitemapsRead": {
"type": "integer"
},
"source": {
"type": "string"
},
"sources": {
"type": "object"
},
"total": {
"type": "integer"
},
"truncated": {
"type": "boolean"
},
"url": {
"type": "string"
},
"urls": {
"type": "array"
},
"warnings": {
"type": "array"
}
},
"required": [
"url",
"host",
"total",
"urls",
"sources",
"sitemapsRead",
"truncated",
"fetches",
"warnings",
"source",
"fetchedAt"
],
"type": "object"
},
"type": {
"type": "string"
}
},
"required": [
"type"
],
"type": "object"
}
},
"required": [
"input"
],
"type": "object"
}