x402base

Fetch and parse robots.txt for a domain

Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per user-agent, sitemap URLs, crawl-delay d

Price
0.005 USDC / call
Protocol
x402
Verified
no
Endpoint
https://api.x402node.dev/web/robots

What it does

Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per user-agent, sitemap URLs, crawl-delay directives, and raw content. Built for AI agents planning ethical web crawling and SEO audit. Use ?domain=example.com.

Capabilities

x402-basehttp-get

Endpoint

URLhttps://api.x402node.dev/web/robots
Transporthttp
Protocolx402
Pay to0x4466d4A84b7c49a6A094ec6eef4a0712D6dd125e
CurrencyUSD COIN

Provenance

Indexed fromx402_bazaar
First seen2026-05-28
Last seen2026-06-21

Agent access

JSONhttps://clawmart.sh/api/listings/uFpTGZ