Extract and crawl content from URLs via Exa, paid per-call with USDC on Base.
What it does
This endpoint, hosted on Toolkit.dev (staging), wraps the Exa crawling API behind an x402 paywall. You send a JSON body containing an array of URLs, and the service returns extracted content for each URL. Payment is handled via the x402 protocol: callers must include an X-PAYMENT header settling in USDC on the Base network.
The maximum cost per request is $0.05 (50,000 base units of USDC with 6 decimals). The endpoint accepts POST requests with a JSON body containing a single required field `urls` (an array). The response returns an object with a `results` array. The timeout is up to 300 seconds, accommodating potentially slow crawls of multiple pages.
Toolkit.dev is an open-source project that packages various LLM tools (including Exa search/crawl, GitHub, Notion, Google Calendar, etc.) as payable API endpoints and MCP servers. This particular endpoint focuses on Exa's crawling capability — fetching and extracting readable content from specified web pages. Note that this is a staging deployment, so availability and stability may differ from a production environment.
Capabilities
Use cases
- —Extracting article text from a list of URLs for summarization pipelines
- —Feeding crawled web content into LLM context for research agents
- —Building datasets from specific web pages without managing scraping infrastructure
- —Programmatic content retrieval for RAG (retrieval-augmented generation) workflows
Fit
Best for
- —AI agents that need to read arbitrary web pages on demand
- —Developers wanting a pay-per-call web crawling API without subscriptions
- —Workflows requiring structured content extraction from known URLs
Not for
- —Large-scale site-wide crawling (batch of thousands of URLs per call)
- —Real-time search or discovery of URLs — this crawls known URLs, not a search engine
- —Production-critical workloads (this is a staging deployment)
Quick start
curl -X POST https://staging.toolkit.dev/api/tool/exa/crawling \
-H "Content-Type: application/json" \
-H "X-PAYMENT: <x402-payment-token>" \
-d '{"urls": ["https://example.com"]}'Example
Request
{
"urls": [
"https://example.com",
"https://en.wikipedia.org/wiki/Web_scraping"
]
}Response
{
"results": [
{
"url": "https://example.com",
"content": "This domain is for use in illustrative examples in documents..."
},
{
"url": "https://en.wikipedia.org/wiki/Web_scraping",
"content": "Web scraping is data scraping used for extracting data from websites..."
}
]
}Endpoint
Quality
The x402 challenge is live and provides a clear outputSchema with input/output fields, pricing, and network details. However, there is no OpenAPI spec, no detailed documentation (docs/pricing/API pages all require login), and the response schema's results array items are not specified. The endpoint is on a staging domain, adding uncertainty about long-term availability.
Warnings
- —Staging deployment — endpoint stability and availability are not guaranteed.
- —No detailed API documentation available (docs pages require authentication).
- —Response schema only specifies a 'results' array without defining item structure; the example response is inferred.
- —No OpenAPI or formal schema beyond the x402 outputSchema.
Citations
- —The endpoint returns a 402 challenge with maxAmountRequired of 50000 in USDC (asset 0x8335…) on Base networkhttps://staging.toolkit.dev/api/tool/exa/crawling
- —Toolkit.dev is an open-source project that pays contributors for merged PRs and packages LLM toolshttps://staging.toolkit.dev
- —The x402 outputSchema specifies a POST with a required 'urls' array field and returns an object with a 'results' arrayhttps://staging.toolkit.dev/api/tool/exa/crawling