Auto-detect and extract structured data from any URL — articles, products, discussions, images, videos, events, and more.
What it does
Diffbot Analyze is a pay-per-call web data extraction endpoint exposed via the Locus MPP (Micropayment Protocol). It automatically classifies a given URL and extracts structured data appropriate to the page type (article, product, discussion, image, video, list, event, or job posting). You can also force a specific extraction mode or set a fallback if auto-classification fails.
This specific endpoint (`/diffbot/analyze`) is part of a broader Diffbot service suite hosted on the Locus MPP gateway, which also exposes dedicated endpoints for articles, products, discussions, images, videos, events, lists, and job postings. Each call costs 4,200 base units of pathUSD (6 decimals), which equals $0.0042 per request, settled on Tempo L2. The endpoint accepts POST requests with a JSON body containing at minimum a `url` field.
Additional parameters let you control timeout, restrict extraction mode, request extra fields (links, meta, breadcrumb), and toggle discussion/comment extraction. NLP features like entity extraction, sentiment analysis, and summarization are available on the article-specific endpoint. The service is documented at https://docs.diffbot.com and the MPP-specific LLM docs are at https://beta.paywithlocus.com/mpp/diffbot.md.
Capabilities
Use cases
- —Extracting structured article content (title, author, body, date) from news URLs for an AI pipeline
- —Pulling product details (price, availability, specs) from e-commerce pages for price monitoring
- —Auto-detecting page type and extracting relevant structured data without manual configuration
- —Extracting discussion threads and comments from forum pages for sentiment analysis
- —Parsing job postings to populate a job aggregation database
Fit
Best for
- —AI agents that need structured web data on-demand without managing scraping infrastructure
- —Pay-per-use extraction without subscription commitments
- —Automated pipelines that process diverse URL types and need auto-classification
Not for
- —High-volume bulk crawling where per-call pricing becomes expensive
- —Extracting data from pages behind authentication or paywalls that Diffbot cannot access
- —Real-time streaming data needs
Quick start
curl -X POST https://diffbot.mpp.paywithlocus.com/diffbot/analyze \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <MPP_TOKEN>" \
-d '{"url": "https://example.com/article"}'Example
Request
{
"url": "https://techcrunch.com/2024/01/15/example-article",
"mode": "article",
"fields": "links,meta",
"timeout": 30000,
"discussion": true
}Endpoint
Quality
Full OpenAPI schema with clear request bodies and payment info is available. However, the probe did not capture a live 402 challenge (HEAD/GET returned 404; endpoint requires POST), no response schema or example responses are documented, and crawled pages returned only error JSON. Pricing and capabilities are well-specified from the OpenAPI spec.
Warnings
- —Probe returned 404 on HEAD/GET — endpoint likely only responds to POST requests, so liveness could not be confirmed via the probe methods used.
- —No response schema documented in the OpenAPI spec — only '200: Successful response' without body definition.
- —No example responses available from crawl or spec.
Citations
- —The /diffbot/analyze endpoint costs 4200 base units of pathUSD per call, settled via Tempohttps://diffbot.mpp.paywithlocus.com
- —Diffbot API reference is at docs.diffbot.comhttps://docs.diffbot.com
- —MPP-specific LLM documentation at beta.paywithlocus.com/mpp/diffbot.mdhttps://beta.paywithlocus.com/mpp/diffbot.md