{"id":"252e68fe-5700-423b-a737-682808e78ea5","shortId":"cGJNBF","kind":"skill","title":"fetch-url-as-markdown","tagline":"Fetch a web page (URL) and return clean Markdown via local trafilatura, with Exa MCP as a fallback for JS-rendered or anti-bot pages. Use when the user asks to read, fetch, scrape, summarize, or quote a URL — prefer this over the built-in WebFetch tool. Don't use for binary files","description":"# URL to Markdown\n\nFetch any web URL and get clean, readable Markdown — main content only, no\nnavigation/footer/ads. Local + free by default; smart fallback to Exa MCP\nwhen the page can't be extracted locally.\n\n## Workflow (the only thing the agent needs to remember)\n\n1. **Try trafilatura first**:\n\n   ```bash\n   python3 ~/.claude/skills/fetch-url-as-markdown/scripts/fetch_url.py \"<URL>\"\n   ```\n\n2. **If exit code is 1 or 2 → fall back to Exa MCP** with the same URL:\n\n   ```\n   mcp__exa__web_search_advanced_exa(\n       query=\"<URL>\",\n       includeDomains=[\"<host of URL>\"],\n       numResults=1,\n       textMaxCharacters=50000,\n       type=\"auto\"\n   )\n   ```\n\n   (`mcp__exa__crawling` works too if the server exposes it; the `web_search_advanced_exa`\n   call above is the always-available variant — pin the host with `includeDomains` and\n   use the URL itself as the query.)\n\n3. Exit code `3` means trafilatura is not installed — install once:\n\n   ```bash\n   python3 -m pip install --break-system-packages trafilatura\n   ```\n\n## Exit codes (what they mean for the fallback decision)\n\n| Code | Meaning | Action |\n|---|---|---|\n| 0 | Markdown printed to stdout | done |\n| 1 | DownloadError — network/HTTP/timeout/anti-bot block at fetch | fall back to Exa |\n| 2 | ExtractionError — empty extract, JS/Cloudflare wall, or stub body (<200 chars) | fall back to Exa |\n| 3 | trafilatura missing | install (see above), then retry |\n| 4 | UnsupportedContentTypeError — URL is binary (PDF, image, archive) | **don't** fall back to Exa; use the right specialized skill (e.g. `pdf` for PDFs) |\n\n## Defaults baked into the script\n\n- `output_format=\"markdown\"`, `include_formatting=True` — keeps headings/lists/code structure where the source HTML uses real `<h1..h6>` etc.\n- `include_links=True`, `include_tables=True`\n- `with_metadata=True` → emits a YAML frontmatter (`title`, `author`, `date`, `url`, `hostname`)\n- `favor_recall=True`, `deduplicate=True` — readable but trims duplicates\n- Real-browser User-Agent + 30s timeout configured in `scripts/settings.cfg`\n- Anti-stub guards (built into the script):\n  - rejects `Content-Type` other than `text/html|application/xhtml+xml|text/plain|application/xml|text/xml` → exit `4`\n  - sniffs raw HTML for Cloudflare / \"Please enable JavaScript\" / Imperva / DataDome wall markers → exit `2`\n  - rejects extracted bodies under 50 chars (configurable via `--min-body N`, `0` to disable) → exit `2`\n\n## Useful flags\n\n```bash\n... fetch_url.py \"<URL>\" --no-links     # strip hyperlinks\n... fetch_url.py \"<URL>\" --no-tables    # strip tables\n... fetch_url.py \"<URL>\" --no-metadata  # omit YAML header\n... fetch_url.py \"<URL>\" --comments     # include user comments (off by default — usually noise)\n... fetch_url.py \"<URL>\" --images       # include image refs (experimental)\n... fetch_url.py \"<URL>\" --precision    # terser output, drops borderline content\n```\n\n## When to choose what\n\n| Situation | Tool |\n|---|---|\n| Article, blog post, docs, README, wiki | trafilatura (default) — local, free |\n| JS-heavy SPA, login-walled, Cloudflare | Exa fallback (the script will signal exit 2) |\n| Bulk / many URLs | trafilatura — no quota, no API key |\n| Already failed twice on a domain | Exa directly |","tags":["fetch","url","markdown","driven","development","codealive-ai","agent-safety","agent-skills","ai-coding","ai-driven-development","ai-safety","antigravity"],"capabilities":["skill","source-codealive-ai","skill-fetch-url-as-markdown","topic-agent-safety","topic-agent-skills","topic-ai-coding","topic-ai-driven-development","topic-ai-safety","topic-antigravity","topic-bash","topic-claude-code","topic-codex-cli","topic-cursor","topic-developer-tools","topic-gemini-cli"],"categories":["ai-driven-development"],"synonyms":[],"warnings":[],"endpointUrl":"https://skills.sh/CodeAlive-AI/ai-driven-development/fetch-url-as-markdown","protocol":"skill","transport":"skills-sh","auth":{"type":"none","details":{"cli":"npx skills add CodeAlive-AI/ai-driven-development","source_repo":"https://github.com/CodeAlive-AI/ai-driven-development","install_from":"skills.sh"}},"qualityScore":"0.483","qualityRationale":"deterministic score 0.48 from registry signals: · indexed on github topic:agent-skills · 67 github stars · SKILL.md body (3,229 chars)","verified":false,"liveness":"unknown","lastLivenessCheck":null,"agentReviews":{"count":0,"score_avg":null,"cost_usd_avg":null,"success_rate":null,"latency_p50_ms":null,"narrative_summary":null,"summary_updated_at":null},"enrichmentModel":"deterministic:skill-github:v1","enrichmentVersion":1,"enrichedAt":"2026-05-18T18:57:06.403Z","embedding":null,"createdAt":"2026-05-04T06:56:22.997Z","updatedAt":"2026-05-18T18:57:06.403Z","lastSeenAt":"2026-05-18T18:57:06.403Z","tsv":"'/.claude/skills/fetch-url-as-markdown/scripts/fetch_url.py':111 '0':212,381 '1':105,117,138,218 '2':112,119,228,368,385,462 '200':237 '3':179,182,243 '30s':328 '4':251,354 '50':373 '50000':140 'action':211 'advanc':133,156 'agent':101,327 'alreadi':472 'alway':163 'always-avail':162 'anti':30,334 'anti-bot':29 'anti-stub':333 'api':470 'application/xhtml':348 'application/xml':351 'archiv':258 'articl':437 'ask':37 'author':309 'auto':142 'avail':164 'back':121,225,240,262 'bake':275 'bash':109,190,388 'binari':60,255 'block':221 'blog':438 'bodi':236,371,379 'borderlin':429 'bot':31 'break':196 'break-system-packag':195 'browser':324 'built':52,337 'built-in':51 'bulk':463 'call':158 'char':238,374 'choos':433 'clean':13,71 'cloudflar':359,454 'code':115,181,201,209 'comment':409,412 'configur':330,375 'content':75,343,430 'content-typ':342 'crawl':145 'datadom':364 'date':310 'decis':208 'dedupl':316 'default':82,274,415,444 'direct':479 'disabl':383 'doc':440 'domain':477 'done':217 'downloaderror':219 'drop':428 'duplic':321 'e.g':270 'emit':304 'empti':230 'enabl':361 'etc':294 'exa':19,86,123,130,134,144,157,227,242,264,455,478 'exit':114,180,200,353,367,384,461 'experiment':423 'expos':151 'extract':94,231,370 'extractionerror':229 'fail':473 'fall':120,224,239,261 'fallback':23,84,207,456 'favor':313 'fetch':2,6,40,65,223 'fetch-url-as-markdown':1 'fetch_url.py':389,395,401,408,418,424 'file':61 'first':108 'flag':387 'format':280,283 'free':80,446 'frontmatt':307 'get':70 'guard':336 'header':407 'headings/lists/code':286 'heavi':449 'host':168 'hostnam':312 'html':291,357 'hyperlink':394 'imag':257,419,421 'imperva':363 'includ':282,295,298,410,420 'includedomain':136,170 'instal':187,188,194,246 'javascript':362 'js':26,448 'js-heavi':447 'js-render':25 'js/cloudflare':232 'keep':285 'key':471 'link':296,392 'local':16,79,95,445 'login':452 'login-wal':451 'm':192 'main':74 'mani':464 'markdown':5,14,64,73,213,281 'marker':366 'mcp':20,87,124,129,143 'mean':183,204,210 'metadata':302,404 'min':378 'min-bodi':377 'miss':245 'n':380 'navigation/footer/ads':78 'need':102 'network/http/timeout/anti-bot':220 'no-link':390 'no-metadata':402 'no-tabl':396 'nois':417 'numresult':137 'omit':405 'output':279,427 'packag':198 'page':9,32,90 'pdf':256,271 'pdfs':273 'pin':166 'pip':193 'pleas':360 'post':439 'precis':425 'prefer':47 'print':214 'python3':110,191 'queri':135,178 'quot':44 'quota':468 'raw':356 'read':39 'readabl':72,318 'readm':441 'real':293,323 'real-brows':322 'recal':314 'ref':422 'reject':341,369 'rememb':104 'render':27 'retri':250 'return':12 'right':267 'scrape':41 'script':278,340,458 'scripts/settings.cfg':332 'search':132,155 'see':247 'server':150 'signal':460 'situat':435 'skill':269 'skill-fetch-url-as-markdown' 'smart':83 'sniff':355 'sourc':290 'source-codealive-ai' 'spa':450 'special':268 'stdout':216 'strip':393,399 'structur':287 'stub':235,335 'summar':42 'system':197 'tabl':299,398,400 'terser':426 'text/html':347 'text/plain':350 'text/xml':352 'textmaxcharact':139 'thing':99 'timeout':329 'titl':308 'tool':55,436 'topic-agent-safety' 'topic-agent-skills' 'topic-ai-coding' 'topic-ai-driven-development' 'topic-ai-safety' 'topic-antigravity' 'topic-bash' 'topic-claude-code' 'topic-codex-cli' 'topic-cursor' 'topic-developer-tools' 'topic-gemini-cli' 'trafilatura':17,107,184,199,244,443,466 'tri':106 'trim':320 'true':284,297,300,303,315,317 'twice':474 'type':141,344 'unsupportedcontenttypeerror':252 'url':3,10,46,62,68,128,174,253,311,465 'use':33,58,172,265,292,386 'user':36,326,411 'user-ag':325 'usual':416 'variant':165 'via':15,376 'wall':233,365,453 'web':8,67,131,154 'webfetch':54 'wiki':442 'work':146 'workflow':96 'xml':349 'yaml':306,406","prices":[{"id":"48b43f93-8d45-47f3-9665-3ff996dd8479","listingId":"252e68fe-5700-423b-a737-682808e78ea5","amountUsd":"0","unit":"free","nativeCurrency":null,"nativeAmount":null,"chain":null,"payTo":null,"paymentMethod":"skill-free","isPrimary":true,"details":{"org":"CodeAlive-AI","category":"ai-driven-development","install_from":"skills.sh"},"createdAt":"2026-05-04T06:56:22.997Z"}],"sources":[{"listingId":"252e68fe-5700-423b-a737-682808e78ea5","source":"github","sourceId":"CodeAlive-AI/ai-driven-development/fetch-url-as-markdown","sourceUrl":"https://github.com/CodeAlive-AI/ai-driven-development/tree/main/skills/fetch-url-as-markdown","isPrimary":false,"firstSeenAt":"2026-05-04T06:56:22.997Z","lastSeenAt":"2026-05-18T18:57:06.403Z"}],"details":{"listingId":"252e68fe-5700-423b-a737-682808e78ea5","quickStartSnippet":null,"exampleRequest":null,"exampleResponse":null,"schema":null,"openapiUrl":null,"agentsTxtUrl":null,"citations":[],"useCases":[],"bestFor":[],"notFor":[],"kindDetails":{"org":"CodeAlive-AI","slug":"fetch-url-as-markdown","github":{"repo":"CodeAlive-AI/ai-driven-development","stars":67,"topics":["agent-safety","agent-skills","ai-coding","ai-driven-development","ai-safety","antigravity","bash","claude-code","codex-cli","cursor","developer-tools","gemini-cli","hooks","mcp","multi-agent","opencode","plugins","prompt-engineering","skills","subagents"],"license":"mit","html_url":"https://github.com/CodeAlive-AI/ai-driven-development","pushed_at":"2026-05-12T20:04:46Z","description":"Practices, protocols, and skills for AI-driven software development. 18 skills + 1 Bash safety hook for Claude Code, Codex CLI, OpenCode, Cursor, Gemini CLI, Antigravity, and any agent supporting the Agent Skills standard.","skill_md_sha":"1564269da7c6fa11e28e47b8ab1628b3c55e74a3","skill_md_path":"skills/fetch-url-as-markdown/SKILL.md","default_branch":"main","skill_tree_url":"https://github.com/CodeAlive-AI/ai-driven-development/tree/main/skills/fetch-url-as-markdown"},"layout":"multi","source":"github","category":"ai-driven-development","frontmatter":{"name":"fetch-url-as-markdown","description":"Fetch a web page (URL) and return clean Markdown via local trafilatura, with Exa MCP as a fallback for JS-rendered or anti-bot pages. Use when the user asks to read, fetch, scrape, summarize, or quote a URL — prefer this over the built-in WebFetch tool. Don't use for binary files (PDFs, images, archives) or for fetching API/JSON endpoints."},"skills_sh_url":"https://skills.sh/CodeAlive-AI/ai-driven-development/fetch-url-as-markdown"},"updatedAt":"2026-05-18T18:57:06.403Z"}}