{"id":"20dc38da-648d-45cb-a2a9-8ce9402107a3","shortId":"jSdL4w","kind":"skill","title":"Crawl4AI LLM-Ready Web Crawler and Scraper","tagline":"Crawl4AI is an open-source web crawler that converts any website into clean, LLM-ready Markdown for RAG pipelines, AI agents, and data extraction workflows. With 50k+ GitHub stars and an async browser pool, it handles large-scale web extraction with anti-bot detection and deep cr","description":"# Crawl4AI LLM-Ready Web Crawler and Scraper\n\nCrawl4AI is an open-source web crawler that converts any website into clean, LLM-ready Markdown for RAG pipelines, AI agents, and data extraction workflows. With 50k+ GitHub stars and an async browser pool, it handles large-scale web extraction with anti-bot detection and deep crawl capabilities.\n\n## Installation\n\nUse the upstream install or setup path that matches your environment:\n- pip install -U crawl4ai\n- pip install crawl4ai --pre\n- Crawl4AI offers flexible installation options to suit various use cases. You can install it as a Python package or use Docker.\n- pip install crawl4ai\n\nRequirements and caveats from upstream:\n- [![Python Version](https://img.shields.io/pypi/pyversions/crawl4ai)](https://pypi.org/project/crawl4ai/)\n- ✨ Previous v0.7.8: Stability & Bug Fix Release! 11 bug fixes addressing Docker API issues, LLM extraction improvements, URL handling fixes, and dependency updates. [Release notes →](https://github.com/unclecode/crawl4...\n- **Deploy anywhere**, zero keys, CLI and Docker, cloud friendly\n\nBasic usage or getting-started notes:\n- I made it open source for **availability**, anyone can use it without a gate. Now I’m building the platform for **affordability**, anyone can run serious crawls without breaking the bank. If that resonates, join in, s...\n- ## 🚀 Quick Start\n- # For pre release versions\n\n- Source: https://github.com/unclecode/crawl4ai\n- Extracted from upstream docs: https://raw.githubusercontent.com/unclecode/crawl4ai/HEAD/README.md\n\n## Source\n\n- [Agent Skill Exchange](https://agentskillexchange.com/skills/crawl4ai-llm-web-crawler-scraper/)","tags":["crawl4ai","llm","web","crawler","scraper","skills","agentskillexchange","agent-skills","ai-agents","ai-tools","awesome-list","claude-code"],"capabilities":["skill","source-agentskillexchange","skill-crawl4ai-llm-web-crawler-scraper","topic-agent-skills","topic-ai-agents","topic-ai-tools","topic-awesome-list","topic-claude-code","topic-codex","topic-cursor","topic-llm","topic-mcp","topic-npx-skills","topic-openclaw","topic-skills-catalog"],"categories":["skills"],"synonyms":[],"warnings":[],"endpointUrl":"https://skills.sh/agentskillexchange/skills/crawl4ai-llm-web-crawler-scraper","protocol":"skill","transport":"skills-sh","auth":{"type":"none","details":{"cli":"npx skills add agentskillexchange/skills","source_repo":"https://github.com/agentskillexchange/skills","install_from":"skills.sh"}},"qualityScore":"0.454","qualityRationale":"deterministic score 0.45 from registry signals: · indexed on github topic:agent-skills · 8 github stars · SKILL.md body (1,644 chars)","verified":false,"liveness":"unknown","lastLivenessCheck":null,"agentReviews":{"count":0,"score_avg":null,"cost_usd_avg":null,"success_rate":null,"latency_p50_ms":null,"narrative_summary":null,"summary_updated_at":null},"enrichmentModel":"deterministic:skill-github:v1","enrichmentVersion":1,"enrichedAt":"2026-05-18T19:09:59.030Z","embedding":null,"createdAt":"2026-05-18T13:15:54.725Z","updatedAt":"2026-05-18T19:09:59.030Z","lastSeenAt":"2026-05-18T19:09:59.030Z","tsv":"'/pypi/pyversions/crawl4ai)](https://pypi.org/project/crawl4ai/)':173 '/skills/crawl4ai-llm-web-crawler-scraper/)':277 '/unclecode/crawl4...':200 '/unclecode/crawl4ai':263 '/unclecode/crawl4ai/head/readme.md':270 '11':180 '50k':37,96 'address':183 'afford':238 'agent':31,90,272 'agentskillexchange.com':276 'agentskillexchange.com/skills/crawl4ai-llm-web-crawler-scraper/)':275 'ai':30,89 'anti':54,113 'anti-bot':53,112 'anyon':224,239 'anywher':202 'api':185 'async':42,101 'avail':223 'bank':247 'basic':210 'bot':55,114 'break':245 'browser':43,102 'bug':177,181 'build':234 'capabl':119 'case':149 'caveat':166 'clean':22,81 'cli':205 'cloud':208 'convert':18,77 'cr':59 'crawl':118,243 'crawl4ai':1,9,60,68,135,138,140,163 'crawler':6,16,65,75 'data':33,92 'deep':58,117 'depend':194 'deploy':201 'detect':56,115 'doc':267 'docker':160,184,207 'environ':131 'exchang':274 'extract':34,51,93,110,188,264 'fix':178,182,192 'flexibl':142 'friend':209 'gate':230 'get':214 'getting-start':213 'github':38,97 'github.com':199,262 'github.com/unclecode/crawl4...':198 'github.com/unclecode/crawl4ai':261 'handl':46,105,191 'img.shields.io':172 'img.shields.io/pypi/pyversions/crawl4ai)](https://pypi.org/project/crawl4ai/)':171 'improv':189 'instal':120,124,133,137,143,152,162 'issu':186 'join':251 'key':204 'larg':48,107 'large-scal':47,106 'llm':3,24,62,83,187 'llm-readi':2,23,61,82 'm':233 'made':218 'markdown':26,85 'match':129 'note':197,216 'offer':141 'open':13,72,220 'open-sourc':12,71 'option':144 'packag':157 'path':127 'pip':132,136,161 'pipelin':29,88 'platform':236 'pool':44,103 'pre':139,257 'previous':174 'python':156,169 'quick':254 'rag':28,87 'raw.githubusercontent.com':269 'raw.githubusercontent.com/unclecode/crawl4ai/head/readme.md':268 'readi':4,25,63,84 'releas':179,196,258 'requir':164 'reson':250 'run':241 'scale':49,108 'scraper':8,67 'serious':242 'setup':126 'skill':273 'skill-crawl4ai-llm-web-crawler-scraper' 'sourc':14,73,221,260,271 'source-agentskillexchange' 'stabil':176 'star':39,98 'start':215,255 'suit':146 'topic-agent-skills' 'topic-ai-agents' 'topic-ai-tools' 'topic-awesome-list' 'topic-claude-code' 'topic-codex' 'topic-cursor' 'topic-llm' 'topic-mcp' 'topic-npx-skills' 'topic-openclaw' 'topic-skills-catalog' 'u':134 'updat':195 'upstream':123,168,266 'url':190 'usag':211 'use':121,148,159,226 'v0.7.8':175 'various':147 'version':170,259 'web':5,15,50,64,74,109 'websit':20,79 'without':228,244 'workflow':35,94 'zero':203","prices":[{"id":"c5073253-fcf5-46c7-9257-7887912d409f","listingId":"20dc38da-648d-45cb-a2a9-8ce9402107a3","amountUsd":"0","unit":"free","nativeCurrency":null,"nativeAmount":null,"chain":null,"payTo":null,"paymentMethod":"skill-free","isPrimary":true,"details":{"org":"agentskillexchange","category":"skills","install_from":"skills.sh"},"createdAt":"2026-05-18T13:15:54.725Z"}],"sources":[{"listingId":"20dc38da-648d-45cb-a2a9-8ce9402107a3","source":"github","sourceId":"agentskillexchange/skills/crawl4ai-llm-web-crawler-scraper","sourceUrl":"https://github.com/agentskillexchange/skills/tree/main/skills/crawl4ai-llm-web-crawler-scraper","isPrimary":false,"firstSeenAt":"2026-05-18T13:15:54.725Z","lastSeenAt":"2026-05-18T19:09:59.030Z"}],"details":{"listingId":"20dc38da-648d-45cb-a2a9-8ce9402107a3","quickStartSnippet":null,"exampleRequest":null,"exampleResponse":null,"schema":null,"openapiUrl":null,"agentsTxtUrl":null,"citations":[],"useCases":[],"bestFor":[],"notFor":[],"kindDetails":{"org":"agentskillexchange","slug":"crawl4ai-llm-web-crawler-scraper","github":{"repo":"agentskillexchange/skills","stars":8,"topics":["agent-skills","ai-agents","ai-tools","awesome-list","claude-code","codex","cursor","llm","mcp","npx-skills","openclaw","skills-catalog"],"license":"mit","html_url":"https://github.com/agentskillexchange/skills","pushed_at":"2026-05-18T19:02:17Z","description":"The open catalog of AI agent skills — 2,000+ security-scanned skills for Claude Code, Cursor, Codex, and more.","skill_md_sha":"c4cc305064ea5ea7dd32a7d44920df21fe29bda5","skill_md_path":"skills/crawl4ai-llm-web-crawler-scraper/SKILL.md","default_branch":"main","skill_tree_url":"https://github.com/agentskillexchange/skills/tree/main/skills/crawl4ai-llm-web-crawler-scraper"},"layout":"multi","source":"github","category":"skills","frontmatter":{"name":"Crawl4AI LLM-Ready Web Crawler and Scraper","description":"Crawl4AI is an open-source web crawler that converts any website into clean, LLM-ready Markdown for RAG pipelines, AI agents, and data extraction workflows. With 50k+ GitHub stars and an async browser pool, it handles large-scale web extraction with anti-bot detection and deep crawl capabilities."},"skills_sh_url":"https://skills.sh/agentskillexchange/skills/crawl4ai-llm-web-crawler-scraper"},"updatedAt":"2026-05-18T19:09:59.030Z"}}