{"id":"44dbd703-9859-450e-a836-69b3973d19e2","shortId":"pGULjm","kind":"skill","title":"Scrapy Python Web Crawling and Structured Data Extraction Framework","tagline":"Scrapy is a high-level Python framework for web crawling and structured data extraction. It is a strong fit for agent workflows that need repeatable scraping, asynchronous crawling, feed exports, and extensible pipelines for transforming or storing collected data.","description":"# Scrapy Python Web Crawling and Structured Data Extraction Framework\n\nScrapy is a high-level Python framework for web crawling and structured data extraction. It is a strong fit for agent workflows that need repeatable scraping, asynchronous crawling, feed exports, and extensible pipelines for transforming or storing collected data.\n\n## Installation\n\nUse the upstream install or setup path that matches your environment:\n- pip install scrapy\n\nRequirements and caveats from upstream:\n- :alt: Supported Python Versions\n- It is cross-platform, and requires Python 3.10+. It is maintained by Zyte_\n\nBasic usage or getting-started notes:\n- .. code:: bash\n- And follow the documentation_ to learn how to use it.\n- .. _documentation: https://docs.scrapy.org/en/latest/\n\n- Source: https://github.com/scrapy/scrapy\n- Extracted from upstream docs: https://raw.githubusercontent.com/scrapy/scrapy/HEAD/README.rst\n\n## Source\n\n- [Agent Skill Exchange](https://agentskillexchange.com/skills/scrapy-python-web-crawling-structured-data-extraction-framework/)","tags":["scrapy","python","web","crawling","structured","data","extraction","framework","skills","agentskillexchange","agent-skills","ai-agents"],"capabilities":["skill","source-agentskillexchange","skill-scrapy-python-web-crawling-structured-data-extraction-framework","topic-agent-skills","topic-ai-agents","topic-ai-tools","topic-awesome-list","topic-claude-code","topic-codex","topic-cursor","topic-llm","topic-mcp","topic-npx-skills","topic-openclaw","topic-skills-catalog"],"categories":["skills"],"synonyms":[],"warnings":[],"endpointUrl":"https://skills.sh/agentskillexchange/skills/scrapy-python-web-crawling-structured-data-extraction-framework","protocol":"skill","transport":"skills-sh","auth":{"type":"none","details":{"cli":"npx skills add agentskillexchange/skills","source_repo":"https://github.com/agentskillexchange/skills","install_from":"skills.sh"}},"qualityScore":"0.454","qualityRationale":"deterministic score 0.45 from registry signals: · indexed on github topic:agent-skills · 8 github stars · SKILL.md body (1,045 chars)","verified":false,"liveness":"unknown","lastLivenessCheck":null,"agentReviews":{"count":0,"score_avg":null,"cost_usd_avg":null,"success_rate":null,"latency_p50_ms":null,"narrative_summary":null,"summary_updated_at":null},"enrichmentModel":"deterministic:skill-github:v1","enrichmentVersion":1,"enrichedAt":"2026-05-18T19:12:20.420Z","embedding":null,"createdAt":"2026-05-18T13:19:12.258Z","updatedAt":"2026-05-18T19:12:20.420Z","lastSeenAt":"2026-05-18T19:12:20.420Z","tsv":"'/en/latest/':159 '/scrapy/scrapy':163 '/scrapy/scrapy/head/readme.rst':170 '/skills/scrapy-python-web-crawling-structured-data-extraction-framework/)':177 '3.10':131 'agent':31,80,172 'agentskillexchange.com':176 'agentskillexchange.com/skills/scrapy-python-web-crawling-structured-data-extraction-framework/)':175 'alt':119 'asynchron':37,86 'bash':145 'basic':137 'caveat':116 'code':144 'collect':48,97 'crawl':4,20,38,53,69,87 'cross':126 'cross-platform':125 'data':7,23,49,56,72,98 'doc':167 'docs.scrapy.org':158 'docs.scrapy.org/en/latest/':157 'document':149,156 'environ':110 'exchang':174 'export':40,89 'extens':42,91 'extract':8,24,57,73,164 'feed':39,88 'fit':29,78 'follow':147 'framework':9,17,58,66 'get':141 'getting-start':140 'github.com':162 'github.com/scrapy/scrapy':161 'high':14,63 'high-level':13,62 'instal':99,103,112 'learn':151 'level':15,64 'maintain':134 'match':108 'need':34,83 'note':143 'path':106 'pip':111 'pipelin':43,92 'platform':127 'python':2,16,51,65,121,130 'raw.githubusercontent.com':169 'raw.githubusercontent.com/scrapy/scrapy/head/readme.rst':168 'repeat':35,84 'requir':114,129 'scrape':36,85 'scrapi':1,10,50,59,113 'setup':105 'skill':173 'skill-scrapy-python-web-crawling-structured-data-extraction-framework' 'sourc':160,171 'source-agentskillexchange' 'start':142 'store':47,96 'strong':28,77 'structur':6,22,55,71 'support':120 'topic-agent-skills' 'topic-ai-agents' 'topic-ai-tools' 'topic-awesome-list' 'topic-claude-code' 'topic-codex' 'topic-cursor' 'topic-llm' 'topic-mcp' 'topic-npx-skills' 'topic-openclaw' 'topic-skills-catalog' 'transform':45,94 'upstream':102,118,166 'usag':138 'use':100,154 'version':122 'web':3,19,52,68 'workflow':32,81 'zyte':136","prices":[{"id":"3504aa2b-2d65-421e-83a8-5635ffb35bb6","listingId":"44dbd703-9859-450e-a836-69b3973d19e2","amountUsd":"0","unit":"free","nativeCurrency":null,"nativeAmount":null,"chain":null,"payTo":null,"paymentMethod":"skill-free","isPrimary":true,"details":{"org":"agentskillexchange","category":"skills","install_from":"skills.sh"},"createdAt":"2026-05-18T13:19:12.258Z"}],"sources":[{"listingId":"44dbd703-9859-450e-a836-69b3973d19e2","source":"github","sourceId":"agentskillexchange/skills/scrapy-python-web-crawling-structured-data-extraction-framework","sourceUrl":"https://github.com/agentskillexchange/skills/tree/main/skills/scrapy-python-web-crawling-structured-data-extraction-framework","isPrimary":false,"firstSeenAt":"2026-05-18T13:19:12.258Z","lastSeenAt":"2026-05-18T19:12:20.420Z"}],"details":{"listingId":"44dbd703-9859-450e-a836-69b3973d19e2","quickStartSnippet":null,"exampleRequest":null,"exampleResponse":null,"schema":null,"openapiUrl":null,"agentsTxtUrl":null,"citations":[],"useCases":[],"bestFor":[],"notFor":[],"kindDetails":{"org":"agentskillexchange","slug":"scrapy-python-web-crawling-structured-data-extraction-framework","github":{"repo":"agentskillexchange/skills","stars":8,"topics":["agent-skills","ai-agents","ai-tools","awesome-list","claude-code","codex","cursor","llm","mcp","npx-skills","openclaw","skills-catalog"],"license":"mit","html_url":"https://github.com/agentskillexchange/skills","pushed_at":"2026-05-18T19:02:17Z","description":"The open catalog of AI agent skills — 2,000+ security-scanned skills for Claude Code, Cursor, Codex, and more.","skill_md_sha":"63795f71690cee625386cdff9f8563040c25ee07","skill_md_path":"skills/scrapy-python-web-crawling-structured-data-extraction-framework/SKILL.md","default_branch":"main","skill_tree_url":"https://github.com/agentskillexchange/skills/tree/main/skills/scrapy-python-web-crawling-structured-data-extraction-framework"},"layout":"multi","source":"github","category":"skills","frontmatter":{"name":"Scrapy Python Web Crawling and Structured Data Extraction Framework","description":"Scrapy is a high-level Python framework for web crawling and structured data extraction. It is a strong fit for agent workflows that need repeatable scraping, asynchronous crawling, feed exports, and extensible pipelines for transforming or storing collected data."},"skills_sh_url":"https://skills.sh/agentskillexchange/skills/scrapy-python-web-crawling-structured-data-extraction-framework"},"updatedAt":"2026-05-18T19:12:20.420Z"}}