Scrapy Spider Data Pipeline

Builds and manages Scrapy web scraping spiders with custom item pipelines. Supports Splash rendering for JavaScript pages, rotating proxies via scrapy-rotating-proxies, and export to MongoDB or Elasticsearch.

Installation

Use the upstream install or setup path that matches your environment:

• pip install scrapy

Requirements and caveats from upstream:

• :alt: Supported Python Versions
• It is cross-platform, and requires Python 3.10+. It is maintained by Zyte_

Basic usage or getting-started notes:

• .. code:: bash

• And follow the documentation_ to learn how to use it.

• .. _documentation: https://docs.scrapy.org/en/latest/

• Source: https://github.com/scrapy/scrapy

• Extracted from upstream docs: https://raw.githubusercontent.com/scrapy/scrapy/HEAD/README.rst

Source

• Agent Skill Exchange