Scrapy Pipeline Manager

Manages Scrapy spider deployments via Scrapyd API with custom item pipelines for MongoDB ingestion, deduplication via MinHash LSH, and rotating proxy middleware configuration.

Installation

Use the upstream install or setup path that matches your environment:

• pip install scrapy

Requirements and caveats from upstream:

• :alt: Supported Python Versions
• It is cross-platform, and requires Python 3.10+. It is maintained by Zyte_

Basic usage or getting-started notes:

• .. code:: bash

• And follow the documentation_ to learn how to use it.

• .. _documentation: https://docs.scrapy.org/en/latest/

• Source: https://github.com/scrapy/scrapy

• Extracted from upstream docs: https://raw.githubusercontent.com/scrapy/scrapy/HEAD/README.rst

Source

• Agent Skill Exchange