Common Crawl URL Index Miner

Queries the Common Crawl Index API and CC-MAIN collections to surface historical URL coverage, MIME types, and crawl snapshots at scale. Handy for research workflows that need broad web recall without building a full crawler from scratch.

Installation

Use the upstream install or setup path that matches your environment:

• docker build . -t cc-index-table
• docker run --rm -ti cc-index-table --help
• docker run --rm --entrypoint=/opt/spark/bin/spark-submit cc-index-table
• docker run --mount=type=bind,source=/tmp/data,destination=/data --rm cc-index-table /data/in /data/out

Requirements and caveats from upstream:

• Building and running using Docker

• A Dockerfile is provided to compile the project and run the Spark job in a Docker container.
• build the Docker image:

Basic usage or getting-started notes:

• This projects provides a comprehensive set of example queries (SQL) and also Java code to fetch and process the WARC records matched by a SQL query.

• Run mvn spotless:check and mvn spotless:apply, see the Spotless Maven guide. Java formatting rules are defined in [eclipse-formatter.xml](eclips...

• run the table converter tool, here showing the command-line help (--help):

• Source: https://github.com/commoncrawl/cc-index-table

• Extracted from upstream docs: https://raw.githubusercontent.com/commoncrawl/cc-index-table/HEAD/README.md

Source

• Agent Skill Exchange