Crawl4AI RAG

Integrates web crawling with RAG functionality to enable website content retrieval, storage in vector databases, and semantic searching over crawled data for enhanced knowledge access

This MCP server integrates web crawling capabilities from Crawl4AI with RAG (Retrieval Augmented Generation) functionality, enabling AI agents and coding assistants to crawl websites, store content in Supabase vector databases, and perform semantic searches over the crawled data. Developed by Cole Medin, it provides four essential tools: single page crawling, intelligent recursive website crawling with automatic URL type detection, source retrieval, and vector-based RAG queries with optional source filtering. The implementation is designed to be incorporated into Archon as a knowledge engine for AI coding assistants, with future plans to support different embedding models and local deployment options using Ollama.