Document Crawler & Search

Crawls websites to generate searchable Markdown documentation with vector embeddings for semantic search capabilities beyond training cutoffs

MCPDocSearch is a tool for crawling websites, generating Markdown documentation, and making it searchable via an MCP server. It consists of two main components: a crawler CLI that extracts content from documentation sites and converts it to Markdown, and an MCP server that loads these documents, chunks them into searchable sections, and generates vector embeddings using sentence-transformers. The server exposes tools for listing documents, retrieving document structures, and performing semantic searches over the content, making it particularly useful for AI agents that need to access up-to-date documentation beyond their training cutoff.