Jina Supabase RAG

Crawls and indexes documentation websites using Jina AI Reader API and Crawl4AI for intelligent content extraction, automatically discovering URLs through sitemaps and recursive crawling, then chunks and embeds the content into Supabase with pgvector for semantic search and retrieval-augmented generation workflows.

A documentation crawling and indexing server by Marty Martin that combines Jina AI's Reader API with Crawl4AI for intelligent content extraction and stores processed documents in Supabase with pgvector for semantic search. The implementation features multi-strategy URL discovery (sitemap parsing, recursive crawling), dual extraction methods with automatic fallback, intelligent text chunking by headers and paragraphs, OpenAI embedding generation, and vector similarity search with project-based filtering.