Airflow ETL Pipeline Generator

Integrates with Apache Airflow, GitLab, Kafka, PostgreSQL, S3, and SMB file systems to enable conversational ETL pipeline creation, data source connectivity testing, schema extraction, and automatic generation of production-ready Airflow DAGs with pre-built Spark application templates for database-to-datalake, file-to-datalake, and API-to-datalake workflows.

Data Engineering Agent MCP Server provides tools for building and managing ETL pipelines in enterprise data environments. Built by Anatoliy A Aksenov, it integrates with Apache Airflow, GitLab, Kafka, PostgreSQL, S3-compatible storage, and SMB file systems to enable end-to-end data pipeline creation through conversational interfaces. The server offers connectivity testing, schema extraction, file sampling across multiple data sources, form-based pipeline configuration for database-to-datalake, file-to-datalake, and API-to-datalake workflows, plus automatic GitLab issue creation for pipeline requests with pre-built Spark application templates. Particularly valuable for data engineers and analysts who need to rapidly prototype ETL processes, validate data source connectivity, and generate production-ready Airflow DAGs without manual coding, supporting common enterprise scenarios like PostgreSQL to data lake migrations, S3 file processing, and real-time Kafka stream ingestion.