{"id":"2bc614c4-c3a2-4ecd-803d-7baf4fc8de91","shortId":"3gCGkh","kind":"skill","title":"engineering-data-pipelines","tagline":"Data engineering knowledge reference covering Airflow, Dagster, Kafka Streams, Flink, dbt, and data quality patterns. Use when building data pipelines, ETL workflows, stream processing, or data quality checks.","description":"# 数据工程域 · Data Engineering\n\n```\n编排：Airflow(调度) | Dagster(资产) | Prefect(现代流)\n流处理：Kafka Streams(嵌入式) | Flink(集群) | Spark Streaming\n质量：Great Expectations | dbt tests | Soda Core\n```\n\n## 编排检查项\n\n幂等(UPSERT/分区覆盖) | 增量(`WHERE updated_at > last_run`) | 事件驱动触发 | 跨 DAG 依赖 | 数据血缘(`ref()`/Asset deps)\n\n## 流处理检查项\n\n时间语义选择 | Watermark 乱序容忍 | 状态 TTL 防膨胀 | Checkpoint 间隔 | 端到端 Exactly-Once | 背压监控\n\n## 质量检查项\n\n分层验证(源→转换→目标) | 完整性+准确性+一致性 | 及时性阈值 | 加权评分 | 告警(Slack/PagerDuty)\n\n工具对比、API 用法、质量维度详见 [references/details.md](references/details.md)\n\n## 触发词\n\n数据管道、Airflow、Dagster、Prefect、ETL、流处理、Kafka Streams、Flink、数据质量、dbt、数据血缘","tags":["engineering","data","pipelines","code","abyss","telagod","agent-skills","ai-agent","ai-assistant","ai-personality","blue-team","character-card"],"capabilities":["skill","source-telagod","skill-engineering-data-pipelines","topic-agent-skills","topic-ai-agent","topic-ai-assistant","topic-ai-personality","topic-blue-team","topic-character-card","topic-claude-code","topic-cli","topic-codex","topic-codex-cli","topic-configuration","topic-developer-tools"],"categories":["code-abyss"],"synonyms":[],"warnings":[],"endpointUrl":"https://skills.sh/telagod/code-abyss/engineering-data-pipelines","protocol":"skill","transport":"skills-sh","auth":{"type":"none","details":{"cli":"npx skills add telagod/code-abyss","source_repo":"https://github.com/telagod/code-abyss","install_from":"skills.sh"}},"qualityScore":"0.555","qualityRationale":"deterministic score 0.56 from registry signals: · indexed on github topic:agent-skills · 211 github stars · SKILL.md body (602 chars)","verified":false,"liveness":"unknown","lastLivenessCheck":null,"agentReviews":{"count":0,"score_avg":null,"cost_usd_avg":null,"success_rate":null,"latency_p50_ms":null,"narrative_summary":null,"summary_updated_at":null},"enrichmentModel":"deterministic:skill-github:v1","enrichmentVersion":1,"enrichedAt":"2026-05-18T18:55:06.326Z","embedding":null,"createdAt":"2026-05-16T12:54:50.359Z","updatedAt":"2026-05-18T18:55:06.326Z","lastSeenAt":"2026-05-18T18:55:06.326Z","tsv":"'/asset':74 'airflow':10,37,110 'api':103 'build':22 'check':32 'checkpoint':83 'core':57 'cover':9 'dag':70 'dagster':11,39,111 'data':3,5,17,23,30,34 'dbt':15,54,119 'dep':75 'engin':2,6,35 'engineering-data-pipelin':1 'etl':25,113 'exact':87 'exactly-onc':86 'expect':53 'flink':14,47,117 'great':52 'kafka':12,44,115 'knowledg':7 'last':66 'pattern':19 'pipelin':4,24 'prefect':41,112 'process':28 'qualiti':18,31 'ref':73 'refer':8 'references/details.md':106,107 'run':67 'skill' 'skill-engineering-data-pipelines' 'slack/pagerduty':101 'soda':56 'source-telagod' 'spark':49 'stream':13,27,45,50,116 'test':55 'topic-agent-skills' 'topic-ai-agent' 'topic-ai-assistant' 'topic-ai-personality' 'topic-blue-team' 'topic-character-card' 'topic-claude-code' 'topic-cli' 'topic-codex' 'topic-codex-cli' 'topic-configuration' 'topic-developer-tools' 'ttl':81 'updat':64 'upsert':60 'use':20 'watermark':78 'workflow':26 '一致性':97 '乱序容忍':79 '事件驱动触发':68 '依赖':71 '准确性':96 '分区覆盖':61 '分层验证':91 '加权评分':99 '及时性阈值':98 '告警':100 '增量':62 '完整性':95 '嵌入式':46 '工具对比':102 '幂等':59 '数据工程域':33 '数据管道':109 '数据血缘':72,120 '数据质量':118 '时间语义选择':77 '流处理':43,114 '流处理检查项':76 '源':92 '状态':80 '现代流':42 '用法':104 '目标':94 '端到端':85 '编排':36 '编排检查项':58 '背压监控':89 '触发词':108 '调度':38 '质量':51 '质量检查项':90 '质量维度详见':105 '资产':40 '跨':69 '转换':93 '间隔':84 '防膨胀':82 '集群':48","prices":[{"id":"9bf806cf-7fa5-4f09-925f-ea3ef8fa6312","listingId":"2bc614c4-c3a2-4ecd-803d-7baf4fc8de91","amountUsd":"0","unit":"free","nativeCurrency":null,"nativeAmount":null,"chain":null,"payTo":null,"paymentMethod":"skill-free","isPrimary":true,"details":{"org":"telagod","category":"code-abyss","install_from":"skills.sh"},"createdAt":"2026-05-16T12:54:50.359Z"}],"sources":[{"listingId":"2bc614c4-c3a2-4ecd-803d-7baf4fc8de91","source":"github","sourceId":"telagod/code-abyss/engineering-data-pipelines","sourceUrl":"https://github.com/telagod/code-abyss/tree/main/skills/engineering-data-pipelines","isPrimary":false,"firstSeenAt":"2026-05-16T12:54:50.359Z","lastSeenAt":"2026-05-18T18:55:06.326Z"}],"details":{"listingId":"2bc614c4-c3a2-4ecd-803d-7baf4fc8de91","quickStartSnippet":null,"exampleRequest":null,"exampleResponse":null,"schema":null,"openapiUrl":null,"agentsTxtUrl":null,"citations":[],"useCases":[],"bestFor":[],"notFor":[],"kindDetails":{"org":"telagod","slug":"engineering-data-pipelines","github":{"repo":"telagod/code-abyss","stars":211,"topics":["agent-skills","ai-agent","ai-assistant","ai-personality","blue-team","character-card","claude-code","cli","codex","codex-cli","configuration","developer-tools","devops","gemini-cli","persona","prompt-engineering","red-team","security","skills"],"license":"mit","html_url":"https://github.com/telagod/code-abyss","pushed_at":"2026-05-16T10:42:04Z","description":"Give your AI coding agent a personality. Composable persona + style + skills for Claude Code, Codex, Gemini CLI & OpenClaw. Ships Tech Persona Card v1.0 spec.","skill_md_sha":"bb22997148c132153250cca684ffe3fa601a4783","skill_md_path":"skills/engineering-data-pipelines/SKILL.md","default_branch":"main","skill_tree_url":"https://github.com/telagod/code-abyss/tree/main/skills/engineering-data-pipelines"},"layout":"multi","source":"github","category":"code-abyss","frontmatter":{"name":"engineering-data-pipelines","description":"Data engineering knowledge reference covering Airflow, Dagster, Kafka Streams, Flink, dbt, and data quality patterns. Use when building data pipelines, ETL workflows, stream processing, or data quality checks."},"skills_sh_url":"https://skills.sh/telagod/code-abyss/engineering-data-pipelines"},"updatedAt":"2026-05-18T18:55:06.326Z"}}