{"id":"22462ed6-e0f7-4ffb-a57b-4ab569418c73","shortId":"MbY4Q3","kind":"skill","title":"pilot-etl-data-pipeline-setup","tagline":"Deploy a five-stage ETL data pipeline with 5 agents.  Use this skill when: 1. User wants to set up an ETL or data processing pipeline 2. User is configuring an ingestion, transform, validation, load, or reporting agent 3. User asks about data pipeline orchestration across agents ","description":"# ETL Data Pipeline Setup\n\nDeploy 5 agents: ingest, transform, validate, load, and report.\n\n## Roles\n\n| Role | Hostname | Skills | Purpose |\n|------|----------|--------|---------|\n| ingest | `<prefix>-ingest` | pilot-s3-bridge, pilot-database-bridge, pilot-task-chain, pilot-cron | Pulls raw data on schedule |\n| transform | `<prefix>-transform` | pilot-task-router, pilot-stream-data, pilot-task-parallel | Parallel data processing |\n| validate | `<prefix>-validate` | pilot-task-router, pilot-audit-log, pilot-alert, pilot-quarantine | Quality checks, quarantine |\n| loader | `<prefix>-loader` | pilot-database-bridge, pilot-task-chain, pilot-receipt | Writes to target stores |\n| reporter | `<prefix>-reporter` | pilot-webhook-bridge, pilot-metrics, pilot-slack-bridge, pilot-cron | Dashboards and reports |\n\n## Setup Procedure\n\n**Step 1:** Ask the user which role and prefix.\n\n**Step 2:** Install skills:\n```bash\n# ingest:\nclawhub install pilot-s3-bridge pilot-database-bridge pilot-task-chain pilot-cron\n# transform:\nclawhub install pilot-task-router pilot-stream-data pilot-task-parallel\n# validate:\nclawhub install pilot-task-router pilot-audit-log pilot-alert pilot-quarantine\n# loader:\nclawhub install pilot-database-bridge pilot-task-chain pilot-receipt\n# reporter:\nclawhub install pilot-webhook-bridge pilot-metrics pilot-slack-bridge pilot-cron\n```\n\n**Step 3:** Set hostname and write manifest to `~/.pilot/setups/etl-data-pipeline.json`.\n\n**Step 4:** Handshake with adjacent pipeline stages.\n\n## Manifest Templates Per Role\n\n### ingest\n```json\n{\n  \"setup\": \"etl-data-pipeline\", \"role\": \"ingest\", \"role_name\": \"Data Ingestion\",\n  \"hostname\": \"<prefix>-ingest\",\n  \"skills\": {\n    \"pilot-s3-bridge\": \"Pull raw data from S3 buckets.\",\n    \"pilot-database-bridge\": \"Extract from source databases.\",\n    \"pilot-task-chain\": \"Chain ingestion steps sequentially.\",\n    \"pilot-cron\": \"Schedule periodic data pulls.\"\n  },\n  \"data_flows\": [{ \"direction\": \"send\", \"peer\": \"<prefix>-transform\", \"port\": 1001, \"topic\": \"ingest-batch\", \"description\": \"Raw data batches\" }],\n  \"handshakes_needed\": [\"<prefix>-transform\"]\n}\n```\n\n### transform\n```json\n{\n  \"setup\": \"etl-data-pipeline\", \"role\": \"transform\", \"role_name\": \"Data Transformer\",\n  \"hostname\": \"<prefix>-transform\",\n  \"skills\": {\n    \"pilot-task-router\": \"Accept transform tasks from ingest.\",\n    \"pilot-stream-data\": \"Stream transformed records to validator.\",\n    \"pilot-task-parallel\": \"Process data in parallel for throughput.\"\n  },\n  \"data_flows\": [\n    { \"direction\": \"receive\", \"peer\": \"<prefix>-ingest\", \"port\": 1001, \"topic\": \"ingest-batch\", \"description\": \"Raw data\" },\n    { \"direction\": \"send\", \"peer\": \"<prefix>-validate\", \"port\": 1001, \"topic\": \"transform-complete\", \"description\": \"Transformed records\" }\n  ],\n  \"handshakes_needed\": [\"<prefix>-ingest\", \"<prefix>-validate\"]\n}\n```\n\n### validate\n```json\n{\n  \"setup\": \"etl-data-pipeline\", \"role\": \"validate\", \"role_name\": \"Data Validator\",\n  \"hostname\": \"<prefix>-validate\",\n  \"skills\": {\n    \"pilot-task-router\": \"Accept validation tasks.\",\n    \"pilot-audit-log\": \"Log validation results.\",\n    \"pilot-alert\": \"Alert on high error rates.\",\n    \"pilot-quarantine\": \"Quarantine invalid records.\"\n  },\n  \"data_flows\": [\n    { \"direction\": \"receive\", \"peer\": \"<prefix>-transform\", \"port\": 1001, \"topic\": \"transform-complete\", \"description\": \"Transformed records\" },\n    { \"direction\": \"send\", \"peer\": \"<prefix>-loader\", \"port\": 1001, \"topic\": \"validation-passed\", \"description\": \"Validated records\" },\n    { \"direction\": \"send\", \"peer\": \"<prefix>-reporter\", \"port\": 1002, \"topic\": \"validation-metrics\", \"description\": \"Error rates\" }\n  ],\n  \"handshakes_needed\": [\"<prefix>-transform\", \"<prefix>-loader\", \"<prefix>-reporter\"]\n}\n```\n\n### loader\n```json\n{\n  \"setup\": \"etl-data-pipeline\", \"role\": \"loader\", \"role_name\": \"Data Loader\",\n  \"hostname\": \"<prefix>-loader\",\n  \"skills\": {\n    \"pilot-database-bridge\": \"Write validated data to target databases.\",\n    \"pilot-task-chain\": \"Chain load steps.\",\n    \"pilot-receipt\": \"Issue receipts for every load batch.\"\n  },\n  \"data_flows\": [\n    { \"direction\": \"receive\", \"peer\": \"<prefix>-validate\", \"port\": 1001, \"topic\": \"validation-passed\", \"description\": \"Validated records\" },\n    { \"direction\": \"send\", \"peer\": \"<prefix>-reporter\", \"port\": 1002, \"topic\": \"load-receipt\", \"description\": \"Load receipts\" }\n  ],\n  \"handshakes_needed\": [\"<prefix>-validate\", \"<prefix>-reporter\"]\n}\n```\n\n### reporter\n```json\n{\n  \"setup\": \"etl-data-pipeline\", \"role\": \"reporter\", \"role_name\": \"Pipeline Reporter\",\n  \"hostname\": \"<prefix>-reporter\",\n  \"skills\": {\n    \"pilot-webhook-bridge\": \"Forward pipeline alerts to external services.\",\n    \"pilot-metrics\": \"Aggregate pipeline metrics.\",\n    \"pilot-slack-bridge\": \"Post daily pipeline summaries to Slack.\",\n    \"pilot-cron\": \"Schedule hourly/daily report generation.\"\n  },\n  \"data_flows\": [\n    { \"direction\": \"receive\", \"peer\": \"<prefix>-validate\", \"port\": 1002, \"topic\": \"validation-metrics\", \"description\": \"Error rates\" },\n    { \"direction\": \"receive\", \"peer\": \"<prefix>-loader\", \"port\": 1002, \"topic\": \"load-receipt\", \"description\": \"Load receipts\" }\n  ],\n  \"handshakes_needed\": [\"<prefix>-validate\", \"<prefix>-loader\"]\n}\n```\n\n## Data Flows\n\n- `ingest → transform` : raw data batches (port 1001)\n- `transform → validate` : transformed records (port 1001)\n- `validate → loader` : validated records (port 1001)\n- `loader → reporter` : load receipts (port 1002)\n- `validate → reporter` : validation metrics (port 1002)\n\n## Workflow Example\n\n```bash\n# On ingest:\npilotctl --json send-file <prefix>-transform ./data/raw/orders.csv\npilotctl --json publish <prefix>-transform ingest-batch '{\"source\":\"s3://data/orders\",\"rows\":50000}'\n# On validate:\npilotctl --json publish <prefix>-loader validation-passed '{\"batch_id\":\"B-1042\",\"valid\":49700,\"quarantined\":123}'\n# On loader:\npilotctl --json publish <prefix>-reporter load-receipt '{\"batch_id\":\"B-1042\",\"rows_loaded\":49700}'\n```\n\n## Dependencies\n\nRequires `pilot-protocol` skill, `pilotctl` binary, `clawhub` binary, and a running daemon.","tags":["pilot","etl","data","pipeline","setup","skills","teoslayer","agent-skills","ai-agents","clawhub","networking","openclaw"],"capabilities":["skill","source-teoslayer","skill-pilot-etl-data-pipeline-setup","topic-agent-skills","topic-ai-agents","topic-clawhub","topic-networking","topic-openclaw","topic-overlay-network","topic-p2p","topic-pilot-protocol"],"categories":["pilot-skills"],"synonyms":[],"warnings":[],"endpointUrl":"https://skills.sh/TeoSlayer/pilot-skills/pilot-etl-data-pipeline-setup","protocol":"skill","transport":"skills-sh","auth":{"type":"none","details":{"cli":"npx skills add TeoSlayer/pilot-skills","source_repo":"https://github.com/TeoSlayer/pilot-skills","install_from":"skills.sh"}},"qualityScore":"0.453","qualityRationale":"deterministic score 0.45 from registry signals: · indexed on github topic:agent-skills · 6 github stars · SKILL.md body (6,077 chars)","verified":false,"liveness":"unknown","lastLivenessCheck":null,"agentReviews":{"count":0,"score_avg":null,"cost_usd_avg":null,"success_rate":null,"latency_p50_ms":null,"narrative_summary":null,"summary_updated_at":null},"enrichmentModel":"deterministic:skill-github:v1","enrichmentVersion":1,"enrichedAt":"2026-05-18T19:14:55.166Z","embedding":null,"createdAt":"2026-05-18T13:22:39.656Z","updatedAt":"2026-05-18T19:14:55.166Z","lastSeenAt":"2026-05-18T19:14:55.166Z","tsv":"'-1042':742,759 '/.pilot/setups/etl-data-pipeline.json':272 '/data/orders':727 '/data/raw/orders.csv':717 '1':22,170 '1001':340,403,416,479,492,567,681,687,693 '1002':505,580,648,661,699,705 '123':746 '2':34,179 '3':46,265 '4':274 '49700':744,762 '5':16,60 '50000':729 'accept':372,448 'across':53 'adjac':277 'agent':17,45,54,61 'aggreg':621 'alert':124,229,460,461,614 'ask':48,171 'audit':120,225,453 'b':741,758 'bash':182,708 'batch':344,348,407,559,679,724,739,756 'binari':770,772 'bridg':78,82,136,153,160,189,193,239,253,260,303,313,537,611,627 'bucket':309 'chain':86,140,197,243,321,322,547,548 'check':129 'clawhub':184,202,217,234,248,771 'complet':420,483 'configur':37 'cron':89,163,200,263,328,636 'daemon':776 'daili':629 'dashboard':164 'data':4,13,31,50,56,92,104,110,211,289,295,306,331,333,347,357,363,380,391,396,410,433,439,472,523,529,540,560,597,641,673,678 'databas':81,135,192,238,312,317,536,543 'depend':763 'deploy':7,59 'descript':345,408,421,484,497,510,572,585,653,666 'direct':335,398,411,474,487,500,562,575,643,656 'error':464,511,654 'etl':3,12,29,55,288,356,432,522,596 'etl-data-pipelin':287,355,431,521,595 'everi':557 'exampl':707 'extern':616 'extract':314 'file':715 'five':10 'five-stag':9 'flow':334,397,473,561,642,674 'forward':612 'generat':640 'handshak':275,349,424,513,588,669 'high':463 'hostnam':70,267,297,365,441,531,605 'hourly/daily':638 'id':740,757 'ingest':39,62,73,74,183,284,292,296,298,323,343,376,401,406,426,675,710,723 'ingest-batch':342,405,722 'instal':180,185,203,218,235,249 'invalid':470 'issu':554 'json':285,353,429,519,593,712,719,733,750 'load':42,65,549,558,583,586,664,667,696,754,761 'load-receipt':582,663,753 'loader':131,132,233,490,516,518,526,530,532,659,672,689,694,735,748 'log':121,226,454,455 'manifest':270,280 'metric':156,256,509,620,623,652,703 'name':294,362,438,528,602 'need':350,425,514,589,670 'orchestr':52 'parallel':108,109,215,389,393 'pass':496,571,738 'peer':337,400,413,476,489,502,564,577,645,658 'per':282 'period':330 'pilot':2,76,80,84,88,98,102,106,115,119,123,126,134,138,142,151,155,158,162,187,191,195,199,205,209,213,220,224,228,231,237,241,245,251,255,258,262,301,311,319,327,369,378,387,445,452,459,467,535,545,552,609,619,625,635,766 'pilot-alert':122,227,458 'pilot-audit-log':118,223,451 'pilot-cron':87,161,198,261,326,634 'pilot-database-bridg':79,133,190,236,310,534 'pilot-etl-data-pipeline-setup':1 'pilot-metr':154,254,618 'pilot-protocol':765 'pilot-quarantin':125,230,466 'pilot-receipt':141,244,551 'pilot-s3-bridge':75,186,300 'pilot-slack-bridg':157,257,624 'pilot-stream-data':101,208,377 'pilot-task-chain':83,137,194,240,318,544 'pilot-task-parallel':105,212,386 'pilot-task-rout':97,114,204,219,368,444 'pilot-webhook-bridg':150,250,608 'pilotctl':711,718,732,749,769 'pipelin':5,14,33,51,57,278,290,358,434,524,598,603,613,622,630 'port':339,402,415,478,491,504,566,579,647,660,680,686,692,698,704 'post':628 'prefix':177 'procedur':168 'process':32,111,390 'protocol':767 'publish':720,734,751 'pull':90,304,332 'purpos':72 'qualiti':128 'quarantin':127,130,232,468,469,745 'rate':465,512,655 'raw':91,305,346,409,677 'receipt':143,246,553,555,584,587,665,668,697,755 'receiv':399,475,563,644,657 'record':383,423,471,486,499,574,685,691 'report':44,67,148,149,166,247,503,517,578,591,592,600,604,606,639,695,701,752 'requir':764 'result':457 'role':68,69,175,283,291,293,359,361,435,437,525,527,599,601 'router':100,117,207,222,371,447 'row':728,760 'run':775 's3':77,188,302,308,726 'schedul':94,329,637 'send':336,412,488,501,576,714 'send-fil':713 'sequenti':325 'servic':617 'set':26,266 'setup':6,58,167,286,354,430,520,594 'skill':20,71,181,299,367,443,533,607,768 'skill-pilot-etl-data-pipeline-setup' 'slack':159,259,626,633 'sourc':316,725 'source-teoslayer' 'stage':11,279 'step':169,178,264,273,324,550 'store':147 'stream':103,210,379,381 'summari':631 'target':146,542 'task':85,99,107,116,139,196,206,214,221,242,320,370,374,388,446,450,546 'templat':281 'throughput':395 'topic':341,404,417,480,493,506,568,581,649,662 'topic-agent-skills' 'topic-ai-agents' 'topic-clawhub' 'topic-networking' 'topic-openclaw' 'topic-overlay-network' 'topic-p2p' 'topic-pilot-protocol' 'transform':40,63,95,96,201,338,351,352,360,364,366,373,382,419,422,477,482,485,515,676,682,684,716,721 'transform-complet':418,481 'use':18 'user':23,35,47,173 'valid':41,64,112,113,216,385,414,427,428,436,440,442,449,456,495,498,508,539,565,570,573,590,646,651,671,683,688,690,700,702,731,737,743 'validation-metr':507,650 'validation-pass':494,569,736 'want':24 'webhook':152,252,610 'workflow':706 'write':144,269,538","prices":[{"id":"e43e7032-d2ff-4882-ad8c-3bf14d757cf3","listingId":"22462ed6-e0f7-4ffb-a57b-4ab569418c73","amountUsd":"0","unit":"free","nativeCurrency":null,"nativeAmount":null,"chain":null,"payTo":null,"paymentMethod":"skill-free","isPrimary":true,"details":{"org":"TeoSlayer","category":"pilot-skills","install_from":"skills.sh"},"createdAt":"2026-05-18T13:22:39.656Z"}],"sources":[{"listingId":"22462ed6-e0f7-4ffb-a57b-4ab569418c73","source":"github","sourceId":"TeoSlayer/pilot-skills/pilot-etl-data-pipeline-setup","sourceUrl":"https://github.com/TeoSlayer/pilot-skills/tree/main/skills/pilot-etl-data-pipeline-setup","isPrimary":false,"firstSeenAt":"2026-05-18T13:22:39.656Z","lastSeenAt":"2026-05-18T19:14:55.166Z"}],"details":{"listingId":"22462ed6-e0f7-4ffb-a57b-4ab569418c73","quickStartSnippet":null,"exampleRequest":null,"exampleResponse":null,"schema":null,"openapiUrl":null,"agentsTxtUrl":null,"citations":[],"useCases":[],"bestFor":[],"notFor":[],"kindDetails":{"org":"TeoSlayer","slug":"pilot-etl-data-pipeline-setup","github":{"repo":"TeoSlayer/pilot-skills","stars":6,"topics":["agent-skills","ai-agents","clawhub","networking","openclaw","overlay-network","p2p","pilot-protocol"],"license":"agpl-3.0","html_url":"https://github.com/TeoSlayer/pilot-skills","pushed_at":"2026-05-13T06:08:49Z","description":"80+ agent skills for Pilot Protocol — communication, file transfer, trust, task routing, swarm coordination, and more","skill_md_sha":"54266df1a9039854f55d06bd7f3e2a5230de68f1","skill_md_path":"skills/pilot-etl-data-pipeline-setup/SKILL.md","default_branch":"main","skill_tree_url":"https://github.com/TeoSlayer/pilot-skills/tree/main/skills/pilot-etl-data-pipeline-setup"},"layout":"multi","source":"github","category":"pilot-skills","frontmatter":{"name":"pilot-etl-data-pipeline-setup","license":"AGPL-3.0","description":"Deploy a five-stage ETL data pipeline with 5 agents.  Use this skill when: 1. User wants to set up an ETL or data processing pipeline 2. User is configuring an ingestion, transform, validation, load, or reporting agent 3. User asks about data pipeline orchestration across agents  Do NOT use this skill when: - User wants to transfer a single dataset (use pilot-dataset instead) - User wants to stream data once (use pilot-stream-data instead)"},"skills_sh_url":"https://skills.sh/TeoSlayer/pilot-skills/pilot-etl-data-pipeline-setup"},"updatedAt":"2026-05-18T19:14:55.166Z"}}