{"id":"2e1fbdbd-79ef-429d-873e-d125439093ae","shortId":"6Tekru","kind":"skill","title":"ml-pipeline-workflow","tagline":"Complete end-to-end MLOps pipeline orchestration from data preparation through model deployment.","description":"# ML Pipeline Workflow\n\nComplete end-to-end MLOps pipeline orchestration from data preparation through model deployment.\n\n## Do not use this skill when\n\n- The task is unrelated to ml pipeline workflow\n- You need a different domain or tool outside this scope\n\n## Instructions\n\n- Clarify goals, constraints, and required inputs.\n- Apply relevant best practices and validate outcomes.\n- Provide actionable steps and verification.\n- If detailed examples are required, open `resources/implementation-playbook.md`.\n\n## Overview\n\nThis skill provides comprehensive guidance for building production ML pipelines that handle the full lifecycle: data ingestion → preparation → training → validation → deployment → monitoring.\n\n## Use this skill when\n\n- Building new ML pipelines from scratch\n- Designing workflow orchestration for ML systems\n- Implementing data → model → deployment automation\n- Setting up reproducible training workflows\n- Creating DAG-based ML orchestration\n- Integrating ML components into production systems\n\n## What This Skill Provides\n\n### Core Capabilities\n\n1. **Pipeline Architecture**\n   - End-to-end workflow design\n   - DAG orchestration patterns (Airflow, Dagster, Kubeflow)\n   - Component dependencies and data flow\n   - Error handling and retry strategies\n\n2. **Data Preparation**\n   - Data validation and quality checks\n   - Feature engineering pipelines\n   - Data versioning and lineage\n   - Train/validation/test splitting strategies\n\n3. **Model Training**\n   - Training job orchestration\n   - Hyperparameter management\n   - Experiment tracking integration\n   - Distributed training patterns\n\n4. **Model Validation**\n   - Validation frameworks and metrics\n   - A/B testing infrastructure\n   - Performance regression detection\n   - Model comparison workflows\n\n5. **Deployment Automation**\n   - Model serving patterns\n   - Canary deployments\n   - Blue-green deployment strategies\n   - Rollback mechanisms\n\n### Reference Documentation\n\nSee the `references/` directory for detailed guides:\n- **data-preparation.md** - Data cleaning, validation, and feature engineering\n- **model-training.md** - Training workflows and best practices\n- **model-validation.md** - Validation strategies and metrics\n- **model-deployment.md** - Deployment patterns and serving architectures\n\n### Assets and Templates\n\nThe `assets/` directory contains:\n- **pipeline-dag.yaml.template** - DAG template for workflow orchestration\n- **training-config.yaml** - Training configuration template\n- **validation-checklist.md** - Pre-deployment validation checklist\n\n## Usage Patterns\n\n### Basic Pipeline Setup\n\n```python\n# 1. Define pipeline stages\nstages = [\n    \"data_ingestion\",\n    \"data_validation\",\n    \"feature_engineering\",\n    \"model_training\",\n    \"model_validation\",\n    \"model_deployment\"\n]\n\n# 2. Configure dependencies\n# See assets/pipeline-dag.yaml.template for full example\n```\n\n### Production Workflow\n\n1. **Data Preparation Phase**\n   - Ingest raw data from sources\n   - Run data quality checks\n   - Apply feature transformations\n   - Version processed datasets\n\n2. **Training Phase**\n   - Load versioned training data\n   - Execute training jobs\n   - Track experiments and metrics\n   - Save trained models\n\n3. **Validation Phase**\n   - Run validation test suite\n   - Compare against baseline\n   - Generate performance reports\n   - Approve for deployment\n\n4. **Deployment Phase**\n   - Package model artifacts\n   - Deploy to serving infrastructure\n   - Configure monitoring\n   - Validate production traffic\n\n## Best Practices\n\n### Pipeline Design\n\n- **Modularity**: Each stage should be independently testable\n- **Idempotency**: Re-running stages should be safe\n- **Observability**: Log metrics at every stage\n- **Versioning**: Track data, code, and model versions\n- **Failure Handling**: Implement retry logic and alerting\n\n### Data Management\n\n- Use data validation libraries (Great Expectations, TFX)\n- Version datasets with DVC or similar tools\n- Document feature engineering transformations\n- Maintain data lineage tracking\n\n### Model Operations\n\n- Separate training and serving infrastructure\n- Use model registries (MLflow, Weights & Biases)\n- Implement gradual rollouts for new models\n- Monitor model performance drift\n- Maintain rollback capabilities\n\n### Deployment Strategies\n\n- Start with shadow deployments\n- Use canary releases for validation\n- Implement A/B testing infrastructure\n- Set up automated rollback triggers\n- Monitor latency and throughput\n\n## Integration Points\n\n### Orchestration Tools\n\n- **Apache Airflow**: DAG-based workflow orchestration\n- **Dagster**: Asset-based pipeline orchestration\n- **Kubeflow Pipelines**: Kubernetes-native ML workflows\n- **Prefect**: Modern dataflow automation\n\n### Experiment Tracking\n\n- MLflow for experiment tracking and model registry\n- Weights & Biases for visualization and collaboration\n- TensorBoard for training metrics\n\n### Deployment Platforms\n\n- AWS SageMaker for managed ML infrastructure\n- Google Vertex AI for GCP deployments\n- Azure ML for Azure cloud\n- Kubernetes + KServe for cloud-agnostic serving\n\n## Progressive Disclosure\n\nStart with the basics and gradually add complexity:\n\n1. **Level 1**: Simple linear pipeline (data → train → deploy)\n2. **Level 2**: Add validation and monitoring stages\n3. **Level 3**: Implement hyperparameter tuning\n4. **Level 4**: Add A/B testing and gradual rollouts\n5. **Level 5**: Multi-model pipelines with ensemble strategies\n\n## Common Patterns\n\n### Batch Training Pipeline\n\n```yaml\n# See assets/pipeline-dag.yaml.template\nstages:\n  - name: data_preparation\n    dependencies: []\n  - name: model_training\n    dependencies: [data_preparation]\n  - name: model_evaluation\n    dependencies: [model_training]\n  - name: model_deployment\n    dependencies: [model_evaluation]\n```\n\n### Real-time Feature Pipeline\n\n```python\n# Stream processing for real-time features\n# Combined with batch training\n# See references/data-preparation.md\n```\n\n### Continuous Training\n\n```python\n# Automated retraining on schedule\n# Triggered by data drift detection\n# See references/model-training.md\n```\n\n## Troubleshooting\n\n### Common Issues\n\n- **Pipeline failures**: Check dependencies and data availability\n- **Training instability**: Review hyperparameters and data quality\n- **Deployment issues**: Validate model artifacts and serving config\n- **Performance degradation**: Monitor data drift and model metrics\n\n### Debugging Steps\n\n1. Check pipeline logs for each stage\n2. Validate input/output data at boundaries\n3. Test components in isolation\n4. Review experiment tracking metrics\n5. Inspect model artifacts and metadata\n\n## Next Steps\n\nAfter setting up your pipeline:\n\n1. Explore **hyperparameter-tuning** skill for optimization\n2. Learn **experiment-tracking-setup** for MLflow/W&B\n3. Review **model-deployment-patterns** for serving strategies\n4. Implement monitoring with observability tools\n\n## Related Skills\n\n- **experiment-tracking-setup**: MLflow and Weights & Biases integration\n- **hyperparameter-tuning**: Automated hyperparameter optimization\n- **model-deployment-patterns**: Advanced deployment strategies\n\n## Limitations\n- Use this skill only when the task clearly matches the scope described above.\n- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.\n- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.","tags":["pipeline","workflow","antigravity","awesome","skills","sickn33","agent-skills","agentic-skills","ai-agent-skills","ai-agents","ai-coding","ai-workflows"],"capabilities":["skill","source-sickn33","skill-ml-pipeline-workflow","topic-agent-skills","topic-agentic-skills","topic-ai-agent-skills","topic-ai-agents","topic-ai-coding","topic-ai-workflows","topic-antigravity","topic-antigravity-skills","topic-claude-code","topic-claude-code-skills","topic-codex-cli","topic-codex-skills"],"categories":["antigravity-awesome-skills"],"synonyms":[],"warnings":[],"endpointUrl":"https://skills.sh/sickn33/antigravity-awesome-skills/ml-pipeline-workflow","protocol":"skill","transport":"skills-sh","auth":{"type":"none","details":{"cli":"npx skills add sickn33/antigravity-awesome-skills","source_repo":"https://github.com/sickn33/antigravity-awesome-skills","install_from":"skills.sh"}},"qualityScore":"0.700","qualityRationale":"deterministic score 0.70 from registry signals: · indexed on github topic:agent-skills · 34666 github stars · SKILL.md body (7,431 chars)","verified":false,"liveness":"unknown","lastLivenessCheck":null,"agentReviews":{"count":0,"score_avg":null,"cost_usd_avg":null,"success_rate":null,"latency_p50_ms":null,"narrative_summary":null,"summary_updated_at":null},"enrichmentModel":"deterministic:skill-github:v1","enrichmentVersion":1,"enrichedAt":"2026-04-23T06:51:34.298Z","embedding":null,"createdAt":"2026-04-18T21:40:51.794Z","updatedAt":"2026-04-23T06:51:34.298Z","lastSeenAt":"2026-04-23T06:51:34.298Z","tsv":"'1':153,303,330,593,595,734,770 '2':178,320,349,602,604,741,778 '3':196,366,610,612,747,787 '4':210,382,616,618,752,796 '5':226,625,627,757 'a/b':217,498,620 'action':75 'add':591,605,619 'advanc':823 'agnost':581 'ai':567 'airflow':165,515 'alert':435 'apach':514 'appli':67,343 'approv':379 'architectur':155,273 'artifact':387,720,760 'ask':859 'asset':274,278,523 'asset-bas':522 'assets/pipeline-dag.yaml.template':324,642 'autom':129,228,503,537,688,816 'avail':708 'aw':559 'azur':571,574 'b':786 'base':138,518,524 'baselin':375 'basic':299,588 'batch':637,681 'best':69,261,397 'bias':472,548,811 'blue':235 'blue-green':234 'boundari':746,867 'build':93,113 'canari':232,493 'capabl':152,485 'check':185,342,704,735 'checklist':296 'clarif':861 'clarifi':61 'clean':252 'clear':834 'cloud':575,580 'cloud-agnost':579 'code':425 'collabor':552 'combin':679 'common':635,700 'compar':373 'comparison':224 'complet':5,22 'complex':592 'compon':143,168,749 'comprehens':90 'config':723 'configur':289,321,392 'constraint':63 'contain':280 'continu':685 'core':151 'creat':135 'criteria':870 'dag':137,162,282,517 'dag-bas':136,516 'dagster':166,521 'data':14,31,102,126,171,179,181,189,251,308,310,331,336,340,355,424,436,439,457,599,645,652,694,707,714,727,744 'data-preparation.md':250 'dataflow':536 'dataset':348,446 'debug':732 'defin':304 'degrad':725 'depend':169,322,647,651,657,663,705 'deploy':18,35,107,128,227,233,237,269,294,319,381,383,388,486,491,557,570,601,662,716,791,821,824 'describ':838 'design':119,161,400 'detail':80,248 'detect':222,696 'differ':53 'directori':246,279 'disclosur':584 'distribut':207 'document':242,452 'domain':54 'drift':482,695,728 'dvc':448 'end':7,9,24,26,157,159 'end-to-end':6,23,156 'engin':187,256,313,454 'ensembl':633 'environ':850 'environment-specif':849 'error':173 'evalu':656,665 'everi':420 'exampl':81,327 'execut':356 'expect':443 'experi':204,360,538,542,754,781,805 'experiment-tracking-setup':780,804 'expert':855 'explor':771 'failur':429,703 'featur':186,255,312,344,453,669,678 'flow':172 'framework':214 'full':100,326 'gcp':569 'generat':376 'goal':62 'googl':565 'gradual':474,590,623 'great':442 'green':236 'guid':249 'guidanc':91 'handl':98,174,430 'hyperparamet':202,614,712,773,814,817 'hyperparameter-tun':772,813 'idempot':408 'implement':125,431,473,497,613,797 'independ':406 'infrastructur':219,391,466,500,564 'ingest':103,309,334 'input':66,864 'input/output':743 'inspect':758 'instabl':710 'instruct':60 'integr':141,206,510,812 'isol':751 'issu':701,717 'job':200,358 'kserv':577 'kubeflow':167,527 'kubernet':530,576 'kubernetes-n':529 'latenc':507 'learn':779 'level':594,603,611,617,626 'librari':441 'lifecycl':101 'limit':826 'lineag':192,458 'linear':597 'load':352 'log':417,737 'logic':433 'maintain':456,483 'manag':203,437,562 'match':835 'mechan':240 'metadata':762 'metric':216,267,362,418,556,731,756 'miss':872 'ml':2,19,47,95,115,123,139,142,532,563,572 'ml-pipeline-workflow':1 'mlflow':470,540,808 'mlflow/w':785 'mlop':10,27 'model':17,34,127,197,211,223,229,314,316,318,365,386,427,460,468,478,480,545,630,649,655,658,661,664,719,730,759,790,820 'model-deployment-pattern':789,819 'model-deployment.md':268 'model-training.md':257 'model-validation.md':263 'modern':535 'modular':401 'monitor':108,393,479,506,608,726,798 'multi':629 'multi-model':628 'name':644,648,654,660 'nativ':531 'need':51 'new':114,477 'next':763 'observ':416,800 'open':84 'oper':461 'optim':777,818 'orchestr':12,29,121,140,163,201,286,512,520,526 'outcom':73 'output':844 'outsid':57 'overview':86 'packag':385 'pattern':164,209,231,270,298,636,792,822 'perform':220,377,481,724 'permiss':865 'phase':333,351,368,384 'pipelin':3,11,20,28,48,96,116,154,188,300,305,399,525,528,598,631,639,670,702,736,769 'pipeline-dag.yaml.template':281 'platform':558 'point':511 'practic':70,262,398 'pre':293 'pre-deploy':292 'prefect':534 'prepar':15,32,104,180,332,646,653 'process':347,673 'product':94,145,328,395 'progress':583 'provid':74,89,150 'python':302,671,687 'qualiti':184,341,715 'raw':335 're':410 're-run':409 'real':667,676 'real-tim':666,675 'refer':241,245 'references/data-preparation.md':684 'references/model-training.md':698 'registri':469,546 'regress':221 'relat':802 'releas':494 'relev':68 'report':378 'reproduc':132 'requir':65,83,863 'resources/implementation-playbook.md':85 'retrain':689 'retri':176,432 'review':711,753,788,856 'rollback':239,484,504 'rollout':475,624 'run':339,369,411 'safe':415 'safeti':866 'sagemak':560 'save':363 'schedul':691 'scope':59,837 'scratch':118 'see':243,323,641,683,697 'separ':462 'serv':230,272,390,465,582,722,794 'set':130,501,766 'setup':301,783,807 'shadow':490 'similar':450 'simpl':596 'skill':40,88,111,149,775,803,829 'skill-ml-pipeline-workflow' 'sourc':338 'source-sickn33' 'specif':851 'split':194 'stage':306,307,403,412,421,609,643,740 'start':488,585 'step':76,733,764 'stop':857 'strategi':177,195,238,265,487,634,795,825 'stream':672 'substitut':847 'success':869 'suit':372 'system':124,146 'task':43,833 'templat':276,283,290 'tensorboard':553 'test':218,371,499,621,748,853 'testabl':407 'tfx':444 'throughput':509 'time':668,677 'tool':56,451,513,801 'topic-agent-skills' 'topic-agentic-skills' 'topic-ai-agent-skills' 'topic-ai-agents' 'topic-ai-coding' 'topic-ai-workflows' 'topic-antigravity' 'topic-antigravity-skills' 'topic-claude-code' 'topic-claude-code-skills' 'topic-codex-cli' 'topic-codex-skills' 'track':205,359,423,459,539,543,755,782,806 'traffic':396 'train':105,133,198,199,208,258,288,315,350,354,357,364,463,555,600,638,650,659,682,686,709 'train/validation/test':193 'training-config.yaml':287 'transform':345,455 'treat':842 'trigger':505,692 'troubleshoot':699 'tune':615,774,815 'unrel':45 'usag':297 'use':38,109,438,467,492,827 'valid':72,106,182,212,213,253,264,295,311,317,367,370,394,440,496,606,718,742,852 'validation-checklist.md':291 'verif':78 'version':190,346,353,422,428,445 'vertex':566 'visual':550 'weight':471,547,810 'workflow':4,21,49,120,134,160,225,259,285,329,519,533 'yaml':640","prices":[{"id":"c78da222-0795-4f01-a2f9-1b0773ee493d","listingId":"2e1fbdbd-79ef-429d-873e-d125439093ae","amountUsd":"0","unit":"free","nativeCurrency":null,"nativeAmount":null,"chain":null,"payTo":null,"paymentMethod":"skill-free","isPrimary":true,"details":{"org":"sickn33","category":"antigravity-awesome-skills","install_from":"skills.sh"},"createdAt":"2026-04-18T21:40:51.794Z"}],"sources":[{"listingId":"2e1fbdbd-79ef-429d-873e-d125439093ae","source":"github","sourceId":"sickn33/antigravity-awesome-skills/ml-pipeline-workflow","sourceUrl":"https://github.com/sickn33/antigravity-awesome-skills/tree/main/skills/ml-pipeline-workflow","isPrimary":false,"firstSeenAt":"2026-04-18T21:40:51.794Z","lastSeenAt":"2026-04-23T06:51:34.298Z"}],"details":{"listingId":"2e1fbdbd-79ef-429d-873e-d125439093ae","quickStartSnippet":null,"exampleRequest":null,"exampleResponse":null,"schema":null,"openapiUrl":null,"agentsTxtUrl":null,"citations":[],"useCases":[],"bestFor":[],"notFor":[],"kindDetails":{"org":"sickn33","slug":"ml-pipeline-workflow","github":{"repo":"sickn33/antigravity-awesome-skills","stars":34666,"topics":["agent-skills","agentic-skills","ai-agent-skills","ai-agents","ai-coding","ai-workflows","antigravity","antigravity-skills","claude-code","claude-code-skills","codex-cli","codex-skills","cursor","cursor-skills","developer-tools","gemini-cli","gemini-skills","kiro","mcp","skill-library"],"license":"mit","html_url":"https://github.com/sickn33/antigravity-awesome-skills","pushed_at":"2026-04-23T06:41:03Z","description":"Installable GitHub library of 1,400+ agentic skills for Claude Code, Cursor, Codex CLI, Gemini CLI, Antigravity, and more. Includes installer CLI, bundles, workflows, and official/community skill collections.","skill_md_sha":"14c9b4a17cfa7ad71ed157fb8a5946c02a63aafc","skill_md_path":"skills/ml-pipeline-workflow/SKILL.md","default_branch":"main","skill_tree_url":"https://github.com/sickn33/antigravity-awesome-skills/tree/main/skills/ml-pipeline-workflow"},"layout":"multi","source":"github","category":"antigravity-awesome-skills","frontmatter":{"name":"ml-pipeline-workflow","description":"Complete end-to-end MLOps pipeline orchestration from data preparation through model deployment."},"skills_sh_url":"https://skills.sh/sickn33/antigravity-awesome-skills/ml-pipeline-workflow"},"updatedAt":"2026-04-23T06:51:34.298Z"}}