{"id":"5bb17dbc-4276-4792-8b7f-86cacac580d9","shortId":"z8qYWu","kind":"skill","title":"dspy-output-refinement-constraints","tagline":"This skill should be used when the user asks to \"refine DSPy outputs\", \"enforce constraints\", \"use dspy.Refine\", \"select best output\", \"use dspy.BestOfN\", mentions \"output validation\", \"constraint checking\", \"multi-attempt generation\", \"reward function\", or needs to improve outpu","description":"# DSPy Output Refinement & Constraints\n\n## Goal\n\nImprove output quality using iterative refinement (dspy.Refine) and best-of-N selection (dspy.BestOfN) with custom constraint validation.\n\n## When to Use\n\n- Outputs need format validation (JSON, specific structure)\n- Length constraints (max tokens, word count)\n- Content requirements (must include X, avoid Y)\n- Quality improvement through multiple attempts\n- Replacing deprecated Assert/Suggest patterns\n\n## Related Skills\n\n- Design signatures: [dspy-signature-designer](../dspy-signature-designer/SKILL.md)\n- Optimize programs: [dspy-miprov2-optimizer](../dspy-miprov2-optimizer/SKILL.md)\n- Evaluate quality: [dspy-evaluation-suite](../dspy-evaluation-suite/SKILL.md)\n\n## Inputs\n\n| Input | Type | Description |\n|-------|------|-------------|\n| `module` | `dspy.Module` | Module to refine |\n| `reward_fn` | `callable` | Constraint validation function |\n| `N` | `int` | Number of attempts |\n| `threshold` | `float` | Minimum reward to accept |\n\n## Outputs\n\n| Output | Type | Description |\n|--------|------|-------------|\n| `refined_output` | `dspy.Prediction` | Validated, refined result |\n\n## Workflow\n\n### Phase 1: dspy.Refine for Iterative Improvement\n\nRefine iteratively improves outputs across multiple attempts:\n\n```python\nimport dspy\n\ndspy.configure(lm=dspy.LM(\"openai/gpt-4o-mini\"))\n\n# Base module\nsummarizer = dspy.ChainOfThought(\"document -> summary: str\")\n\n# Reward function: checks constraints\ndef summary_reward(args, pred):\n    summary = pred.summary\n    word_count = len(summary.split())\n\n    if word_count > 100 or len(summary) < 50:\n        return 0.0\n    if \"important\" not in summary.lower():\n        return 0.5\n    return 1.0\n\n# Refine module\nrefined_summarizer = dspy.Refine(\n    module=summarizer,\n    reward_fn=summary_reward,\n    N=3,\n    threshold=1.0\n)\n\n# Use it\nresult = refined_summarizer(document=\"Long document text here...\")\nprint(result.summary)\n```\n\n### Phase 2: dspy.BestOfN for Selection\n\nGenerate N outputs and pick the best:\n\n```python\nimport dspy\n\ndef json_reward(args, pred):\n    \"\"\"Validate JSON format and fields.\"\"\"\n    import json\n    try:\n        data = json.loads(pred.output)\n        if not {'name', 'age', 'email'}.issubset(data.keys()):\n            return 0.3\n        if '@' not in data.get('email', ''):\n            return 0.5\n        return 1.0\n    except json.JSONDecodeError:\n        return 0.0\n\n# BestOfN: try 5 times, pick best\nextractor = dspy.Predict(\"text -> output: str\")\nbest_extractor = dspy.BestOfN(module=extractor, reward_fn=json_reward, N=5, threshold=1.0)\n\nresult = best_extractor(text=\"John Doe, 30 years old, john@example.com\")\nprint(result.output)  # Best valid JSON\n```\n\n### Phase 3: Multi-Constraint Reward Functions\n\nComplex validation with scoring:\n\n```python\nimport dspy\nimport re\n\ndef comprehensive_reward(args, pred):\n    \"\"\"Validate format, length, and content.\"\"\"\n    text = pred.answer\n    score = 0.0\n\n    # Length: 50-150 words (33%)\n    word_count = len(text.split())\n    if 50 <= word_count <= 150:\n        score += 0.33\n\n    # Format: capitalized, ends with period (33%)\n    if re.match(r'^[A-Z]', text) and text.endswith('.'):\n        score += 0.33\n\n    # Content: required terms present (34%)\n    if all(term in text.lower() for term in ['data', 'analysis']):\n        score += 0.34\n\n    return score\n\n# Use with Refine\nqa = dspy.ChainOfThought(\"question -> answer: str\")\nrefined_qa = dspy.Refine(module=qa, reward_fn=comprehensive_reward, N=4, threshold=0.9)\n\nresult = refined_qa(question=\"What is data science?\")\n```\n\n## Production Example\n\n```python\nimport dspy\nimport json\nimport logging\n\nlogger = logging.getLogger(__name__)\n\nclass StructuredExtractor(dspy.Module):\n    \"\"\"Extract structured data with validation.\"\"\"\n\n    def __init__(self):\n        self.extractor = dspy.Predict(\n            \"text -> json_output: str\"\n        )\n        self.refined = dspy.Refine(\n            module=self.extractor,\n            reward_fn=self.validation_reward,\n            N=3,\n            threshold=0.9\n        )\n\n    def validation_reward(self, args, pred):\n        \"\"\"Validate JSON structure and business logic.\"\"\"\n        try:\n            data = json.loads(pred.json_output)\n            score = 0.0\n\n            # Required fields\n            if {'product', 'price', 'quantity'}.issubset(data.keys()):\n                score += 0.4\n\n            # Type validation\n            if isinstance(data.get('price'), (int, float)) and data['price'] > 0:\n                score += 0.3\n            if isinstance(data.get('quantity'), int) and data['quantity'] > 0:\n                score += 0.3\n\n            return score\n        except (json.JSONDecodeError, TypeError) as e:\n            logger.warning(f\"Validation failed: {e}\")\n            return 0.0\n\n    def forward(self, text: str):\n        try:\n            return self.refined(text=text)\n        except Exception as e:\n            logger.error(f\"Extraction failed: {e}\")\n            return dspy.Prediction(json_output='{}')\n\n# Usage\nextractor = StructuredExtractor()\nresult = extractor(text=\"iPhone 15, $999, quantity: 50\")\nprint(result.json_output)\n```\n\n## Migration from Assert/Suggest\n\nDSPy 2.6+ deprecates `dspy.Assert`/`dspy.Suggest`. Use Refine with reward functions:\n\n```python\n# Old: dspy.Assert(len(output) < 100, \"Too long\")\n# New:\ndef reward(args, pred):\n    return 1.0 if len(pred.output) < 100 else 0.0\n\nrefined = dspy.Refine(module=module, reward_fn=reward, N=3, threshold=1.0)\n```\n\n## Best Practices\n\n1. **Score gradually** - Use 0.0-1.0 range, not binary pass/fail\n2. **Multiple constraints** - Weight each constraint (e.g., 25% each for 4 checks)\n3. **Handle exceptions** - Reward functions should never raise, return 0.0 on error\n4. **Limit attempts** - 3-5 attempts for Refine, 5-10 for BestOfN\n5. **Log failures** - Track which constraints fail most often\n\n## Limitations\n\n- Each attempt costs an additional LLM call\n- Reward functions don't receive feedback prompts (unlike GEPA)\n- BestOfN is expensive (N × cost)\n- No automatic constraint learning (manual reward design)\n- Refine may not improve if base module is fundamentally wrong\n\n## Official Documentation\n\n- **DSPy Documentation**: https://dspy.ai/\n- **DSPy GitHub**: https://github.com/stanfordnlp/dspy\n- **Refine Module**: https://dspy.ai/api/modules/Refine/","tags":["dspy","output","refinement","constraints","skills","omidzamani","agent-skills","claude-code","claude-skills","llm","prompt-optimization","rag"],"capabilities":["skill","source-omidzamani","skill-dspy-output-refinement-constraints","topic-agent-skills","topic-claude-code","topic-claude-skills","topic-dspy","topic-llm","topic-prompt-optimization","topic-rag"],"categories":["dspy-skills"],"synonyms":[],"warnings":[],"endpointUrl":"https://skills.sh/OmidZamani/dspy-skills/dspy-output-refinement-constraints","protocol":"skill","transport":"skills-sh","auth":{"type":"none","details":{"cli":"npx skills add OmidZamani/dspy-skills","source_repo":"https://github.com/OmidZamani/dspy-skills","install_from":"skills.sh"}},"qualityScore":"0.487","qualityRationale":"deterministic score 0.49 from registry signals: · indexed on github topic:agent-skills · 74 github stars · SKILL.md body (6,307 chars)","verified":false,"liveness":"unknown","lastLivenessCheck":null,"agentReviews":{"count":0,"score_avg":null,"cost_usd_avg":null,"success_rate":null,"latency_p50_ms":null,"narrative_summary":null,"summary_updated_at":null},"enrichmentModel":"deterministic:skill-github:v1","enrichmentVersion":1,"enrichedAt":"2026-05-02T06:55:44.648Z","embedding":null,"createdAt":"2026-04-18T22:14:15.731Z","updatedAt":"2026-05-02T06:55:44.648Z","lastSeenAt":"2026-05-02T06:55:44.648Z","tsv":"'-1.0':648 '-10':686 '-150':371 '-5':681 '/api/modules/refine/':751 '/dspy-evaluation-suite/skill.md':121 '/dspy-miprov2-optimizer/skill.md':114 '/dspy-signature-designer/skill.md':107 '/stanfordnlp/dspy':746 '0':531,542 '0.0':210,299,368,509,558,629,647,674 '0.3':286,533,544 '0.33':384,401 '0.34':418 '0.4':519 '0.5':217,293 '0.9':441,490 '1':160,643 '1.0':219,234,295,323,623,640 '100':204,614,627 '15':589 '150':382 '2':248,653 '2.6':600 '25':660 '3':232,340,488,638,665,680 '30':330 '33':373,390 '34':406 '4':439,663,677 '5':302,321,685,689 '50':208,370,379,592 '999':590 'a-z':394 'accept':147 'across':169 'addit':703 'age':281 'analysi':416 'answer':427 'arg':193,265,358,495,620 'ask':14 'assert/suggest':97,598 'attempt':35,94,141,171,679,682,700 'automat':721 'avoid':88 'base':179,732 'best':24,58,258,305,311,325,336,641 'best-of-n':57 'bestofn':300,688,715 'binari':651 'busi':501 'call':705 'callabl':133 'capit':386 'check':32,188,664 'class':462 'complex':346 'comprehens':356,436 'constraint':5,20,31,47,65,78,134,189,343,655,658,694,722 'content':83,364,402 'cost':701,719 'count':82,198,203,375,381 'custom':64 'data':275,415,448,467,504,529,540 'data.get':290,524,536 'data.keys':284,517 'def':190,262,355,470,491,559,618 'deprec':96,601 'descript':125,151 'design':101,106,726 'document':183,240,242,738,740 'doe':329 'dspi':2,17,44,104,111,118,174,261,352,454,599,739,742 'dspy-evaluation-suit':117 'dspy-miprov2-optimizer':110 'dspy-output-refinement-constraint':1 'dspy-signature-design':103 'dspy.ai':741,750 'dspy.ai/api/modules/refine/':749 'dspy.assert':602,611 'dspy.bestofn':27,62,249,313 'dspy.chainofthought':182,425 'dspy.configure':175 'dspy.lm':177 'dspy.module':127,464 'dspy.predict':307,474 'dspy.prediction':154,579 'dspy.refine':22,55,161,224,431,480,631 'dspy.suggest':603 'e':551,556,572,577 'e.g':659 'els':628 'email':282,291 'end':387 'enforc':19 'error':676 'evalu':115,119 'exampl':451 'except':296,547,569,570,667 'expens':717 'extract':465,575 'extractor':306,312,315,326,583,586 'f':553,574 'fail':555,576,695 'failur':691 'feedback':711 'field':271,511 'float':143,527 'fn':132,228,317,435,484,635 'format':72,269,361,385 'forward':560 'function':38,136,187,345,608,669,707 'fundament':735 'generat':36,252 'gepa':714 'github':743 'github.com':745 'github.com/stanfordnlp/dspy':744 'goal':48 'gradual':645 'handl':666 'import':173,212,260,272,351,353,453,455,457 'improv':42,49,91,164,167,730 'includ':86 'init':471 'input':122,123 'int':138,526,538 'iphon':588 'isinst':523,535 'issubset':283,516 'iter':53,163,166 'john':328 'john@example.com':333 'json':74,263,268,273,318,338,456,476,498,580 'json.jsondecodeerror':297,548 'json.loads':276,505 'learn':723 'len':199,206,376,612,625 'length':77,362,369 'limit':678,698 'llm':704 'lm':176 'log':458,690 'logger':459 'logger.error':573 'logger.warning':552 'logging.getlogger':460 'logic':502 'long':241,616 'manual':724 'max':79 'may':728 'mention':28 'migrat':596 'minimum':144 'miprov2':112 'modul':126,128,180,221,225,314,432,481,632,633,733,748 'multi':34,342 'multi-attempt':33 'multi-constraint':341 'multipl':93,170,654 'must':85 'n':60,137,231,253,320,438,487,637,718 'name':280,461 'need':40,71 'never':671 'new':617 'number':139 'offici':737 'often':697 'old':332,610 'openai/gpt-4o-mini':178 'optim':108,113 'outpu':43 'output':3,18,25,29,45,50,70,148,149,153,168,254,309,477,507,581,595,613 'pass/fail':652 'pattern':98 'period':389 'phase':159,247,339 'pick':256,304 'practic':642 'pred':194,266,359,496,621 'pred.answer':366 'pred.json':506 'pred.output':277,626 'pred.summary':196 'present':405 'price':514,525,530 'print':245,334,593 'product':450,513 'program':109 'prompt':712 'python':172,259,350,452,609 'qa':424,430,433,444 'qualiti':51,90,116 'quantiti':515,537,541,591 'question':426,445 'r':393 'rais':672 'rang':649 're':354 're.match':392 'receiv':710 'refin':4,16,46,54,130,152,156,165,220,222,238,423,429,443,605,630,684,727,747 'relat':99 'replac':95 'requir':84,403,510 'result':157,237,324,442,585 'result.json':594 'result.output':335 'result.summary':246 'return':209,216,218,285,292,294,298,419,545,557,565,578,622,673 'reward':37,131,145,186,192,227,230,264,316,319,344,357,434,437,483,486,493,607,619,634,636,668,706,725 'scienc':449 'score':349,367,383,400,417,420,508,518,532,543,546,644 'select':23,61,251 'self':472,494,561 'self.extractor':473,482 'self.refined':479,566 'self.validation':485 'signatur':102,105 'skill':7,100 'skill-dspy-output-refinement-constraints' 'source-omidzamani' 'specif':75 'str':185,310,428,478,563 'structur':76,466,499 'structuredextractor':463,584 'suit':120 'summar':181,223,226,239 'summari':184,191,195,207,229 'summary.lower':215 'summary.split':200 'term':404,409,413 'text':243,308,327,365,397,475,562,567,568,587 'text.endswith':399 'text.lower':411 'text.split':377 'threshold':142,233,322,440,489,639 'time':303 'token':80 'topic-agent-skills' 'topic-claude-code' 'topic-claude-skills' 'topic-dspy' 'topic-llm' 'topic-prompt-optimization' 'topic-rag' 'track':692 'tri':274,301,503,564 'type':124,150,520 'typeerror':549 'unlik':713 'usag':582 'use':10,21,26,52,69,235,421,604,646 'user':13 'valid':30,66,73,135,155,267,337,347,360,469,492,497,521,554 'weight':656 'word':81,197,202,372,374,380 'workflow':158 'wrong':736 'x':87 'y':89 'year':331 'z':396","prices":[{"id":"01079c67-e676-49e0-a4c5-9bb1a2f31c0e","listingId":"5bb17dbc-4276-4792-8b7f-86cacac580d9","amountUsd":"0","unit":"free","nativeCurrency":null,"nativeAmount":null,"chain":null,"payTo":null,"paymentMethod":"skill-free","isPrimary":true,"details":{"org":"OmidZamani","category":"dspy-skills","install_from":"skills.sh"},"createdAt":"2026-04-18T22:14:15.731Z"}],"sources":[{"listingId":"5bb17dbc-4276-4792-8b7f-86cacac580d9","source":"github","sourceId":"OmidZamani/dspy-skills/dspy-output-refinement-constraints","sourceUrl":"https://github.com/OmidZamani/dspy-skills/tree/master/skills/dspy-output-refinement-constraints","isPrimary":false,"firstSeenAt":"2026-04-18T22:14:15.731Z","lastSeenAt":"2026-05-02T06:55:44.648Z"}],"details":{"listingId":"5bb17dbc-4276-4792-8b7f-86cacac580d9","quickStartSnippet":null,"exampleRequest":null,"exampleResponse":null,"schema":null,"openapiUrl":null,"agentsTxtUrl":null,"citations":[],"useCases":[],"bestFor":[],"notFor":[],"kindDetails":{"org":"OmidZamani","slug":"dspy-output-refinement-constraints","github":{"repo":"OmidZamani/dspy-skills","stars":74,"topics":["agent-skills","claude-code","claude-skills","dspy","llm","prompt-optimization","rag"],"license":"mit","html_url":"https://github.com/OmidZamani/dspy-skills","pushed_at":"2026-02-21T12:49:43Z","description":"Collection of Claude Skills for DSPy framework - program language models, optimize prompts, and build RAG pipelines systematically","skill_md_sha":"f0c8bfb2497c372fdc8a9da8c9b138d18f4155d5","skill_md_path":"skills/dspy-output-refinement-constraints/SKILL.md","default_branch":"master","skill_tree_url":"https://github.com/OmidZamani/dspy-skills/tree/master/skills/dspy-output-refinement-constraints"},"layout":"multi","source":"github","category":"dspy-skills","frontmatter":{"name":"dspy-output-refinement-constraints","description":"This skill should be used when the user asks to \"refine DSPy outputs\", \"enforce constraints\", \"use dspy.Refine\", \"select best output\", \"use dspy.BestOfN\", mentions \"output validation\", \"constraint checking\", \"multi-attempt generation\", \"reward function\", or needs to improve output quality through iterative refinement or best-of-N selection with custom constraints."},"skills_sh_url":"https://skills.sh/OmidZamani/dspy-skills/dspy-output-refinement-constraints"},"updatedAt":"2026-05-02T06:55:44.648Z"}}