{"id":"f29430f3-30ee-45ba-8160-e45fc99d7694","shortId":"6VPdcd","kind":"skill","title":"systematic-debugging","tagline":"Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes","description":"# Systematic Debugging\n\n## Overview\n\n**Announce at start:** \"Systematic Debugging skill activated.\"\n\nRandom fixes waste time and create new bugs. Quick patches mask underlying issues.\n\n**Core principle:** ALWAYS find root cause before attempting fixes. Symptom fixes are failure.\n\n**Violating the letter of this process is violating the spirit of debugging.**\n\n## The Iron Law\n\n```\nNO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST\n```\n\nIf you haven't completed Phase 1, you cannot propose fixes.\n\n## When to Use\n\nUse for ANY technical issue:\n- Test failures\n- Bugs in production\n- Unexpected behavior\n- Performance problems\n- Build failures\n- Integration issues\n\n**Use this ESPECIALLY when:**\n- Under time pressure (emergencies make guessing tempting)\n- \"Just one quick fix\" seems obvious\n- You've already tried multiple fixes\n- Previous fix didn't work\n- You don't fully understand the issue\n\n**Don't skip when:**\n- Issue seems simple (simple bugs have root causes too)\n- You're in a hurry (rushing guarantees rework)\n- Manager wants it fixed NOW (systematic is faster than thrashing)\n\n## The Four Phases\n\nYou MUST complete each phase before proceeding to the next.\n\n### Phase 1: Root Cause Investigation\n\n**BEFORE attempting ANY fix:**\n\n1. **Read Error Messages Carefully**\n   - Don't skip past errors or warnings\n   - They often contain the exact solution\n   - Read stack traces completely\n   - Note line numbers, file paths, error codes\n\n2. **Reproduce Consistently**\n   - Can you trigger it reliably?\n   - What are the exact steps?\n   - Does it happen every time?\n   - If not reproducible → gather more data, don't guess\n\n3. **Check Recent Changes**\n   - What changed that could cause this?\n   - Git diff, recent commits\n   - New dependencies, config changes\n   - Environmental differences\n\n4. **Gather Evidence in Multi-Component Systems**\n\n   **WHEN system has multiple components (CI → build → signing, API → service → database):**\n\n   **BEFORE proposing fixes, add diagnostic instrumentation:**\n   ```\n   For EACH component boundary:\n     - Log what data enters component\n     - Log what data exits component\n     - Verify environment/config propagation\n     - Check state at each layer\n\n   Run once to gather evidence showing WHERE it breaks\n   THEN analyze evidence to identify failing component\n   THEN investigate that specific component\n   ```\n\n   **Example (multi-layer system):**\n   ```bash\n   # Layer 1: Workflow\n   echo \"=== Secrets available in workflow: ===\"\n   echo \"IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}\"\n\n   # Layer 2: Build script\n   echo \"=== Env vars in build script: ===\"\n   env | grep IDENTITY || echo \"IDENTITY not in environment\"\n\n   # Layer 3: Signing script\n   echo \"=== Keychain state: ===\"\n   security list-keychains\n   security find-identity -v\n\n   # Layer 4: Actual signing\n   codesign --sign \"$IDENTITY\" --verbose=4 \"$APP\"\n   ```\n\n   **This reveals:** Which layer fails (secrets → workflow ✓, workflow → build ✗)\n\n5. **Trace Data Flow**\n\n   **WHEN error is deep in call stack:**\n\n   See `root-cause-tracing.md` in this directory for the complete backward tracing technique.\n\n   **Quick version:**\n   - Where does bad value originate?\n   - What called this with bad value?\n   - Keep tracing up until you find the source\n   - Fix at source, not at symptom\n\n### Phase 2: Pattern Analysis\n\n**Find the pattern before fixing:**\n\n1. **Find Working Examples**\n   - Locate similar working code in same codebase\n   - What works that's similar to what's broken?\n\n2. **Compare Against References**\n   - If implementing pattern, read reference implementation COMPLETELY\n   - Don't skim - read every line\n   - Understand the pattern fully before applying\n\n3. **Identify Differences**\n   - What's different between working and broken?\n   - List every difference, however small\n   - Don't assume \"that can't matter\"\n\n4. **Understand Dependencies**\n   - What other components does this need?\n   - What settings, config, environment?\n   - What assumptions does it make?\n\n### Phase 3: Hypothesis and Testing\n\n**Scientific method:**\n\n1. **Form Single Hypothesis**\n   - State clearly: \"I think X is the root cause because Y\"\n   - Write it down\n   - Be specific, not vague\n\n2. **Test Minimally**\n   - Make the SMALLEST possible change to test hypothesis\n   - One variable at a time\n   - Don't fix multiple things at once\n\n3. **Verify Before Continuing**\n   - Did it work? Yes → Phase 4\n   - Didn't work? Form NEW hypothesis\n   - DON'T add more fixes on top\n\n4. **When You Don't Know**\n   - Say \"I don't understand X\"\n   - Don't pretend to know\n   - Ask for help\n   - Research more\n\n### Phase 4: Implementation\n\n**Fix the root cause, not the symptom:**\n\n1. **Create Failing Test Case**\n   - Simplest possible reproduction\n   - Automated test if possible\n   - One-off test script if no framework\n   - MUST have before fixing\n   - Use the `superpowers:test-driven-development` skill for writing proper failing tests\n\n2. **Implement Single Fix**\n   - Address the root cause identified\n   - ONE change at a time\n   - No \"while I'm here\" improvements\n   - No bundled refactoring\n\n3. **Verify Fix**\n   - Test passes now?\n   - No other tests broken?\n   - Issue actually resolved?\n\n4. **If Fix Doesn't Work**\n   - STOP\n   - Count: How many fixes have you tried?\n   - If < 3: Return to Phase 1, re-analyze with new information\n   - **If ≥ 3: STOP and question the architecture (step 5 below)**\n   - DON'T attempt Fix #4 without architectural discussion\n\n5. **If 3+ Fixes Failed: Question Architecture**\n\n   **Pattern indicating architectural problem:**\n   - Each fix reveals new shared state/coupling/problem in different place\n   - Fixes require \"massive refactoring\" to implement\n   - Each fix creates new symptoms elsewhere\n\n   **STOP and question fundamentals:**\n   - Is this pattern fundamentally sound?\n   - Are we \"sticking with it through sheer inertia\"?\n   - Should we refactor architecture vs. continue fixing symptoms?\n\n   **Discuss with your human partner before attempting more fixes**\n\n   This is NOT a failed hypothesis - this is a wrong architecture.\n\n## Red Flags - STOP and Follow Process\n\nIf you catch yourself thinking:\n- \"Quick fix for now, investigate later\"\n- \"Just try changing X and see if it works\"\n- \"Add multiple changes, run tests\"\n- \"Skip the test, I'll manually verify\"\n- \"It's probably X, let me fix that\"\n- \"I don't fully understand but this might work\"\n- \"Pattern says X but I'll adapt it differently\"\n- \"Here are the main problems: [lists fixes without investigation]\"\n- Proposing solutions before tracing data flow\n- **\"One more fix attempt\" (when already tried 2+)**\n- **Each fix reveals new problem in different place**\n\n**ALL of these mean: STOP. Return to Phase 1.**\n\n**If 3+ fixes failed:** Question the architecture (see Phase 4.5)\n\n## your human partner's Signals You're Doing It Wrong\n\n**Watch for these redirections:**\n- \"Is that not happening?\" - You assumed without verifying\n- \"Will it show us...?\" - You should have added evidence gathering\n- \"Stop guessing\" - You're proposing fixes without understanding\n- \"Ultrathink this\" - Question fundamentals, not just symptoms\n- \"We're stuck?\" (frustrated) - Your approach isn't working\n\n**When you see these:** STOP. Return to Phase 1.\n\n## Common Rationalizations\n\n| Excuse | Reality |\n|--------|---------|\n| \"Issue is simple, don't need process\" | Simple issues have root causes too. Process is fast for simple bugs. |\n| \"Emergency, no time for process\" | Systematic debugging is FASTER than guess-and-check thrashing. |\n| \"Just try this first, then investigate\" | First fix sets the pattern. Do it right from the start. |\n| \"I'll write test after confirming fix works\" | Untested fixes don't stick. Test first proves it. |\n| \"Multiple fixes at once saves time\" | Can't isolate what worked. Causes new bugs. |\n| \"Reference too long, I'll adapt the pattern\" | Partial understanding guarantees bugs. Read it completely. |\n| \"I see the problem, let me fix it\" | Seeing symptoms ≠ understanding root cause. |\n| \"One more fix attempt\" (after 2+ failures) | 3+ failures = architectural problem. Question pattern, don't fix again. |\n\n## Quick Reference\n\n| Phase | Key Activities | Success Criteria |\n|-------|---------------|------------------|\n| **1. Root Cause** | Read errors, reproduce, check changes, gather evidence | Understand WHAT and WHY |\n| **2. Pattern** | Find working examples, compare | Identify differences |\n| **3. Hypothesis** | Form theory, test minimally | Confirmed or new hypothesis |\n| **4. Implementation** | Create test, fix, verify | Bug resolved, tests pass |\n\n## When Process Reveals \"No Root Cause\"\n\nIf systematic investigation reveals issue is truly environmental, timing-dependent, or external:\n\n1. You've completed the process\n2. Document what you investigated\n3. Implement appropriate handling (retry, timeout, error message)\n4. Add monitoring/logging for future investigation\n\n**But:** 95% of \"no root cause\" cases are incomplete investigation.\n\n## Supporting Techniques\n\nThese techniques are part of systematic debugging and available in this directory:\n\n- **`root-cause-tracing.md`** - Trace bugs backward through call stack to find original trigger\n- **`defense-in-depth.md`** - Add validation at multiple layers after finding root cause\n- **`condition-based-waiting.md`** - Replace arbitrary timeouts with condition polling\n\n**Related skills:**\n- **superpowers:test-driven-development** - For creating failing test case (Phase 4, Step 1)\n- **superpowers:verification-before-completion** - Verify fix worked before claiming success\n\n## Real-World Impact\n\nFrom debugging sessions:\n- Systematic approach: 15-30 minutes to fix\n- Random fixes approach: 2-3 hours of thrashing\n- First-time fix rate: 95% vs 40%\n- New bugs introduced: Near zero vs common","tags":["systematic","debugging","coco","rkz91","agent-skills","agents-md","ai-agents","claude-code","codex","cursor","developer-tools","llm-tools"],"capabilities":["skill","source-rkz91","skill-systematic-debugging","topic-agent-skills","topic-agents-md","topic-ai-agents","topic-claude-code","topic-codex","topic-cursor","topic-developer-tools","topic-llm-tools","topic-mcp","topic-pm-tools","topic-product-management","topic-productivity"],"categories":["coco"],"synonyms":[],"warnings":[],"endpointUrl":"https://skills.sh/rkz91/coco/systematic-debugging","protocol":"skill","transport":"skills-sh","auth":{"type":"none","details":{"cli":"npx skills add rkz91/coco","source_repo":"https://github.com/rkz91/coco","install_from":"skills.sh"}},"qualityScore":"0.453","qualityRationale":"deterministic score 0.45 from registry signals: · indexed on github topic:agent-skills · 7 github stars · SKILL.md body (9,782 chars)","verified":false,"liveness":"unknown","lastLivenessCheck":null,"agentReviews":{"count":0,"score_avg":null,"cost_usd_avg":null,"success_rate":null,"latency_p50_ms":null,"narrative_summary":null,"summary_updated_at":null},"enrichmentModel":"deterministic:skill-github:v1","enrichmentVersion":1,"enrichedAt":"2026-05-18T19:14:09.320Z","embedding":null,"createdAt":"2026-05-18T13:21:42.662Z","updatedAt":"2026-05-18T19:14:09.320Z","lastSeenAt":"2026-05-18T19:14:09.320Z","tsv":"'-3':1354 '-30':1346 '1':81,187,195,346,470,560,660,752,957,1032,1171,1232,1324 '15':1345 '2':224,360,462,490,582,697,940,1152,1185,1238,1353 '3':251,378,513,554,605,720,748,760,779,959,1154,1193,1243 '4':271,394,401,535,614,628,651,733,773,1203,1251,1322 '4.5':967 '40':1365 '5':412,767,777 '95':1258,1363 'activ':26,1168 'actual':395,731 'ad':997 'adapt':915,1124 'add':293,623,880,1252,1293 'address':701 'alreadi':126,938 'alway':42 'analysi':464 'analyz':328,755 'announc':20 'api':287 'app':402 'appli':512 'approach':1020,1344,1352 'appropri':1245 'arbitrari':1304 'architectur':765,775,783,786,829,853,964,1156 'ask':645 'assum':530,987 'assumpt':549 'attempt':47,192,771,840,936,1150 'autom':668 'avail':350,1277 'backward':431,1284 'bad':438,445 'bash':344 'behavior':13,100 'boundari':299 'break':326 'broken':489,522,729 'bug':8,34,96,150,1055,1118,1130,1209,1283,1367 'build':103,285,361,367,411 'bundl':718 'call':421,442,1286 'cannot':83 'care':199 'case':664,1263,1320 'catch':862 'caus':45,72,153,189,259,572,656,704,1048,1116,1146,1173,1218,1262,1301 'chang':254,256,268,589,707,873,882,1178 'check':252,313,1069,1177 'ci':284 'claim':1334 'clear':565 'code':223,477 'codebas':480 'codesign':397 'commit':264 'common':1033,1372 'compar':491,1190 'complet':79,178,216,430,500,1133,1235,1329 'compon':277,283,298,304,309,333,338,540 'condit':1307 'condition-based-waiting.md':1302 'config':267,546 'confirm':1093,1199 'consist':226 'contain':209 'continu':608,831 'core':40 'could':258 'count':740 'creat':32,661,805,1205,1317 'criteria':1170 'data':247,302,307,414,931 'databas':289 'debug':3,18,24,64,1062,1275,1341 'deep':419 'defense-in-depth.md':1292 'depend':266,537,1229 'develop':690,1315 'diagnost':294 'didn':132,615 'diff':262 'differ':270,515,518,525,795,917,947,1192 'directori':427,1280 'discuss':776,834 'document':1239 'doesn':736 'driven':689,1314 'echo':348,353,363,372,381 'elsewher':808 'emerg':114,1056 'encount':6 'enter':303 'env':364,369 'environ':376,547 'environment':269,1226 'environment/config':311 'error':197,204,222,417,1175,1249 'especi':109 'everi':240,505,524 'evid':273,322,329,998,1180 'exact':211,235 'exampl':339,473,1189 'excus':1035 'exit':308 'extern':1231 'fail':332,407,662,695,781,847,961,1318 'failur':10,52,95,104,1153,1155 'fast':1052 'faster':170,1064 'file':220 'find':43,390,452,465,471,1187,1289,1299 'find-ident':389 'first':74,1074,1077,1102,1359 'first-tim':1358 'fix':16,28,48,50,69,85,121,129,131,166,194,292,455,469,600,625,653,683,700,722,735,743,772,780,789,797,804,832,842,866,898,924,935,942,960,1005,1078,1094,1097,1106,1140,1149,1162,1207,1331,1349,1351,1361 'flag':855 'flow':415,932 'follow':858 'form':561,618,1195 'four':174 'framework':679 'frustrat':1018 'fulli':138,510,903 'fundament':812,816,1011 'futur':1255 'gather':245,272,321,999,1179 'git':261 'grep':370 'guarante':161,1129 'guess':116,250,1001,1067 'guess-and-check':1066 'handl':1246 'happen':239,985 'haven':77 'help':647 'hour':1355 'howev':526 'human':837,969 'hurri':159 'hypothesi':555,563,592,620,848,1194,1202 'ident':354,355,357,371,373,391,399 'identifi':331,514,705,1191 'impact':1339 'implement':495,499,652,698,802,1204,1244 'improv':716 'incomplet':1265 'indic':785 'inertia':825 'inform':758 'instrument':295 'integr':105 'introduc':1368 'investig':73,190,335,869,926,1076,1221,1242,1256,1266 'iron':66 'isn':1021 'isol':1113 'issu':39,93,106,141,146,730,1037,1045,1223 'keep':447 'key':1167 'keychain':382,387 'know':633,644 'later':870 'law':67 'layer':317,342,345,359,377,393,406,1297 'let':896,1138 'letter':55 'line':218,506 'list':386,523,923 'list-keychain':385 'll':889,914,1089,1123 'locat':474 'log':300,305 'long':1121 'm':714 'main':921 'make':115,552,585 'manag':163 'mani':742 'manual':890 'mask':37 'massiv':799 'matter':534 'mean':952 'messag':198,1250 'method':559 'might':907 'minim':584,1198 'minut':1347 'monitoring/logging':1253 'multi':276,341 'multi-compon':275 'multi-lay':340 'multipl':128,282,601,881,1105,1296 'must':177,680 'near':1369 'need':543,1042 'new':33,265,619,757,791,806,944,1117,1201,1366 'next':185 'note':217 'number':219 'obvious':123 'often':208 'one':119,593,673,706,933,1147 'one-off':672 'origin':440,1290 'overview':19 'part':1272 'partial':1127 'partner':838,970 'pass':724,1212 'past':203 'patch':36 'path':221 'pattern':463,467,496,509,784,815,909,1081,1126,1159,1186 'perform':101 'phase':80,175,180,186,461,553,613,650,751,956,966,1031,1166,1321 'place':796,948 'poll':1308 'possibl':588,666,671 'pressur':113 'pretend':642 'previous':130 'principl':41 'probabl':894 'problem':102,787,922,945,1137,1157 'proceed':182 'process':58,859,1043,1050,1060,1214,1237 'product':98 'propag':312 'proper':694 'propos':15,84,291,927,1004 'prove':1103 'question':763,782,811,962,1010,1158 'quick':35,120,434,865,1164 'random':27,1350 'rate':1362 'ration':1034 're':156,754,974,1003,1016 're-analyz':753 'read':196,213,497,504,1131,1174 'real':1337 'real-world':1336 'realiti':1036 'recent':253,263 'red':854 'redirect':981 'refactor':719,800,828 'refer':493,498,1119,1165 'relat':1309 'reliabl':231 'replac':1303 'reproduc':225,244,1176 'reproduct':667 'requir':798 'research':648 'resolv':732,1210 'retri':1247 'return':749,954,1029 'reveal':404,790,943,1215,1222 'rework':162 'right':1084 'root':44,71,152,188,571,655,703,1047,1145,1172,1217,1261,1300 'root-cause-tracing.md':424,1281 'run':318,883 'rush':160 'save':1109 'say':634,910 'scientif':558 'script':362,368,380,676 'secret':349,408 'secur':384,388 'see':423,876,965,1026,1135,1142 'seem':122,147 'servic':288 'session':1342 'set':356,545,1079 'share':792 'sheer':824 'show':323,992 'sign':286,379,396,398 'signal':972 'similar':475,485 'simpl':148,149,1039,1044,1054 'simplest':665 'singl':562,699 'skill':25,691,1310 'skill-systematic-debugging' 'skim':503 'skip':144,202,885 'small':527 'smallest':587 'solut':212,928 'sound':817 'sourc':454,457 'source-rkz91' 'specif':337,579 'spirit':62 'stack':214,422,1287 'start':22,1087 'state':314,383,564 'state/coupling/problem':793 'step':236,766,1323 'stick':820,1100 'stop':739,761,809,856,953,1000,1028 'stuck':1017 'success':1169,1335 'superpow':686,1311,1325 'support':1267 'symptom':49,460,659,807,833,1014,1143 'system':278,280,343 'systemat':2,17,23,168,1061,1220,1274,1343 'systematic-debug':1 'technic':92 'techniqu':433,1268,1270 'tempt':117 'test':9,94,557,583,591,663,669,675,688,696,723,728,884,887,1091,1101,1197,1206,1211,1313,1319 'test-driven-develop':687,1312 'theori':1196 'thing':602 'think':567,864 'thrash':172,1070,1357 'time':30,112,241,597,710,1058,1110,1228,1360 'timeout':1248,1305 'timing-depend':1227 'top':627 'topic-agent-skills' 'topic-agents-md' 'topic-ai-agents' 'topic-claude-code' 'topic-codex' 'topic-cursor' 'topic-developer-tools' 'topic-llm-tools' 'topic-mcp' 'topic-pm-tools' 'topic-product-management' 'topic-productivity' 'trace':215,413,432,448,930,1282 'tri':127,746,872,939,1072 'trigger':229,1291 'truli':1225 'ultrathink':1008 'under':38 'understand':139,507,536,638,904,1007,1128,1144,1181 'unexpect':12,99 'unset':358 'untest':1096 'us':993 'use':4,88,89,107,684 'v':392 'vagu':581 'valid':1294 'valu':439,446 'var':365 'variabl':594 've':125,1234 'verbos':400 'verif':1327 'verifi':310,606,721,891,989,1208,1330 'verification-before-complet':1326 'version':435 'violat':53,60 'vs':830,1364,1371 'want':164 'warn':206 'wast':29 'watch':978 'without':70,774,925,988,1006 'work':134,472,476,482,520,611,617,738,879,908,1023,1095,1115,1188,1332 'workflow':347,352,409,410 'world':1338 'write':575,693,1090 'wrong':852,977 'x':568,639,874,895,911 'y':574 'yes':612 'zero':1370","prices":[{"id":"90c846e9-5dbf-4af7-a340-270bb7edc302","listingId":"f29430f3-30ee-45ba-8160-e45fc99d7694","amountUsd":"0","unit":"free","nativeCurrency":null,"nativeAmount":null,"chain":null,"payTo":null,"paymentMethod":"skill-free","isPrimary":true,"details":{"org":"rkz91","category":"coco","install_from":"skills.sh"},"createdAt":"2026-05-18T13:21:42.662Z"}],"sources":[{"listingId":"f29430f3-30ee-45ba-8160-e45fc99d7694","source":"github","sourceId":"rkz91/coco/systematic-debugging","sourceUrl":"https://github.com/rkz91/coco/tree/main/skills/systematic-debugging","isPrimary":false,"firstSeenAt":"2026-05-18T13:21:42.662Z","lastSeenAt":"2026-05-18T19:14:09.320Z"}],"details":{"listingId":"f29430f3-30ee-45ba-8160-e45fc99d7694","quickStartSnippet":null,"exampleRequest":null,"exampleResponse":null,"schema":null,"openapiUrl":null,"agentsTxtUrl":null,"citations":[],"useCases":[],"bestFor":[],"notFor":[],"kindDetails":{"org":"rkz91","slug":"systematic-debugging","github":{"repo":"rkz91/coco","stars":7,"topics":["agent-skills","agents-md","ai","ai-agents","claude-code","codex","cursor","developer-tools","llm-tools","mcp","pm-tools","product-management","productivity","prompt-engineering","workflow-automation"],"license":"mit","html_url":"https://github.com/rkz91/coco","pushed_at":"2026-04-26T01:51:27Z","description":"Open-source library of AI superpowers — 59 skills, 34 commands, 10 agents + 24 GSD subagents, 3 system bundles. An entire team, wherever your AI lives. Vendor-neutral across Claude Code, Cursor, Codex, and any AGENTS.md tool.","skill_md_sha":"051799f4c5a501db6f638172041b01b8be0b6b7b","skill_md_path":"skills/systematic-debugging/SKILL.md","default_branch":"main","skill_tree_url":"https://github.com/rkz91/coco/tree/main/skills/systematic-debugging"},"layout":"multi","source":"github","category":"coco","frontmatter":{"name":"systematic-debugging","description":"Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes"},"skills_sh_url":"https://skills.sh/rkz91/coco/systematic-debugging"},"updatedAt":"2026-05-18T19:14:09.320Z"}}