{"id":"bd0df933-e530-439c-a690-995ffbab0cf5","shortId":"URDtac","kind":"skill","title":"baoyu-url-to-markdown","tagline":"Fetch any URL and convert to markdown using baoyu-fetch CLI (Chrome CDP with site-specific adapters). Built-in adapters for X/Twitter, YouTube transcripts, Hacker News threads, and generic pages via Defuddle. Handles login/CAPTCHA via interaction wait modes. Use when user wants t","description":"# URL to Markdown\n\nFetches any URL via `baoyu-fetch` CLI (Chrome CDP + site-specific adapters) and converts it to clean markdown.\n\n## User Input Tools\n\nWhen this skill prompts the user, follow this tool-selection rule (priority order):\n\n1. **Prefer built-in user-input tools** exposed by the current agent runtime — e.g., `AskUserQuestion`, `request_user_input`, `clarify`, `ask_user`, or any equivalent.\n2. **Fallback**: if no such tool exists, emit a numbered plain-text message and ask the user to reply with the chosen number/answer for each question.\n3. **Batching**: if the tool supports multiple questions per call, combine all applicable questions into a single call; if only single-question, ask them one at a time in priority order.\n\nConcrete `AskUserQuestion` references below are examples — substitute the local equivalent in other runtimes.\n\n## CLI Setup\n\n**Important**: The CLI source is vendored in `{baseDir}/scripts/lib`. `scripts/package.json` installs only third-party runtime dependencies.\n\n**Agent Execution Instructions**:\n1. Determine this SKILL.md file's directory path as `{baseDir}`\n2. Resolve `${BUN}` runtime: if `bun` installed → `bun`; else suggest installing Bun\n3. If `{baseDir}/scripts/node_modules` does not exist, run `${BUN} install --cwd {baseDir}/scripts`\n4. `${READER}` = `{baseDir}/scripts/baoyu-fetch`\n5. Replace all `${READER}` in this document with the resolved value\n\n## Preferences (EXTEND.md)\n\nCheck EXTEND.md in priority order — the first one found wins:\n\n| Priority | Path | Scope |\n|----------|------|-------|\n| 1 | `.baoyu-skills/baoyu-url-to-markdown/EXTEND.md` | Project |\n| 2 | `${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/baoyu-url-to-markdown/EXTEND.md` | XDG |\n| 3 | `$HOME/.baoyu-skills/baoyu-url-to-markdown/EXTEND.md` | User home |\n\n| Result | Action |\n|--------|--------|\n| Found | Read, parse, apply settings |\n| Not found | **MUST** run first-time setup (see below) — do NOT silently create defaults |\n\n**EXTEND.md supports**: download media by default, default output directory.\n\n### First-Time Setup ⛔ BLOCKING\n\nWhen EXTEND.md is not found, you **MUST** use `AskUserQuestion` to gather preferences before creating EXTEND.md. **NEVER** create EXTEND.md with silent defaults. Generation is BLOCKED until setup completes. Batch all three questions into a single call:\n\n- **Q1 — Media** (header \"Media\"): \"How to handle images and videos in pages?\"\n  - \"Ask each time (Recommended)\" — Prompt after each save\n  - \"Always download\" — Download to local `imgs/` and `videos/`\n  - \"Never download\" — Keep remote URLs\n- **Q2 — Output** (header \"Output\"): \"Default output directory?\"\n  - \"url-to-markdown (Recommended)\" — Save to `./url-to-markdown/{domain}/{slug}.md`\n  - User may pick \"Other\" and type a custom path\n- **Q3 — Save** (header \"Save\"): \"Where to save preferences?\"\n  - \"User (Recommended)\" — `~/.baoyu-skills/` (all projects)\n  - \"Project\" — `.baoyu-skills/` (this project only)\n\nAfter answers, write EXTEND.md, confirm \"Preferences saved to [path]\", then continue.\n\nFull template: [references/config/first-time-setup.md](references/config/first-time-setup.md).\n\n### Supported Keys\n\n| Key | Default | Values | Description |\n|-----|---------|--------|-------------|\n| `download_media` | `ask` | `ask` / `1` / `0` | `ask` = prompt each time, `1` = always, `0` = never |\n| `default_output_dir` | empty | path or empty | Default output directory (empty = `./url-to-markdown/`) |\n\n**EXTEND.md → CLI mapping**:\n\n| EXTEND.md key | CLI argument | Notes |\n|---------------|-------------|-------|\n| `download_media: 1` | `--download-media` | Requires `--output` to be set |\n| `default_output_dir: ./posts/` | Agent constructs `--output ./posts/{domain}/{slug}.md` | Agent generates path, not a direct flag |\n\n**Value priority**: CLI arguments → EXTEND.md → skill defaults.\n\n## Usage\n\n```bash\n# Default: headless capture, markdown to stdout\n${READER} <url>\n\n# Save to file\n${READER} <url> --output article.md\n\n# Save with media download\n${READER} <url> --output article.md --download-media\n\n# Wait for interaction (login/CAPTCHA) — auto-detect and continue\n${READER} <url> --wait-for interaction --output article.md\n\n# Wait for interaction — manual control (Enter to continue)\n${READER} <url> --wait-for force --output article.md\n\n# JSON output\n${READER} <url> --format json --output article.json\n\n# Force specific adapter\n${READER} <url> --adapter youtube --output transcript.md\n```\n\n## Options\n\n| Option | Description |\n|--------|-------------|\n| `<url>` | URL to fetch |\n| `--output <path>` | Output file path (default: stdout) |\n| `--format <type>` | Output format: `markdown` (default) or `json` |\n| `--json` | Shorthand for `--format json` |\n| `--adapter <name>` | Force adapter: `x`, `youtube`, `hn`, or `generic` (default: auto-detect) |\n| `--headless` | Force headless Chrome (no visible window) |\n| `--wait-for <mode>` | Interaction wait mode: `none` (default), `interaction`, or `force` |\n| `--wait-for-interaction` | Alias for `--wait-for interaction` |\n| `--wait-for-login` | Alias for `--wait-for interaction` |\n| `--timeout <ms>` | Page load timeout (default: 30000) |\n| `--interaction-timeout <ms>` | Login/CAPTCHA wait timeout (default: 600000 = 10 min) |\n| `--interaction-poll-interval <ms>` | Poll interval for interaction checks (default: 1500) |\n| `--download-media` | Download images/videos to local `imgs/` and `videos/`, rewrite markdown links. Requires `--output` |\n| `--media-dir <dir>` | Base directory for downloaded media (default: same as `--output` directory) |\n| `--cdp-url <url>` | Reuse existing Chrome DevTools Protocol endpoint |\n| `--browser-path <path>` | Custom Chrome/Chromium binary path |\n| `--chrome-profile-dir <path>` | Chrome user data directory (default: `BAOYU_CHROME_PROFILE_DIR` env or `./baoyu-skills/chrome-profile`) |\n| `--debug-dir <dir>` | Write debug artifacts (document.json, markdown.md, page.html, network.json) |\n\n## Agent Quality Gate\n\n**CRITICAL**: treat default headless capture as provisional. Some sites render differently in headless mode and can silently return low-quality content without failing the CLI.\n\nAfter every headless run, inspect the saved markdown. See [references/quality-gate.md](references/quality-gate.md) for the full checklist, recovery workflow, and capture-mode table. Read it whenever a run looks suspicious or the user asks about login/CAPTCHA handling.\n\n## Output Path Generation\n\nThe agent must construct the output file path — `baoyu-fetch` does not auto-generate paths.\n\n**Algorithm**:\n1. Determine base directory from EXTEND.md `default_output_dir` or default `./url-to-markdown/`\n2. Extract domain from URL (e.g., `example.com`)\n3. Generate slug from URL path or page title (kebab-case, 2-6 words)\n4. Construct: `{base_dir}/{domain}/{slug}/{slug}.md` — each URL gets its own directory so media files stay isolated\n5. Conflict resolution: append timestamp `{slug}-YYYYMMDD-HHMMSS/{slug}-YYYYMMDD-HHMMSS.md`\n\nPass the constructed path to `--output`. Media files (`--download-media`) are saved into subdirectories next to the markdown file, keeping each URL's assets self-contained.\n\n## Adapters & Media\n\nSee [references/adapters.md](references/adapters.md) for the adapter catalog (X, YouTube, Hacker News, generic), per-adapter notes, the media download flow (`ask` / always / never), and the JSON output schema. Read it before answering adapter-specific questions or handling media prompts.\n\n## Environment Variables\n\n| Variable | Description |\n|----------|-------------|\n| `BAOYU_CHROME_PROFILE_DIR` | Chrome user data directory (can also use `--chrome-profile-dir`) |\n\n**Troubleshooting**: Chrome not found → use `--browser-path`. Timeout → increase `--timeout`. Login/CAPTCHA → `--wait-for interaction`. Debug → `--debug-dir` to inspect captured HTML and network logs.\n\n## Extension Support\n\nCustom configurations via EXTEND.md. See **Preferences** section above for paths and supported keys.","tags":["baoyu","url","markdown","skills","jimliu","agent-skills","claude-skills","codex-skills","openclaw-skills"],"capabilities":["skill","source-jimliu","skill-baoyu-url-to-markdown","topic-agent-skills","topic-claude-skills","topic-codex-skills","topic-openclaw-skills"],"categories":["baoyu-skills"],"synonyms":[],"warnings":[],"endpointUrl":"https://skills.sh/JimLiu/baoyu-skills/baoyu-url-to-markdown","protocol":"skill","transport":"skills-sh","auth":{"type":"none","details":{"cli":"npx skills add JimLiu/baoyu-skills","source_repo":"https://github.com/JimLiu/baoyu-skills","install_from":"skills.sh"}},"qualityScore":"0.700","qualityRationale":"deterministic score 0.70 from registry signals: · indexed on github topic:agent-skills · 16958 github stars · SKILL.md body (7,951 chars)","verified":false,"liveness":"unknown","lastLivenessCheck":null,"agentReviews":{"count":0,"score_avg":null,"cost_usd_avg":null,"success_rate":null,"latency_p50_ms":null,"narrative_summary":null,"summary_updated_at":null},"enrichmentModel":"deterministic:skill-github:v1","enrichmentVersion":1,"enrichedAt":"2026-05-03T00:52:27.979Z","embedding":null,"createdAt":"2026-04-18T21:53:38.072Z","updatedAt":"2026-05-03T00:52:27.979Z","lastSeenAt":"2026-05-03T00:52:27.979Z","tsv":"'-6':896 '/.baoyu-skills':435 '/baoyu-skills/baoyu-url-to-markdown/extend.md':288 '/baoyu-skills/chrome-profile':767 '/baoyu-url-to-markdown/extend.md':281 '/posts':514,518 '/scripts':246 '/scripts/baoyu-fetch':250 '/scripts/lib':200 '/scripts/node_modules':237 '/url-to-markdown':412,491,875 '0':471,478 '1':92,212,277,470,476,502,864 '10':695 '1500':707 '2':118,222,283,876,895 '3':145,234,290,883 '30000':686 '4':247,898 '5':251,917 '600000':694 'action':295 'adapt':24,28,68,601,603,631,633,956,963,972,991 'adapter-specif':990 'agent':105,209,515,522,778,847 'algorithm':863 'alia':665,675 'also':1011 'alway':385,477,979 'answer':446,989 'append':920 'appli':299 'applic':157 'argument':498,532 'article.json':598 'article.md':550,557,576,591 'artifact':773 'ask':113,133,168,377,468,469,472,839,978 'askuserquest':108,178,338 'asset':952 'auto':566,641,860 'auto-detect':565,640 'auto-gener':859 'baoyu':2,15,60,279,440,761,855,1002 'baoyu-fetch':14,59,854 'baoyu-skil':278,439 'baoyu-url-to-markdown':1 'base':726,866,900 'basedir':199,221,236,245,249 'bash':537 'batch':146,357 'binari':750 'block':329,353 'browser':746,1023 'browser-path':745,1022 'built':26,95 'built-in':25,94 'bun':224,227,229,233,242 'call':154,162,364 'captur':540,785,826,1039 'capture-mod':825 'case':894 'catalog':964 'cdp':19,64,737 'cdp-url':736 'check':264,705 'checklist':821 'chosen':140 'chrome':18,63,646,741,753,756,762,1003,1006,1014,1018 'chrome-profile-dir':752,1013 'chrome/chromium':749 'clarifi':112 'clean':73 'cli':17,62,190,194,493,497,531,806 'combin':155 'complet':356 'concret':177 'config':285 'configur':1047 'confirm':449 'conflict':918 'construct':516,849,899,930 'contain':955 'content':802 'continu':455,569,584 'control':581 'convert':10,70 'creat':314,343,346 'critic':781 'current':104 'custom':423,748,1046 'cwd':244 'data':758,1008 'debug':769,772,1033,1035 'debug-dir':768,1034 'default':315,321,322,350,402,463,480,487,511,535,538,617,623,639,657,685,693,706,731,760,783,870,874 'defuddl':40 'depend':208 'descript':465,609,1001 'detect':567,642 'determin':213,865 'devtool':742 'differ':791 'dir':482,513,725,755,764,770,872,901,1005,1016,1036 'direct':527 'directori':218,324,404,489,727,735,759,867,911,1009 'document':257 'document.json':774 'domain':413,519,878,902 'download':318,386,387,394,466,500,504,554,559,709,711,729,937,976 'download-media':503,558,708,936 'e.g':107,881 'els':230 'emit':125 'empti':483,486,490 'endpoint':744 'enter':582 'env':765 'environ':998 'equival':117,186 'everi':808 'exampl':182 'example.com':882 'execut':210 'exist':124,240,740 'expos':101 'extend.md':263,265,316,331,344,347,448,492,495,533,869,1049 'extens':1044 'extract':877 'fail':804 'fallback':119 'fetch':6,16,55,61,612,856 'file':216,547,615,852,914,935,947 'first':270,306,326 'first-tim':305,325 'flag':528 'flow':977 'follow':84 'forc':589,599,632,644,660 'format':595,619,621,629 'found':272,296,302,334,1020 'full':456,820 'gate':780 'gather':340 'generat':351,523,845,861,884 'generic':37,638,969 'get':908 'hacker':33,967 'handl':41,371,842,995 'header':367,400,427 'headless':539,643,645,784,793,809 'hhmmss':925 'hn':636 'home':286,293 'home/.baoyu-skills/baoyu-url-to-markdown/extend.md':291 'home/.config':287 'html':1040 'imag':372 'images/videos':712 'img':390,715 'import':192 'increas':1026 'input':76,99,111 'inspect':811,1038 'instal':202,228,232,243 'instruct':211 'interact':44,563,574,579,653,658,664,670,680,688,698,704,1032 'interaction-poll-interv':697 'interaction-timeout':687 'interv':700,702 'isol':916 'json':592,596,625,626,630,983 'kebab':893 'kebab-cas':892 'keep':395,948 'key':461,462,496,1058 'link':720 'load':683 'local':185,389,714 'log':1043 'login':674 'login/captcha':42,564,690,841,1028 'look':834 'low':800 'low-qual':799 'manual':580 'map':494 'markdown':5,12,54,74,408,541,622,719,814,946 'markdown.md':775 'may':417 'md':415,521,905 'media':319,366,368,467,501,505,553,560,710,724,730,913,934,938,957,975,996 'media-dir':723 'messag':131 'min':696 'mode':46,655,794,827 'multipl':151 'must':303,336,848 'network':1042 'network.json':777 'never':345,393,479,980 'news':34,968 'next':943 'none':656 'note':499,973 'number':127 'number/answer':141 'one':170,271 'option':607,608 'order':91,176,268 'output':323,399,401,403,481,488,507,512,517,549,556,575,590,593,597,605,613,614,620,722,734,843,851,871,933,984 'page':38,376,682,890 'page.html':776 'pars':298 'parti':206 'pass':928 'path':219,275,424,453,484,524,616,747,751,844,853,862,888,931,1024,1055 'per':153,971 'per-adapt':970 'pick':418 'plain':129 'plain-text':128 'poll':699,701 'prefer':93,262,341,432,450,1051 'prioriti':90,175,267,274,530 'profil':754,763,1004,1015 'project':282,437,438,443 'prompt':81,381,473,997 'protocol':743 'provision':787 'q1':365 'q2':398 'q3':425 'qualiti':779,801 'question':144,152,158,167,360,993 'read':297,829,986 'reader':248,254,544,548,555,570,585,594,602 'recommend':380,409,434 'recoveri':822 'refer':179 'references/adapters.md':959,960 'references/config/first-time-setup.md':458,459 'references/quality-gate.md':816,817 'remot':396 'render':790 'replac':252 'repli':137 'request':109 'requir':506,721 'resolut':919 'resolv':223,260 'result':294 'return':798 'reus':739 'rewrit':718 'rule':89 'run':241,304,810,833 'runtim':106,189,207,225 'save':384,410,426,428,431,451,545,551,813,940 'schema':985 'scope':276 'scripts/package.json':201 'section':1052 'see':309,815,958,1050 'select':88 'self':954 'self-contain':953 'set':300,510 'setup':191,308,328,355 'shorthand':627 'silent':313,349,797 'singl':161,166,363 'single-quest':165 'site':22,66,789 'site-specif':21,65 'skill':80,280,441,534 'skill-baoyu-url-to-markdown' 'skill.md':215 'slug':414,520,885,903,904,922,926 'sourc':195 'source-jimliu' 'specif':23,67,600,992 'stay':915 'stdout':543,618 'subdirectori':942 'substitut':183 'suggest':231 'support':150,317,460,1045,1057 'suspici':835 'tabl':828 'templat':457 'text':130 'third':205 'third-parti':204 'thread':35 'three':359 'time':173,307,327,379,475 'timeout':681,684,689,692,1025,1027 'timestamp':921 'titl':891 'tool':77,87,100,123,149 'tool-select':86 'topic-agent-skills' 'topic-claude-skills' 'topic-codex-skills' 'topic-openclaw-skills' 'transcript':32 'transcript.md':606 'treat':782 'troubleshoot':1017 'type':421 'url':3,8,52,57,397,406,610,738,880,887,907,950 'url-to-markdown':405 'usag':536 'use':13,47,337,1012,1021 'user':49,75,83,98,110,114,135,292,416,433,757,838,1007 'user-input':97 'valu':261,464,529 'variabl':999,1000 'vendor':197 'via':39,43,58,1048 'video':374,392,717 'visibl':648 'wait':45,561,572,577,587,651,654,662,668,672,678,691,1030 'wait-for':571,586,650,667,677,1029 'wait-for-interact':661 'wait-for-login':671 'want':50 'whenev':831 'win':273 'window':649 'without':803 'word':897 'workflow':823 'write':447,771 'x':634,965 'x/twitter':30 'xdg':284,289 'youtub':31,604,635,966 'yyyymmdd':924 'yyyymmdd-hhmmss':923 'yyyymmdd-hhmmss.md':927","prices":[{"id":"9e8f3037-fdb6-4daa-a7a5-d7b4fd9b32d9","listingId":"bd0df933-e530-439c-a690-995ffbab0cf5","amountUsd":"0","unit":"free","nativeCurrency":null,"nativeAmount":null,"chain":null,"payTo":null,"paymentMethod":"skill-free","isPrimary":true,"details":{"org":"JimLiu","category":"baoyu-skills","install_from":"skills.sh"},"createdAt":"2026-04-18T21:53:38.072Z"}],"sources":[{"listingId":"bd0df933-e530-439c-a690-995ffbab0cf5","source":"github","sourceId":"JimLiu/baoyu-skills/baoyu-url-to-markdown","sourceUrl":"https://github.com/JimLiu/baoyu-skills/tree/main/skills/baoyu-url-to-markdown","isPrimary":false,"firstSeenAt":"2026-04-18T21:53:38.072Z","lastSeenAt":"2026-05-03T00:52:27.979Z"}],"details":{"listingId":"bd0df933-e530-439c-a690-995ffbab0cf5","quickStartSnippet":null,"exampleRequest":null,"exampleResponse":null,"schema":null,"openapiUrl":null,"agentsTxtUrl":null,"citations":[],"useCases":[],"bestFor":[],"notFor":[],"kindDetails":{"org":"JimLiu","slug":"baoyu-url-to-markdown","github":{"repo":"JimLiu/baoyu-skills","stars":16958,"topics":["agent-skills","claude-skills","codex-skills","openclaw-skills"],"license":null,"html_url":"https://github.com/JimLiu/baoyu-skills","pushed_at":"2026-04-25T20:03:31Z","description":null,"skill_md_sha":"adad2b70be26c97934b338ca5d2b7248956a4dbf","skill_md_path":"skills/baoyu-url-to-markdown/SKILL.md","default_branch":"main","skill_tree_url":"https://github.com/JimLiu/baoyu-skills/tree/main/skills/baoyu-url-to-markdown"},"layout":"multi","source":"github","category":"baoyu-skills","frontmatter":{"name":"baoyu-url-to-markdown","description":"Fetch any URL and convert to markdown using baoyu-fetch CLI (Chrome CDP with site-specific adapters). Built-in adapters for X/Twitter, YouTube transcripts, Hacker News threads, and generic pages via Defuddle. Handles login/CAPTCHA via interaction wait modes. Use when user wants to save a webpage as markdown."},"skills_sh_url":"https://skills.sh/JimLiu/baoyu-skills/baoyu-url-to-markdown"},"updatedAt":"2026-05-03T00:52:27.979Z"}}