{"id":"82dd2bef-1bb2-471b-bd50-e93c3d20616a","shortId":"6Uxc29","kind":"skill","title":"dt-obs-tracing","tagline":">-","description":"# Application Tracing Skill\n\n## Overview\n\nDistributed traces in Dynatrace consist of spans - building blocks representing units of work. With Traces in Grail, every span is accessible via DQL with full-text searchability on all attributes. This skill covers trace fundamentals, common analysis patterns, and span-type specific queries.\n\n---\n\n## Use Cases\n\n### 1. Investigate Slow Requests\n- **Goal:** Find and diagnose requests exceeding a latency threshold\n- **Trigger:** \"slow requests\", \"high latency\", \"p99 response time\", \"find traces over 5 seconds\"\n- **Done:** List of slow traces with duration, endpoint, service, and trace IDs for drilldown\n\n### 2. Analyze Request Failures\n- **Goal:** Identify failed requests, failure reasons, and exception patterns\n- **Trigger:** \"failed spans\", \"HTTP 500 errors\", \"exception analysis\", \"failure rate by service\"\n- **Done:** Failure breakdown by reason (HTTP code, exception, gRPC status) with exemplar traces\n\n### 3. Map Service Dependencies\n- **Goal:** Understand service-to-service communication patterns and external API calls\n- **Trigger:** \"service dependencies\", \"what services does X call\", \"outgoing HTTP calls\"\n- **Done:** Dependency map showing call counts, latency, and error rates between services\n\n---\n\n## Core Concepts\n\n### Understanding Traces and Spans\n\n**Spans** represent logical units of work in distributed traces:\n- HTTP requests, RPC calls, database operations\n- Messaging system interactions\n- Internal function invocations\n- Custom instrumentation points\n\n**Span kinds**:\n- `span.kind: server` - Incoming call to a service\n- `span.kind: client` - Outgoing call from a service\n- `span.kind: consumer` - Incoming message consumption call to a service\n- `span.kind: producer` - Outgoing message production call from a service\n- `span.kind: internal` - Internal operation within a service\n\n**Root spans**: A request root span (`request.is_root_span == true`) represents an incoming call to a service. Use this to analyze end-to-end request performance.\n\n### Key Trace Attributes\n\nEssential attributes for trace analysis:\n\n| Attribute | Description |\n|-----------|-------------|\n| `trace.id` | Unique trace identifier |\n| `span.id` | Unique span identifier |\n| `span.parent_id` | Parent span ID (null for root spans) |\n| `request.is_root_span` | Boolean, true for request entry points |\n| `request.is_failed` | Boolean, true if request failed |\n| `duration` | Span duration in nanoseconds |\n| `span.timing.cpu` | Overall CPU time of the span (stable) |\n| `span.timing.cpu_self` | CPU time excluding child spans (stable) |\n| `dt.smartscape.service` | Service Smartscape node ID |\n| `dt.service.name` | Dynatrace service name derived from service detection rules. It is equal to the Smartscape service node name.  |\n| `endpoint.name` | Endpoint/route name |\n\n### Service Context\n\nSpans reference services via Smartscape node IDs and the detected service name `dt.service.name` which is also present on every span.\n\n```dql\nfetch spans\n| summarize spans=count(), by: { dt.smartscape.service, dt.service.name }\n```\n\n**Node functions**:\n- `getNodeName(dt.smartscape.service)` - Adds `dt.smartscape.service.name` field with the human-readable service name\n- `getNodeField(dt.smartscape.service, \"attribute_name\")` - Access specific node attributes\n\n**📖 Learn more**: See [Entity Lookups](references/entity-lookups.md) for advanced entity selectors, infrastructure correlation, and hardware analysis.\n\n### Sampling and Extrapolation\n\nOne span can represent multiple real operations due to:\n- **Aggregation**: Multiple operations in one span (`aggregation.count`)\n- **ATM (Adaptive Traffic Management)**: Head-based sampling by agent\n- **ALR (Adaptive Load Reduction)**: Server-side sampling\n- **Read Sampling**: Query-time sampling via `samplingRatio` parameter\n\n**When to extrapolate**: Always extrapolate when counting actual operations (not just spans). Use the multiplicity factor:\n\n```dql\nfetch spans\n| fieldsAdd sampling.probability = (power(2, 56) - coalesce(sampling.threshold, 0)) * power(2, -56)\n| fieldsAdd sampling.multiplicity = 1 / sampling.probability\n| fieldsAdd multiplicity = coalesce(sampling.multiplicity, 1)\n                         * coalesce(aggregation.count, 1)\n                         * dt.system.sampling_ratio\n| summarize operation_count = sum(multiplicity)\n```\n\n**📖 Learn more**: See [Sampling and Extrapolation](references/sampling-extrapolation.md) for detailed formulas and examples.\n\n## Common Query Patterns\n\n### Basic Span Access\n\nFetch spans and explore by type:\n\n```dql\nfetch spans | limit 1\n```\n\nExplore spans by function and type:\n\n```dql\nfetch spans\n| summarize count(), by: { span.kind, code.namespace, code.function }\n```\n\n### Request Root Filtering\n\nList request root spans (incoming service calls):\n\n```dql\nfetch spans\n| filter request.is_root_span == true\n| fields trace.id, span.id, start_time, response_time = duration, endpoint.name\n| limit 100\n```\n\n### Service Performance Summary\n\nAnalyze service performance with error rates:\n\n```dql\nfetch spans\n| filter request.is_root_span == true\n| summarize\n    total_requests = count(),\n    failed_requests = countIf(request.is_failed == true),\n    avg_duration = avg(duration),\n    p95_duration = percentile(duration, 95),\n    by: {dt.service.name}\n| fieldsAdd error_rate = (failed_requests * 100.0) / total_requests\n| sort error_rate desc\n```\n\n### Trace ID Lookup\n\nFind all spans in a specific trace:\n\n```dql\nfetch spans\n| filter trace.id == toUid(\"abc123def456\")\n| fields span.name, duration, dt.service.name\n```\n\n## Performance Analysis\n\n### Response Time Percentiles\n\nCalculate percentiles by endpoint:\n\n```dql\nfetch spans\n| filter request.is_root_span == true\n| summarize {\n    requests=count(),\n    avg_duration=avg(duration),\n    p95=percentile(duration, 95),\n    p99=percentile(duration, 99)\n  }, by: { endpoint.name }\n| sort p99 desc\n```\n\n**💡 Best practice**: Use percentiles (p95, p99) over averages for performance insights.\n\n### Slow Trace Detection\n\nFind requests exceeding a threshold:\n\n```dql\nfetch spans, from:now() - 2h\n| filter request.is_root_span == true\n| filter duration > 5s\n| fields trace.id, span.name, dt.service.name, duration\n| sort duration desc\n| limit 50\n```\n\n### Duration Buckets with Exemplars\n\n```dql\nfetch spans, from:now() - 24h\n| filter http.route == \"/api/v1/storage/findByISBN\"\n| summarize {\n    spans=count(),\n    trace=takeAny(record(start_time, trace.id))\n  }, by: { bin(duration, 10ms) }\n| fields `bin(duration, 10ms)`, spans, trace.id=trace[trace.id], start_time=trace[start_time]\n```\n\n### Performance Timeseries\n\nExtract response time as timeseries:\n\n```dql\nfetch spans, from:now() - 24h\n| filter request.is_root_span == true\n| makeTimeseries {\n    requests=count(),\n    avg_duration=avg(duration),\n    p95=percentile(duration, 95),\n    p99=percentile(duration, 99)\n  }, by: { endpoint.name }\n```\n\n**📖 Learn more**: See [Performance Analysis](references/performance-analysis.md) for advanced patterns and timeseries techniques.\n\n## Failure Investigation\n\n### Failed Request Summary\n\nSummarize failures by service:\n\n```dql\nfetch spans\n| filter request.is_root_span == true\n| summarize\n    total = count(),\n    failed = countIf(request.is_failed == true),\n  by: { dt.service.name }\n| fieldsAdd failure_rate = (failed * 100.0) / total\n| sort failure_rate desc\n```\n\n### Failure Reason Analysis\n\nBreakdown by failure detection reason:\n\n```dql\nfetch spans\n| filter request.is_failed == true and isNotNull(dt.failure_detection.results)\n| expand dt.failure_detection.results\n| summarize count(), by: { dt.failure_detection.results[reason] }\n```\n\n**Failure reasons**:\n- `http_code` - HTTP response code triggered failure\n- `grpc_code` - gRPC status code triggered failure\n- `exception` - Exception caused failure\n- `span_status` - Span status indicated failure\n- `custom_rule` - Custom failure detection rule matched\n\n### HTTP Code Failures\n\nFind failures by HTTP status code:\n\n```dql\nfetch spans\n| filter request.is_failed == true\n| filter iAny(dt.failure_detection.results[][reason] == \"http_code\")\n| summarize count(), by: { http.response.status_code, endpoint.name }\n| sort `count()` desc\n```\n\n### Recent Failed Requests\n\nList recent failures with details:\n\n```dql\nfetch spans\n| filter request.is_root_span == true and request.is_failed == true\n| fields\n    start_time,\n    trace.id,\n    endpoint.name,\n    http.response.status_code,\n    duration\n| sort start_time desc\n| limit 100\n```\n\n**📖 Learn more**: See [Failure Detection](references/failure-detection.md) for exception analysis and custom rule investigation.\n\n## Service Dependencies\n\n### Service-to-Service Analysis\n\nAnalyze service communication patterns:\n\n```dql\nfetch spans, from:now() - 1h\n| filter isNotNull(server.address)\n| fieldsAdd\n    remote_side = server.address\n| summarize\n    call_count = count(),\n    avg_duration = avg(duration),\n    by: {dt.service.name, remote_side}\n| sort call_count desc\n```\n\n### Outgoing HTTP Calls\n\nIdentify external API dependencies:\n\n```dql\nfetch spans\n| filter span.kind == \"client\" and isNotNull(http.request.method)\n| summarize\n    calls = count(),\n    avg_latency = avg(duration),\n    p99_latency = percentile(duration, 99),\n  by: { dt.service.name, server.address, server.port }\n| sort calls desc\n```\n\n## Trace Aggregation\n\n### Complete Trace Analysis\n\nAggregate all spans in a trace to understand full request flow:\n\n```dql\nfetch spans, from:now() - 30m\n| summarize {\n    spans = count(),\n    client_spans = countIf(span.kind == \"client\"),\n\n    // Endpoints involved in the trace\n    endpoints = toString(arrayRemoveNulls(collectDistinct(endpoint.name))),\n\n    // Extract the first request root in the trace\n    trace_root = takeMin(record(\n        root_detection_helper = coalesce(\n            if(request.is_root_span, 1),\n            if(isNull(span.parent_id), 2),\n            3),\n        start_time, endpoint.name, duration\n      ))\n}, by: { trace.id }\n\n| fieldsFlatten trace_root\n| fieldsRemove trace_root.root_detection_helper, trace_root\n\n| fields\n    start_time = trace_root.start_time,\n    endpoint = trace_root.endpoint.name,\n    response_time = trace_root.duration,\n    spans,\n    client_spans,\n    endpoints,\n    trace.id\n| sort start_time\n| limit 100\n```\n\n**Root detection strategy**: Use `takeMin(record(...))` with a detection helper to reliably find the root request:\n1. Priority 1: Spans with `request.is_root_span == true`\n2. Priority 2: Spans without parent (root spans)\n3. Priority 3: All other spans\n\n### Multi-Service Traces\n\nFind traces spanning multiple services:\n\n```dql\nfetch spans, from:now() - 1h\n| summarize {\n    services = collectDistinct(dt.service.name),\n    trace_root = takeMin(record(root_detection_helper = coalesce(if(request.is_root_span, 1), 2), endpoint.name))\n}, by: { trace.id }\n| fieldsAdd service_count = arraySize(services)\n| filter service_count > 1\n| fields endpoint = trace_root[endpoint.name], service_count, services = toString(services), trace.id\n| sort service_count desc\n| limit 50\n```\n\n## Request-Level Analysis\n\n### Request Attributes\n\nAccess custom request attributes captured by OneAgent on request root spans:\n\n```dql\nfetch spans\n| filter request.is_root_span == true\n| filter isNotNull(request_attribute.PaidAmount)\n| makeTimeseries sum(request_attribute.PaidAmount)\n```\n\n**Field patterns**: `request_attribute.<name>`, `captured_attribute.<name>` (always arrays)\n\n→ [Request Attributes](references/request-attributes.md) — full patterns for request attributes, captured attributes, and request ID aggregation\n\n## Span Types\n\n| Span Type | Detection | Key Fields | Reference |\n|-----------|-----------|------------|-----------|\n| HTTP server (incoming) | `span.kind == \"server\" and isNotNull(http.request.method)` | `http.route`, `http.request.method`, `http.response.status_code` | [http-spans.md](references/http-spans.md) |\n| HTTP client (outgoing) | `span.kind == \"client\" and isNotNull(http.request.method)` | `server.address`, `server.port` | [http-spans.md](references/http-spans.md) |\n| Database | `span.kind == \"client\" and isNotNull(db.system)` | `db.system`, `db.namespace`, `db.statement` | [database-spans.md](references/database-spans.md) |\n| Messaging | `isNotNull(messaging.system)` | `messaging.system`, `messaging.destination.name`, `messaging.operation.type` | [messaging-spans.md](references/messaging-spans.md) |\n| RPC / gRPC | `isNotNull(rpc.system)` | `rpc.system`, `rpc.service`, `rpc.method`, `rpc.grpc.status_code` | [rpc-spans.md](references/rpc-spans.md) |\n| Serverless / FaaS | `isNotNull(faas.name) and span.kind == \"server\"` | `faas.name`, `faas.trigger.type`, `cloud.provider` | [serverless-spans.md](references/serverless-spans.md) |\n\n**⚠️ Database spans**: Can be aggregated (one span = multiple calls). Always use `aggregation.count` extrapolation for accurate operation counts.\n\n**📖 Detailed patterns per span type:** See the reference files above.\n\n## Advanced Topics\n\n### Exception Analysis\n\nExceptions are stored as `span.events` within spans:\n\n```dql\nfetch spans\n| filter iAny(span.events[][span_event.name] == \"exception\")\n| expand span.events\n| fieldsFlatten span.events, fields: { exception.type }\n| summarize {\n    count(),\n    trace=takeAny(record(start_time, trace.id))\n  }, by: { exception.type }\n| fields exception.type, `count()`, trace.id=trace[trace.id], start_time=trace[start_time]\n```\n\n**💡 Tip**: Use `iAny()` to check conditions within span event arrays.\n\n→ [Logs Correlation](references/logs-correlation.md) — joining logs and traces, filtering traces by log content\n→ [Network Analysis](references/networking-analysis.md) — client IPs, DNS resolution, subnet analysis\n\n## Best Practices\n\n| Area | Rule |\n|------|------|\n| **Filtering** | Apply `request.is_root_span == true` and endpoint filters first |\n| **Sampling** | Use `samplingRatio` (e.g., `100` = read 1%) for performance |\n| **Percentiles** | Use p95/p99 over averages for performance analysis |\n| **Root spans** | Use `request.is_root_span == true` for end-to-end analysis |\n| **Trace grouping** | Group by `trace.id` for complete trace metrics |\n| **Request grouping** | Group by `request.id` for OneAgent-only request metrics |\n| **Extrapolation** | Always apply multiplicity for accurate operation counts |\n| **Exemplars** | Use `takeAny(record(start_time, trace.id))` to enable UI drilldown |\n\n---\n\n## Troubleshooting\n\n| Problem | Cause | Solution |\n|---------|-------|----------|\n| Duration values seem wrong (too large) | `duration` is in nanoseconds, not milliseconds | Divide by `1000000` or compare with `5s` (DQL duration literal) |\n| Span counts don't match expected request volume | Sampling or aggregation not accounted for | Use multiplicity extrapolation — see Sampling and Extrapolation reference |\n| `getNodeName(dt.smartscape.service)` returns null | Service not yet resolved or OneAgent not monitoring | Verify OneAgent monitors the service; entity resolution may have a short delay |\n| `request.is_root_span` filter returns nothing | Querying OpenTelemetry-only traces without OneAgent | Use `isNull(span.parent_id)` as fallback for root span detection |\n| `trace.id` filter returns no results | Trace ID not converted to UID format | Use `filter trace.id == toUid(\"abc123...\")` for string-based trace IDs |\n| Database span counts are too low | Database spans are aggregated (one span = N calls) | Always use `aggregation.count` extrapolation for database operation counts |\n\n## Related Skills\n\n- **dt-dql-essentials** — Core DQL syntax for querying trace data\n- **dt-app-dashboards** — Embed trace queries in dashboards\n- **dt-migration** — Smartscape entity model and relationship navigation\n\n---\n\n## References\n\nDetailed documentation for specific topics:\n\n- **[Performance Analysis](references/performance-analysis.md)** - Advanced timeseries, duration buckets, endpoint ranking\n- **[Failure Detection](references/failure-detection.md)** - Failure reasons, exception investigation, custom rules\n- **[Sampling and Extrapolation](references/sampling-extrapolation.md)** - Multiplicity calculation, database extrapolation\n- **[Request Attributes](references/request-attributes.md)** - Request attributes, captured attributes, request ID aggregation\n- **[Entity Lookups](references/entity-lookups.md)** - Advanced node lookups, infrastructure correlation, hardware analysis\n- **[HTTP Span Analysis](references/http-spans.md)** - Status codes, payload analysis, client IPs\n- **[Database Span Analysis](references/database-spans.md)** - Extrapolated counts, slow queries, statement analysis\n- **[Messaging Span Analysis](references/messaging-spans.md)** - Kafka, RabbitMQ, SQS throughput and latency\n- **[RPC Span Analysis](references/rpc-spans.md)** - gRPC, SOAP, service dependencies\n- **[Serverless Span Analysis](references/serverless-spans.md)** - Lambda, Azure Functions, cold start analysis\n- **[Logs Correlation](references/logs-correlation.md)** - Joining logs and traces, correlation patterns\n- **[Network Analysis](references/networking-analysis.md)** - IP addresses, DNS resolution, communication mapping","tags":["obs","tracing","dynatrace","for","agent-skills","ai-agents","claude-code","devops","dql","mcp","observability"],"capabilities":["skill","source-dynatrace","skill-dt-obs-tracing","topic-agent-skills","topic-ai-agents","topic-claude-code","topic-devops","topic-dql","topic-dynatrace","topic-mcp","topic-observability"],"categories":["dynatrace-for-ai"],"synonyms":[],"warnings":[],"endpointUrl":"https://skills.sh/Dynatrace/dynatrace-for-ai/dt-obs-tracing","protocol":"skill","transport":"skills-sh","auth":{"type":"none","details":{"cli":"npx skills add Dynatrace/dynatrace-for-ai","source_repo":"https://github.com/Dynatrace/dynatrace-for-ai","install_from":"skills.sh"}},"qualityScore":"0.489","qualityRationale":"deterministic score 0.49 from registry signals: · indexed on github topic:agent-skills · 78 github stars · SKILL.md body (17,113 chars)","verified":false,"liveness":"unknown","lastLivenessCheck":null,"agentReviews":{"count":0,"score_avg":null,"cost_usd_avg":null,"success_rate":null,"latency_p50_ms":null,"narrative_summary":null,"summary_updated_at":null},"enrichmentModel":"deterministic:skill-github:v1","enrichmentVersion":1,"enrichedAt":"2026-05-18T18:56:48.954Z","embedding":null,"createdAt":"2026-05-11T18:57:14.984Z","updatedAt":"2026-05-18T18:56:48.954Z","lastSeenAt":"2026-05-18T18:56:48.954Z","tsv":"'-56':504 '/api/v1/storage/findbyisbn':760 '0':501 '1':56,507,513,516,552,1142,1200,1202,1254,1267,1538 '100':596,993,1183,1536 '100.0':640,865 '1000000':1619 '10ms':773,777 '1h':1023,1237 '2':96,497,503,1147,1209,1211,1255 '24h':757,799 '2h':729 '3':134,1148,1217,1219 '30m':1103 '5':80 '50':747,1284 '500':113 '56':498 '5s':737,1623 '95':632,695,815 '99':699,819,1074 'abc123':1712 'abc123def456':663 'access':29,410,541,1291 'account':1639 'accur':1428,1587 'actual':482 'adapt':449,459 'add':396 'address':1885 'advanc':421,829,1441,1781,1817 'agent':457 'aggreg':441,1083,1087,1337,1418,1637,1728,1813 'aggregation.count':447,515,1425,1735 'alr':458 'also':378 'alway':478,1322,1423,1583,1733 'analysi':46,116,278,428,669,826,873,1002,1013,1086,1288,1444,1510,1517,1548,1561,1779,1823,1826,1831,1836,1843,1846,1856,1864,1871,1882 'analyz':97,264,600,1014 'api':148,1052 'app':1756 'appli':1523,1584 'applic':5 'area':1520 'array':1323,1496 'arrayremovenul':1119 'arrays':1262 'atm':448 'attribut':39,273,275,279,408,413,1290,1294,1319,1321,1325,1331,1333,1805,1808,1810 'averag':712,1545 'avg':624,626,688,690,808,810,1035,1037,1066,1068 'azur':1867 'base':454,1716 'basic':539 'best':705,1518 'bin':771,775 'block':17 'boolean':301,309 'breakdown':123,874 'bucket':749,1784 'build':16 'calcul':673,1801 'call':149,157,160,165,191,208,215,224,233,257,577,1032,1044,1049,1064,1080,1422,1732 'captur':1295,1320,1332,1809 'case':55 'caus':914,1603 'check':1491 'child':332 'client':213,1059,1107,1111,1175,1361,1364,1374,1512,1832 'cloud.provider':1411 'coalesc':499,511,514,1137,1249 'code':127,899,902,906,909,930,937,950,955,986,1357,1399,1829 'code.function':567 'code.namespace':566 'cold':1869 'collectdistinct':1120,1240 'common':45,536 'communic':144,1016,1888 'compar':1621 'complet':1084,1568 'concept':174 'condit':1492 'consist':13 'consum':220 'consumpt':223 'content':1508 'context':362 'convert':1704 'core':173,1747 'correl':425,1498,1821,1873,1879 'count':166,388,481,521,563,617,687,763,807,853,892,952,958,1033,1034,1045,1065,1106,1261,1266,1274,1281,1430,1467,1478,1589,1628,1721,1740,1839 'countif':620,855,1109 'cover':42 'cpu':321,329 'custom':200,922,924,1004,1292,1794 'dashboard':1757,1762 'data':1753 'databas':192,1372,1414,1719,1725,1738,1802,1834 'database-spans.md':1381 'db.namespace':1379 'db.statement':1380 'db.system':1377,1378 'delay':1672 'depend':137,152,162,1008,1053,1861 'deriv':344 'desc':646,704,745,870,959,991,1046,1081,1282 'descript':280 'detail':532,967,1431,1773 'detect':347,372,718,877,926,998,1135,1160,1185,1192,1247,1342,1695,1788 'diagnos':63 'distribut':9,186 'divid':1617 'dns':1514,1886 'document':1774 'done':82,121,161 'dql':31,383,491,548,559,578,606,657,677,724,752,794,843,879,938,968,1018,1054,1098,1232,1302,1452,1624,1745,1748 'drilldown':95,1600 'dt':2,1744,1755,1764 'dt-app-dashboard':1754 'dt-dql-essenti':1743 'dt-migrat':1763 'dt-obs-trac':1 'dt.failure_detection.results':888,890,894,947 'dt.service.name':340,375,391,634,667,741,860,1040,1076,1241 'dt.smartscape.service':335,390,395,407,1650 'dt.smartscape.service.name':397 'dt.system.sampling':517 'due':439 'durat':88,314,316,593,625,627,629,631,666,689,691,694,698,736,742,744,748,772,776,809,811,814,818,987,1036,1038,1069,1073,1152,1605,1611,1625,1783 'dynatrac':12,341 'e.g':1535 'emb':1758 'enabl':1598 'end':266,268,1558,1560 'end-to-end':265,1557 'endpoint':89,676,1112,1117,1169,1177,1269,1529,1785 'endpoint.name':358,594,701,821,956,984,1121,1151,1256,1272 'endpoint/route':359 'entiti':417,422,1666,1767,1814 'entri':305 'equal':351 'error':114,169,604,636,644 'essenti':274,1746 'event':1495 'everi':26,381 'exampl':535 'exceed':65,721 'except':107,115,128,912,913,1001,1443,1445,1459,1792 'exception.type':1465,1475,1477 'exclud':331 'exemplar':132,751,1590 'expand':889,1460 'expect':1632 'explor':545,553 'extern':147,1051 'extract':789,1122 'extrapol':431,477,479,529,1426,1582,1643,1647,1736,1798,1803,1838 'faa':1403 'faas.name':1405,1409 'faas.trigger.type':1410 'factor':490 'fail':102,110,308,313,618,622,638,836,854,857,864,884,943,961,978 'failur':99,104,117,122,834,840,862,868,871,876,896,904,911,915,921,925,931,933,965,997,1787,1790 'fallback':1691 'fetch':384,492,542,549,560,579,607,658,678,725,753,795,844,880,939,969,1019,1055,1099,1233,1303,1453 'field':398,586,664,738,774,980,1164,1268,1316,1344,1464,1476 'fieldsadd':494,505,509,635,861,1027,1259 'fieldsflatten':1155,1462 'fieldsremov':1158 'file':1439 'filter':570,581,609,660,680,730,735,758,800,846,882,941,945,971,1024,1057,1264,1305,1310,1455,1504,1522,1530,1676,1697,1709 'find':61,77,650,719,932,1196,1227 'first':1124,1531 'flow':1097 'format':1707 'formula':533 'full':34,1095,1327 'full-text':33 'function':198,393,556,1868 'fundament':44 'getnodefield':406 'getnodenam':394,1649 'goal':60,100,138 'grail':25 'group':1563,1564,1572,1573 'grpc':129,905,907,1392,1858 'hardwar':427,1822 'head':453 'head-bas':452 'helper':1136,1161,1193,1248 'high':72 'http':112,126,159,188,898,900,929,935,949,1048,1346,1360,1824 'http-spans.md':1358,1370 'http.request.method':1062,1353,1355,1367 'http.response.status':954,985,1356 'http.route':759,1354 'human':402 'human-read':401 'iani':946,1456,1489 'id':93,290,293,339,369,648,1146,1336,1689,1702,1718,1812 'identifi':101,284,288,1050 'incom':207,221,256,575,1348 'indic':920 'infrastructur':424,1820 'insight':715 'instrument':201 'interact':196 'intern':197,238,239 'investig':57,835,1006,1793 'invoc':199 'involv':1113 'ip':1513,1833,1884 'isnotnul':887,1025,1061,1311,1352,1366,1376,1384,1393,1404 'isnul':1144,1687 'join':1500,1875 'kafka':1848 'key':271,1343 'kind':204 'lambda':1866 'larg':1610 'latenc':67,73,167,1067,1071,1853 'learn':414,524,822,994 'level':1287 'limit':551,595,746,992,1182,1283 'list':83,571,963 'liter':1626 'load':460 'log':1497,1501,1507,1872,1876 'logic':181 'lookup':418,649,1815,1819 'low':1724 'maketimeseri':805,1313 'manag':451 'map':135,163,1889 'match':928,1631 'may':1668 'messag':194,222,231,1383,1844 'messaging-spans.md':1389 'messaging.destination.name':1387 'messaging.operation.type':1388 'messaging.system':1385,1386 'metric':1570,1581 'migrat':1765 'millisecond':1616 'model':1768 'monitor':1660,1663 'multi':1224 'multi-servic':1223 'multipl':436,442,489,510,523,1230,1421,1585,1642,1800 'n':1731 'name':343,357,360,374,405,409 'nanosecond':318,1614 'navig':1771 'network':1509,1881 'node':338,356,368,392,412,1818 'noth':1678 'null':294,1652 'ob':3 'one':432,445,1419,1729 'oneag':1297,1578,1658,1662,1685 'oneagent-on':1577 'opentelemetri':1681 'opentelemetry-on':1680 'oper':193,240,438,443,483,520,1429,1588,1739 'outgo':158,214,230,1047,1362 'overal':320 'overview':8 'p95':628,692,709,812 'p95/p99':1543 'p99':74,696,703,710,816,1070 'paramet':474 'parent':291,1214 'pattern':47,108,145,538,830,1017,1317,1328,1432,1880 'payload':1830 'per':1433 'percentil':630,672,674,693,697,708,813,817,1072,1541 'perform':270,598,602,668,714,787,825,1540,1547,1778 'point':202,306 'power':496,502 'practic':706,1519 'present':379 'prioriti':1201,1210,1218 'problem':1602 'produc':229 'product':232 'queri':53,469,537,1679,1751,1760,1841 'query-tim':468 'rabbitmq':1849 'rank':1786 'rate':118,170,605,637,645,863,869 'ratio':518 'read':466,1537 'readabl':403 'real':437 'reason':105,125,872,878,895,897,948,1791 'recent':960,964 'record':766,1133,1189,1245,1470,1593 'reduct':461 'refer':364,1345,1438,1648,1772 'references/database-spans.md':1382,1837 'references/entity-lookups.md':419,1816 'references/failure-detection.md':999,1789 'references/http-spans.md':1359,1371,1827 'references/logs-correlation.md':1499,1874 'references/messaging-spans.md':1390,1847 'references/networking-analysis.md':1511,1883 'references/performance-analysis.md':827,1780 'references/request-attributes.md':1326,1806 'references/rpc-spans.md':1401,1857 'references/sampling-extrapolation.md':530,1799 'references/serverless-spans.md':1413,1865 'relat':1741 'relationship':1770 'reliabl':1195 'remot':1028,1041 'repres':18,180,254,435 'request':59,64,71,98,103,189,247,269,304,312,568,572,616,619,639,642,686,720,806,837,962,1096,1125,1199,1286,1289,1293,1299,1318,1324,1330,1335,1571,1580,1633,1804,1807,1811 'request-level':1285 'request.id':1575 'request.is':250,298,307,582,610,621,681,731,801,847,856,883,942,972,977,1139,1205,1251,1306,1524,1552,1673 'request_attribute.paidamount':1312,1315 'resolut':1515,1667,1887 'resolv':1656 'respons':75,591,670,790,901,1171 'result':1700 'return':1651,1677,1698 'root':244,248,251,296,299,569,573,583,611,682,732,802,848,973,1126,1131,1134,1140,1157,1163,1184,1198,1206,1215,1243,1246,1252,1271,1300,1307,1525,1549,1553,1674,1693 'rpc':190,1391,1854 'rpc-spans.md':1400 'rpc.grpc.status':1398 'rpc.method':1397 'rpc.service':1396 'rpc.system':1394,1395 'rule':348,923,927,1005,1521,1795 'sampl':429,455,465,467,471,527,1532,1635,1645,1796 'sampling.multiplicity':506,512 'sampling.probability':495,508 'sampling.threshold':500 'samplingratio':473,1534 'searchabl':36 'second':81 'see':416,526,824,996,1436,1644 'seem':1607 'selector':423 'self':328 'server':206,463,1347,1350,1408 'server-sid':462 'server.address':1026,1030,1077,1368 'server.port':1078,1369 'serverless':1402,1862 'serverless-spans.md':1412 'servic':90,120,136,141,143,151,154,172,211,218,227,236,243,260,336,342,346,355,361,365,373,404,576,597,601,842,1007,1010,1012,1015,1225,1231,1239,1260,1263,1265,1273,1275,1277,1280,1653,1665,1860 'service-to-servic':140,1009 'short':1671 'show':164 'side':464,1029,1042 'skill':7,41,1742 'skill-dt-obs-tracing' 'slow':58,70,85,716,1840 'smartscap':337,354,367,1766 'soap':1859 'solut':1604 'sort':643,702,743,867,957,988,1043,1079,1179,1279 'source-dynatrace' 'span':15,27,50,111,178,179,203,245,249,252,287,292,297,300,315,325,333,363,382,385,387,433,446,486,493,540,543,550,554,561,574,580,584,608,612,652,659,679,683,726,733,754,762,778,796,803,845,849,881,916,918,940,970,974,1020,1056,1089,1100,1105,1108,1141,1174,1176,1203,1207,1212,1216,1222,1229,1234,1253,1301,1304,1308,1338,1340,1415,1420,1434,1451,1454,1494,1526,1550,1554,1627,1675,1694,1720,1726,1730,1825,1835,1845,1855,1863 'span-typ':49 'span.events':1449,1457,1461,1463 'span.id':285,588 'span.kind':205,212,219,228,237,565,1058,1110,1349,1363,1373,1407 'span.name':665,740 'span.parent':289,1145,1688 'span.timing.cpu':319,327 'span_event.name':1458 'specif':52,411,655,1776 'sqs':1850 'stabl':326,334 'start':589,767,782,785,981,989,1149,1165,1180,1471,1482,1485,1594,1870 'statement':1842 'status':130,908,917,919,936,1828 'store':1447 'strategi':1186 'string':1715 'string-bas':1714 'subnet':1516 'sum':522,1314 'summar':386,519,562,614,685,761,839,851,891,951,1031,1063,1104,1238,1466 'summari':599,838 'syntax':1749 'system':195 'takeani':765,1469,1592 'takemin':1132,1188,1244 'techniqu':833 'text':35 'threshold':68,723 'throughput':1851 'time':76,322,330,470,590,592,671,768,783,786,791,982,990,1150,1166,1168,1172,1181,1472,1483,1486,1595 'timeseri':788,793,832,1782 'tip':1487 'topic':1442,1777 'topic-agent-skills' 'topic-ai-agents' 'topic-claude-code' 'topic-devops' 'topic-dql' 'topic-dynatrace' 'topic-mcp' 'topic-observability' 'tostr':1118,1276 'total':615,641,852,866 'touid':662,1711 'trace':4,6,10,23,43,78,86,92,133,176,187,272,277,283,647,656,717,764,780,784,1082,1085,1092,1116,1129,1130,1156,1162,1226,1228,1242,1270,1468,1480,1484,1503,1505,1562,1569,1683,1701,1717,1752,1759,1878 'trace.id':281,587,661,739,769,779,781,983,1154,1178,1258,1278,1473,1479,1481,1566,1596,1696,1710 'trace_root.duration':1173 'trace_root.endpoint.name':1170 'trace_root.root':1159 'trace_root.start':1167 'traffic':450 'trigger':69,109,150,903,910 'troubleshoot':1601 'true':253,302,310,585,613,623,684,734,804,850,858,885,944,975,979,1208,1309,1527,1555 'type':51,547,558,1339,1341,1435 'ui':1599 'uid':1706 'understand':139,175,1094 'uniqu':282,286 'unit':19,182 'use':54,261,487,707,1187,1424,1488,1533,1542,1551,1591,1641,1686,1708,1734 'valu':1606 'verifi':1661 'via':30,366,472 'volum':1634 'within':241,1450,1493 'without':1213,1684 'work':21,184 'wrong':1608 'x':156 'yet':1655","prices":[{"id":"b05173c3-2170-409f-9d89-4b245f75df98","listingId":"82dd2bef-1bb2-471b-bd50-e93c3d20616a","amountUsd":"0","unit":"free","nativeCurrency":null,"nativeAmount":null,"chain":null,"payTo":null,"paymentMethod":"skill-free","isPrimary":true,"details":{"org":"Dynatrace","category":"dynatrace-for-ai","install_from":"skills.sh"},"createdAt":"2026-05-11T18:57:14.984Z"}],"sources":[{"listingId":"82dd2bef-1bb2-471b-bd50-e93c3d20616a","source":"github","sourceId":"Dynatrace/dynatrace-for-ai/dt-obs-tracing","sourceUrl":"https://github.com/Dynatrace/dynatrace-for-ai/tree/main/skills/dt-obs-tracing","isPrimary":false,"firstSeenAt":"2026-05-11T18:57:14.984Z","lastSeenAt":"2026-05-18T18:56:48.954Z"}],"details":{"listingId":"82dd2bef-1bb2-471b-bd50-e93c3d20616a","quickStartSnippet":null,"exampleRequest":null,"exampleResponse":null,"schema":null,"openapiUrl":null,"agentsTxtUrl":null,"citations":[],"useCases":[],"bestFor":[],"notFor":[],"kindDetails":{"org":"Dynatrace","slug":"dt-obs-tracing","github":{"repo":"Dynatrace/dynatrace-for-ai","stars":78,"topics":["agent-skills","ai-agents","claude-code","devops","dql","dynatrace","mcp","observability"],"license":"apache-2.0","html_url":"https://github.com/Dynatrace/dynatrace-for-ai","pushed_at":"2026-05-15T16:06:09Z","description":"Skills, prompts, and instructions for building AI agents on top of Dynatrace production context","skill_md_sha":"16a4f8fa6ac6f024fca00f31f397a43775f6a45e","skill_md_path":"skills/dt-obs-tracing/SKILL.md","default_branch":"main","skill_tree_url":"https://github.com/Dynatrace/dynatrace-for-ai/tree/main/skills/dt-obs-tracing"},"layout":"multi","source":"github","category":"dynatrace-for-ai","frontmatter":{"name":"dt-obs-tracing","license":"Apache-2.0","description":">-"},"skills_sh_url":"https://skills.sh/Dynatrace/dynatrace-for-ai/dt-obs-tracing"},"updatedAt":"2026-05-18T18:56:48.954Z"}}