A table of recent traces with nested spans (root, command id, span count, what nests in it) and Grafana's trace panel over otel_traces in the ClickHouse plugin's trace format. Trace ids in the tables link to the view through a Trace id variable; empty, it shows the latest nested trace.
- commit
- 6ff76bbe14ff0983e0a2adf76f4bb2199a7d60ab
- parent
- 2205e51f199081aba631629fdc57f2c9f9169551
- author
- Dami <dami.etoile@gmail.com>
| M | deploy/ | 122 | +++++++++++++++++++++++++++++- |
| M | docs/ | 9 | ++- |
@@ -409,7 +409,7 @@
409409 "id": 9,410410 "type": "table",411411 "title": "Slowest spans (OTLP traces)",412- "description": "The slowest spans 012 sent over OTLP; each is the root of its own trace, and the log record of the same event carries its TraceId",412+ "description": "The slowest spans 012 sent over OTLP, roots and nested ones; a trace id opens that trace in the trace view below",413413 "datasource": {414414 "type": "grafana-clickhouse-datasource",415415 "uid": "clickhouse"@@ -435,17 +435,131 @@
435435 ],436436 "fieldConfig": {437437 "defaults": {},438- "overrides": []438+ "overrides": [439+ {440+ "matcher": {441+ "id": "byName",442+ "options": "trace"443+ },444+ "properties": [445+ {446+ "id": "links",447+ "value": [448+ {449+ "title": "Show this trace",450+ "url": "/d/o12-runtime/012-runtime?var-trace=${__value.raw}&${__url_time_range}"451+ }452+ ]453+ }454+ ]455+ }456+ ]457+ },458+ "options": {459+ "showHeader": true460+ }461+ },462+ {463+ "id": 10,464+ "type": "table",465+ "title": "Recent traces with nested spans",466+ "description": "Traces of more than one span, newest first: the root span, its command id, how many spans it holds, which spans nest in it and how long the root took. A trace id opens it in the trace view below",467+ "datasource": {468+ "type": "grafana-clickhouse-datasource",469+ "uid": "clickhouse"470+ },471+ "gridPos": {472+ "x": 0,473+ "y": 39,474+ "w": 24,475+ "h": 8476+ },477+ "targets": [478+ {479+ "datasource": {480+ "type": "grafana-clickhouse-datasource",481+ "uid": "clickhouse"482+ },483+ "editorType": "sql",484+ "format": 1,485+ "queryType": "table",486+ "refId": "A",487+ "rawSql": "SELECT min(Timestamp) AS time, anyIf(SpanName, ParentSpanId = '') AS root, anyIf(SpanAttributes['id'], ParentSpanId = '') AS command, count() AS spans, arrayStringConcat(arraySort(groupUniqArrayIf(SpanName, ParentSpanId != '')), ', ') AS nested, round(maxIf(Duration, ParentSpanId = '') / 1e6, 2) AS ms, countIf(StatusCode = 'Error') AS errors, TraceId AS trace FROM otel.otel_traces WHERE $__timeFilter(Timestamp) AND ServiceName = '012' GROUP BY TraceId HAVING spans > 1 ORDER BY time DESC LIMIT 100"488+ }489+ ],490+ "fieldConfig": {491+ "defaults": {},492+ "overrides": [493+ {494+ "matcher": {495+ "id": "byName",496+ "options": "trace"497+ },498+ "properties": [499+ {500+ "id": "links",501+ "value": [502+ {503+ "title": "Show this trace",504+ "url": "/d/o12-runtime/012-runtime?var-trace=${__value.raw}&${__url_time_range}"505+ }506+ ]507+ }508+ ]509+ }510+ ]439511 },440512 "options": {441513 "showHeader": true442514 }515+ },516+ {517+ "id": 11,518+ "type": "traces",519+ "title": "Trace ${trace}",520+ "description": "The spans of one trace as a tree with their timing: the trace id in the Trace id box at the top, or the latest trace with nested spans when it is empty",521+ "datasource": {522+ "type": "grafana-clickhouse-datasource",523+ "uid": "clickhouse"524+ },525+ "gridPos": {526+ "x": 0,527+ "y": 47,528+ "w": 24,529+ "h": 14530+ },531+ "targets": [532+ {533+ "datasource": {534+ "type": "grafana-clickhouse-datasource",535+ "uid": "clickhouse"536+ },537+ "editorType": "sql",538+ "format": 3,539+ "queryType": "traces",540+ "refId": "A",541+ "rawSql": "SELECT TraceId AS traceID, SpanId AS spanID, ParentSpanId AS parentSpanID, ServiceName AS serviceName, SpanName AS operationName, multiply(toUnixTimestamp64Nano(Timestamp), 0.000001) AS startTime, multiply(Duration, 0.000001) AS duration, arrayMap(key -> map('key', key, 'value', SpanAttributes[key]), mapKeys(SpanAttributes)) AS tags, arrayMap(key -> map('key', key, 'value', ResourceAttributes[key]), mapKeys(ResourceAttributes)) AS serviceTags, if(StatusCode IN ('Error', 'STATUS_CODE_ERROR'), 2, 0) AS statusCode FROM otel.otel_traces WHERE ServiceName = '012' AND TraceId = if('${trace}' != '', '${trace}', (SELECT TraceId FROM otel.otel_traces WHERE $__timeFilter(Timestamp) AND ServiceName = '012' AND ParentSpanId != '' ORDER BY Timestamp DESC LIMIT 1)) ORDER BY Timestamp LIMIT 1000"542+ }543+ ],544+ "options": {}443545 }444546 ],445547 "templating": {446- "list": []548+ "list": [549+ {550+ "name": "trace",551+ "label": "Trace id",552+ "type": "textbox",553+ "query": "",554+ "current": {555+ "text": "",556+ "value": ""557+ },558+ "description": "The trace the trace view shows; empty shows the latest trace with nested spans"559+ }560+ ]447561 },448562 "annotations": {449563 "list": []450564 }451-}\ No newline at end of file565+}@@ -407,8 +407,13 @@ The `012 runtime` dashboard shows recalculation time (p50, p95, max),
407407 frame and key-to-frame p95 per second, heap and cells, JEV answer time408408 and queue, imports and exports with throughput, the slowest operations,409409 and events per minute (all from the logs, so either way in works), and410-from OTLP the key-to-frame latency summary and the slowest spans with411-their trace ids. `012 stress runs` shows edit latency per410+from OTLP the key-to-frame latency summary, the slowest spans, the411+recent traces with nested spans (root span, command id, span count and412+which spans nest in it), and a trace view: the spans of one trace as a413+tree on a timeline, in Grafana's trace panel (the ClickHouse plugin's414+trace format). A trace id in either table links to it, through the415+dashboard's Trace id box; with the box empty the view shows the latest416+trace with nested spans. `012 stress runs` shows edit latency per412417 topology, frame and keystroke times, full recalc and file operations,413418 memory per cell (log scales where the spread is wide), the latest run414419 against the previous one, and the runs.