Reading summary.json in a shell script — which keys are stable?

Asked 17 May 202210,400 views2 answers

31votes

I want the CI job to post totals to Slack without scraping console output. .sdods/runs/<id>/summary.json looks like exactly what I need:

json

{
  "runId": "0180c7d4-9a11-7f02-b3c8-6d10a5e2f477",
  "status": "failed",
  "totals": {
    "total": 20,
    "passed": 18,
    "failed": 1,
    "skipped": 0,
    "timedOut": 0,
    "flaky": 0,
    "healed": 1,
    "durationMs": 9825,
    "workers": 4
  },
  "byProject": { "demo-shop--ui--chromium": { "total": 20, "passed": 18 } }
}

Before I build on it: is this file part of the contract or an implementation detail that is going to move under me? And what is byProject keyed on — I want to split the message per browser.

reporting ci cli

asked by Gio Castellano521 · 17 May 2022

2 answers

27votes✓
Accepted answer
It is the contract. summary.json is a RunSummary: runId, status, totals, byProject, failed, flaky and reportPaths. status is passed, failed or cancelled and is derived from the exit code, so it agrees with what CI saw rather than being computed a second way.
byProject is keyed on the runner project name, which is <project-slug>--<layer>--<browser>:
bash
```
jq -r '.byProject | keys[]' .sdods/runs/<id>/summary.json
```
text
```
rwa-bank--api--chromium
rwa-bank--hybrid--chromium
rwa-bank--ui--firefox
```
For the Slack message you probably want failed and flaky rather than counts: both are arrays of { fingerprint, title, runnerProject }, and failed carries error as well. Three scenario titles read better than "1 failed".
One caveat before you make the job depend on it: summary.json is written at the end from the dashboard reporter's metrics.json. If the process was killed mid-run — an agent timeout, a cancelled build — there is no metrics.json and therefore no summary.json. Handle the file not existing; do not treat it as "0 failures".
answered by Reeta Nandalmaintainer41,280 · 18 May 2022
14votes
If you would rather not know the path at all:
bash
```
sdods report --last --json | jq '.summary.totals'
```
The same object also carries the manifest, which is where git and command live. Putting the sha and the branch in the Slack message is worth the extra two lines — otherwise the first reply is always "which commit is this".
answered by Lena Hartwig10,388 · 19 May 2022

Have the same question?

Ask it with the command you ran and the output you got, and it will be answered here.

Ask a question

Reading summary.json in a shell script — which keys are stable?

2 answers

Related