Server ingest takes the ndjson but the run viewer has no screenshots

Asked 5 Aug 20245,940 views2 answers

21votes

Our CI agents have no database credentials, so we ingest to the server with a scoped token instead. Messages go up fine. Screenshots do not.

bash

sdods report ingest --run-id 019103ab-7d5e-7c40-9b12-8ae0f4d3c221 \
  --server https://sdods.internal --token $SDODS_TOKEN \
  .sdods/runs/019103ab-.../messages.ndjson artifacts.tgz

First attempt was my own fault:

text

Server ingest failed (401): {"error":{...}}

With the right token the upload reports success, the run appears in the UI and the totals are correct — and every scenario has no images at all. Is the tarball being ignored, or is it landing somewhere the viewer does not look?

reporting ci cli

asked by Bruno Teixeira694 · 5 Aug 2024

2 answers

26votes✓
Accepted answer
Do not trust the limitations table on this one. It says POST /api/runs/:id/ingest takes run.json, NDJSON and runner JSON only and that an artifacts.tgz is not accepted yet — but the route does accept it. It looks for a part named exactly artifacts.tgz or artifacts.tar.gz, extracts it under the run directory, skips any entry that would escape that directory, and returns an extracted count of files and bytes alongside the usual result.
So the question is why yours came back with nothing extracted. Two things to check, in this order:
- **The part name.** It is matched literally. artifacts.tgz works; 019103ab-artifacts.tgz or artifacts.tar does not, and an unrecognised part is dropped without complaint. Your command line passes a path, so whatever basename that path ends in is the name the server sees.
- **The size ceiling.** The multipart limit is SDODS_INGEST_MAX_MB, and the extracted bytes are capped at four times that. A run with a few hundred screenshots goes past a modest default quickly, and the extraction stops rather than filling the disk.
Look at the JSON the ingest call returns — if extracted is missing entirely the part was not recognised, and if it is present with a skipped count the cap is what bit you.
answered by Ify Adeyemi6,584 · 5 Aug 2024
12votes
Worth writing down what the 401 was so the next person does not chase it: the token needs the runs:ingest scope. Read scopes get you a 401 from that route and nothing else, and the body is not especially forthcoming about it.
The other one to recognise:
text
```
--server needs --token (or SDODS_TOKEN).
```
That is an exit 2 before any request is made, so if you see it, nothing was uploaded and there is no half-ingested run to clean up.
answered by Priya Venkateshmaintainer29,604 · 7 Aug 2024

Have a different question? Ask it.

Server ingest takes the ndjson but the run viewer has no screenshots

2 answers

Know the answer?

Related