Possibly an unpopular opinion, but: if a developer renames a button and my test still passes, my test did not test anything.
Yesterday a whole feature shipped with the wrong label on the submit button and the suite was green, because healing quietly found it by role. I want the suite to go red on a locator that no longer matches. Is there a switch, or am I holding this wrong?
Accepted answer
You are not holding it wrong, you are describing the trade the feature makes. Healing keeps a suite running through a refactor; it is not there to certify the DOM.
Nothing about a heal is quiet, though. Every one is attached to the scenario as sdods/heal/<step>/<n>, appended to heal.jsonl in the run directory, added to the test annotations, and logged with the selector that won and a suggested rewrite. The failure mode you hit is that nobody read any of it.
If you want it off, the honest switches are:
heal: { enabled: false } in envs/<env>.yaml — the env layer may override healSDODS_HEAL=false for a single runheal.actions to the actions you accept, remembering that arrays replace rather than merge.raw on a specific locator when you want exactly that element and no substituteWhat I would actually do is leave it on and gate on the report instead. A green run with fourteen heals in it is a signal, and sdods heal report --last puts it on one screen.
We landed on the middle position: healing on everywhere, and the pipeline fails the job if sdods heal report --last --json comes back with any events for the smoke suite. Regression is allowed to heal, smoke is not.
Took about ten lines of shell, and it caught exactly the rename you are describing — the day after, rather than never.
One more angle. The case that actually bit us was not a rename, it was a duplicate: two elements matched, uniqueness dragged the score down but not below the threshold, and the healer took the first one. It passed. It was the wrong button.
So "healing hid a bug" is real, but so is "our locator was ambiguous all along". Worth checking which of the two you have before you turn the feature off.
Late to this, but the annotations are the bit that made it visible for us. They show up on the test in the HTML report, so a reviewer looking at a green run sees sdods:heal next to the scenario name without going anywhere near a JSON file.
Have the same question?
Ask it with the command you ran and the output you got, and it will be answered here.
Ask a question