Skip to main content

Run History & Regressions

Every run created from scratch is standalone. But a run created by clicking Re-run, or by triggering from CI/CD, clones the exact same steps as its source, and VeriWasp tracks that lineage, so it can tell you not just "did this pass," but "did this used to pass and now doesn't."

How lineage works​

The first time a flow is run, it has no history yet. Every time it's re-run (via the Re-run button or a CI trigger, not "Edit & Re-run", since editing changes the steps and breaks the comparison), the new run is linked back to that same original flow. This chain has no length limit: re-run a flow ten times, and all ten runs share the same lineage.

A Run history section appears on the report and run detail pages once a flow has actually been re-run at least once, listing every run in the lineage with its status, most recent first. It doesn't show up on a run that's never been re-run; there'd be nothing to compare.

Regression detection​

Every time a run in a lineage finishes, VeriWasp compares it against the most recent previous completed or failed run of the same flow, step by step. If a step passed last time and fails now, that's flagged as a regression, a distinct issue category from the usual bug/a11y/perf/usability findings, with a summary like:

Step 2 (click) passed 3 days ago and fails now

This is the difference between "this run failed" and "this used to work and something broke it." The second one is what actually tells you a recent deploy caused a problem.

Regression detection only compares steps that are identical between the two runs (same position, same action, same target description). If the step plan changed since the previous run, that step is skipped rather than producing a misleading comparison.

Flaky vs. regressed​

A step that passed last run and fails now isn't always a real regression. Sometimes it's a flaky third-party dependency, a rate-limited vendor API, or test data that got deleted or expired since the last run. Flagging every one of those as a high-severity "you broke this" finding trains you to stop trusting the alerts.

So before framing a passed-to-failed flip as a regression, VeriWasp checks the step's stability over its last 5 runs (not counting the one that just passed). If the step has been passing less than 80% of the time even before this run, it's shown as a low-severity, flaky finding instead:

Step 2 (click) is flaky, passed only 3 of the last 5 runs before this one

If the step has a clean, stable history of passing and only just started failing, it keeps the normal high-severity regression framing described above. That's still the common case, and it doesn't get slower or noisier. A step with no comparable history yet (fewer than 5 prior runs) is treated as a regular regression by default, since there isn't enough data to call it flaky.

Console errors, network errors, and performance​

Beyond pass/fail and pixel content, VeriWasp also compares three more things between a step and its counterpart in the previous run:

  • New console errors: if a step logs a browser console error that wasn't there last time, that's its own finding, even if the step itself still passes.
  • New failed network requests: same idea, for requests that failed during the step.
  • Slower steps: if a step takes meaningfully longer than it did last run (both at least 50% slower and at least 500ms slower in absolute terms, so a 10ms-to-25ms blip doesn't trigger a finding), it's flagged as a perf issue: "Step 3 (click) took 2.1s, up from 0.8s 3 days ago." This is different from the existing "this step took over 3 seconds" usability finding. That one flags a step that's slow in isolation, this one flags a step that got slower than its own baseline, even if it's still fast in absolute terms.

Accessibility drift​

Every run includes an accessibility scan of the page the flow ends on. If a run in a lineage turns up more accessibility issues than the previous run, that's surfaced as its own finding, "2 new accessibility issue(s) since 3 days ago", so a regression in accessibility doesn't quietly hide inside the general issues list.

Visual regressions​

A step can pass (the button was found and clicked) while the page around it looks completely different: a broken image, a font that failed to load, a layout that shifted. Functional pass/fail doesn't catch any of that, so for every matched step pair in a lineage, VeriWasp also compares the two screenshots pixel by pixel.

If enough of the image changed, you'll see:

  • A visual diff badge on the step, showing the percentage of pixels that changed (colored neutral under 3%, amber from 3%, red from 15%).
  • A visual issue in the Issues Found section, distinct from regression (which is about pass/fail), something like "Step 1 (navigate) looks 42.0% different from 3 days ago."
  • A diff image rendered right below the step's normal screenshot, with every changed region highlighted in magenta, so you can see exactly what moved or broke without eyeballing two separate screenshots side by side.

The percentage and diff image are recorded for every comparable step regardless of whether it crosses the "worth an issue" threshold. A step that's 0.4% different won't get a finding, but you can still see that number if you're looking closely.

Two screenshots are only compared when they're the same pixel dimensions (same viewport, same device pixel ratio); a size mismatch is treated as "not comparable" and silently skipped, not flagged as 100% different.

What isn't covered yet​

Console errors, network errors, and performance are all compared against the single most recent previous run, not a multi-run trend. VeriWasp won't yet tell you "this step has been getting a little slower for the last 5 runs" the way the flaky-vs-regressed check looks back further for pass/fail. Accessibility drift is a whole-run issue count, not a per-violation diff, so "2 new issues" doesn't yet tell you which 2 without checking the findings list yourself.