Skip to main content

API Testing

Everything else VeriWasp tests goes through a real browser. API testing is different: it's a sequence of plain HTTP requests, no browser involved, for checking the things that only happen at the API layer. A field silently dropped from a response, a status code that changed, an endpoint that started requiring an undocumented header. Each request's assertion is a plain-English description of what a passing response looks like, judged by Claude Haiku against the actual response, the same "AI decides" philosophy already used for finding elements in a UI playtest.

Creating an API test​

Go to My Runs → New API Test, or API Tests in the top nav. Fill in:

  • Base URL: every request's URL is resolved against this unless the request's own URL is already absolute (starts with http:// or https://).
  • Label (optional): a name for this test, e.g. "Signup + login flow."
  • For each request: a method, a URL (relative or absolute), an optional block of Header: value lines, an optional body, and an assertion, a plain-English sentence describing what a passing response looks like, e.g. "the response confirms the user was created" or "should return a 401 without an auth header."

Creating an API test costs nothing, the same two-step model as a regular playtest. Nothing is charged until you trigger it.

Chaining requests​

Real API flows usually need to carry a value from one response into a later request: log in, then use the token; create a resource, then use its ID. Give a request an extract as name (e.g. token) and a response path (e.g. data.token, a dot-separated path into the response's JSON body), and every later request in the same test can reference it as {{token}} anywhere in its URL, headers, or body.

This is also the practical fix for a common source of flaky API tests: depending on a hardcoded ID that was valid when the test was written but got deleted or changed since. A test that creates its own fixture data at the start and chains that value forward doesn't have this problem. It's never depending on data outside its own run.

Mocking a response​

Retries handle a request that's slow or briefly flaky. They don't help when a dependency is actually down, or when the data it expects has expired or belongs to another team and is no longer valid. For exactly those cases, any request can be set to mock this response instead of actually sending it: you supply a status code, headers, and a body, and the step is judged against those instead of a real network call.

A mocked step still goes through the same evaluation as a real one. Claude checks the mocked response against your assertion, and if the request was set to capture a value, extraction runs on the mocked body exactly like it would on a real one, so a later request can still reference {{that_value}}. Mock response bodies also support {{name}} substitution themselves, so a mocked step can reflect a value a real, earlier step actually captured.

Typical uses:

  • A vendor you depend on is down or rate-limited. Mock the response it would normally return so the rest of the test can still run.
  • Another team's test data has expired or was removed. Mock the shape their endpoint should return instead of depending on fixture data you don't control.
  • Testing your own error handling. Deliberately mock a failure, e.g. a 503, to check that your app handles it correctly, independent of whether the real dependency happens to be failing right now.

An optional note (e.g. "vendor X is down") is shown next to the step in the report, so anyone reading it later understands why that request didn't hit the network. Mocked steps are marked with a mocked badge in the report.

Triggering​

The API test detail page shows a Run this test button and its credit cost (currently the same flat rate as a standard playtest, regardless of how many requests it has). Triggering debits the credit and runs every request in order, sequentially, not concurrently, since a later request may depend on an earlier one's captured value.

A request that fails at the transport level (a timeout, a connection refused) or gets a 5xx response is retried automatically, up to twice, before being marked failed. This is aimed squarely at flaky third-party dependencies rather than genuine bugs. A 4xx response is never retried: that's a deterministic client-side outcome (bad request, unauthorized, not found), not flakiness.

Reading the results​

Each request in the report shows its method and URL, the response status code, the assertion you wrote, and Claude's one-sentence reasoning for why it passed or failed. If a request has an assertion evaluation error (the AI call itself failed, rather than the assertion failing) the request is marked failed rather than silently passed. There's no deterministic fallback for a plain-English assertion the way there is for finding an element on a page, so an unverifiable assertion is treated as a failed one, not a passed one.

If a request was set to capture a value, the captured name and value are shown directly on that request so you can see exactly what got passed forward. Response bodies are available in a collapsible section on each request.

Sharing​

Every API test gets a public link at /api-report/{slug}, shown once it's been run: read-only, no trigger button, same content as the private detail page.

Triggering from CI/CD​

Like a regular playtest, every API test gets its own secret trigger URL (shown on the detail page) that a deploy pipeline can POST to, at /api-test-trigger/{token}, a separate endpoint from playtests' own CI trigger URL, since they're different run types with their own credit cost and their own report.

Gate mode​

Add ?wait=<seconds> to block until the run finishes, same as a playtest's CI/CD gate mode:

curl -X POST "https://veriwasp.com/api-test-trigger/{token}?wait=60"

You get the real result back once the run settles, up to a 90 second maximum wait (same cap as playtests):

{
  "run_id": "3f1e2c9a-...",
  "status": "completed",
  "report_url": "https://veriwasp.com/api-report/f8a2c1",
  "steps_total": 3,
  "steps_passed": 3,
  "steps_failed": 0,
  "issues_found": 0,
  "regressions_found": 0
}

If the wait budget runs out first, you get the same fire-and-forget response as not passing ?wait= at all, plus a status_url you can poll yourself at /api/api-tests/{slug}/status.

regressions_found is always 0 today. API test runs don't have regression detection yet the way playtests do (see Run History & Regressions), so ?gate=regression is currently a no-op for this run kind. Once API test regression detection ships, this field and the gate flag will activate automatically, no API changes needed.