Skip to main content

Variants & combinations in executions

A test case with variants (parameter combinations) runs the same step definition multiple times with different data — for example "Order book in EN", "Order book in DE" and "Order book in FR" from the same parent test definition. This page explains how those multi-variant runs show up in the Test Execution views.

If you are new to defining variants, start with Parameters and test-data — this page only covers the execution-side view.

One execution, many test executions​

The platform's data model has two layers:

  • Execution — the parent object created by clicking Execute Test (or by a test set / pipeline trigger). One row in the Executions list.
  • Test execution — one concrete run of one test with one specific variant. There is one test execution per variant chosen at start time.

So when you run a 3-variant test in All variants mode, you get one execution containing three test executions.

Both layers carry their own status. The parent execution status is derived: Passed iff every test execution passed; otherwise Failed (or Cancelled / etc. depending on how the run ended).

Choosing which variants to run​

When you click Execute Test on a test that has variants, a dialog asks which combination(s) to execute. You can:

  • Run a single variant — useful for re-trying a specific failure or debugging one parameter set.
  • Run multiple selected variants — useful for running a chosen subset.
  • Run all variants — the typical "regression" mode.

For interactive flows (Record Test, Pause to Interactive) you can only pick one variant at a time, since interactive runs need a single live browser session.

How it shows up in the list​

In the Executions list, a multi-variant run appears as one row — the parent execution. The columns and the row's status reflect the aggregate:

  • Execution column shows the test or test set name with the parent's status tag.
  • Tokens consumed, Cache hit ratio and Duration are summed (or aggregated) across all variants.
  • The tooltip on the Execution column lists the first ten member tests with their individual statuses — so you can tell at a glance whether the failure was in one variant or many.

The per-variant breakdown lives on the detail page.

The detail page with variants​

Open a multi-variant parent execution and you'll see the per-variant tests in the left panel:

  • Each variant appears as its own row, with the test title followed by (combination name) in parentheses — e.g. Order book (EN), Order book (DE), Order book (FR).
  • Each row has its own status icon (so you can immediately see which variant passed and which failed).
  • The URL /tenant/:tenant/executions/display/:executionId/test/:testId encodes the selected variant via :testId — different variants of the same parent execution all share the same executionId but have different testId values.

Click any row to see the steps, screenshots and video for that specific variant.

tip

Use the URL to share a link to exactly one variant of a multi-variant run — for example when filing a bug against a specific language or currency. Copy the URL while the variant is selected.

Reading parameter values per variant​

The variant's parameter values are visible in two places on the detail page:

  1. The test's ⓘ info popover in the left panel shows an Execution Variant field with the combination name, and a timeline of resolved parameter and macro values for that variant.
  2. The step detail popover has a Resolved parameters & calculations field that shows the step description with every [[parameter]] and ${ ... } and ${= ... } resolved against the current variant's values.

When you compare two variants of the same test — say one that passed and one that failed — those two fields are usually where you find the difference.

Re-running a multi-variant run​

The toolbar of the parent execution has the standard rerun options:

  • Rerun — re-runs all variants of the parent. The new execution will again have one test execution per variant.
  • Rerun failed — re-runs only the variants whose status is Failed, Cancelled or Initial. The already-passed variants are reused (their results are carried over).
  • Start recording from here — only meaningful for a single variant. Open the variant you want to fork from, then use the option from a step's popover.

Re-runs always create a new parent execution; the old one stays in the list.

Re-running just one variant from a multi-variant parent​

There is no toolbar shortcut to re-run a single variant from a parent execution that ran all variants. Either:

  • Go to the test definition (the edit pencil icon next to the variant's title in the left panel) and click Execute Test, then pick the single variant — this creates a fresh execution.
  • Or use Start recording from here on a step inside that variant — this gives you an interactive session.

What if the variant set changed since the original run?​

The test execution row stores the variant name it was started with, not a live reference to the variant definition. If you delete or rename a variant after a run, the old execution keeps showing the original combination name. Rerunning the old execution with Rerun picks up the current variant set: variants that were deleted are skipped, new variants are not added automatically.

In other words: Rerun preserves the original definition of "which variants to run", as long as those variants still exist. If you want a strictly-up-to-date run of all current variants, start a new execution from the test definition.

Logs, cache and assertions across variants​

Each variant is a fully independent test execution. That means:

  • Logs are scoped per variant. The Log popover switches view when you select a different variant in the left panel.
  • Cache is shared at the test-definition level — variants share the same XPath cache for the test. Clearing the cache for one variant clears it for the entire test definition.
  • Assertions that depend on a parameter (e.g. "I expect the page to show [[expectedTitle]]") are resolved against each variant's values independently, so different variants can legitimately reach different verdicts on the same step.

What variants do not provide​

A few things that come up in practice but are not what variants are for:

  • Same test in different browsers — variants share one browser context per run. To test across browsers, configure separate execution targets at the tenant level.
  • Different user/role per variant — the user/role is set on the test as a whole, not per variant. To exercise multiple roles, create separate test definitions.
  • Long-running data setup — variants don't have their own setup hooks. For per-variant setup, use Scripting or add setup steps to the test that read from parameters.

See also​