Running scenario tests¶

Info

Learn the concepts and fundamentals of scenario tests in the Explanation page.

Scenario tests are offline tests used to compare the output from one or more scenarios.

This how-to guide explains how to interact with scenario tests using the /v1/applications/{application_id}/experiments/batch endpoints, passing "type": "scenario" in the request payload to distinguish them from regular batch experiments. Go to the reference section to see all the available parameters for each endpoint.

Create a scenario test¶

Use the POST /v1/applications/{application_id}/experiments/batch endpoint to create a new scenario test, passing "type": "scenario" in the request payload. The endpoint requires an id and a name in the payload, so generate an ID yourself if you don't want to choose a custom one.

Unlike a regular batch experiment, a scenario test's runs and option_sets are organized around individual scenarios. For each scenario, you need:

An instance to run against: set its ID directly in the instance_id field of every run belonging to the scenario.
Input data, which can come from:
- An existing input set: use its ID directly as the input_set_id of every run, and one of its input IDs as input_id.
- A list of existing managed inputs: first create a new input set from those inputs, then reference it as above.
- New raw data: first upload it as managed inputs, create a new input set from them, then reference it as above.
A scenario_id to group the scenario's runs together, such as scenario-1.
One or more option variations: for every option you want to vary, list its values. The cross product of all these values becomes the scenario's option sets, which are named <scenario_id>_<index> and placed under the top-level option_sets field of the payload.

Consider the following example where two scenarios are defined, using an input set (burrowing-hares) and a managed input (baxter-burrow), respectively. Both scenarios vary the details option between "true" and "false", and both are repeated 3 times (4 executions each).

Because the second scenario uses a list of managed inputs rather than an existing input set, an input set must be created for it first.

BashOutput

curl -s -X POST "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/inputsets" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"id": "baxter-burrow-set", "name": "Input set for baxter-burrow", "inputs": [{"id": "baxter-burrow"}]}' \
  | jq '.'

{
  "id": "baxter-burrow-set",
  "name": "Input set for baxter-burrow",
  "description": "",
  "app_id": "uncanny-rodent",
  "created_at": "2026-07-29T14:38:48.243781577Z",
  "input_ids": [],
  "updated_at": "2026-07-29T14:38:48.243781577Z",
  "inputs": [
    {
      "id": "baxter-burrow",
      "name": "The managed input of Baxter's Burrow",
      "description": ""
    }
  ]
}

With the input set in place for the second scenario, the two scenarios can be expressed as option_sets and runs in the scenario test's request payload:

BashOutput

SCENARIO_TEST_ID="scenario-$(openssl rand -hex 4)"

RESPONSE=$(curl -s -X POST "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/batch" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  -H "Content-Type: application/json" \
  -d @- <<EOF
{
  "id": "${SCENARIO_TEST_ID}",
  "name": "${SCENARIO_TEST_ID}",
  "type": "scenario",
  "option_sets": {
    "scenario-1_0": {"details": "true"},
    "scenario-1_1": {"details": "false"},
    "scenario-2_0": {"details": "true"},
    "scenario-2_1": {"details": "false"}
  },
  "runs": [
    {"input_id": "input.json-8MFSJpPDg", "input_set_id": "burrowing-hares", "instance_id": "production", "option_set": "scenario-1_0", "scenario_id": "scenario-1", "repetition": 0},
    {"input_id": "input.json-8MFSJpPDg", "input_set_id": "burrowing-hares", "instance_id": "production", "option_set": "scenario-1_0", "scenario_id": "scenario-1", "repetition": 1},
    {"input_id": "input.json-8MFSJpPDg", "input_set_id": "burrowing-hares", "instance_id": "production", "option_set": "scenario-1_0", "scenario_id": "scenario-1", "repetition": 2},
    {"input_id": "input.json-8MFSJpPDg", "input_set_id": "burrowing-hares", "instance_id": "production", "option_set": "scenario-1_0", "scenario_id": "scenario-1", "repetition": 3},
    {"input_id": "input.json-8MFSJpPDg", "input_set_id": "burrowing-hares", "instance_id": "production", "option_set": "scenario-1_1", "scenario_id": "scenario-1", "repetition": 0},
    {"input_id": "input.json-8MFSJpPDg", "input_set_id": "burrowing-hares", "instance_id": "production", "option_set": "scenario-1_1", "scenario_id": "scenario-1", "repetition": 1},
    {"input_id": "input.json-8MFSJpPDg", "input_set_id": "burrowing-hares", "instance_id": "production", "option_set": "scenario-1_1", "scenario_id": "scenario-1", "repetition": 2},
    {"input_id": "input.json-8MFSJpPDg", "input_set_id": "burrowing-hares", "instance_id": "production", "option_set": "scenario-1_1", "scenario_id": "scenario-1", "repetition": 3},
    {"input_id": "baxter-burrow", "input_set_id": "baxter-burrow-set", "instance_id": "production", "option_set": "scenario-2_0", "scenario_id": "scenario-2", "repetition": 0},
    {"input_id": "baxter-burrow", "input_set_id": "baxter-burrow-set", "instance_id": "production", "option_set": "scenario-2_0", "scenario_id": "scenario-2", "repetition": 1},
    {"input_id": "baxter-burrow", "input_set_id": "baxter-burrow-set", "instance_id": "production", "option_set": "scenario-2_0", "scenario_id": "scenario-2", "repetition": 2},
    {"input_id": "baxter-burrow", "input_set_id": "baxter-burrow-set", "instance_id": "production", "option_set": "scenario-2_0", "scenario_id": "scenario-2", "repetition": 3},
    {"input_id": "baxter-burrow", "input_set_id": "baxter-burrow-set", "instance_id": "production", "option_set": "scenario-2_1", "scenario_id": "scenario-2", "repetition": 0},
    {"input_id": "baxter-burrow", "input_set_id": "baxter-burrow-set", "instance_id": "production", "option_set": "scenario-2_1", "scenario_id": "scenario-2", "repetition": 1},
    {"input_id": "baxter-burrow", "input_set_id": "baxter-burrow-set", "instance_id": "production", "option_set": "scenario-2_1", "scenario_id": "scenario-2", "repetition": 2},
    {"input_id": "baxter-burrow", "input_set_id": "baxter-burrow-set", "instance_id": "production", "option_set": "scenario-2_1", "scenario_id": "scenario-2", "repetition": 3}
  ]
}
EOF
)

echo "$RESPONSE" | jq '.'

{
  "id": "scenario-f8667f6e"
}

The call above uses a randomly generated ID, and the same identifier for the scenario test's name. The name of the test is used as a human-readable label. You can pass different values for the id and name fields in the payload to specify a custom ID and name for the scenario test. For example:

BashOutput

RESPONSE=$(curl -s -X POST "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/batch" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  -H "Content-Type: application/json" \
  -d @- <<EOF
{
  "id": "fluffy-scenario-test",
  "name": "Scenario test for a fluffy bunny",
  "type": "scenario",
  "option_sets": {
    "scenario-1_0": {"details": "true"},
    "scenario-1_1": {"details": "false"},
    "scenario-2_0": {"details": "true"},
    "scenario-2_1": {"details": "false"}
  },
  "runs": [
    {"input_id": "input.json-8MFSJpPDg", "input_set_id": "burrowing-hares", "instance_id": "production", "option_set": "scenario-1_0", "scenario_id": "scenario-1", "repetition": 0},
    {"input_id": "input.json-8MFSJpPDg", "input_set_id": "burrowing-hares", "instance_id": "production", "option_set": "scenario-1_0", "scenario_id": "scenario-1", "repetition": 1},
    {"input_id": "input.json-8MFSJpPDg", "input_set_id": "burrowing-hares", "instance_id": "production", "option_set": "scenario-1_0", "scenario_id": "scenario-1", "repetition": 2},
    {"input_id": "input.json-8MFSJpPDg", "input_set_id": "burrowing-hares", "instance_id": "production", "option_set": "scenario-1_0", "scenario_id": "scenario-1", "repetition": 3},
    {"input_id": "input.json-8MFSJpPDg", "input_set_id": "burrowing-hares", "instance_id": "production", "option_set": "scenario-1_1", "scenario_id": "scenario-1", "repetition": 0},
    {"input_id": "input.json-8MFSJpPDg", "input_set_id": "burrowing-hares", "instance_id": "production", "option_set": "scenario-1_1", "scenario_id": "scenario-1", "repetition": 1},
    {"input_id": "input.json-8MFSJpPDg", "input_set_id": "burrowing-hares", "instance_id": "production", "option_set": "scenario-1_1", "scenario_id": "scenario-1", "repetition": 2},
    {"input_id": "input.json-8MFSJpPDg", "input_set_id": "burrowing-hares", "instance_id": "production", "option_set": "scenario-1_1", "scenario_id": "scenario-1", "repetition": 3},
    {"input_id": "baxter-burrow", "input_set_id": "baxter-burrow-set", "instance_id": "production", "option_set": "scenario-2_0", "scenario_id": "scenario-2", "repetition": 0},
    {"input_id": "baxter-burrow", "input_set_id": "baxter-burrow-set", "instance_id": "production", "option_set": "scenario-2_0", "scenario_id": "scenario-2", "repetition": 1},
    {"input_id": "baxter-burrow", "input_set_id": "baxter-burrow-set", "instance_id": "production", "option_set": "scenario-2_0", "scenario_id": "scenario-2", "repetition": 2},
    {"input_id": "baxter-burrow", "input_set_id": "baxter-burrow-set", "instance_id": "production", "option_set": "scenario-2_0", "scenario_id": "scenario-2", "repetition": 3},
    {"input_id": "baxter-burrow", "input_set_id": "baxter-burrow-set", "instance_id": "production", "option_set": "scenario-2_1", "scenario_id": "scenario-2", "repetition": 0},
    {"input_id": "baxter-burrow", "input_set_id": "baxter-burrow-set", "instance_id": "production", "option_set": "scenario-2_1", "scenario_id": "scenario-2", "repetition": 1},
    {"input_id": "baxter-burrow", "input_set_id": "baxter-burrow-set", "instance_id": "production", "option_set": "scenario-2_1", "scenario_id": "scenario-2", "repetition": 2},
    {"input_id": "baxter-burrow", "input_set_id": "baxter-burrow-set", "instance_id": "production", "option_set": "scenario-2_1", "scenario_id": "scenario-2", "repetition": 3}
  ]
}
EOF
)

echo "$RESPONSE" | jq '.'

{
  "id": "fluffy-scenario-test"
}

Get a scenario test¶

Info

The best way to view and interact with scenario test results is in the Nextmv Console.

Use the GET /v1/applications/{application_id}/experiments/batch/{batch_id}/metadata endpoint to retrieve the metadata for a scenario test, using the scenario test ID.

BashOutput

curl -s -X GET "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/batch/fluffy-scenario-test/metadata" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  | jq '.'

{
  "id": "fluffy-scenario-test",
  "name": "Scenario test for a fluffy bunny",
  "description": "",
  "app_id": "uncanny-rodent",
  "created_at": "2026-07-29T14:42:39Z",
  "updated_at": "2026-07-29T14:42:53Z",
  "status": "completed",
  "number_of_requested_runs": 16,
  "number_of_runs": 16,
  "number_of_completed_runs": 16,
  "type": "scenario"
}

Once the status of the scenario test is completed, you can get its summary and distributional statistics using the GET /v1/applications/{application_id}/experiments/batch/{batch_id} endpoint.

BashOutput

curl -s -X GET "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/batch/fluffy-scenario-test" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  | jq '.'

{
  "id": "fluffy-scenario-test",
  "name": "Scenario test for a fluffy bunny",
  "description": "",
  "status": "completed",
  "created_at": "2026-07-29T14:42:39.114354903Z",
  "updated_at": "2026-07-29T14:42:53.507794402Z",
  "input_set_id": "",
  "instance_ids": [
    "production"
  ],
  "option_sets": {
    "scenario-1_0": {
      "details": "true"
    },
    "scenario-1_1": {
      "details": "false"
    },
    "scenario-2_0": {
      "details": "true"
    },
    "scenario-2_1": {
      "details": "false"
    }
  },
  "number_of_requested_runs": 16,
  "number_of_runs": 16,
  "number_of_completed_runs": 16,
  "type": "scenario",
  "grouped_distributional_summaries": [
    {
      "group_keys": [
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "production",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 16,
          "mean": 1.2300000000000002,
          "std": 2.293266818639604e-16,
          "shifted_geometric_mean": {
            "value": 1.230000000000011,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 16
    },
    {
      "group_keys": [
        "inputID",
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "baxter-burrow",
        "production",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 8,
          "mean": 1.2300000000000002,
          "std": 2.3737566748887e-16,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 8
    },
    {
      "group_keys": [
        "inputID",
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "input.json-8MFSJpPDg",
        "production",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 8,
          "mean": 1.2300000000000002,
          "std": 2.3737566748887e-16,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 8
    },
    {
      "group_keys": [
        "inputID"
      ],
      "group_values": [
        "baxter-burrow"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 8,
          "mean": 1.2300000000000002,
          "std": 2.3737566748887e-16,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 8
    },
    {
      "group_keys": [
        "inputID"
      ],
      "group_values": [
        "input.json-8MFSJpPDg"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 8,
          "mean": 1.2300000000000002,
          "std": 2.3737566748887e-16,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 8
    }
  ]
}

Note that this endpoint does not include the individual runs of the scenario test. Use the GET /v1/applications/{application_id}/experiments/batch/{batch_id}/runs endpoint to retrieve them instead. This endpoint is paginated, so the snippet below always uses pagination: it passes pagereturn=true to receive a next_page_token in the response, and keeps requesting pages by passing that token back as pagetoken until no token is returned.

BashOutput

RUNS="[]"
PAGE_TOKEN=""

while :; do
  RESPONSE=$(curl -s -G "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/batch/fluffy-scenario-test/runs" \
    -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
    --data-urlencode "pagereturn=true" \
    --data-urlencode "pagetoken=${PAGE_TOKEN}")

  RUNS=$(jq -n --argjson existing "$RUNS" --argjson page "$(echo "$RESPONSE" | jq '.runs')" '$existing + $page')
  PAGE_TOKEN=$(echo "$RESPONSE" | jq -r '.next_page_token // empty')

  [ -z "$PAGE_TOKEN" ] && break
done

echo "$RUNS" | jq '.'

[
  {
    "id": "production-lq6TCQPvg",
    "user_email": "sebastian@nextmv.io",
    "name": "",
    "description": "",
    "created_at": "2026-07-29T14:42:39.194833544Z",
    "application_id": "uncanny-rodent",
    "application_instance_id": "production",
    "application_version_id": "v0.0.2",
    "run_type": {
      "type": "",
      "definition_id": "",
      "reference_id": ""
    },
    "execution_class": "6c9500mb870s",
    "queuing_priority": 6,
    "queuing_disabled": false,
    "runtime": "python-3_11",
    "status": "succeeded",
    "status_v2": "succeeded",
    "experiment_id": "fluffy-scenario-test",
    "metrics": {
      "status": "succeeded",
      "indicators": [
        {
          "name": "value",
          "value": 1.23
        },
        {
          "name": "metadata.duration",
          "value": 4.448
        }
      ]
    },
    "input_id": "baxter-burrow",
    "option_set": "scenario-2_1",
    "options": {
      "details": "false"
    },
    "options_summary": [
      {
        "name": "details",
        "value": "false",
        "source": "run"
      }
    ],
    "scenario_id": "scenario-2",
    "repetition": 3,
    "input_set_id": "baxter-burrow-set"
  },
  ...
]

You can list all scenario tests in the application using the GET /v1/applications/{application_id}/experiments/batch endpoint, passing type=scenario as a query parameter. This endpoint is paginated, so the snippet below always uses pagination: it passes pagereturn=true to receive a next_page_token in the response, and keeps requesting pages by passing that token back as pagetoken until no token is returned.

BashOutput

ITEMS="[]"
PAGE_TOKEN=""

while :; do
  RESPONSE=$(curl -s -G "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/batch" \
    -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
    --data-urlencode "type=scenario" \
    --data-urlencode "pagereturn=true" \
    --data-urlencode "pagetoken=${PAGE_TOKEN}")

  ITEMS=$(jq -n --argjson existing "$ITEMS" --argjson page "$(echo "$RESPONSE" | jq '.items')" '$existing + $page')
  PAGE_TOKEN=$(echo "$RESPONSE" | jq -r '.next_page_token // empty')

  [ -z "$PAGE_TOKEN" ] && break
done

echo "$ITEMS" | jq '.[:2]'

[
  {
    "id": "fluffy-scenario-test",
    "name": "Scenario test for a fluffy bunny",
    "description": "",
    "status": "completed",
    "created_at": "2026-07-29T14:42:39.114354903Z",
    "updated_at": "2026-07-29T14:42:53.507794402Z",
    "input_set_id": "",
    "instance_ids": [
      "production"
    ],
    "option_sets": {
      "scenario-1_0": {
        "details": "true"
      },
      "scenario-1_1": {
        "details": "false"
      },
      "scenario-2_0": {
        "details": "true"
      },
      "scenario-2_1": {
        "details": "false"
      }
    },
    "number_of_requested_runs": 16,
    "number_of_runs": 16,
    "number_of_completed_runs": 16,
    "type": "scenario"
  },
  {
    "id": "scenario-f8667f6e",
    "name": "scenario-f8667f6e",
    "description": "",
    "status": "completed",
    "created_at": "2026-07-29T14:42:36.320232518Z",
    "updated_at": "2026-07-29T14:42:49.360138327Z",
    "input_set_id": "",
    "instance_ids": [
      "production"
    ],
    "option_sets": {
      "scenario-1_0": {
        "details": "true"
      },
      "scenario-1_1": {
        "details": "false"
      },
      "scenario-2_0": {
        "details": "true"
      },
      "scenario-2_1": {
        "details": "false"
      }
    },
    "number_of_requested_runs": 16,
    "number_of_runs": 16,
    "number_of_completed_runs": 16,
    "type": "scenario"
  }
]

Update a scenario test¶

You can update attributes of a scenario test with the PATCH /v1/applications/{application_id}/experiments/batch/{batch_id} endpoint, such as its:

Name
Description

Unlike updating an application or an instance, this endpoint only expects the fields you want to change, so there is no need to fetch the scenario test first. You cannot update the ID of a scenario test.

BashOutput

curl -s -X PATCH "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/batch/fluffy-scenario-test" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"name": "Updated Scenario Test Name", "description": "Updated description for the scenario test"}' \
  | jq '.'

{
  "id": "fluffy-scenario-test",
  "name": "Updated Scenario Test Name",
  "description": "Updated description for the scenario test",
  "created_at": "2026-07-29T14:42:39.114354903Z",
  "updated_at": "2026-07-29T14:43:16.700686849Z"
}

Delete a scenario test¶

Warning

Deleting a scenario test is irreversible. All the runs associated with the scenario test will be permanently deleted.

Delete a scenario test using the DELETE /v1/applications/{application_id}/experiments/batch/{batch_id} endpoint.

Bash

curl -s -X DELETE "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/batch/fluffy-scenario-test" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}"