Skip to content

Running scenario tests

Info

Learn the concepts and fundamentals of scenario tests in the Explanation page.

Scenario tests are offline tests used to compare the output from one or more scenarios.

Included in this how-to guide are instructions on how to interact with scenario tests using methods on the Application class. Go the reference section to see all the available parameters for each method.

Create a scenario test

Use the Application.new_scenario_test method (or Application.new_scenario_test_with_result, which additionally polls for the result) to create a new scenario test for an application. Pass the scenarios keyword argument as a list of Scenario objects.

Each scenario has the following attributes:

  • instance_id: ID of the instance to use for this scenario (required).
  • scenario_input: A ScenarioInput object (required), with:
    • scenario_input_type: A ScenarioInputType value (required). Allowed values: ScenarioInputType.INPUT_SET, ScenarioInputType.INPUT, and ScenarioInputType.NEW.
    • scenario_input_data: Data for the scenario input (required).
      • For ScenarioInputType.INPUT_SET: a str (the input set ID).
      • For ScenarioInputType.INPUT: a list[str] (list of input IDs).
      • For ScenarioInputType.NEW: a list[dict] (raw data).
  • scenario_id: ID of the scenario (optional). The default value will be set as scenario-<index> if not set.
  • configuration: A list of ScenarioConfiguration objects (optional). Use this attribute to configure variation of options for the scenario. Each object requires:
    • name: Name of the configuration option.
    • values: List of values for the configuration option.

Consider the following example where two scenarios are defined, using an input set and a managed input, respectively.

from nextmv.cloud import Scenario, ScenarioConfiguration, ScenarioInput, ScenarioInputType

scenarios = [
    Scenario(
        instance_id="production",
        scenario_input=ScenarioInput(
            scenario_input_type=ScenarioInputType.INPUT_SET,
            scenario_input_data="burrowing-hares",
        ),
        configuration=[ScenarioConfiguration(name="details", values=["true", "false"])],
    ),
    Scenario(
        instance_id="production",
        scenario_input=ScenarioInput(
            scenario_input_type=ScenarioInputType.INPUT,
            scenario_input_data=["baxter-burrow"],
        ),
        configuration=[ScenarioConfiguration(name="details", values=["true", "false"])],
    ),
]

Here is an example that uses the scenarios defined above to create a new scenario test for an application with repetitions.

import os

import nextmv
from nextmv import cloud
from nextmv.cloud import Scenario, ScenarioConfiguration, ScenarioInput, ScenarioInputType

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
scenarios = [
    Scenario(
        instance_id="production",
        scenario_input=ScenarioInput(
            scenario_input_type=ScenarioInputType.INPUT_SET,
            scenario_input_data="burrowing-hares",
        ),
        configuration=[ScenarioConfiguration(name="details", values=["true", "false"])],
    ),
    Scenario(
        instance_id="production",
        scenario_input=ScenarioInput(
            scenario_input_type=ScenarioInputType.INPUT,
            scenario_input_data=["baxter-burrow"],
        ),
        configuration=[ScenarioConfiguration(name="details", values=["true", "false"])],
    ),
]
scenario_test_id = app.new_scenario_test(scenarios=scenarios, repetitions=3)

nextmv.write({"scenario_test_id": scenario_test_id})

uv run main.py

{
  "scenario_test_id": "scenario-y72aeh0h"
}

The call above will create a random ID, and use the same identifier for the scenario test's name. The name of the test is used as a human-readable label. You can use the id and/or name keyword arguments to specify a custom ID and name for the scenario test. For example:

import os

import nextmv
from nextmv import cloud
from nextmv.cloud import Scenario, ScenarioConfiguration, ScenarioInput, ScenarioInputType

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
scenarios = [
    Scenario(
        instance_id="production",
        scenario_input=ScenarioInput(
            scenario_input_type=ScenarioInputType.INPUT_SET,
            scenario_input_data="burrowing-hares",
        ),
        configuration=[ScenarioConfiguration(name="details", values=["true", "false"])],
    ),
    Scenario(
        instance_id="production",
        scenario_input=ScenarioInput(
            scenario_input_type=ScenarioInputType.INPUT,
            scenario_input_data=["baxter-burrow"],
        ),
        configuration=[ScenarioConfiguration(name="details", values=["true", "false"])],
    ),
]
scenario_test_id = app.new_scenario_test(
    scenarios=scenarios,
    repetitions=3,
    id="fluffy-scenario-test",
    name="Scenario test for a fluffy bunny",
)

nextmv.write({"scenario_test_id": scenario_test_id})

uv run main.py

{
  "scenario_test_id": "fluffy-scenario-test"
}

Get a scenario test

Info

The best way to view and interact with scenario test results is in the Nextmv Console.

Use the Application.scenario_test_metadata method to retrieve the metadata for a scenario test, using the scenario test ID. The method returns a BatchExperimentMetadata object.

import os

import nextmv
from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
metadata = app.scenario_test_metadata(scenario_test_id="fluffy-scenario-test")

nextmv.write(metadata.to_dict())

uv run main.py

{
  "id": "fluffy-scenario-test",
  "name": "Scenario test for a fluffy bunny",
  "created_at": "2026-07-28T20:16:41Z",
  "updated_at": "2026-07-28T20:16:56Z",
  "status": "completed",
  "description": "",
  "number_of_requested_runs": 16,
  "number_of_runs": 16,
  "number_of_completed_runs": 16,
  "type": "scenario",
  "app_id": "uncanny-rodent"
}

Once the status of the scenario test is completed, you can get the results using the Application.scenario_test method (or Application.scenario_test_with_polling, which polls until the test finishes). The method returns a BatchExperiment object, whose output includes the runs that were made for the scenario test.

import os

import nextmv
from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
scenario_test = app.scenario_test(scenario_test_id="fluffy-scenario-test")

nextmv.write(scenario_test.to_dict())

uv run main.py

{
  "id": "fluffy-scenario-test",
  "name": "Scenario test for a fluffy bunny",
  "created_at": "2026-07-28T20:16:41.889417Z",
  "updated_at": "2026-07-28T20:16:56.096654Z",
  "status": "completed",
  "description": "",
  "number_of_requested_runs": 16,
  "number_of_runs": 16,
  "number_of_completed_runs": 16,
  "type": "scenario",
  "option_sets": {
    "scenario-1_0": {
      "details": "true"
    },
    "scenario-1_1": {
      "details": "false"
    },
    "scenario-2_0": {
      "details": "true"
    },
    "scenario-2_1": {
      "details": "false"
    }
  },
  "input_set_id": "",
  "instance_ids": [
    "production"
  ],
  "grouped_distributional_summaries": [
    {
      "group_keys": [
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "production",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 16,
          "mean": 1.2300000000000002,
          "std": 2.293266818639604e-16,
          "shifted_geometric_mean": {
            "value": 1.230000000000011,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 16
    },
    {
      "group_keys": [
        "inputID",
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "baxter-burrow",
        "production",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 8,
          "mean": 1.2300000000000002,
          "std": 2.3737566748887e-16,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 8
    },
    {
      "group_keys": [
        "inputID",
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "input.json-8MFSJpPDg",
        "production",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 8,
          "mean": 1.2300000000000002,
          "std": 2.3737566748887e-16,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 8
    },
    {
      "group_keys": [
        "inputID"
      ],
      "group_values": [
        "baxter-burrow"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 8,
          "mean": 1.2300000000000002,
          "std": 2.3737566748887e-16,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 8
    },
    {
      "group_keys": [
        "inputID"
      ],
      "group_values": [
        "input.json-8MFSJpPDg"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 8,
          "mean": 1.2300000000000002,
          "std": 2.3737566748887e-16,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 8
    }
  ],
  "runs": [
    {
      "id": "production-EeC4qyPvg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-28T20:16:41.974212Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-scenario-test",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 5.582
          }
        ]
      },
      "input_id": "baxter-burrow",
      "option_set": "scenario-2_1",
      "options": {
        "details": "false"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "false",
          "source": "run"
        }
      ],
      "scenario_id": "scenario-2",
      "repetition": 3,
      "input_set_id": "inpset-scenario-2-nkb0whcn"
    },
    {
      "id": "production-J6CV3yEDR",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-28T20:16:41.969549Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-scenario-test",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.528
          }
        ]
      },
      "input_id": "baxter-burrow",
      "option_set": "scenario-2_1",
      "options": {
        "details": "false"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "false",
          "source": "run"
        }
      ],
      "scenario_id": "scenario-2",
      "repetition": 2,
      "input_set_id": "inpset-scenario-2-nkb0whcn"
    },
    {
      "id": "production-pej4qsPvg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-28T20:16:41.965183Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-scenario-test",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 5.616
          }
        ]
      },
      "input_id": "baxter-burrow",
      "option_set": "scenario-2_1",
      "options": {
        "details": "false"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "false",
          "source": "run"
        }
      ],
      "scenario_id": "scenario-2",
      "repetition": 1,
      "input_set_id": "inpset-scenario-2-nkb0whcn"
    },
    {
      "id": "production-H6j43yPDR",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-28T20:16:41.960639Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-scenario-test",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.451
          }
        ]
      },
      "input_id": "baxter-burrow",
      "option_set": "scenario-2_1",
      "options": {
        "details": "false"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "false",
          "source": "run"
        }
      ],
      "scenario_id": "scenario-2",
      "repetition": 0,
      "input_set_id": "inpset-scenario-2-nkb0whcn"
    },
    {
      "id": "production-Wej4qyPDg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-28T20:16:41.955913Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-scenario-test",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.38
          }
        ]
      },
      "input_id": "baxter-burrow",
      "option_set": "scenario-2_0",
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ],
      "scenario_id": "scenario-2",
      "repetition": 3,
      "input_set_id": "inpset-scenario-2-nkb0whcn"
    },
    {
      "id": "production-6qjVqsPvg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-28T20:16:41.951583Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-scenario-test",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.325
          }
        ]
      },
      "input_id": "baxter-burrow",
      "option_set": "scenario-2_0",
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ],
      "scenario_id": "scenario-2",
      "repetition": 2,
      "input_set_id": "inpset-scenario-2-nkb0whcn"
    },
    {
      "id": "production-Q3jV3sPvR",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-28T20:16:41.945880Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-scenario-test",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.425
          }
        ]
      },
      "input_id": "baxter-burrow",
      "option_set": "scenario-2_0",
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ],
      "scenario_id": "scenario-2",
      "repetition": 1,
      "input_set_id": "inpset-scenario-2-nkb0whcn"
    },
    {
      "id": "production-D3jVqsPDR",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-28T20:16:41.929910Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-scenario-test",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 3.752
          }
        ]
      },
      "input_id": "baxter-burrow",
      "option_set": "scenario-2_0",
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ],
      "scenario_id": "scenario-2",
      "repetition": 0,
      "input_set_id": "inpset-scenario-2-nkb0whcn"
    },
    {
      "id": "production-n3C43sPvg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-28T20:16:41.925588Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-scenario-test",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.394
          }
        ]
      },
      "input_id": "input.json-8MFSJpPDg",
      "option_set": "scenario-1_1",
      "options": {
        "details": "false"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "false",
          "source": "run"
        }
      ],
      "scenario_id": "scenario-1",
      "repetition": 3,
      "input_set_id": "burrowing-hares"
    },
    {
      "id": "production-R3CVqsPvg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-28T20:16:41.920076Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-scenario-test",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 5.616
          }
        ]
      },
      "input_id": "input.json-8MFSJpPDg",
      "option_set": "scenario-1_1",
      "options": {
        "details": "false"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "false",
          "source": "run"
        }
      ],
      "scenario_id": "scenario-1",
      "repetition": 2,
      "input_set_id": "burrowing-hares"
    },
    {
      "id": "production-XCCVqsEDR",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-28T20:16:41.915319Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-scenario-test",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.44
          }
        ]
      },
      "input_id": "input.json-8MFSJpPDg",
      "option_set": "scenario-1_1",
      "options": {
        "details": "false"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "false",
          "source": "run"
        }
      ],
      "scenario_id": "scenario-1",
      "repetition": 1,
      "input_set_id": "burrowing-hares"
    },
    {
      "id": "production-sjCV3yPvg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-28T20:16:41.911123Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-scenario-test",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 5.679
          }
        ]
      },
      "input_id": "input.json-8MFSJpPDg",
      "option_set": "scenario-1_1",
      "options": {
        "details": "false"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "false",
          "source": "run"
        }
      ],
      "scenario_id": "scenario-1",
      "repetition": 0,
      "input_set_id": "burrowing-hares"
    },
    {
      "id": "production-bCCVqyPDR",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-28T20:16:41.906574Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-scenario-test",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.455
          }
        ]
      },
      "input_id": "input.json-8MFSJpPDg",
      "option_set": "scenario-1_0",
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ],
      "scenario_id": "scenario-1",
      "repetition": 3,
      "input_set_id": "burrowing-hares"
    },
    {
      "id": "production-pjj4qsEvg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-28T20:16:41.901241Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-scenario-test",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 5.58
          }
        ]
      },
      "input_id": "input.json-8MFSJpPDg",
      "option_set": "scenario-1_0",
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ],
      "scenario_id": "scenario-1",
      "repetition": 2,
      "input_set_id": "burrowing-hares"
    },
    {
      "id": "production-SCjV3yEDg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-28T20:16:41.895495Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-scenario-test",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.46
          }
        ]
      },
      "input_id": "input.json-8MFSJpPDg",
      "option_set": "scenario-1_0",
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ],
      "scenario_id": "scenario-1",
      "repetition": 1,
      "input_set_id": "burrowing-hares"
    },
    {
      "id": "production-kCCV3yPvR",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-28T20:16:41.889667Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-scenario-test",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 5.707
          }
        ]
      },
      "input_id": "input.json-8MFSJpPDg",
      "option_set": "scenario-1_0",
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ],
      "scenario_id": "scenario-1",
      "repetition": 0,
      "input_set_id": "burrowing-hares"
    }
  ]
}

You can list all scenario tests in the application using the Application.list_scenario_tests method.

import json
import os

from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
scenario_tests = app.list_scenario_tests()

print(json.dumps([scenario_test.to_dict() for scenario_test in scenario_tests[:2]], indent=2))

uv run main.py

[
  {
    "id": "fluffy-scenario-test",
    "name": "Scenario test for a fluffy bunny",
    "created_at": "2026-07-28T20:16:41.889417Z",
    "updated_at": "2026-07-28T20:16:56.096654Z",
    "status": "completed",
    "description": "",
    "number_of_requested_runs": 16,
    "number_of_runs": 16,
    "number_of_completed_runs": 16,
    "type": "scenario",
    "option_sets": {
      "scenario-1_0": {
        "details": "true"
      },
      "scenario-1_1": {
        "details": "false"
      },
      "scenario-2_0": {
        "details": "true"
      },
      "scenario-2_1": {
        "details": "false"
      }
    }
  },
  {
    "id": "scenario-y72aeh0h",
    "name": "scenario-y72aeh0h",
    "created_at": "2026-07-28T20:15:32.509435Z",
    "updated_at": "2026-07-28T20:15:47.579269Z",
    "status": "completed",
    "description": "",
    "number_of_requested_runs": 16,
    "number_of_runs": 16,
    "number_of_completed_runs": 16,
    "type": "scenario",
    "option_sets": {
      "scenario-1_0": {
        "details": "true"
      },
      "scenario-1_1": {
        "details": "false"
      },
      "scenario-2_0": {
        "details": "true"
      },
      "scenario-2_1": {
        "details": "false"
      }
    }
  }
]

Update a scenario test

You can update attributes of a scenario test with the Application.update_scenario_test method, such as its:

  • Name
  • Description

The method returns a BatchExperimentInformation object. You cannot update the ID of a scenario test.

import os

import nextmv
from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
info = app.update_scenario_test(
    scenario_test_id="fluffy-scenario-test",
    name="Updated Scenario Test Name",
    description="Updated description for the scenario test",
)

nextmv.write(info.to_dict())

uv run main.py

{
  "id": "fluffy-scenario-test",
  "name": "Updated Scenario Test Name",
  "created_at": "2026-07-28T20:16:41.889417Z",
  "updated_at": "2026-07-28T20:17:35.079082Z",
  "description": "Updated description for the scenario test"
}

Delete a scenario test

Warning

Deleting a scenario test is irreversible. All the runs associated with the scenario test will be permanently deleted.

Delete a scenario test using the Application.delete_scenario_test method.

import os

from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
app.delete_scenario_test(scenario_test_id="fluffy-scenario-test")

uv run main.py