Scenario

Scenario tests

A tutorial for creating and managing scenario tests on Nextmv Cloud.

Scenario tests are offline tests used to compare the output from one or more scenarios. A scenario is composed of:

  • An instance: the executable code of the model being tested.
  • An input set: a collection of inputs.
  • One or more options: a configuration for the run. Also known as parameters or constants.

On top of scenarios, you can configure repetitions, to test for variability in the results.

Scenario tests are generally used as an exploratory test to understand the impacts to business metrics (or KPIs) on situations such as:

  • updating a model with a new feature, such as an additional constraint;
  • comparing how the same model performs in different conditions, such as low demand vs. high demand;
  • doing a sensitivity analysis to understand how the model behaves when changing a parameter.

They can also be used to validate that a model is ready for further testing and likely to make an intended business impact.

To be concise, the test creates a run for each scenario, for all the combinations of repetitions, inputs in the input set, and option sets. For example, imagine you configure these two scenarios:

  • Scenario 1: instance i-1, input set s-1 (with 3 inputs)
  • Scenario 2: instance i-2, input set s-2 (with 2 inputs)

Both scenarios use the same options. If this scenario test was applied 2 repetitions, it would produce the following 15 runs:

Run #InstanceInput setInput IDRepetition
1i-1s-1input_s-1_10
2i-1s-1input_s-1_11
3i-1s-1input_s-1_12
4i-1s-1input_s-1_20
5i-1s-1input_s-1_21
6i-1s-1input_s-1_22
7i-1s-1input_s-1_30
8i-1s-1input_s-1_31
9i-1s-1input_s-1_32
10i-2s-2input_s-2_10
11i-2s-2input_s-2_11
12i-2s-2input_s-2_12
13i-2s-2input_s-2_20
14i-2s-2input_s-2_21
15i-2s-2input_s-2_22

Now, consider that you are adding two options to the scenario:

  • duration: with three possible values of 1, 2, and 3 seconds.
  • iterations: also with three possible values of 32, 35, and 40.

The possible combination of these two options is displayed in the table below:

durationiterations
132
135
140
232
235
240
332
335
340

If we apply these 9 combinations to the 15 runs, we will have a total of 135 runs that can be created for a scenario test.

Once the runs are finished, the results are then aggregated and analyzed. The decision model must produce an output that contains a statistics block that uses the statistics convention.

Scenario tests are designed to be started and visualized from the Console web interface. Go to the app, Experiments > Scenario tab.

Scenario test

We highly recommend using the Console for creating and managing scenario tests. However, you can also use the Cloud API for this purpose.

When using the Console, you will see a view similar to the following when creating a new scenario test. Please direct your attention to the configuration of options for the scenario. This is where you can add the different variations of options that will be applied to each run.

Scenario test create

Cloud API

Use the following endpoint to create a scenario test. Define the desired experiment ID and name. After, create the scenario test.

POSThttps://api.cloud.nextmv.io/v1/applications/{application_id}/experiments/batch

Create and start batch experiment.

Create and start batch experiment.

curl -L -X POST \
    "https://api.cloud.nextmv.io/v1/applications/$APP_ID/experiments/batch" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $NEXTMV_API_KEY" \
    -d "{
        \"id\": \"<YOUR-SCENARIO-TEST-ID>\",
        \"name\": \"<YOUR-SCENARIO-TEST-NAME>\",
        \"option_sets\": {
          \"set-a\": {
            \"option-a-1\": \"value-a-1\",
          },
          \"set-b\": {
            \"option-b-1\": \"value-b-1\",
          },
        },
        \"runs\": [
          {
            \"instance_id\": \"<INSTANCE-ID>\",
            \"version_id\": \"<VERSION-ID>\",
            \"option_set\": \"set-a\",
            \"input_id\": \"<INPUT-ID>\",
            \"scenario_id\": \"<SCENARIO-ID>\",
            \"repetition\": 0
          }
        ],
        \"type\": \"scenario\",
        \"description\": \"An optional description\"
    }"
Copy

Please take into account the following considerations for the payload:

  • You don’t need to specify input_set_id or instance_ids.

  • The type must be scenario.

  • option_sets is a dictionary of objects that specify the different variations of options that will be applied to each run.

    "option_sets": {
      "option_set_id": {
        "string": "string"
      }
    }
    
    Copy

    An option set can be referenced in the runs list via its option_set_id.

  • runs is a list of objects that allows you to specify each run that the scenario test will create. In the example described in a previous section, we addressed how different combinations would result in 135 runs. For that same example, this list would contain 135 objects.

    This list is created automatically when using the Console. You must always define runs when creating a scenario test.

    "runs": [
      {
        "instance_id": "string",
        "version_id": "string",
        "option_set": "string",
        "input_id": "string",
        "scenario_id": "string",
        "repetition": 0
      }
    ]
    
    Copy

    The input_id of a run can be either a previous run_id or a managed input_id. A run can specify an option_set to be used for that run.

Experiment runs

Runs made for the experiment can be retreived with this endpoint.

GEThttps://api.cloud.nextmv.io/v1/applications/{application_id}/experiments/batch/{batch_id}/runs

List batch experiment runs.

List runs used for batch experiment specified by application and batch ID.

This will return the runs of the experiment in the following format:

{
  "runs": [
    {
      "id": "staging-dehnyogSg",
      "input_id": "staging-wZ8xcIQ4R",
      "created_at": "2023-08-21T21:09:41.48243386Z",
      "application_id": "pizza-delivery",
      "experiment_id": "pizza-delivery-region-compare",
      "application_instance_id": "nyc-pizza-region",
      "application_version_id": "nyc-v1.2.13",
      "status": "succeeded",
      "statistics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "result.value",
            "value": 2175064,
          },
          ...
        ]
      }
    },
    ...
  ]
}
Copy

Where runs is an array that contains the runs made for the experiment. Each run object includes the run metadata plus any summary statistics that were specified in the excecutable binaries used for the experiment (see table below).

Experiment runs must be retrieved independent of the experiment details. However, in Console this run history table can be viewed at the bottom of the experiment details. Each run history item can be clicked to view the details of the run. If the app is a routing app (using either the routing or nextroute template) the run details will also include a visualization of the results.

The run history data can also can be downloaded as a CSV file in Console. Click the Download CSV link in the upper right area of the experiment run history table to download the data as a CSV file.

Note that experiment runs are not shown in your app’s run history.

Delete a scenario test

Deleting a scenario test will also delete all of the associated information such as the udnerlying app runs.

This action is permanent and cannot be undone.

To delete a scenario test, you can use the following interfaces:

  • Console: use the web interface. Use the Delete button in the scenario test details.

  • Cloud API: use this HTTP endpoint.

    DELETEhttps://api.cloud.nextmv.io/v1/applications/{application_id}/experiments/batch/{batch_id}

    Delete a batch experiment.

    Delete a batch experiment specified by application and batch ID.

    curl -L -X DELETE \
        "https://api.cloud.nextmv.io/v1/applications/$APP_ID/experiments/batch/$EXPERIMENT_ID" \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer $NEXTMV_API_KEY"
    
    Copy

    You will not be prompted to confirm the deletion.

Page last updated

Go to on-page nav menu