Batch experiments

Batch experiments are offline tests used to analyze the output from one or more decision models on a fixed input set. They are generally used as an exploratory test to understand the impacts to business metrics (or KPIs) when updating a model with a new feature, such as an additional constraint. They can also be used to validate that a model is ready for further testing and likely to make an intended business impact.

To be concise, batch experiments take the inputs from the input set and the specified instances and do a run for each input-instance pair. The results are then aggregated and analyzed. The decision model must produce an output that contains a statistics block that uses the statistics convention.

Batch experiments are designed to be visualized in the Console web interface. Go to the app, Experiments > Batch tab.

There are several interfaces for creating batch experiments:

• Console: use the web interface.
• Nextmv CLI: use your terminal.
• Python SDK: use Python.
• Cloud API: use the HTTP endpoints.

In addition to an input_set_id and instance_ids, a batch experiment can process runs and option_sets.

• runs. A list of objects.

  • JSON

  "runs": [
    {
      "instance_id": "string",
      "version_id": "string",
      "option_set": "string",
      "input_id": "string"
    }
  ]
  

  Copy

  Without runs, the batch experiment will run all of the inputs in the input set on all of the instances. With runs, the batch experiment will only run the specified inputs on the specified instances. The runs list can also be used to run the same input multiple times on specific instances or versions.

  A run can specify an option_set to be used for that run.

• option_sets. A dictionary of objects.

  • JSON

  "option_sets": {
    "option_set_id": {
      "string": "string"
    }
  }
  

  Copy

  option_sets is a dictionary of options that will be used for a run. An option set can be referenced in the runs list via its option_set_id. This can be useful for example if:

  • You want to compare the results of different solvers that have conflicting options. E.g. one uses solve.duration for the maximum runtime while the other uses duration.
  • You want to run specific inputs with different options within the same experiment.

Console

Go to the Console web interface, and open your app. Go to the Experiments > Batch tab. Click on New Experiment. Fill in the fields.

Nextmv CLI

Define the desired batch experiment ID and name. After, create the batch experiment.

nextmv experiment batch start \
    --app-id $APP_ID \
    --experiment-id $EXPERIMENT_ID \
    --name $EXPERIMENT_ID \
    --input-set-id $INPUT_SET_ID \
    --instance-ids "latest,latest-2" \
    --description "An optional description" \
    --confirm

Alternatively, you can use a JSON file that contains all of the information needed to start a batch experiment. The file must be valid JSON and equal to the payload of the POST request when using the Cloud API.

Using the definitions file you can fine tune the batch experiment by specifying runs and option_sets, for example.

Consider this example using a JSON file named batch-experiment.json.

{
  "id": "experiment-1",
  "name": "experiment-1",
  "description": "An optional description",
  "input_set_id": "input-set-1",
  "option_sets": {
    "long_running": {
      "solve.duration": "10s"
    },
    "short_running": {
      "solve.duration": "5s"
    }
  },
  "runs": [
    {
      "instance_id": "latest",
      "input_id": "latest-RNBs7AKSg",
      "option_set": "long_running"
    },
    {
      "instance_id": "latest-2",
      "input_id": "latest-RNBs7AKSg",
      "option_set": "short_running"
    },
    {
      "instance_id": "latest",
      "input_id": "latest-fK_wZ0FSg",
      "option_set": "long_running"
    },
    {
      "instance_id": "latest-2",
      "input_id": "latest-fK_wZ0FSg",
      "option_set": "short_running"
    }
  ]
}

Python SDK

Define the desired batch experiment ID and name. After, create the batch experiment.

import os

from nextmv.cloud import Application, Client

client = Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = Application(client=client, id=os.getenv("APP_ID"))
batch_experiment_id = app.new_batch_experiment(
    id=os.getenv("BATCH_EXPERIMENT_ID"),
    name=os.getenv("BATCH_EXPERIMENT_ID"),
    input_set_id=os.getenv("INPUT_SET_ID"),
    instance_ids=["latest", "latest-2"],
    description="An optional description",
)
print(batch_experiment_id)

Defining runs and option_sets is supported as well.

Cloud API

Define the desired batch experiment ID and name. After, create the batch experiment.

curl -sS -L -X POST \
    "https://api.cloud.nextmv.io/v1/applications/$APP_ID/experiments/batch" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $NEXTMV_API_KEY" \
    -d "{
      \"id\": \"$EXPERIMENT_ID\",
      \"name\": \"YOUR_EXPERIMENT_NAME\",
      \"input_set_id\": \"$INPUT_SET_ID\",
      \"instance_ids\": [\"$INSTANCE_ID\", \"$INSTANCE_ID\"],
      \"description\": \"An optional description\"
    }" | jq

Result of a batch experiment - Console

Nextmv Console displays the results grouped by indicator keys and for each one includes:

• a summary table
• the percentile values
• a box plot chart displaying summary values from the instances compared

Result of a batch experiment - CLI

The results can also be retrieved with the Nextmv CLI, which returns the raw data in JSON format and is useful if you would like to perform your own operations on the experiment results.

nextmv experiment batch result \
    --app-id $APP_ID \
    --experiment-id $EXPERIMENT_ID

The result of a batch experiment is returned as JSON. If you’re viewing the result in Console, the JSON is parsed and displayed in a web view; Nextmv CLI returns the complete JSON response. The response contains the batch experiment metadata and the results in the form of grouped distributional summaries.

Top-level properties

The table below summarizes the top-level properties in the return.

Grouped distributional summaries

There are three types of summaries included in the grouped_distributional_summaries array (there could be more in the future):

1. Version (instance)
2. Version (instance) + input
3. Input

No matter the type, each grouped distributional summary includes the following:

Indicator distributions

Each object property value in the indicator_distributions contains the values in the table below. Note that for some runs, certain values may be missing (a custom statistic for example). If you’re viewing the results in Console and a grouped distributional summary is missing values, a warning message will appear. If you are analyzing the results from the returned JSON, you must handle this check in your own systems.

Experiment runs

Runs made for the experiment can be retreived with this endpoint.

This will return the runs of the experiment in the following format:

{
  "runs": [
    {
      "id": "staging-dehnyogSg",
      "input_id": "staging-wZ8xcIQ4R",
      "created_at": "2023-08-21T21:09:41.48243386Z",
      "application_id": "pizza-delivery",
      "experiment_id": "pizza-delivery-region-compare",
      "application_instance_id": "nyc-pizza-region",
      "application_version_id": "nyc-v1.2.13",
      "status": "succeeded",
      "statistics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "result.value",
            "value": 2175064,
          },
          ...
        ]
      }
    },
    ...
  ]
}

Where runs is an array that contains the runs made for the experiment. Each run object includes the run metadata plus any summary statistics that were specified in the excecutable binaries used for the experiment (see table below).

Experiment runs must be retrieved independent of the batch experiment details. However, in Console this run history table can be viewed at the bottom of the experiment details. Each run history item can be clicked to view the details of the run. If the app is a routing app (using either the routing or nextroute template) the run details will also include a visualization of the results.

The run history data can also can be downloaded as a CSV file in Console. Click the Download CSV link in the upper right area of the experiment run history table to download the data as a CSV file.

Experiment run summary data

Run history objects can include the items specified in the table below.