Skip to content

Running batch experiments

Info

Learn the concepts and fundamentals of batch experiments in the Explanation page.

Batch experiments are used to analyze the output from one or more decision models on a fixed input set.

Included in this how-to guide are instructions on how to interact with the actions under the nextmv cloud batch command tree. Go the reference section or use the --help option to see all the available options for each command.

Create a batch experiment

Use the nextmv cloud batch create command to create a new batch experiment for an application. You can specify multiple runs to use for the batch experiment with the --runs flag in two ways:

  • Using the --runs flag multiple times, or
  • Providing an array of objects in valid JSON format.

Each run is defined as a JSON object with the following attributes:

  • input_id: ID of the input to use for this run (required). If a managed input is used, this should be the ID of the managed input. If input_set_id is provided for the run, this should be the ID of an input within that input set.
  • instance_id or version_id: Either an instance ID or version ID must be provided (at least one required).
  • option_set: ID of the option set to use (optional). Make sure to define the option sets using the --option-sets flag.
  • input_set_id: ID of the input set (optional).
  • scenario_id: Scenario ID if part of a scenario test (optional).
  • repetition: Repetition number (optional).

Consider the following example where runs are defined using an input set. and a combination of instances and option sets.

[
  {
    "instance_id": "production",
    "input_set_id": "burrowing-hares",
    "input_id": "input.json-8MFSJpPDg",
    "option_set": "details-on"
  },
  {
    "instance_id": "production",
    "input_set_id": "burrowing-hares",
    "input_id": "input.json-8MFSJpPDg",
    "option_set": "details-off"
  },
  {
    "instance_id": "staging",
    "input_set_id": "burrowing-hares",
    "input_id": "input.json-8MFSJpPDg",
    "option_set": "details-on"
  },
  {
    "instance_id": "staging",
    "input_set_id": "burrowing-hares",
    "input_id": "input.json-8MFSJpPDg",
    "option_set": "details-off"
  }
]

The above runs use the details option set. Option sets define named collections of runtime options that can be referenced by runs. Let's also define the option sets to use for the batch experiment.

{
  "details-on": {"details": "true"},
  "details-off": {"details": "false"}
}

Here is an example command that uses the runs and option sets defined above to create a new batch experiment for an application.

RUNS='[
  {
    "instance_id": "production",
    "input_set_id": "burrowing-hares",
    "input_id": "input.json-8MFSJpPDg",
    "option_set": "details-on"
  },
  {
    "instance_id": "production",
    "input_set_id": "burrowing-hares",
    "input_id": "input.json-8MFSJpPDg",
    "option_set": "details-off"
  },
  {
    "instance_id": "staging",
    "input_set_id": "burrowing-hares",
    "input_id": "input.json-8MFSJpPDg",
    "option_set": "details-on"
  },
  {
    "instance_id": "staging",
    "input_set_id": "burrowing-hares",
    "input_id": "input.json-8MFSJpPDg",
    "option_set": "details-off"
  }
]'
OPTION_SETS='{
  "details-on": {"details": "true"},
  "details-off": {"details": "false"}
}'
nextmv cloud batch create \
    --app-id uncanny-rodent \
    --runs "$RUNS" \
    --option-sets "$OPTION_SETS"

{
  "batch_experiment_id": "batch-ruxbf384"
}

The command above will create a random ID, and use the same identifier for batch experiment's name. The name of the experiment is used as a human-readable You can use the --batch-experiment-id and/or --name options to specify a custom ID and name for the batch experiment. For example:

nextmv cloud batch create \
    --app-id uncanny-rodent \
    --runs "$RUNS" \
    --option-sets "$OPTION_SETS" \
    --batch-experiment-id fluffy-batch-experiment \
    --name "Batch experiment for a fluffy bunny"

{
  "batch_experiment_id": "fluffy-batch-experiment"
}

Get a batch experiment

Info

The best way to view and interact with batch experiment results is in the Nextmv Console.

Use the nextmv cloud batch metadata command to retrieve the metadata for a batch experiment, using the batch experiment ID.

nextmv cloud batch metadata --app-id uncanny-rodent --batch-experiment-id fluffy-batch-experiment

⏳ Getting batch experiment metadata...
{
  "id": "fluffy-batch-experiment",
  "name": "Batch experiment for a fluffy bunny",
  "created_at": "2026-07-25T18:18:13Z",
  "updated_at": "2026-07-25T18:18:20Z",
  "status": "completed",
  "description": "",
  "number_of_requested_runs": 4,
  "number_of_runs": 4,
  "number_of_completed_runs": 4,
  "type": "batch",
  "app_id": "uncanny-rodent"
}

Once the status of the batch experiment is completed, you can get the results using the nextmv cloud batch get command. The output includes the runs that were made for the batch experiment.

nextmv cloud batch get --app-id uncanny-rodent --batch-experiment-id fluffy-batch-experiment

⏳ Getting batch experiment...
{
  "id": "fluffy-batch-experiment",
  "name": "Batch experiment for a fluffy bunny",
  "created_at": "2026-07-25T18:18:13.917572Z",
  "updated_at": "2026-07-25T18:18:20.596460Z",
  "status": "completed",
  "description": "",
  "number_of_requested_runs": 4,
  "number_of_runs": 4,
  "number_of_completed_runs": 4,
  "type": "batch",
  "option_sets": {
    "details-off": {
      "details": "false"
    },
    "details-on": {
      "details": "true"
    }
  },
  "input_set_id": "",
  "instance_ids": [
    "production",
    "staging"
  ],
  "grouped_distributional_summaries": [
    {
      "group_keys": [
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "production",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 2,
          "mean": 1.23,
          "std": 0,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 2
    },
    {
      "group_keys": [
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "staging",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 2,
          "mean": 1.23,
          "std": 0,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 2
    },
    {
      "group_keys": [
        "inputID",
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "input.json-8MFSJpPDg",
        "staging",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 2,
          "mean": 1.23,
          "std": 0,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 2
    },
    {
      "group_keys": [
        "inputID",
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "input.json-8MFSJpPDg",
        "production",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 2,
          "mean": 1.23,
          "std": 0,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 2
    },
    {
      "group_keys": [
        "inputID"
      ],
      "group_values": [
        "input.json-8MFSJpPDg"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 4,
          "mean": 1.23,
          "std": 0,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 4
    }
  ],
  "runs": [
    {
      "id": "staging-aJXIgAPvR",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-25T18:18:13.939759Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "staging",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-batch-experiment",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.459
          }
        ]
      },
      "input_id": "input.json-8MFSJpPDg",
      "option_set": "details-off",
      "options": {
        "details": "false"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "false",
          "source": "run"
        }
      ],
      "input_set_id": "burrowing-hares"
    },
    {
      "id": "staging-p1uIRAPDg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-25T18:18:13.933295Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "staging",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-batch-experiment",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.447
          }
        ]
      },
      "input_id": "input.json-8MFSJpPDg",
      "option_set": "details-on",
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ],
      "input_set_id": "burrowing-hares"
    },
    {
      "id": "production-41XSg0PDR",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-25T18:18:13.926751Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-batch-experiment",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.428
          }
        ]
      },
      "input_id": "input.json-8MFSJpPDg",
      "option_set": "details-off",
      "options": {
        "details": "false"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "false",
          "source": "run"
        }
      ],
      "input_set_id": "burrowing-hares"
    },
    {
      "id": "production-jAXSgAPvg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-25T18:18:13.917815Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-batch-experiment",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 5.751
          }
        ]
      },
      "input_id": "input.json-8MFSJpPDg",
      "option_set": "details-on",
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ],
      "input_set_id": "burrowing-hares"
    }
  ]
}

You can list all batch experiments in the application using the nextmv cloud batch list command.

nextmv cloud batch list --app-id uncanny-rodent

⏳ Listing batch experiments...
[
  {
    "id": "fluffy-batch-experiment",
    "name": "Batch experiment for a fluffy bunny",
    "created_at": "2026-07-25T18:18:13.917572Z",
    "updated_at": "2026-07-25T18:18:20.596460Z",
    "status": "completed",
    "description": "",
    "number_of_requested_runs": 4,
    "number_of_runs": 4,
    "number_of_completed_runs": 4,
    "type": "batch",
    "option_sets": {
      "details-off": {
        "details": "false"
      },
      "details-on": {
        "details": "true"
      }
    }
  },
  {
    "id": "batch-ruxbf384",
    "name": "batch-ruxbf384",
    "created_at": "2026-07-25T18:16:17.228801Z",
    "updated_at": "2026-07-25T18:16:26.532588Z",
    "status": "completed",
    "description": "",
    "number_of_requested_runs": 4,
    "number_of_runs": 4,
    "number_of_completed_runs": 4,
    "type": "batch",
    "option_sets": {
      "details-off": {
        "details": "false"
      },
      "details-on": {
        "details": "true"
      }
    }
  }
]

Update a batch experiment

You can update attributes of a batch experiment with the nextmv cloud batch update command, such as its:

  • Name
  • Description

Please use the --help option on the nextmv cloud batch update command to see all the available options for updating a batch experiment.

You cannot update the ID of a batch experiment.

nextmv cloud batch update \
    --app-id uncanny-rodent \
    --batch-experiment-id fluffy-batch-experiment \
    --name "Updated Batch Experiment Name" \
    --description "Updated description for the batch experiment"

⏳ Updating batch experiment...
✅ Batch experiment fluffy-batch-experiment updated successfully in application uncanny-rodent.
{
  "id": "fluffy-batch-experiment",
  "name": "Updated Batch Experiment Name",
  "created_at": "2026-07-25T18:18:13.917572Z",
  "updated_at": "2026-07-25T18:22:27.306580Z",
  "description": "Updated description for the batch experiment"
}

Delete a batch experiment

Warning

Deleting a batch experiment is irreversible. All the runs associated with the batch experiment will be permanently deleted.

Delete a batch experiment using the nextmv cloud batch delete command. The CLI will prompt you to confirm deletion but you can override it with the --yes option.

nextmv cloud batch delete --app-id uncanny-rodent --batch-experiment-id fluffy-batch-experiment

Are you sure you want to delete batch experiment fluffy-batch-experiment from application uncanny-rodent? This action cannot be undone.
💡 Confirm Yes
✅ Batch experiment fluffy-batch-experiment deleted successfully from application uncanny-rodent.