Skip to content

Running batch experiments

Info

Learn the concepts and fundamentals of batch experiments in the Explanation page.

Batch experiments are used to analyze the output from one or more decision models on a fixed input set.

Included in this how-to guide are instructions on how to interact with batch experiments using methods on the Application class. Go the reference section to see all the available parameters for each method.

Create a batch experiment

Use the Application.new_batch_experiment method (or Application.new_batch_experiment_with_result, which additionally polls for the result) to create a new batch experiment for an application. Pass the runs keyword argument as a list of BatchExperimentRun objects.

Each run has the following attributes:

  • input_id: ID of the input to use for this run (required). If a managed input is used, this should be the ID of the managed input. If input_set_id is provided for the run, this should be the ID of an input within that input set.
  • instance_id or version_id: Either an instance ID or version ID must be provided (at least one required).
  • option_set: ID of the option set to use (optional). Make sure to define the option sets using the option_sets keyword argument.
  • input_set_id: ID of the input set (optional).
  • scenario_id: Scenario ID if part of a scenario test (optional).
  • repetition: Repetition number (optional).

Consider the following example where runs are defined using an input set and a combination of instances and option sets.

from nextmv.cloud import BatchExperimentRun

runs = [
    BatchExperimentRun(
        instance_id="production",
        input_set_id="burrowing-hares",
        input_id="input.json-8MFSJpPDg",
        option_set="details-on",
    ),
    BatchExperimentRun(
        instance_id="production",
        input_set_id="burrowing-hares",
        input_id="input.json-8MFSJpPDg",
        option_set="details-off",
    ),
    BatchExperimentRun(
        instance_id="staging",
        input_set_id="burrowing-hares",
        input_id="input.json-8MFSJpPDg",
        option_set="details-on",
    ),
    BatchExperimentRun(
        instance_id="staging",
        input_set_id="burrowing-hares",
        input_id="input.json-8MFSJpPDg",
        option_set="details-off",
    ),
]

The runs above use the details option set. Option sets define named collections of runtime options that can be referenced by runs. Let's also define the option sets to use for the batch experiment, passed as the option_sets keyword argument.

option_sets = {
    "details-on": {"details": "true"},
    "details-off": {"details": "false"},
}

Here is an example that uses the runs and option sets defined above to create a new batch experiment for an application.

import os

import nextmv
from nextmv import cloud
from nextmv.cloud import BatchExperimentRun

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
runs = [
    BatchExperimentRun(
        instance_id="production",
        input_set_id="burrowing-hares",
        input_id="input.json-8MFSJpPDg",
        option_set="details-on",
    ),
    BatchExperimentRun(
        instance_id="production",
        input_set_id="burrowing-hares",
        input_id="input.json-8MFSJpPDg",
        option_set="details-off",
    ),
    BatchExperimentRun(
        instance_id="staging",
        input_set_id="burrowing-hares",
        input_id="input.json-8MFSJpPDg",
        option_set="details-on",
    ),
    BatchExperimentRun(
        instance_id="staging",
        input_set_id="burrowing-hares",
        input_id="input.json-8MFSJpPDg",
        option_set="details-off",
    ),
]
option_sets = {
    "details-on": {"details": "true"},
    "details-off": {"details": "false"},
}
batch_experiment_id = app.new_batch_experiment(runs=runs, option_sets=option_sets)

nextmv.write({"batch_experiment_id": batch_experiment_id})

uv run main.py

{
  "batch_experiment_id": "batch-wli4otht"
}

The call above will create a random ID, and use the same identifier for the batch experiment's name. The name of the experiment is used as a human-readable label. You can use the id and/or name keyword arguments to specify a custom ID and name for the batch experiment. For example:

import os

import nextmv
from nextmv import cloud
from nextmv.cloud import BatchExperimentRun

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
runs = [
    BatchExperimentRun(
        instance_id="production",
        input_set_id="burrowing-hares",
        input_id="input.json-8MFSJpPDg",
        option_set="details-on",
    ),
    BatchExperimentRun(
        instance_id="production",
        input_set_id="burrowing-hares",
        input_id="input.json-8MFSJpPDg",
        option_set="details-off",
    ),
    BatchExperimentRun(
        instance_id="staging",
        input_set_id="burrowing-hares",
        input_id="input.json-8MFSJpPDg",
        option_set="details-on",
    ),
    BatchExperimentRun(
        instance_id="staging",
        input_set_id="burrowing-hares",
        input_id="input.json-8MFSJpPDg",
        option_set="details-off",
    ),
]
option_sets = {
    "details-on": {"details": "true"},
    "details-off": {"details": "false"},
}
batch_experiment_id = app.new_batch_experiment(
    runs=runs,
    option_sets=option_sets,
    id="fluffy-batch-experiment",
    name="Batch experiment for a fluffy bunny",
)

nextmv.write({"batch_experiment_id": batch_experiment_id})

uv run main.py

{
  "batch_experiment_id": "fluffy-batch-experiment"
}

Get a batch experiment

Info

The best way to view and interact with batch experiment results is in the Nextmv Console.

Use the Application.batch_experiment_metadata method to retrieve the metadata for a batch experiment, using the batch experiment ID. The method returns a BatchExperimentMetadata object.

import os

import nextmv
from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
metadata = app.batch_experiment_metadata(batch_id="fluffy-batch-experiment")

nextmv.write(metadata.to_dict())

uv run main.py

{
  "id": "fluffy-batch-experiment",
  "name": "Batch experiment for a fluffy bunny",
  "created_at": "2026-07-28T20:50:14Z",
  "updated_at": "2026-07-28T20:50:22Z",
  "status": "completed",
  "description": "",
  "number_of_requested_runs": 4,
  "number_of_runs": 4,
  "number_of_completed_runs": 4,
  "type": "batch",
  "app_id": "uncanny-rodent"
}

Once the status of the batch experiment is completed, you can get the results using the Application.batch_experiment method (or Application.batch_experiment_with_polling, which polls until the experiment finishes). The method returns a BatchExperiment object, whose output includes the runs that were made for the batch experiment.

import os

import nextmv
from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
batch_experiment = app.batch_experiment(batch_id="fluffy-batch-experiment")

nextmv.write(batch_experiment.to_dict())

uv run main.py

{
  "id": "fluffy-batch-experiment",
  "name": "Batch experiment for a fluffy bunny",
  "created_at": "2026-07-28T20:50:14.793634Z",
  "updated_at": "2026-07-28T20:50:22.029489Z",
  "status": "completed",
  "description": "",
  "number_of_requested_runs": 4,
  "number_of_runs": 4,
  "number_of_completed_runs": 4,
  "type": "batch",
  "option_sets": {
    "details-off": {
      "details": "false"
    },
    "details-on": {
      "details": "true"
    }
  },
  "input_set_id": "",
  "instance_ids": [
    "production",
    "staging"
  ],
  "grouped_distributional_summaries": [
    {
      "group_keys": [
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "production",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 2,
          "mean": 1.23,
          "std": 0,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 2
    },
    {
      "group_keys": [
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "staging",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 2,
          "mean": 1.23,
          "std": 0,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 2
    },
    {
      "group_keys": [
        "inputID",
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "input.json-8MFSJpPDg",
        "staging",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 2,
          "mean": 1.23,
          "std": 0,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 2
    },
    {
      "group_keys": [
        "inputID",
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "input.json-8MFSJpPDg",
        "production",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 2,
          "mean": 1.23,
          "std": 0,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 2
    },
    {
      "group_keys": [
        "inputID"
      ],
      "group_values": [
        "input.json-8MFSJpPDg"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 4,
          "mean": 1.23,
          "std": 0,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 4
    }
  ],
  "runs": [
    {
      "id": "staging-wfFGg8PDR",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-28T20:50:14.809034Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "staging",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-batch-experiment",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 3.755
          }
        ]
      },
      "input_id": "input.json-8MFSJpPDg",
      "option_set": "details-off",
      "options": {
        "details": "false"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "false",
          "source": "run"
        }
      ],
      "input_set_id": "burrowing-hares"
    },
    {
      "id": "staging-BfFMg8PDR",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-28T20:50:14.804210Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "staging",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-batch-experiment",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 5.573
          }
        ]
      },
      "input_id": "input.json-8MFSJpPDg",
      "option_set": "details-on",
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ],
      "input_set_id": "burrowing-hares"
    },
    {
      "id": "production-hfKGR8EDg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-28T20:50:14.798866Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-batch-experiment",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.676
          }
        ]
      },
      "input_id": "input.json-8MFSJpPDg",
      "option_set": "details-off",
      "options": {
        "details": "false"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "false",
          "source": "run"
        }
      ],
      "input_set_id": "burrowing-hares"
    },
    {
      "id": "production-vfFGR8EvR",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-28T20:50:14.793801Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "experiment_id": "fluffy-batch-experiment",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 5.537
          }
        ]
      },
      "input_id": "input.json-8MFSJpPDg",
      "option_set": "details-on",
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ],
      "input_set_id": "burrowing-hares"
    }
  ]
}

You can list all batch experiments in the application using the Application.list_batch_experiments method.

import json
import os

from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
batch_experiments = app.list_batch_experiments()

print(json.dumps([batch_experiment.to_dict() for batch_experiment in batch_experiments[:2]], indent=2))

uv run main.py

[
  {
    "id": "fluffy-batch-experiment",
    "name": "Batch experiment for a fluffy bunny",
    "created_at": "2026-07-28T20:50:14.793634Z",
    "updated_at": "2026-07-28T20:50:22.029489Z",
    "status": "completed",
    "description": "",
    "number_of_requested_runs": 4,
    "number_of_runs": 4,
    "number_of_completed_runs": 4,
    "type": "batch",
    "option_sets": {
      "details-off": {
        "details": "false"
      },
      "details-on": {
        "details": "true"
      }
    }
  },
  {
    "id": "batch-wli4otht",
    "name": "batch-wli4otht",
    "created_at": "2026-07-28T20:50:06.431585Z",
    "updated_at": "2026-07-28T20:50:16.067338Z",
    "status": "completed",
    "description": "",
    "number_of_requested_runs": 4,
    "number_of_runs": 4,
    "number_of_completed_runs": 4,
    "type": "batch",
    "option_sets": {
      "details-off": {
        "details": "false"
      },
      "details-on": {
        "details": "true"
      }
    }
  }
]

Update a batch experiment

You can update attributes of a batch experiment with the Application.update_batch_experiment method, such as its:

  • Name
  • Description

The method returns a BatchExperimentInformation object. You cannot update the ID of a batch experiment.

import os

import nextmv
from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
info = app.update_batch_experiment(
    batch_experiment_id="fluffy-batch-experiment",
    name="Updated Batch Experiment Name",
    description="Updated description for the batch experiment",
)

nextmv.write(info.to_dict())

uv run main.py

{
  "id": "fluffy-batch-experiment",
  "name": "Updated Batch Experiment Name",
  "created_at": "2026-07-28T20:50:14.793634Z",
  "updated_at": "2026-07-28T20:51:05.657885Z",
  "description": "Updated description for the batch experiment"
}

Delete a batch experiment

Warning

Deleting a batch experiment is irreversible. All the runs associated with the batch experiment will be permanently deleted.

Delete a batch experiment using the Application.delete_batch_experiment method.

import os

from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
app.delete_batch_experiment(batch_id="fluffy-batch-experiment")

uv run main.py