Running batch experiments¶

Info

Learn the concepts and fundamentals of batch experiments in the Explanation page.

Batch experiments are used to analyze the output from one or more decision models on a fixed input set.

This how-to guide explains how to interact with batch experiments using the /v1/applications/{application_id}/experiments/batch endpoints. Go to the reference section to see all the available parameters for each endpoint.

Create a batch experiment¶

Use the POST /v1/applications/{application_id}/experiments/batch endpoint to create a new batch experiment for an application. Pass the runs field in the request payload as a list of run objects.

Each run object has the following fields:

input_id: ID of the input to use for this run (required). If a managed input is used, this should be the ID of the managed input. If input_set_id is provided for the run, this should be the ID of an input within that input set.
instance_id or version_id: Either an instance ID or version ID must be provided (at least one required).
option_set: ID of the option set to use (optional). Make sure to define the option sets using the option_sets field in the payload.
input_set_id: ID of the input set (optional).
scenario_id: Scenario ID if part of a scenario test (optional).
repetition: Repetition number (optional).

Consider the following example where runs are defined using an input set and a combination of instances and option sets.

The runs below use the details option set. Option sets define named collections of runtime options that can be referenced by runs, and are passed using the option_sets field in the payload.

Here is an example that uses the runs and option sets defined above to create a new batch experiment for an application. The endpoint requires an id in the request payload, so generate one yourself if you don't want to choose a custom one:

BashOutput

BATCH_ID="batch-$(openssl rand -hex 4)"

curl -s -X POST "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/batch" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  -H "Content-Type: application/json" \
  -d "{
    \"id\": \"${BATCH_ID}\",
    \"name\": \"${BATCH_ID}\",
    \"option_sets\": {
      \"details-on\": {\"details\": \"true\"},
      \"details-off\": {\"details\": \"false\"}
    },
    \"runs\": [
      {\"instance_id\": \"production\", \"input_set_id\": \"burrowing-hares\", \"input_id\": \"input.json-8MFSJpPDg\", \"option_set\": \"details-on\"},
      {\"instance_id\": \"production\", \"input_set_id\": \"burrowing-hares\", \"input_id\": \"input.json-8MFSJpPDg\", \"option_set\": \"details-off\"},
      {\"instance_id\": \"staging\", \"input_set_id\": \"burrowing-hares\", \"input_id\": \"input.json-8MFSJpPDg\", \"option_set\": \"details-on\"},
      {\"instance_id\": \"staging\", \"input_set_id\": \"burrowing-hares\", \"input_id\": \"input.json-8MFSJpPDg\", \"option_set\": \"details-off\"}
    ]
  }" \
  | jq '.'

{
  "id": "batch-a1c92fde"
}

The call above uses a randomly generated ID, and the same identifier for the batch experiment's name. The name of the experiment is used as a human-readable label. You can pass different values for the id and name fields in the payload to specify a custom ID and name for the batch experiment. For example:

BashOutput

curl -s -X POST "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/batch" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "fluffy-batch-experiment",
    "name": "Batch experiment for a fluffy bunny",
    "option_sets": {
      "details-on": {"details": "true"},
      "details-off": {"details": "false"}
    },
    "runs": [
      {"instance_id": "production", "input_set_id": "burrowing-hares", "input_id": "input.json-8MFSJpPDg", "option_set": "details-on"},
      {"instance_id": "production", "input_set_id": "burrowing-hares", "input_id": "input.json-8MFSJpPDg", "option_set": "details-off"},
      {"instance_id": "staging", "input_set_id": "burrowing-hares", "input_id": "input.json-8MFSJpPDg", "option_set": "details-on"},
      {"instance_id": "staging", "input_set_id": "burrowing-hares", "input_id": "input.json-8MFSJpPDg", "option_set": "details-off"}
    ]
  }' \
  | jq '.'

{
  "id": "fluffy-batch-experiment"
}

Get a batch experiment¶

Info

The best way to view and interact with batch experiment results is in the Nextmv Console.

Use the GET /v1/applications/{application_id}/experiments/batch/{batch_id}/metadata endpoint to retrieve the metadata for a batch experiment, using the batch experiment ID.

BashOutput

curl -s -X GET "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/batch/fluffy-batch-experiment/metadata" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  | jq '.'

{
  "id": "fluffy-batch-experiment",
  "name": "Batch experiment for a fluffy bunny",
  "description": "",
  "app_id": "uncanny-rodent",
  "created_at": "2026-07-29T14:31:42Z",
  "updated_at": "2026-07-29T14:31:49Z",
  "status": "completed",
  "number_of_requested_runs": 4,
  "number_of_runs": 4,
  "number_of_completed_runs": 4,
  "type": "batch"
}

Once the status of the batch experiment is completed, you can get the results using the GET /v1/applications/{application_id}/experiments/batch/{batch_id} endpoint. The response includes summary statistics for the experiment, but not the individual runs that were made for it. To also get those runs, fetch them separately using the GET /v1/applications/{application_id}/experiments/batch/{batch_id}/runs endpoint and merge them into the result. This endpoint is paginated, so the snippet below always uses pagination: it keeps requesting pages by passing the next_page_token value returned in each response as the pagetoken query parameter, until no token is returned. The runs are then merged into the batch experiment object under the runs key using jq.

BashOutput

BATCH=$(curl -s -X GET "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/batch/fluffy-batch-experiment" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}")

RUNS="[]"
PAGE_TOKEN=""

while :; do
  RESPONSE=$(curl -s -G "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/batch/fluffy-batch-experiment/runs" \
    -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
    --data-urlencode "pagetoken=${PAGE_TOKEN}")

  RUNS=$(jq -n --argjson existing "$RUNS" --argjson page "$(echo "$RESPONSE" | jq '.runs')" '$existing + $page')
  PAGE_TOKEN=$(echo "$RESPONSE" | jq -r '.next_page_token // empty')

  [ -z "$PAGE_TOKEN" ] && break
done

echo "$BATCH" | jq --argjson runs "$RUNS" '. + {runs: $runs}'

{
  "id": "fluffy-batch-experiment",
  "name": "Batch experiment for a fluffy bunny",
  "description": "",
  "status": "completed",
  "created_at": "2026-07-29T14:31:42.900441248Z",
  "updated_at": "2026-07-29T14:31:49.336048958Z",
  "input_set_id": "",
  "instance_ids": [
    "production",
    "staging"
  ],
  "option_sets": {
    "details-off": {
      "details": "false"
    },
    "details-on": {
      "details": "true"
    }
  },
  "number_of_requested_runs": 4,
  "number_of_runs": 4,
  "number_of_completed_runs": 4,
  "type": "batch",
  "grouped_distributional_summaries": [
    {
      "group_keys": [
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "production",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 2,
          "mean": 1.23,
          "std": 0,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 2
    },
    {
      "group_keys": [
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "staging",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 2,
          "mean": 1.23,
          "std": 0,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 2
    },
    {
      "group_keys": [
        "inputID",
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "input.json-8MFSJpPDg",
        "production",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 2,
          "mean": 1.23,
          "std": 0,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 2
    },
    {
      "group_keys": [
        "inputID",
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "input.json-8MFSJpPDg",
        "staging",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 2,
          "mean": 1.23,
          "std": 0,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 2
    },
    {
      "group_keys": [
        "inputID"
      ],
      "group_values": [
        "input.json-8MFSJpPDg"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 4,
          "mean": 1.23,
          "std": 0,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 4
    }
  ],
  "runs": [
    {
      "id": "staging-nieu9wPDR",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-29T14:31:42.917557484Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "staging",
      "application_version_id": "v0.0.2",
      "run_type": {
        "type": "",
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "runtime": "python-3_11",
      "status": "succeeded",
      "status_v2": "succeeded",
      "experiment_id": "fluffy-batch-experiment",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.373
          }
        ]
      },
      "input_id": "input.json-8MFSJpPDg",
      "option_set": "details-off",
      "options": {
        "details": "false"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "false",
          "source": "run"
        }
      ],
      "input_set_id": "burrowing-hares"
    },
    {
      "id": "staging-gmeu9QEDg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-29T14:31:42.912749273Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "staging",
      "application_version_id": "v0.0.2",
      "run_type": {
        "type": "",
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "runtime": "python-3_11",
      "status": "succeeded",
      "status_v2": "succeeded",
      "experiment_id": "fluffy-batch-experiment",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.737
          }
        ]
      },
      "input_id": "input.json-8MFSJpPDg",
      "option_set": "details-on",
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ],
      "input_set_id": "burrowing-hares"
    },
    {
      "id": "production-lz6X9QPvg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-29T14:31:42.906541984Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "type": "",
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "runtime": "python-3_11",
      "status": "succeeded",
      "status_v2": "succeeded",
      "experiment_id": "fluffy-batch-experiment",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.495
          }
        ]
      },
      "input_id": "input.json-8MFSJpPDg",
      "option_set": "details-off",
      "options": {
        "details": "false"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "false",
          "source": "run"
        }
      ],
      "input_set_id": "burrowing-hares"
    },
    {
      "id": "production-Bk6u9QEDg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-29T14:31:42.90061083Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "type": "",
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "runtime": "python-3_11",
      "status": "succeeded",
      "status_v2": "succeeded",
      "experiment_id": "fluffy-batch-experiment",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 5.481
          }
        ]
      },
      "input_id": "input.json-8MFSJpPDg",
      "option_set": "details-on",
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ],
      "input_set_id": "burrowing-hares"
    }
  ]
}

You can list all batch experiments in the application using the GET /v1/applications/{application_id}/experiments/batch endpoint. This endpoint is paginated, so the snippet below always uses pagination: it passes pagereturn=true to receive a next_page_token in the response, and keeps requesting pages by passing that token back as pagetoken until no token is returned. By default this endpoint also returns scenario tests, since they are built on the same underlying resource, so the type query parameter is used to only include batch experiments.

BashOutput

ITEMS="[]"
PAGE_TOKEN=""

while :; do
  RESPONSE=$(curl -s -G "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/batch" \
    -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
    --data-urlencode "type=batch" \
    --data-urlencode "pagereturn=true" \
    --data-urlencode "pagetoken=${PAGE_TOKEN}")

  ITEMS=$(jq -n --argjson existing "$ITEMS" --argjson page "$(echo "$RESPONSE" | jq '.items')" '$existing + $page')
  PAGE_TOKEN=$(echo "$RESPONSE" | jq -r '.next_page_token // empty')

  [ -z "$PAGE_TOKEN" ] && break
done

echo "$ITEMS" | jq '.'

[
  {
    "id": "fluffy-batch-experiment",
    "name": "Batch experiment for a fluffy bunny",
    "description": "",
    "status": "completed",
    "created_at": "2026-07-29T14:31:42.900441248Z",
    "updated_at": "2026-07-29T14:31:49.336048958Z",
    "input_set_id": "",
    "instance_ids": [
      "production",
      "staging"
    ],
    "option_sets": {
      "details-off": {
        "details": "false"
      },
      "details-on": {
        "details": "true"
      }
    },
    "number_of_requested_runs": 4,
    "number_of_runs": 4,
    "number_of_completed_runs": 4,
    "type": "batch"
  },
  {
    "id": "batch-a1c92fde",
    "name": "batch-a1c92fde",
    "description": "",
    "status": "completed",
    "created_at": "2026-07-29T14:31:34.240373897Z",
    "updated_at": "2026-07-29T14:31:40.89321508Z",
    "input_set_id": "",
    "instance_ids": [
      "production",
      "staging"
    ],
    "option_sets": {
      "details-off": {
        "details": "false"
      },
      "details-on": {
        "details": "true"
      }
    },
    "number_of_requested_runs": 4,
    "number_of_runs": 4,
    "number_of_completed_runs": 4,
    "type": "batch"
  },
  ...
]

Update a batch experiment¶

You can update attributes of a batch experiment with the PATCH /v1/applications/{application_id}/experiments/batch/{batch_id} endpoint, such as its:

Name
Description

You cannot update the ID of a batch experiment.

BashOutput

curl -s -X PATCH "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/batch/fluffy-batch-experiment" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"name": "Updated Batch Experiment Name", "description": "Updated description for the batch experiment"}' \
  | jq '.'

{
  "id": "fluffy-batch-experiment",
  "name": "Updated Batch Experiment Name",
  "description": "Updated description for the batch experiment",
  "created_at": "2026-07-29T14:31:42.900441248Z",
  "updated_at": "2026-07-29T14:32:23.926202598Z"
}

Delete a batch experiment¶

Warning

Deleting a batch experiment is irreversible. All the runs associated with the batch experiment will be permanently deleted.

Delete a batch experiment using the DELETE /v1/applications/{application_id}/experiments/batch/{batch_id} endpoint.

Bash

curl -s -X DELETE "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/batch/fluffy-batch-experiment" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}"