Skip to content

Using runs ensembling

Info

Learn the concepts and fundamentals of runs ensembling in the Explanation page.

Run ensembling is a technique where, from a single input, multiple runs are generated in parallel. Once all the runs succeed, the best run is selected based on some criteria.

This how-to guide explains how to interact with ensemble definitions using the /v1/applications/{application_id}/ensembles endpoints. Go to the reference section to see all the available parameters for each endpoint.

Create an ensemble definition

Use the POST /v1/applications/{application_id}/ensembles endpoint to create a new ensemble definition for an application. Ensembles require rules and run_groups in the request payload, both passed as arrays of objects.

Each rule is an object with the following attributes:

  • id: Unique identifier for the rule (required).
  • statistics_path: JSONPath to the metric (e.g., $.result.value) (required).
  • objective: The evaluation objective (required). Allowed values: maximize and minimize.
  • tolerance: An object with the following fields (required):
    • value: Tolerance value (float).
    • type: Allowed values: absolute and relative.
  • index: Evaluation order - lower indices evaluated first (required).

Here is an example:

{
  "id": "rule1",
  "statistics_path": "$.value",
  "objective": "minimize",
  "tolerance": {
    "value": 0.1,
    "type": "relative"
  },
  "index": 0
}

Similarly to rules, each run group is an object with the following attributes:

  • id: Unique identifier for the run group (required).
  • instance_id: The instance to execute runs on (required).
  • options: Runtime options/parameters (optional). Options should be provided as an object with string key-value pairs.
  • repetitions: Number of times to repeat the run (optional).

Consider the following example:

{
  "id": "group1",
  "instance_id": "latest",
  "options": {
    "details": "true"
  },
  "repetitions": 3
}

Here is an example that uses the rule and group above to create an ensemble for the application. The endpoint requires an id in the request payload, so generate one yourself if you don't want to choose a custom one:

ENSEMBLE_ID="ensemble-$(openssl rand -hex 4)"

curl -s -X POST "https://api.cloud.nextmv.io/v1/applications/wandering-otter/ensembles" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  -H "Content-Type: application/json" \
  -d "{
    \"id\": \"${ENSEMBLE_ID}\",
    \"name\": \"${ENSEMBLE_ID}\",
    \"description\": \"${ENSEMBLE_ID}\",
    \"run_groups\": [
      {\"id\": \"group1\", \"instance_id\": \"latest\", \"options\": {\"details\": \"true\"}, \"repetitions\": 3}
    ],
    \"rules\": [
      {
        \"id\": \"rule1\",
        \"statistics_path\": \"\$.value\",
        \"objective\": \"minimize\",
        \"tolerance\": {\"value\": 0.1, \"type\": \"relative\"},
        \"index\": 0
      }
    ]
  }" \
  | jq '.'

{
  "id": "ensemble-3610ce69",
  "application_id": "wandering-otter",
  "name": "ensemble-3610ce69",
  "description": "ensemble-3610ce69",
  "run_groups": [
    {
      "id": "group1",
      "instance_id": "latest",
      "options": {
        "details": "true"
      },
      "repetitions": 3
    }
  ],
  "rules": [
    {
      "id": "rule1",
      "statistics_path": "$.value",
      "tolerance": {
        "value": 0.1,
        "type": "relative"
      },
      "objective": "minimize",
      "index": 0
    }
  ],
  "created_at": "2026-07-29T14:33:19.203592619Z",
  "updated_at": "2026-07-29T14:33:19.203592619Z"
}

The call above uses a randomly generated ID, and the same identifier for the ensemble's name. The name of the ensemble is used as a human-readable label. You can pass different values for the id and name fields in the payload to specify a custom ID and name for the ensemble. For example:

curl -s -X POST "https://api.cloud.nextmv.io/v1/applications/wandering-otter/ensembles" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "jumping-hare",
    "name": "The jumping hare ensemble",
    "description": "The jumping hare ensemble",
    "run_groups": [
      {"id": "group1", "instance_id": "latest", "options": {"details": "true"}, "repetitions": 3}
    ],
    "rules": [
      {
        "id": "rule1",
        "statistics_path": "$.value",
        "objective": "minimize",
        "tolerance": {"value": 0.1, "type": "relative"},
        "index": 0
      }
    ]
  }' \
  | jq '.'

{
  "id": "jumping-hare",
  "application_id": "wandering-otter",
  "name": "The jumping hare ensemble",
  "description": "The jumping hare ensemble",
  "run_groups": [
    {
      "id": "group1",
      "instance_id": "latest",
      "options": {
        "details": "true"
      },
      "repetitions": 3
    }
  ],
  "rules": [
    {
      "id": "rule1",
      "statistics_path": "$.value",
      "tolerance": {
        "value": 0.1,
        "type": "relative"
      },
      "objective": "minimize",
      "index": 0
    }
  ],
  "created_at": "2026-07-29T14:33:23.42390815Z",
  "updated_at": "2026-07-29T14:33:23.42390815Z"
}

Run with an ensemble

Once an ensemble has been created you can run it by using the POST /v1/applications/{application_id}/runs endpoint. Set the configuration.run_type.type field of the request payload to ensemble, and configuration.run_type.definition_id to the ensemble definition ID (see the ExecutionConfigurationRunType schema). Because the run groups of the ensemble definition already specify the instance to run on, the instance_id query parameter is omitted from this request. For example:

curl -s -X POST "https://api.cloud.nextmv.io/v1/applications/wandering-otter/runs" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "input": {"name": "world", "radius": 6378, "distance": 147.6},
    "configuration": {
      "run_type": {"type": "ensemble", "definition_id": "jumping-hare"}
    }
  }' \
  | jq '.'

{
  "run_id": "latest-hR56rwEvR"
}

Retrieve the run information using the run_id with the GET /v1/applications/{application_id}/runs/{run_id}/metadata endpoint. The result includes important metadata, such as the status.

curl -s -X GET "https://api.cloud.nextmv.io/v1/applications/wandering-otter/runs/latest-hR56rwEvR/metadata" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  | jq '.'

{
  "id": "latest-hR56rwEvR",
  "user_email": "sebastian@nextmv.io",
  "name": "",
  "description": "",
  "metadata": {
    "status": "succeeded",
    "status_v2": "succeeded",
    "created_at": "2026-07-29T14:33:34Z",
    "duration": 6189,
    "input_size": 52,
    "output_size": 25099,
    "error": "",
    "application_id": "wandering-otter",
    "application_instance_id": "latest",
    "application_version_id": "",
    "execution_class": "",
    "runtime": "",
    "run_type": {
      "type": "ensemble",
      "definition_id": "jumping-hare",
      "reference_id": ""
    },
    "format": {
      "input": {
        "type": "json"
      },
      "output": {
        "type": "json"
      }
    },
    "metrics": {
      "message": "Hello, world",
      "value": 1.23
    }
  }
}

Use the .metadata.status_v2 field to determine the status of the run. Please read our documentation on run polling to learn how to wait for a run to finish before retrieving its results.

Once the run completes, you can retrieve the results using the run_id with the GET /v1/applications/{application_id}/runs/{run_id} endpoint. The response includes the output of the run, along with the run information.

curl -s -X GET "https://api.cloud.nextmv.io/v1/applications/wandering-otter/runs/latest-hR56rwEvR" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  | jq '.'

{
  "id": "latest-hR56rwEvR",
  "user_email": "sebastian@nextmv.io",
  "name": "",
  "description": "",
  "metadata": {
    "status": "succeeded",
    "status_v2": "succeeded",
    "created_at": "2026-07-29T14:33:34Z",
    "duration": 6189,
    "input_size": 52,
    "output_size": 25099,
    "error": "",
    "application_id": "wandering-otter",
    "application_instance_id": "latest",
    "application_version_id": "",
    "execution_class": "",
    "runtime": "",
    "run_type": {
      "type": "ensemble",
      "definition_id": "jumping-hare",
      "reference_id": ""
    },
    "format": {
      "input": {
        "type": "json"
      },
      "output": {
        "type": "json"
      }
    },
    "metrics": {
      "message": "Hello, world",
      "value": 1.23
    }
  },
  "output": {
    "options": {
      "details": true
    },
    "solution": {
      "message": "Hello, world"
    },
    "metrics": {
      "value": 1.23,
      "message": "Hello, world"
    }
  }
}

For ensemble runs, an additional endpoint is available to inspect the specifics of how the ensemble and its rules were evaluated. Use the GET /v1/applications/{application_id}/runs/{run_id}/ensemble endpoint to retrieve the ensemble's child runs and the decisions made by its rules.

curl -s -X GET "https://api.cloud.nextmv.io/v1/applications/wandering-otter/runs/latest-hR56rwEvR/ensemble" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  | jq '.'

{
  "id": "latest-hR56rwEvR",
  "account_id": "4b6bb68d-73a1-45ce-b2d1-7b3ace5225e7",
  "application_id": "wandering-otter",
  "evaluation_type": "rules",
  "definition_id": "jumping-hare",
  "status_v2": "succeeded",
  "error": "",
  "child_runs": [
    {
      "id": "latest-egc69QPvg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-29T14:33:34.495484256Z",
      "application_id": "wandering-otter",
      "application_instance_id": "latest",
      "application_version_id": "",
      "run_type": {
        "type": "ensemble-child",
        "definition_id": "jumping-hare",
        "reference_id": "group1"
      },
      "execution_class": "6c9500mb870s",
      "queuing_priority": 6,
      "queuing_disabled": true,
      "runtime": "python-3_11",
      "status": "succeeded",
      "status_v2": "succeeded",
      "options": {
        "details": "true"
      },
      "request_options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ]
    },
    {
      "id": "latest-Bgce9QPDR",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-29T14:33:34.484108395Z",
      "application_id": "wandering-otter",
      "application_instance_id": "latest",
      "application_version_id": "",
      "run_type": {
        "type": "ensemble-child",
        "definition_id": "jumping-hare",
        "reference_id": "group1"
      },
      "execution_class": "6c9500mb870s",
      "queuing_priority": 6,
      "queuing_disabled": true,
      "runtime": "python-3_11",
      "status": "succeeded",
      "status_v2": "succeeded",
      "options": {
        "details": "true"
      },
      "request_options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ]
    },
    {
      "id": "latest-7kce9QEDg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-29T14:33:34.501217753Z",
      "application_id": "wandering-otter",
      "application_instance_id": "latest",
      "application_version_id": "",
      "run_type": {
        "type": "ensemble-child",
        "definition_id": "jumping-hare",
        "reference_id": "group1"
      },
      "execution_class": "6c9500mb870s",
      "queuing_priority": 6,
      "queuing_disabled": true,
      "runtime": "python-3_11",
      "status": "succeeded",
      "status_v2": "succeeded",
      "options": {
        "details": "true"
      },
      "request_options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ]
    },
    {
      "id": "latest-wgc69wPDR",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-29T14:33:34.49001425Z",
      "application_id": "wandering-otter",
      "application_instance_id": "latest",
      "application_version_id": "",
      "run_type": {
        "type": "ensemble-child",
        "definition_id": "jumping-hare",
        "reference_id": "group1"
      },
      "execution_class": "6c9500mb870s",
      "queuing_priority": 6,
      "queuing_disabled": true,
      "runtime": "python-3_11",
      "status": "succeeded",
      "status_v2": "succeeded",
      "options": {
        "details": "true"
      },
      "request_options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ]
    }
  ],
  "rules_result": {
    "best_run_id": "latest-egc69QPvg",
    "decisions": [
      {
        "rule": {
          "id": "rule1",
          "statistics_path": "$.value",
          "tolerance": {
            "value": 0.1,
            "type": "relative"
          },
          "objective": "minimize",
          "index": 0
        },
        "best_run": {
          "index": 0,
          "run_id": "latest-Bgce9QPDR",
          "metric_value": 1.23
        },
        "runs_in_tolerance": [
          {
            "index": 1,
            "run_id": "latest-wgc69wPDR",
            "metric_value": 1.23
          },
          {
            "index": 2,
            "run_id": "latest-egc69QPvg",
            "metric_value": 1.23
          },
          {
            "index": 3,
            "run_id": "latest-7kce9QEDg",
            "metric_value": 1.23
          }
        ],
        "discarded_runs": []
      }
    ]
  }
}

The child_runs array lists the individual runs that were executed as part of the ensemble, and the rules_result object details the decision made by each rule, including which run was selected as the best one.

Get an ensemble definition

Use the GET /v1/applications/{application_id}/ensembles/{ensemble_id} endpoint to retrieve an existing ensemble definition for an application by its ID.

curl -s -X GET "https://api.cloud.nextmv.io/v1/applications/wandering-otter/ensembles/jumping-hare" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  | jq '.'

{
  "id": "jumping-hare",
  "application_id": "wandering-otter",
  "name": "The jumping hare ensemble",
  "description": "The jumping hare ensemble",
  "run_groups": [
    {
      "id": "group1",
      "instance_id": "latest",
      "options": {
        "details": "true"
      },
      "repetitions": 3
    }
  ],
  "rules": [
    {
      "id": "rule1",
      "statistics_path": "$.value",
      "tolerance": {
        "value": 0.1,
        "type": "relative"
      },
      "objective": "minimize",
      "index": 0
    }
  ],
  "created_at": "2026-07-29T14:33:23.42390815Z",
  "updated_at": "2026-07-29T14:33:23.42390815Z"
}

You can list all ensemble definitions in the application using the GET /v1/applications/{application_id}/ensembles endpoint. The same information is displayed as for the individual ensemble definitions, but as an array of objects. This endpoint is paginated, so the snippet below always uses pagination: it passes pagereturn=true to receive a next_page_token in the response, and keeps requesting pages by passing that token back as pagetoken until no token is returned.

ITEMS="[]"
PAGE_TOKEN=""

while :; do
  RESPONSE=$(curl -s -G "https://api.cloud.nextmv.io/v1/applications/wandering-otter/ensembles" \
    -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
    --data-urlencode "pagereturn=true" \
    --data-urlencode "pagetoken=${PAGE_TOKEN}")

  ITEMS=$(jq -n --argjson existing "$ITEMS" --argjson page "$(echo "$RESPONSE" | jq '.items')" '$existing + $page')
  PAGE_TOKEN=$(echo "$RESPONSE" | jq -r '.next_page_token // empty')

  [ -z "$PAGE_TOKEN" ] && break
done

echo "$ITEMS" | jq '.'

[
  {
    "id": "jumping-hare",
    "application_id": "wandering-otter",
    "name": "The jumping hare ensemble",
    "description": "The jumping hare ensemble",
    "run_groups": [
      {
        "id": "group1",
        "instance_id": "latest",
        "options": {
          "details": "true"
        },
        "repetitions": 3
      }
    ],
    "rules": [
      {
        "id": "rule1",
        "statistics_path": "$.value",
        "tolerance": {
          "value": 0.1,
          "type": "relative"
        },
        "objective": "minimize",
        "index": 0
      }
    ],
    "created_at": "2026-07-29T14:33:23.42390815Z",
    "updated_at": "2026-07-29T14:33:23.42390815Z"
  },
  {
    "id": "ensemble-3610ce69",
    "application_id": "wandering-otter",
    "name": "ensemble-3610ce69",
    "description": "ensemble-3610ce69",
    "run_groups": [
      {
        "id": "group1",
        "instance_id": "latest",
        "options": {
          "details": "true"
        },
        "repetitions": 3
      }
    ],
    "rules": [
      {
        "id": "rule1",
        "statistics_path": "$.value",
        "tolerance": {
          "value": 0.1,
          "type": "relative"
        },
        "objective": "minimize",
        "index": 0
      }
    ],
    "created_at": "2026-07-29T14:33:19.203592619Z",
    "updated_at": "2026-07-29T14:33:19.203592619Z"
  }
]

Update an ensemble definition

You can update attributes of an ensemble definition with the PATCH /v1/applications/{application_id}/ensembles/{ensemble_id} endpoint, such as its:

  • Name
  • Description

You cannot update the ID, rules or run groups of an ensemble definition. If you need to change the rules or run groups, you must create a new ensemble definition.

curl -s -X PATCH "https://api.cloud.nextmv.io/v1/applications/wandering-otter/ensembles/jumping-hare" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"name": "The jumping hare ensemble v2", "description": "The jumping hare ensemble v2"}' \
  | jq '.'

{
  "id": "jumping-hare",
  "application_id": "wandering-otter",
  "name": "The jumping hare ensemble v2",
  "description": "The jumping hare ensemble v2",
  "run_groups": [
    {
      "id": "group1",
      "instance_id": "latest",
      "options": {
        "details": "true"
      },
      "repetitions": 3
    }
  ],
  "rules": [
    {
      "id": "rule1",
      "statistics_path": "$.value",
      "tolerance": {
        "value": 0.1,
        "type": "relative"
      },
      "objective": "minimize",
      "index": 0
    }
  ],
  "created_at": "2026-07-29T14:33:23.42390815Z",
  "updated_at": "2026-07-29T14:33:23.42390815Z"
}

Delete an ensemble definition

Warning

Deleting an ensemble definition is irreversible. All the information associated with the ensemble definition will be permanently deleted.

Delete an ensemble definition using the DELETE /v1/applications/{application_id}/ensembles/{ensemble_id} endpoint.

curl -s -X DELETE "https://api.cloud.nextmv.io/v1/applications/wandering-otter/ensembles/jumping-hare" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}"