Skip to content

Using runs ensembling

Info

Learn the concepts and fundamentals of runs ensembling in the Explanation page.

Run ensembling is a technique where, from a single input, multiple runs are generated in parallel. Once all the runs succeed, the best run is selected based on some criteria.

Included in this how-to guide are instructions on how to interact with ensemble definitions using methods on the Application class. Go the reference section to see all the available parameters for each method.

Create an ensemble definition

Use the Application.new_ensemble_definition method to create a new ensemble definition for an application. Ensembles require rules and run_groups, both passed as lists of objects.

Each rule is a EvaluationRule object with the following attributes:

  • id: Unique identifier for the rule (required).
  • statistics_path: JSONPath to the metric (e.g., $.result.value) (required).
  • objective: A RuleObjective value for the evaluation (required). Allowed values: RuleObjective.MAXIMIZE and RuleObjective.MINIMIZE.
  • tolerance: A RuleTolerance object with the following fields (required):
    • value: Tolerance value (float).
    • type: A RuleToleranceType value. Allowed values: RuleToleranceType.ABSOLUTE and RuleToleranceType.RELATIVE.
  • index: Evaluation order - lower indices evaluated first (required).

Here is an example:

EvaluationRule(
    id="rule1",
    statistics_path="$.value",
    objective=RuleObjective.MINIMIZE,
    tolerance=RuleTolerance(value=0.1, type=RuleToleranceType.RELATIVE),
    index=0,
)

Similarly to rules, each run group is a RunGroup object with the following attributes:

  • id: Unique identifier for the run group (required).
  • instance_id: The instance to execute runs on (required).
  • options: Runtime options/parameters (optional). Options should be provided as a dict with string key-value pairs.
  • repetitions: Number of times to repeat the run (optional).

Consider the following example:

RunGroup(
    id="group1",
    instance_id="latest",
    options={"details": "true"},
    repetitions=3,
)

Here is an example that uses the rule and group above to create an ensemble for the application.

import os

import nextmv
from nextmv import cloud
from nextmv.cloud import EvaluationRule, RuleObjective, RuleTolerance, RuleToleranceType, RunGroup

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="prancing-marmot")
ensemble_definition = app.new_ensemble_definition(
    run_groups=[
        RunGroup(id="group1", instance_id="latest", options={"details": "true"}, repetitions=3),
    ],
    rules=[
        EvaluationRule(
            id="rule1",
            statistics_path="$.value",
            objective=RuleObjective.MINIMIZE,
            tolerance=RuleTolerance(value=0.1, type=RuleToleranceType.RELATIVE),
            index=0,
        ),
    ],
)

nextmv.write(ensemble_definition.to_dict())

uv run main.py

{
  "id": "ensemble-bxazq8z5",
  "application_id": "prancing-marmot",
  "name": "ensemble-bxazq8z5",
  "description": "ensemble-bxazq8z5",
  "run_groups": [
    {
      "id": "group1",
      "instance_id": "latest",
      "options": {
        "details": "true"
      },
      "repetitions": 3
    }
  ],
  "rules": [
    {
      "id": "rule1",
      "statistics_path": "$.value",
      "objective": "minimize",
      "tolerance": {
        "value": 0.1,
        "type": "relative"
      },
      "index": 0
    }
  ],
  "created_at": "2026-07-28T19:36:23.327927Z",
  "updated_at": "2026-07-28T19:36:23.327927Z"
}

The call above will create a random ID, and use the same identifier for the ensemble's name. The name of the ensemble is used as a human-readable label. You can use the id and/or name keyword arguments to specify a custom ID and name for the ensemble. For example:

import os

import nextmv
from nextmv import cloud
from nextmv.cloud import EvaluationRule, RuleObjective, RuleTolerance, RuleToleranceType, RunGroup

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="prancing-marmot")
ensemble_definition = app.new_ensemble_definition(
    run_groups=[
        RunGroup(id="group1", instance_id="latest", options={"details": "true"}, repetitions=3),
    ],
    rules=[
        EvaluationRule(
            id="rule1",
            statistics_path="$.value",
            objective=RuleObjective.MINIMIZE,
            tolerance=RuleTolerance(value=0.1, type=RuleToleranceType.RELATIVE),
            index=0,
        ),
    ],
    id="jumping-hare",
    name="The jumping hare ensemble",
)

nextmv.write(ensemble_definition.to_dict())

uv run main.py

{
  "id": "jumping-hare",
  "application_id": "prancing-marmot",
  "name": "The jumping hare ensemble",
  "description": "The jumping hare ensemble",
  "run_groups": [
    {
      "id": "group1",
      "instance_id": "latest",
      "options": {
        "details": "true"
      },
      "repetitions": 3
    }
  ],
  "rules": [
    {
      "id": "rule1",
      "statistics_path": "$.value",
      "objective": "minimize",
      "tolerance": {
        "value": 0.1,
        "type": "relative"
      },
      "index": 0
    }
  ],
  "created_at": "2026-07-28T19:36:32.107981Z",
  "updated_at": "2026-07-28T19:36:32.107981Z"
}

Run with an ensemble

Once an ensemble has been created you can use it with the Application.new_run_with_result method (or Application.new_run) by passing a RunConfiguration object whose run_type attribute is a RunTypeConfiguration object. Set the run_type attribute of the latter to RunType.ENSEMBLE, and the definition_id attribute to the ensemble definition ID. For example:

import os

import nextmv
from nextmv import cloud
from nextmv import RunConfiguration, RunType, RunTypeConfiguration

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="prancing-marmot")
run_result = app.new_run_with_result(
    input={"name": "world", "radius": 6378, "distance": 147.6},
    configuration=RunConfiguration(
        run_type=RunTypeConfiguration(run_type=RunType.ENSEMBLE, definition_id="jumping-hare"),
    ),
)

result = run_result.to_dict()
result["output"].pop("assets", None)  # Assets are omitted here for a cleaner display.

nextmv.write(result)

uv run main.py

{
  "description": "",
  "id": "latest-CvYCXyEvR",
  "metadata": {
    "application_id": "prancing-marmot",
    "application_instance_id": "latest",
    "application_version_id": "",
    "created_at": "2026-07-28T19:36:41Z",
    "duration": 6131.0,
    "error": "",
    "execution_class": "",
    "format": {
      "input": {
        "type": "json"
      },
      "output": {
        "type": "json"
      }
    },
    "input_size": 47.0,
    "metrics": {
      "message": "Hello, world",
      "value": 1.23
    },
    "output_size": 25099.0,
    "run_type": {
      "type": "ensemble",
      "definition_id": "jumping-hare",
      "reference_id": ""
    },
    "runtime": "",
    "status_v2": "succeeded"
  },
  "name": "",
  "user_email": "sebastian@nextmv.io",
  "console_url": "https://cloud.nextmv.io/app/prancing-marmot/run/latest-CvYCXyEvR?view=details",
  "output": {
    "options": {
      "details": true
    },
    "solution": {
      "message": "Hello, world"
    },
    "metrics": {
      "value": 1.23,
      "message": "Hello, world"
    }
  },
  "ensemble": {
    "id": "latest-CvYCXyEvR",
    "account_id": "4b6bb68d-73a1-45ce-b2d1-7b3ace5225e7",
    "application_id": "prancing-marmot",
    "evaluation_type": "rules",
    "definition_id": "jumping-hare",
    "status_v2": "succeeded",
    "error": "",
    "child_runs": [
      {
        "id": "latest-TdLCXsEvR",
        "user_email": "sebastian@nextmv.io",
        "name": "",
        "description": "",
        "created_at": "2026-07-28T19:36:41.039707861Z",
        "application_id": "prancing-marmot",
        "application_instance_id": "latest",
        "application_version_id": "",
        "run_type": {
          "type": "ensemble-child",
          "definition_id": "jumping-hare",
          "reference_id": "group1"
        },
        "execution_class": "6c9500mb870s",
        "queuing_priority": 6,
        "queuing_disabled": true,
        "runtime": "python-3_11",
        "status": "succeeded",
        "status_v2": "succeeded",
        "options": {
          "details": "true"
        },
        "request_options": {
          "details": "true"
        },
        "options_summary": [
          {
            "name": "details",
            "value": "true",
            "source": "run"
          }
        ]
      },
      {
        "id": "latest-FOLCuyEvR",
        "user_email": "sebastian@nextmv.io",
        "name": "",
        "description": "",
        "created_at": "2026-07-28T19:36:41.03544735Z",
        "application_id": "prancing-marmot",
        "application_instance_id": "latest",
        "application_version_id": "",
        "run_type": {
          "type": "ensemble-child",
          "definition_id": "jumping-hare",
          "reference_id": "group1"
        },
        "execution_class": "6c9500mb870s",
        "queuing_priority": 6,
        "queuing_disabled": true,
        "runtime": "python-3_11",
        "status": "succeeded",
        "status_v2": "succeeded",
        "options": {
          "details": "true"
        },
        "request_options": {
          "details": "true"
        },
        "options_summary": [
          {
            "name": "details",
            "value": "true",
            "source": "run"
          }
        ]
      },
      {
        "id": "latest-4dLCuyPDg",
        "user_email": "sebastian@nextmv.io",
        "name": "",
        "description": "",
        "created_at": "2026-07-28T19:36:41.030758575Z",
        "application_id": "prancing-marmot",
        "application_instance_id": "latest",
        "application_version_id": "",
        "run_type": {
          "type": "ensemble-child",
          "definition_id": "jumping-hare",
          "reference_id": "group1"
        },
        "execution_class": "6c9500mb870s",
        "queuing_priority": 6,
        "queuing_disabled": true,
        "runtime": "python-3_11",
        "status": "succeeded",
        "status_v2": "succeeded",
        "options": {
          "details": "true"
        },
        "request_options": {
          "details": "true"
        },
        "options_summary": [
          {
            "name": "details",
            "value": "true",
            "source": "run"
          }
        ]
      },
      {
        "id": "latest-zdLjXyPDg",
        "user_email": "sebastian@nextmv.io",
        "name": "",
        "description": "",
        "created_at": "2026-07-28T19:36:41.02556087Z",
        "application_id": "prancing-marmot",
        "application_instance_id": "latest",
        "application_version_id": "",
        "run_type": {
          "type": "ensemble-child",
          "definition_id": "jumping-hare",
          "reference_id": "group1"
        },
        "execution_class": "6c9500mb870s",
        "queuing_priority": 6,
        "queuing_disabled": true,
        "runtime": "python-3_11",
        "status": "succeeded",
        "status_v2": "succeeded",
        "options": {
          "details": "true"
        },
        "request_options": {
          "details": "true"
        },
        "options_summary": [
          {
            "name": "details",
            "value": "true",
            "source": "run"
          }
        ]
      }
    ],
    "rules_result": {
      "best_run_id": "latest-FOLCuyEvR",
      "decisions": [
        {
          "rule": {
            "id": "rule1",
            "statistics_path": "$.value",
            "tolerance": {
              "value": 0.1,
              "type": "relative"
            },
            "objective": "minimize",
            "index": 0
          },
          "best_run": {
            "index": 0,
            "run_id": "latest-zdLjXyPDg",
            "metric_value": 1.23
          },
          "runs_in_tolerance": [
            {
              "index": 1,
              "run_id": "latest-4dLCuyPDg",
              "metric_value": 1.23
            },
            {
              "index": 2,
              "run_id": "latest-FOLCuyEvR",
              "metric_value": 1.23
            },
            {
              "index": 3,
              "run_id": "latest-TdLCXsEvR",
              "metric_value": 1.23
            }
          ],
          "discarded_runs": []
        }
      ]
    }
  }
}

For ensemble runs, an ensemble entry is included in the run result detailing the specifics of how the ensemble and its rules were evaluated.

Get an ensemble definition

Use the Application.ensemble_definition method to retrieve an existing ensemble definition for an application by its ID.

import os

import nextmv
from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="prancing-marmot")
ensemble_definition = app.ensemble_definition(ensemble_definition_id="jumping-hare")

nextmv.write(ensemble_definition.to_dict())

uv run main.py

{
  "id": "jumping-hare",
  "application_id": "prancing-marmot",
  "name": "The jumping hare ensemble",
  "description": "The jumping hare ensemble",
  "run_groups": [
    {
      "id": "group1",
      "instance_id": "latest",
      "options": {
        "details": "true"
      },
      "repetitions": 3
    }
  ],
  "rules": [
    {
      "id": "rule1",
      "statistics_path": "$.value",
      "objective": "minimize",
      "tolerance": {
        "value": 0.1,
        "type": "relative"
      },
      "index": 0
    }
  ],
  "created_at": "2026-07-28T19:36:32.107981Z",
  "updated_at": "2026-07-28T19:36:32.107981Z"
}

You can list all ensemble definitions in the application using the Application.list_ensemble_definitions method.

import json
import os

from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="prancing-marmot")
ensemble_definitions = app.list_ensemble_definitions()

print(json.dumps([ensemble_definition.to_dict() for ensemble_definition in ensemble_definitions], indent=2))

uv run main.py

[
  {
    "id": "jumping-hare",
    "application_id": "prancing-marmot",
    "name": "The jumping hare ensemble",
    "description": "The jumping hare ensemble",
    "run_groups": [
      {
        "id": "group1",
        "instance_id": "latest",
        "options": {
          "details": "true"
        },
        "repetitions": 3
      }
    ],
    "rules": [
      {
        "id": "rule1",
        "statistics_path": "$.value",
        "objective": "minimize",
        "tolerance": {
          "value": 0.1,
          "type": "relative"
        },
        "index": 0
      }
    ],
    "created_at": "2026-07-28T19:36:32.107981Z",
    "updated_at": "2026-07-28T19:36:32.107981Z"
  },
  {
    "id": "ensemble-bxazq8z5",
    "application_id": "prancing-marmot",
    "name": "ensemble-bxazq8z5",
    "description": "ensemble-bxazq8z5",
    "run_groups": [
      {
        "id": "group1",
        "instance_id": "latest",
        "options": {
          "details": "true"
        },
        "repetitions": 3
      }
    ],
    "rules": [
      {
        "id": "rule1",
        "statistics_path": "$.value",
        "objective": "minimize",
        "tolerance": {
          "value": 0.1,
          "type": "relative"
        },
        "index": 0
      }
    ],
    "created_at": "2026-07-28T19:36:23.327927Z",
    "updated_at": "2026-07-28T19:36:23.327927Z"
  }
]

Update an ensemble definition

You can update attributes of an ensemble definition with the Application.update_ensemble_definition method, such as its:

  • Name
  • Description

You cannot update the ID, rules or run groups of an ensemble definition. If you need to change the rules or run groups, you must create a new ensemble definition.

import os

import nextmv
from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="prancing-marmot")
ensemble_definition = app.update_ensemble_definition(
    id="jumping-hare",
    name="The jumping hare ensemble v2",
    description="The jumping hare ensemble v2",
)

nextmv.write(ensemble_definition.to_dict())

uv run main.py

{
  "id": "jumping-hare",
  "application_id": "prancing-marmot",
  "name": "The jumping hare ensemble v2",
  "description": "The jumping hare ensemble v2",
  "run_groups": [
    {
      "id": "group1",
      "instance_id": "latest",
      "options": {
        "details": "true"
      },
      "repetitions": 3
    }
  ],
  "rules": [
    {
      "id": "rule1",
      "statistics_path": "$.value",
      "objective": "minimize",
      "tolerance": {
        "value": 0.1,
        "type": "relative"
      },
      "index": 0
    }
  ],
  "created_at": "2026-07-28T19:36:32.107981Z",
  "updated_at": "2026-07-28T19:36:32.107981Z"
}

Delete an ensemble definition

Warning

Deleting an ensemble definition is irreversible. All the information associated with the ensemble definition will be permanently deleted.

Delete an ensemble definition using the Application.delete_ensemble_definition method.

import os

from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="prancing-marmot")
app.delete_ensemble_definition(ensemble_definition_id="jumping-hare")

uv run main.py