Skip to content

Using runs ensembling

Info

Learn the concepts and fundamentals of runs ensembling in the Explanation page.

Run ensembling is a technique where, from a single input, multiple runs are generated in parallel. Once all the runs succeed, the best run is selected based on some criteria.

Included in this how-to guide are instructions on how to interact with the actions under the nextmv cloud ensemble command tree. Go the reference section or use the --help option to see all the available options for each command.

Create an ensemble definition

Use the nextmv cloud ensemble create command to create a new ensemble definition for an application. Ensembles require rules (--rules) and run groups (--run-groups). For both elements, you can create multiple of them by:

  • Using the --rules/--run-groups flag multiple times, or
  • Providing an array of objects in valid JSON format.

Each rule is defined as a JSON object with the following attributes:

  • id: Unique identifier for the rule (required).
  • statistics_path: JSONPath to the metric (e.g., $.result.value) (required).
  • objective: Objective for the evaluation (required). Allowed values: maximize and minimize.
  • tolerance: Object with the following fields (required):
    • value: Tolerance value (float).
    • type: Tolerance type. Allowed values: absolute and relative.
  • index: Evaluation order - lower indices evaluated first (required).

Here is an example:

{
  "id": "rule1",
  "statistics_path": "$.value",
  "objective": "minimize",
  "tolerance": {
    "value": 0.1,
    "type": "relative"
  },
  "index": 0
}

Similarly to rules, each run group is defined as a JSON object with the following attributes:

  • id: Unique identifier for the run group (required).
  • instance_id: The instance to execute runs on (required).
  • options: Runtime options/parameters (optional). Options should be provided as a JSON object with string key-value pairs.
  • repetitions: Number of times to repeat the run (optional).

Consider the following example:

{
  "id": "group1",
  "instance_id": "latest",
  "options": {
    "details": "true"
  },
  "repetitions": 3
}

Here is an example command that uses the rule and group above to create an ensemble for the application.

RUN_GROUP='{
  "id": "group1",
  "instance_id": "latest",
  "options": {
    "details": "true"
  },
  "repetitions": 3
}'
RULE='{
  "id": "rule1",
  "statistics_path": "$.value",
  "objective": "minimize",
  "tolerance": {"value": 0.1, "type": "relative"},
  "index": 0
}'
nextmv cloud ensemble create --app-id uncanny-rodent --run-groups "$RUN_GROUP" --rules "$RULE"

⏳ Creating ensemble definition...
{
  "id": "ensemble-v23xasrj",
  "application_id": "uncanny-rodent",
  "name": "ensemble-v23xasrj",
  "description": "ensemble-v23xasrj",
  "run_groups": [
    {
      "id": "group1",
      "instance_id": "latest",
      "options": {
        "details": "true"
      },
      "repetitions": 3
    }
  ],
  "rules": [
    {
      "id": "rule1",
      "statistics_path": "$.value",
      "objective": "minimize",
      "tolerance": {
        "value": 0.1,
        "type": "relative"
      },
      "index": 0
    }
  ],
  "created_at": "2026-07-24T14:39:27.208890Z",
  "updated_at": "2026-07-24T14:39:27.208890Z"
}

The command above will create a random ID, and use the same identifier for the ensemble's name. The name of the ensemble is used as a human-readable label. You can use the --ensemble-definition-id and/or --name options to specify a custom ID and name for the ensemble. For example:

RUN_GROUP='{
  "id": "group1",
  "instance_id": "latest",
  "options": {
    "details": "true"
  },
  "repetitions": 3
}'
RULE='{
  "id": "rule1",
  "statistics_path": "$.value",
  "objective": "minimize",
  "tolerance": {"value": 0.1, "type": "relative"},
  "index": 0
}'
nextmv cloud ensemble create \
    --app-id uncanny-rodent \
    --run-groups "$RUN_GROUP" \
    --rules "$RULE" \
    --ensemble-definition-id jumping-hare \
    --name "The jumping hare ensemble"

⏳ Creating ensemble definition...
{
  "id": "jumping-hare",
  "application_id": "uncanny-rodent",
  "name": "The jumping hare ensemble",
  "description": "The jumping hare ensemble",
  "run_groups": [
    {
      "id": "group1",
      "instance_id": "latest",
      "options": {
        "details": "true"
      },
      "repetitions": 3
    }
  ],
  "rules": [
    {
      "id": "rule1",
      "statistics_path": "$.value",
      "objective": "minimize",
      "tolerance": {
        "value": 0.1,
        "type": "relative"
      },
      "index": 0
    }
  ],
  "created_at": "2026-07-24T14:41:24.443315Z",
  "updated_at": "2026-07-24T14:41:24.443315Z"
}

Run with an ensemble

Once an ensemble has been created you can use it with the nextmv cloud run create command by means of the --definition-id and --run-type options. Use the --definition-id option to specify the ensemble definition ID, and the --run-type option to specify the run type as ensemble. For example:

nextmv cloud run create \
    --app-id uncanny-rodent \
    --input inputs \
    --definition-id jumping-hare \
    --run-type ensemble \
    --wait

✅ Run latest-DzbrktPDg created.
⏳ Getting run results...
✅ Run outputs saved to latest-DzbrktPDg-output. Here is the metadata.
{
  "description": "",
  "id": "latest-DzbrktPDg",
  "metadata": {
    "application_id": "uncanny-rodent",
    "application_instance_id": "latest",
    "application_version_id": "",
    "created_at": "2026-07-24T14:49:17Z",
    "duration": 4777.0,
    "error": "",
    "execution_class": "",
    "format": {
      "input": {
        "type": "multi-file"
      },
      "output": {
        "type": "multi-file"
      }
    },
    "input_size": 261.0,
    "metrics": {
      "metrics": {
        "message": "Hello, Patches",
        "value": 1.23
      }
    },
    "output_size": 158.0,
    "run_type": {
      "type": "ensemble",
      "definition_id": "jumping-hare",
      "reference_id": ""
    },
    "runtime": "",
    "status_v2": "succeeded"
  },
  "name": "",
  "user_email": "sebastian@nextmv.io",
  "console_url": "https://cloud.nextmv.io/app/uncanny-rodent/run/latest-DzbrktPDg?view=details",
  "ensemble": {
    "id": "latest-DzbrktPDg",
    "account_id": "4b6bb68d-73a1-45ce-b2d1-7b3ace5225e7",
    "application_id": "uncanny-rodent",
    "evaluation_type": "rules",
    "definition_id": "jumping-hare",
    "status_v2": "succeeded",
    "error": "",
    "child_runs": [
      {
        "id": "latest-yzbrktEvR",
        "user_email": "sebastian@nextmv.io",
        "name": "",
        "description": "",
        "created_at": "2026-07-24T14:49:17.623717373Z",
        "application_id": "uncanny-rodent",
        "application_instance_id": "latest",
        "application_version_id": "",
        "run_type": {
          "type": "ensemble-child",
          "definition_id": "jumping-hare",
          "reference_id": "group1"
        },
        "execution_class": "6c9500mb870s",
        "queuing_priority": 6,
        "queuing_disabled": true,
        "runtime": "python-3_11",
        "status": "succeeded",
        "status_v2": "succeeded",
        "options": {
          "details": "true"
        },
        "request_options": {
          "details": "true"
        },
        "options_summary": [
          {
            "name": "details",
            "value": "true",
            "source": "run"
          }
        ]
      },
      {
        "id": "latest-Mib9ztPvg",
        "user_email": "sebastian@nextmv.io",
        "name": "",
        "description": "",
        "created_at": "2026-07-24T14:49:17.636098109Z",
        "application_id": "uncanny-rodent",
        "application_instance_id": "latest",
        "application_version_id": "",
        "run_type": {
          "type": "ensemble-child",
          "definition_id": "jumping-hare",
          "reference_id": "group1"
        },
        "execution_class": "6c9500mb870s",
        "queuing_priority": 6,
        "queuing_disabled": true,
        "runtime": "python-3_11",
        "status": "succeeded",
        "status_v2": "succeeded",
        "options": {
          "details": "true"
        },
        "request_options": {
          "details": "true"
        },
        "options_summary": [
          {
            "name": "details",
            "value": "true",
            "source": "run"
          }
        ]
      },
      {
        "id": "latest-qkb9kpEDR",
        "user_email": "sebastian@nextmv.io",
        "name": "",
        "description": "",
        "created_at": "2026-07-24T14:49:17.630143048Z",
        "application_id": "uncanny-rodent",
        "application_instance_id": "latest",
        "application_version_id": "",
        "run_type": {
          "type": "ensemble-child",
          "definition_id": "jumping-hare",
          "reference_id": "group1"
        },
        "execution_class": "6c9500mb870s",
        "queuing_priority": 6,
        "queuing_disabled": true,
        "runtime": "python-3_11",
        "status": "succeeded",
        "status_v2": "succeeded",
        "options": {
          "details": "true"
        },
        "request_options": {
          "details": "true"
        },
        "options_summary": [
          {
            "name": "details",
            "value": "true",
            "source": "run"
          }
        ]
      },
      {
        "id": "latest-JzxrkpEDg",
        "user_email": "sebastian@nextmv.io",
        "name": "",
        "description": "",
        "created_at": "2026-07-24T14:49:17.617811808Z",
        "application_id": "uncanny-rodent",
        "application_instance_id": "latest",
        "application_version_id": "",
        "run_type": {
          "type": "ensemble-child",
          "definition_id": "jumping-hare",
          "reference_id": "group1"
        },
        "execution_class": "6c9500mb870s",
        "queuing_priority": 6,
        "queuing_disabled": true,
        "runtime": "python-3_11",
        "status": "succeeded",
        "status_v2": "succeeded",
        "options": {
          "details": "true"
        },
        "request_options": {
          "details": "true"
        },
        "options_summary": [
          {
            "name": "details",
            "value": "true",
            "source": "run"
          }
        ]
      }
    ],
    "rules_result": {
      "best_run_id": "latest-yzbrktEvR",
      "decisions": [
        {
          "rule": {
            "id": "rule1",
            "statistics_path": "$.metrics.value",
            "tolerance": {
              "value": 0.1,
              "type": "relative"
            },
            "objective": "minimize",
            "index": 0
          },
          "best_run": {
            "index": 0,
            "run_id": "latest-JzxrkpEDg",
            "metric_value": 1.23
          },
          "runs_in_tolerance": [
            {
              "index": 1,
              "run_id": "latest-yzbrktEvR",
              "metric_value": 1.23
            },
            {
              "index": 2,
              "run_id": "latest-qkb9kpEDR",
              "metric_value": 1.23
            },
            {
              "index": 3,
              "run_id": "latest-Mib9ztPvg",
              "metric_value": 1.23
            }
          ],
          "discarded_runs": []
        }
      ]
    }
  }
}

For ensemble runs, an ensemble object is included in the run result detailing the specifics of how the ensemble and its rules were evaluated.

Get an ensemble definition

Use the nextmv cloud ensemble get command to retrieve an existing ensemble definition for an application by its ID.

nextmv cloud ensemble get --app-id uncanny-rodent --ensemble-definition-id jumping-hare

⏳ Getting ensemble definition...
{
  "id": "jumping-hare",
  "application_id": "uncanny-rodent",
  "name": "The jumping hare ensemble",
  "description": "The jumping hare ensemble",
  "run_groups": [
    {
      "id": "group1",
      "instance_id": "latest",
      "options": {
        "details": "true"
      },
      "repetitions": 3
    }
  ],
  "rules": [
    {
      "id": "rule1",
      "statistics_path": "$.value",
      "objective": "minimize",
      "tolerance": {
        "value": 0.1,
        "type": "relative"
      },
      "index": 0
    }
  ],
  "created_at": "2026-07-24T14:41:24.443315Z",
  "updated_at": "2026-07-24T14:41:24.443315Z"
}

You can list all ensemble definitions in the application using the nextmv cloud ensemble list command.

nextmv cloud ensemble list --app-id uncanny-rodent

⏳ Listing ensemble definitions...
[
  {
    "id": "jumping-hare",
    "application_id": "uncanny-rodent",
    "name": "The jumping hare ensemble",
    "description": "The jumping hare ensemble",
    "run_groups": [
      {
        "id": "group1",
        "instance_id": "latest",
        "options": {
          "details": "true"
        },
        "repetitions": 3
      }
    ],
    "rules": [
      {
        "id": "rule1",
        "statistics_path": "$.value",
        "objective": "minimize",
        "tolerance": {
          "value": 0.1,
          "type": "relative"
        },
        "index": 0
      }
    ],
    "created_at": "2026-07-24T14:41:24.443315Z",
    "updated_at": "2026-07-24T14:41:24.443315Z"
  },
  {
    "id": "ensemble-v23xasrj",
    "application_id": "uncanny-rodent",
    "name": "ensemble-v23xasrj",
    "description": "ensemble-v23xasrj",
    "run_groups": [
      {
        "id": "group1",
        "instance_id": "latest",
        "options": {
          "details": "true"
        },
        "repetitions": 3
      }
    ],
    "rules": [
      {
        "id": "rule1",
        "statistics_path": "$.value",
        "objective": "minimize",
        "tolerance": {
          "value": 0.1,
          "type": "relative"
        },
        "index": 0
      }
    ],
    "created_at": "2026-07-24T14:39:27.208890Z",
    "updated_at": "2026-07-24T14:39:27.208890Z"
  }
]

Update an ensemble definition

You can update attributes of an ensemble definition with the nextmv cloud ensemble update command, such as its:

  • Name
  • Description

Please use the --help option on the nextmv cloud ensemble update command to see all the available options for updating an ensemble definition. You cannot update the ID, rules or run groups of an ensemble definition. If you need to change the rules or run groups, you must create a new ensemble definition.

nextmv cloud ensemble update \
    --app-id uncanny-rodent \
    --ensemble-definition-id jumping-hare \
    --name "The jumping hare ensemble v2" \
    --description "The jumping hare ensemble v2"

⏳ Updating ensemble definition...
✅ Ensemble definition jumping-hare updated successfully in application uncanny-rodent.
{
  "id": "jumping-hare",
  "application_id": "uncanny-rodent",
  "name": "The jumping hare ensemble v2",
  "description": "The jumping hare ensemble v2",
  "run_groups": [
    {
      "id": "group1",
      "instance_id": "latest",
      "options": {
        "details": "true"
      },
      "repetitions": 3
    }
  ],
  "rules": [
    {
      "id": "rule1",
      "statistics_path": "$.value",
      "objective": "minimize",
      "tolerance": {
        "value": 0.1,
        "type": "relative"
      },
      "index": 0
    }
  ],
  "created_at": "2026-07-24T14:41:24.443315Z",
  "updated_at": "2026-07-24T14:41:24.443315Z"
}

Delete an ensemble definition

Warning

Deleting an ensemble definition is irreversible. All the information associated with the ensemble definition will be permanently deleted.

Delete an ensemble definition using the nextmv cloud ensemble delete command. The CLI will prompt you to confirm deletion but you can override it with the --yes option.

nextmv cloud ensemble delete --app-id uncanny-rodent --ensemble-definition-id jumping-hare

Are you sure you want to delete ensemble definition jumping-hare from application uncanny-rodent? This action cannot be undone.
💡 Confirm Yes
✅ Ensemble definition jumping-hare deleted successfully from application uncanny-rodent.