Runs ensembling¶

Run ensembling is a technique where, from a single input, multiple runs are generated in parallel. Once all the runs succeed, the best run is selected based on some criteria. This technique is useful in situations such as:

• Taking into account randomness in the model, and using repetitions to get a more reliable result.
• Running the same model with different configurations to obtain the best results.
• Comparing different models to determine which one is the best for a given input.

A normal run looks like the following image:

In a normal run, the input is sent to an instance. This creates a unique run within the application. The run is then executed and the results are returned as an output.

An ensemble run works differently, as shown in the following image:

In an ensemble run, the input is processed by an ensemble definition. The ensemble definition contains run groups, and these groups make it possible to create one or more runs from a single group, by means of repetitions.

Once the input is processed by the ensemble definition, one or more runs are created for each of the run groups. All the runs are executed in parallel, and once all the runs succeed, the best run is selected. Finally, the result is returned as an output.

An ensemble run uses the same mechanics as a normal run, this is, it is characterized by a singular run_id and it contains a single input and a single output. The difference is that the ensemble run acts as a parent run that produces multiple child runs.

To work with ensemble runs, please read the following sections:

• Understand how it works. This is a detailed overview of how ensemble runs work and how the results are selected.

• Ensemble definition schema. This is the schema that defines the structure of an ensemble definition.

How it works¶

An ensemble run relies on the ensemble definition to create multiple child runs and then select the best one. As such, the ensemble definition is the key component of the ensemble run. The ensemble definition contains two core parts:

• Run groups: These define how to generate runs from a single input.
• Rules: These define how to select the best run from the generated runs.

After all the runs are executed, as defined by the run groups, the rules are evaluated and a single run is selected as the best run.

Run groups¶

A run group, as its name suggests, is a set of runs. The run group contains an instance which is responsible for running the input. The run group also contains the number of repetitions that will be executed. If the number of repetitions is greater than one, the input will be executed multiple times. A run group can also contain options that are used to configure each run in the group.

Consider the following example of run groups present in an ensemble definition:

{
  "run_groups": [
    {
      "id": "run-group-1",
      "instance_id": "production",
      "options": {
        "duration": "30",
        "threads": "2"
      },
      "repetitions": 3
    },
    {
      "id": "run-group-2",
      "instance_id": "production",
      "options": {
        "duration": "20",
        "threads": "4"
      },
      "repetitions": 2
    }
  ]
}

As a result, the following 5 runs would be created:

Rules¶

Rules are used to select the best run from the generated runs. The input before evaluating the rules is a set of runs. The output after evaluating the rules is a single run, which is the best one. Each rule has an index that gives the rules a priority order. The logic works as follows:

1. Rules are ordered by their index.
2. For the first rule: all the runs are sorted based on the objective and statistics_path of the rules. Each run should have a metric that can be extracted from its statistics, based on the path given by the rule’s statistics_path. If the objective is minimize, the metrics of the runs are sorted in ascending order. If the objective is maximize, the runs are sorted in descending order based on the metric.

3. The first run of the sorted list of runs is the best one for the first rule. For the remaining runs, the ones that fall within a tolerance of the first one (the best) are picked as well. A run falls within the tolerance if the delta (or difference) is less than or equal to the rule’s value. The difference can be absolute or relative.

   Consider the following.

   • \(x^*\) is the metric of the best run (the first run).

   • \(x\) is the metric of the run being compared.

   Here are the formulas used for calculating the delta, depending on the rule.

   • absolute difference and minimize objective:

     \[\Delta_{abs}^{min} = x - x^*\]

   • absolute difference and maximize objective:

     \[\Delta_{abs}^{max} = x^* - x\]

   • relative difference and minimize objective:

     \[\Delta_{rel}^{min} = \frac{x - x^*}{x^*}\]

   • relative difference and maximize objective:

     \[\Delta_{rel}^{max} = \frac{x^* - x}{x^*}\]

4. If there are no runs within the tolerance, it means that the search is complete and we found the best run.

5. If there is more than one run within the tolerance of the best one, move on to the next rule. Now, the list of competing runs is the best run and the runs that fell within the tolerance.
6. The next rule is applied in the same way as the previous one.
7. After all rules are applied, if there is more than one run left, the best one is picked randomly.

Consider the following example. You have the following list of child runs that were created for the parent ensemble run:

{
  "runs": [
    {
      "id": "run-1",
      "statistics": {
        "result": {
          "value": 500,
          "custom": {
            "unplanned_stops": 4,
            "max_travel_duration": 3600
          }
        }
      }
    },
    {
      "id": "run-2",
      "statistics": {
        "result": {
          "value": 298,
          "custom": {
            "unplanned_stops": 3,
            "max_travel_duration": 1200
          }
        }
      }
    },
    {
      "id": "run-3",
      "statistics": {
        "result": {
          "value": 325,
          "custom": {
            "unplanned_stops": 2,
            "max_travel_duration": 7200
          }
        }
      }
    }
  ]
}

Here are some cases of applying different rules and what the expected outcome would be.

Please note that the path in the statistics_path is specified using JSONPath notation.

{
  "rules": [
    {
      "id": "rule-1",
      "statistics_path": "$.result.value",
      "tolerance": {
        "type": "absolute",
        "value": 100
      },
      "objective": "maximize",
      "index": 0
    },
    {
      "id": "rule-2",
      "statistics_path": "$.result.custom.unplanned_stops",
      "tolerance": {
        "type": "relative",
        "value": 0.2
      },
      "objective": "minimize",
      "index": 1
    },
    {
      "id": "rule-3",
      "statistics_path": "$.result.custom.max_travel_duration",
      "tolerance": {
        "type": "relative",
        "value": 0.3
      },
      "objective": "minimize",
      "index": 2
    }
  ]
}

{
  "rules": [
    {
      "id": "rule-1",
      "statistics_path": "$.result.value",
      "tolerance": {
        "type": "absolute",
        "value": 180
      },
      "objective": "maximize",
      "index": 0
    },
      {
      "id": "rule-2",
      "statistics_path": "$.result.custom.unplanned_stops",
      "tolerance": {
        "type": "relative",
        "value": 0.2
      },
      "objective": "minimize",
      "index": 1
    },
    {
      "id": "rule-3",
      "statistics_path": "$.result.custom.max_travel_duration",
      "tolerance": {
        "type": "relative",
        "value": 0.3
      },
      "objective": "minimize",
      "index": 2
    }
  ]
}

{
  "rules": [
    {
      "id": "rule-1",
      "statistics_path": "$.result.value",
      "tolerance": {
        "type": "absolute",
        "value": 250
      },
      "objective": "maximize",
      "index": 0
    },
    {
      "id": "rule-2",
      "statistics_path": "$.result.custom.unplanned_stops",
      "tolerance": {
        "type": "relative",
        "value": 0.7
      },
      "objective": "minimize",
      "index": 1
    },
    {
      "id": "rule-3",
      "statistics_path": "$.result.custom.max_travel_duration",
      "tolerance": {
        "type": "relative",
        "value": 0.8
      },
      "objective": "minimize",
      "index": 2
    }
  ]
}

{
  "rules": [
    {
      "id": "rule-1",
      "statistics_path": "$.result.value",
      "tolerance": {
        "type": "absolute",
        "value": 250
      },
      "objective": "maximize",
      "index": 0
    },
    {
      "id": "rule-2",
      "statistics_path": "$.result.custom.unplanned_stops",
      "tolerance": {
        "type": "relative",
        "value": 0.7
      },
      "objective": "minimize",
      "index": 1
    },
    {
      "id": "rule-3",
      "statistics_path": "$.result.custom.max_travel_duration",
      "tolerance": {
        "type": "relative",
        "value": 7.5
      },
      "objective": "minimize",
      "index": 2
    }
  ]
}

Ensemble definition schema¶

Here is an example of an ensemble definition:

{
  "id": "ensemble-definition-1",
  "name": "First ensemble definition",
  "description": "Ensemble definition description",
  "run_groups": [
    {
      "id": "run-group-1",
      "instance_id": "production",
      "options": {
        "duration": "30",
        "threads": "2"
      },
      "repetitions": 3
    },
    {
      "id": "run-group-2",
      "instance_id": "production",
      "options": {
        "duration": "20",
        "threads": "4"
      },
      "repetitions": 2
    }
  ],
  "rules": [
    {
      "id": "rule-1",
      "statistics_path": "$.result.value",
      "tolerance": {
        "type": "absolute",
        "value": 500
      },
      "objective": "maximize",
      "index": 0
    },
    {
      "id": "rule-1",
      "statistics_path": "$.result.custom.unplanned_stops",
      "tolerance": {
        "type": "relative",
        "value": 0.15
      },
      "objective": "minimize",
      "index": 1
    }
  ]
}

The ensemble definition follows this schema:

A run_group determines how to create a group of runs from a single input. It follows this schema:

A rule performs an evaluation on a set of runs and chooses the runs that best fulfill the given objective on the specified metric. It follows this schema:

The tolerance specifies how to compare a run against the best run in a rule. It follows this schema: