Statistics convention

The statistics convention is a way to surface certain metrics in your system to Nextmv Platform. The custom metrics defined with the statistics convention are displayed on the run summary view and the run history table views, and can be used to compare runs and measure experiment outcomes.

Run format

Statistics can be defined for json format runs by adding them to your JSON output as a top-level statistics key. For multi-file runs, the statistics must be included in a single JSON file that is mapped in the app manifest’s output statistics path.

json

An example of how statistics would be defined for a json format run is given below.

{
  "options": {
    "solve": {
      "control": {
        "bool": [],
        "float": [],
        "int": [],
        "string": []
      },
      "duration": 10000000000,
      "mip": {
        "gap": {
          "absolute": 0.000001,
          "relative": 0.0001
        }
      },
      "verbosity": "off"
    }
  },
  "solutions": [
    {
      "meals": [
        {
          "name": "A",
          "quantity": 2
        },
        {
          "name": "B",
          "quantity": 3
        }
      ]
    }
  ],
  "statistics": {
    "result": {
      "custom": {
        "constraints": 2,
        "provider": "HiGHS",
        "status": "optimal",
        "variables": 2
      },
      "duration": 0.123,
      "value": 27
    },
    "run": {
      "duration": 0.123
    },
    "schema": "v1"
  },
  "version": {
    "go-mip": "VERSION",
    "sdk": "VERSION"
  }
}

Note the top-level statistics block alongside options, solutions, etc.

multi-file

An example of how the same statistics shown above would be defined for a multi-file format run is given below.

First, the file in which the statistics are stored would need to be defined in the app manifest:

configuration:
content:
  format: multi-file
  multi-file:
    input:
      path: inputs
    output:
      solutions: outputs/solutions
      statistics: outputs/statistics.json

Then, the statistics.json file would have the following contents:

{
  "statistics": {
    "result": {
      "custom": {
        "constraints": 2,
        "provider": "HiGHS",
        "status": "optimal",
        "variables": 2
      },
      "duration": 0.123,
      "value": 27
    },
    "run": {
      "duration": 0.123
    },
    "schema": "v1"
  }
}

Note that the top-level key in the JSON file above is statistics; this is a requirement.

Schema

The statistics convention has some requirements and a set hierarchy for defining statistics. There are four potential top-level properties:

• schema (required) is the version of the convention. This is required and indicates which version of the convention is used (v1 is the only accepted value).
• run (optiona) contains statistics about the run itself — for example the total duration of the solver and number of iterations.
• result (optional) contains statistics about the result of the run — for example the objective value.
• series_data (optional) contains time series data for the run, such as the progression of the value function over time.

Other notes:

• All numerical values are interpreted as floating point (IEEE 754) values. This includes the special string values "nan", "inf", "+inf" and "–inf" as JSON does not support these values natively.
• The maximum size of the statistics object without unnecessary whitespace is 10kb.

The four top-level properties of the statistics schema are outlined below in more detail, including how you add custom statistics.

schema

A string that determines the structure of the statistics object. As mentioned above, this property is required and must be "v1".

run

An object that can have the following properties:

result

An object that can have the following properties:

series_data

An object that can have the following properties:

The Series type is defined as such:

type Series = {
  name: string;
  data_points: {
    x: number;
    y: number;
  }[]
}

Example snippet

See the example statistics below for a quick reference of the schema detailed above. The time series included shows a simple progression of the value function over time as well as more complex data like the usage of a custom operator over time.

{
  "schema": "v1",
  "run": {
    "duration": 40,
    "iterations": 100
  },
  "result": {
    "value": 3500,
    "duration": 15.5,
    "custom": {
      "routing": {
        "stops": {
          "assigned": 10,
          "unassigned": 2
        }
      },
      "special_float_values": {
        "inf": "inf",
        "nan": "nan",
        "plus_inf": "+inf",
        "minus_inf": "-inf"
      }
    }
  },
  "series_data": {
    "value": {
      "name": "value",
      "data_points": [
        {
          "x": 5.5,
          "y": 10000
        },
        {
          "x": 15.5,
          "y": 3500
        }
      ]
    },
    "custom": [
      {
        "name": "simple_operator_success",
        "data_points": [
          {
            "x": 5,
            "y": 15
          },
          {
            "x": 10,
            "y": 10
          },
          {
            "x": 12,
            "y": 5
          },
          {
            "x": 15,
            "y": 0
          }
        ]
      },
      {
        "name": "simple_operator_avg_contribution",
        "data_points": [
          {
            "x": 5,
            "y": 300
          },
          {
            "x": 10,
            "y": 200
          },
          {
            "x": 12,
            "y": 150
          },
          {
            "x": 15,
            "y": "nan"
          }
        ]
      }
    ]
  }
}

Adding custom statistics

If you have a custom app, you may wish to add custom statistics to result.custom or run.custom.

Consider an example from vehicle routing. To add the unplanned stop count as a custom statistic, you can:

1. Format the output using the nextroute factory formatter.
2. Add custom unplanned stop count to the formatted solution output.

An example of how to do this is shown below:

output := factory.Format(ctx, options, solver, last)
solution := output.Solutions[0].(schema.SolutionOutput)
output.Statistics.Result.Custom = map[string]any{"unplanned": len(solution.Unplanned)}

How the statistics are used

If the run output conforms to the statistics convention, the system extracts the statistics and makes them available for further processing. For example the statistics are displayed as top-level values in the run history table, and are compared in the compare run views. The statistics can also be referenced and used for analysis with the experiment testing suite.