Running acceptance tests¶

Info

Learn the concepts and fundamentals of acceptance tests in the Explanation page.

Acceptance tests are formal tests that verify if a system satisfies business requirements.

Included in this how-to guide are instructions on how to interact with acceptance tests using methods on the Application class. Go the reference section to see all the available parameters for each method.

Create an acceptance test¶

Use the Application.new_acceptance_test method (or Application.new_acceptance_test_with_result, which additionally polls for the result) to create a new acceptance test for an application. An acceptance test requires metrics to determine the pass/fail criteria. Pass the metrics keyword argument as a list of Metric objects.

Each metric has the following attributes:

field: Field of the metric to measure (e.g., "result.custom.unassigned").
metric_type: A MetricType value. Allowed values: MetricType.direct_comparison.
params: A MetricParams object, with:
- operator: A Comparison value. Allowed values: Comparison.equal_to, Comparison.greater_than, Comparison.greater_than_or_equal_to, Comparison.less_than, Comparison.less_than_or_equal_to, and Comparison.not_equal_to.
- tolerance: A MetricTolerance object, with:
  - type: A MetricToleranceType value. Allowed values: MetricToleranceType.absolute, and MetricToleranceType.relative.
  - value: Tolerance value (numeric).
statistic: A StatisticType value. Allowed values: StatisticType.min, StatisticType.max, StatisticType.mean, StatisticType.std, StatisticType.shifted_geometric_mean, StatisticType.p01, StatisticType.p05, StatisticType.p10, StatisticType.p25, StatisticType.p50, StatisticType.p75, StatisticType.p90, StatisticType.p95, and StatisticType.p99.

Consider the following example where a metric is defined stating that the mean of the value must be equal between two instances, without any tolerance.

from nextmv.cloud import Comparison, Metric, MetricParams, MetricTolerance, MetricToleranceType, MetricType, StatisticType

metrics = [
    Metric(
        field="value",
        metric_type=MetricType.direct_comparison,
        params=MetricParams(
            operator=Comparison.equal_to,
            tolerance=MetricTolerance(type=MetricToleranceType.absolute, value=0),
        ),
        statistic=StatisticType.mean,
    )
]

Here is an example that uses the metric defined above to create a new acceptance test for an application, comparing the production and staging instances. The data is loaded from an input set.

main.pyBashOutput

import os

import nextmv
from nextmv import cloud
from nextmv.cloud import Comparison, Metric, MetricParams, MetricTolerance, MetricToleranceType, MetricType, StatisticType

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
metrics = [
    Metric(
        field="value",
        metric_type=MetricType.direct_comparison,
        params=MetricParams(
            operator=Comparison.equal_to,
            tolerance=MetricTolerance(type=MetricToleranceType.absolute, value=0),
        ),
        statistic=StatisticType.mean,
    )
]
acceptance_test = app.new_acceptance_test(
    candidate_instance_id="staging",
    baseline_instance_id="production",
    metrics=metrics,
    input_set_id="burrowing-hares",
)

nextmv.write({"acceptance_test_id": acceptance_test.id})

uv run main.py

{
  "acceptance_test_id": "acceptance-irm9lp18"
}

The call above will create a random ID, and use the same identifier for the acceptance test's name. The name of the test is used as a human-readable label. You can use the id and/or name keyword arguments to specify a custom ID and name for the acceptance test. For example:

main.pyBashOutput

import os

import nextmv
from nextmv import cloud
from nextmv.cloud import Comparison, Metric, MetricParams, MetricTolerance, MetricToleranceType, MetricType, StatisticType

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
metrics = [
    Metric(
        field="value",
        metric_type=MetricType.direct_comparison,
        params=MetricParams(
            operator=Comparison.equal_to,
            tolerance=MetricTolerance(type=MetricToleranceType.absolute, value=0),
        ),
        statistic=StatisticType.mean,
    )
]
acceptance_test = app.new_acceptance_test(
    candidate_instance_id="staging",
    baseline_instance_id="production",
    metrics=metrics,
    input_set_id="input-set-f8ao98fo",
    id="shaggy-acceptance-test",
    name="Acceptance test for a shaggy bunny",
)

nextmv.write({"acceptance_test_id": acceptance_test.id})

uv run main.py

{
  "acceptance_test_id": "shaggy-acceptance-test"
}

Get an acceptance test¶

Info

The best way to view and interact with acceptance test results is in the Nextmv Console.

You can get the results using the Application.acceptance_test method (or Application.acceptance_test_with_polling, which polls until the test finishes). The method returns an AcceptanceTest object, whose output includes the results for the evaluated metrics.

main.pyBashOutput

import os

import nextmv
from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
acceptance_test = app.acceptance_test(acceptance_test_id="shaggy-acceptance-test")

nextmv.write(acceptance_test.to_dict())

uv run main.py

{
  "id": "shaggy-acceptance-test",
  "name": "Acceptance test for a shaggy bunny",
  "description": "",
  "created_at": "2026-07-29T03:57:51.059472Z",
  "updated_at": "2026-07-29T03:57:51.059472Z",
  "app_id": "uncanny-rodent",
  "experiment_id": "shaggy-acceptance-test",
  "control": {
    "instance_id": "production",
    "version_id": "v0.0.2"
  },
  "candidate": {
    "instance_id": "staging",
    "version_id": "v0.0.2"
  },
  "metrics": [
    {
      "field": "value",
      "metric_type": "direct-comparison",
      "params": {
        "operator": "eq",
        "tolerance": {
          "type": "absolute",
          "value": 0.0
        }
      },
      "statistic": "mean"
    }
  ],
  "status": "completed",
  "results": {
    "passed": true,
    "metric_results": [
      {
        "metric": {
          "field": "value",
          "metric_type": "direct-comparison",
          "params": {
            "operator": "eq",
            "tolerance": {
              "type": "absolute",
              "value": 0.0
            }
          },
          "statistic": "mean"
        },
        "statistics": {
          "control": {
            "instance_id": "production",
            "version_id": "v0.0.2",
            "number_of_runs_total": 3,
            "distribution_summary_statistics": {
              "min": 1.23,
              "max": 1.23,
              "count": 3,
              "mean": 1.23,
              "std": 0.0,
              "shifted_geometric_mean": 1.2300000000000022,
              "shift_parameter": 10.0
            },
            "distribution_percentiles": {
              "p01": 1.23,
              "p05": 1.23,
              "p10": 1.23,
              "p25": 1.23,
              "p50": 1.23,
              "p75": 1.23,
              "p90": 1.23,
              "p95": 1.23,
              "p99": 1.23
            }
          },
          "candidate": {
            "instance_id": "staging",
            "version_id": "v0.0.2",
            "number_of_runs_total": 3,
            "distribution_summary_statistics": {
              "min": 1.23,
              "max": 1.23,
              "count": 3,
              "mean": 1.23,
              "std": 0.0,
              "shifted_geometric_mean": 1.2300000000000022,
              "shift_parameter": 10.0
            },
            "distribution_percentiles": {
              "p01": 1.23,
              "p05": 1.23,
              "p10": 1.23,
              "p25": 1.23,
              "p50": 1.23,
              "p75": 1.23,
              "p90": 1.23,
              "p95": 1.23,
              "p99": 1.23
            }
          }
        },
        "passed": true
      }
    ]
  }
}

You can list all acceptance tests in the application using the Application.list_acceptance_tests method.

main.pyBashOutput

import json
import os

from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
acceptance_tests = app.list_acceptance_tests()

print(json.dumps([acceptance_test.to_dict() for acceptance_test in acceptance_tests[:2]], indent=2))

uv run main.py

[
  {
    "id": "shaggy-acceptance-test",
    "name": "Acceptance test for a shaggy bunny",
    "description": "",
    "created_at": "2026-07-29T03:57:51.059472Z",
    "updated_at": "2026-07-29T03:57:51.059472Z",
    "app_id": "uncanny-rodent",
    "experiment_id": "shaggy-acceptance-test",
    "control": {
      "instance_id": "production",
      "version_id": "v0.0.2"
    },
    "candidate": {
      "instance_id": "staging",
      "version_id": "v0.0.2"
    },
    "metrics": [
      {
        "field": "value",
        "metric_type": "direct-comparison",
        "params": {
          "operator": "eq",
          "tolerance": {
            "type": "absolute",
            "value": 0.0
          }
        },
        "statistic": "mean"
      }
    ],
    "status": "unknown"
  },
  {
    "id": "bushy-acceptance-test",
    "name": "Acceptance test for a bushy bunny",
    "description": "",
    "created_at": "2026-07-29T03:56:40.821842Z",
    "updated_at": "2026-07-29T03:56:40.821842Z",
    "app_id": "uncanny-rodent",
    "experiment_id": "bushy-acceptance-test",
    "control": {
      "instance_id": "production",
      "version_id": "v0.0.2"
    },
    "candidate": {
      "instance_id": "staging",
      "version_id": "v0.0.2"
    },
    "metrics": [
      {
        "field": "value",
        "metric_type": "direct-comparison",
        "params": {
          "operator": "eq",
          "tolerance": {
            "type": "absolute",
            "value": 0.0
          }
        },
        "statistic": "mean"
      }
    ],
    "status": "unknown"
  }
]

Update an acceptance test¶

You can update attributes of an acceptance test with the Application.update_acceptance_test method, such as its:

Name
Description

The method returns an AcceptanceTest object. You cannot update the ID of an acceptance test.

main.pyBashOutput

import os

import nextmv
from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
acceptance_test = app.update_acceptance_test(
    acceptance_test_id="shaggy-acceptance-test",
    name="Updated Acceptance Test Name",
    description="Updated description for the acceptance test",
)

nextmv.write(acceptance_test.to_dict())

uv run main.py

{
  "id": "shaggy-acceptance-test",
  "name": "Updated Acceptance Test Name",
  "description": "Updated description for the acceptance test",
  "created_at": "2026-07-29T03:57:51.059472Z",
  "updated_at": "2026-07-29T03:58:16.295628Z",
  "status": "unknown"
}

Delete an acceptance test¶

Warning

Deleting an acceptance test is irreversible. All the data associated with the acceptance test will be permanently deleted.

Delete an acceptance test using the Application.delete_acceptance_test method.

main.pyBash

import os

from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
app.delete_acceptance_test(acceptance_test_id="shaggy-acceptance-test")

uv run main.py