Running acceptance tests¶

Info

Learn the concepts and fundamentals of acceptance tests in the Explanation page.

Acceptance tests are formal tests that verify if a system satisfies business requirements.

Included in this how-to guide are instructions on how to interact with the actions under the nextmv cloud acceptance command tree. Go the reference section or use the --help option to see all the available options for each command.

Create an acceptance test¶

Use the nextmv cloud acceptance create command to create a new acceptance test for an application. An acceptance test requires metrics to determine the pass/fail criteria. You can pass multiple metrics with the --metrics flag by:

Using the --metrics flag multiple times, or
Providing an array of objects in valid JSON format.

Each metric is defined as a JSON object with the following attributes:

Each metric must have the following fields:

field: Field of the metric to measure (e.g., "result.custom.unassigned").
metric_type: Type of metric comparison. Allowed values: direct-comparison.
params: Parameters of the metric comparison.
- operator: Comparison operator. Allowed values: eq, gt, ge, lt, le, and ne.
- tolerance: Tolerance for the comparison.
  - type: Type of tolerance. Allowed values: absolute, and relative.
  - value: Tolerance value (numeric).
statistic: Statistical method. Allowed values: min, max, mean, std, shifted_geometric_mean, p01, p05, p10, p25, p50, p75, p90, p95, and p99.

Consider the following example where a metric is defined stating that the mean of the value must be equal between two instances, without any tolerance.

{
  "field": "value",
  "metric_type": "direct-comparison",
  "params": {
    "operator": "eq",
    "tolerance": {"type": "absolute", "value": 0}
  },
  "statistic": "mean"
}

Here is an example command that uses the metric defined above to create a new acceptance test for an application, comparing the production and staging instances. The data is loaded from an input set.

BashOutput

METRIC='{
  "field": "value",
  "metric_type": "direct-comparison",
  "params": {
    "operator": "eq",
    "tolerance": {"type": "absolute", "value": 0}
  },
  "statistic": "mean"
}'
nextmv cloud acceptance create \
    --app-id uncanny-rodent \
    --metrics "$METRIC" \
    --baseline-instance-id production \
    --candidate-instance-id staging \
    --input-set-id burrowing-hares

{
  "acceptance_test_id": "acceptance-yy2xolxs"
}

The command above will create a random ID, and use the same identifier for the acceptance test's name. The name of the test is used as a human-readable You can use the --acceptance-test-id and/or --name options to specify a custom ID and name for the acceptance test. For example:

BashOutput

nextmv cloud acceptance create \
    --app-id uncanny-rodent \
    --metrics "$METRIC" \
    --baseline-instance-id production \
    --candidate-instance-id staging \
    --input-set-id input-set-f8ao98fo \
    --acceptance-test-id fluffy-acceptance-test \
    --name "Acceptance test for a fluffy bunny"

{
  "acceptance_test_id": "fluffy-acceptance-test"
}

Get an acceptance test¶

Info

The best way to view and interact with acceptance test results is in the Nextmv Console.

You can get the results using the nextmv cloud acceptance get command. The output includes the results for the evaluated metrics.

BashOutput

nextmv cloud acceptance get --app-id uncanny-rodent --acceptance-test-id fluffy-acceptance-test

⏳ Getting acceptance test...
{
  "id": "fluffy-acceptance-test",
  "name": "Acceptance test for a fluffy bunny",
  "description": "",
  "created_at": "2026-07-26T19:31:49.403359Z",
  "updated_at": "2026-07-26T19:31:49.403359Z",
  "app_id": "uncanny-rodent",
  "experiment_id": "fluffy-acceptance-test",
  "control": {
    "instance_id": "production",
    "version_id": "v0.0.2"
  },
  "candidate": {
    "instance_id": "staging",
    "version_id": "v0.0.2"
  },
  "metrics": [
    {
      "field": "value",
      "metric_type": "direct-comparison",
      "params": {
        "operator": "eq",
        "tolerance": {
          "type": "absolute",
          "value": 0.0
        }
      },
      "statistic": "mean"
    }
  ],
  "status": "completed",
  "results": {
    "passed": true,
    "metric_results": [
      {
        "metric": {
          "field": "value",
          "metric_type": "direct-comparison",
          "params": {
            "operator": "eq",
            "tolerance": {
              "type": "absolute",
              "value": 0.0
            }
          },
          "statistic": "mean"
        },
        "statistics": {
          "control": {
            "instance_id": "production",
            "version_id": "v0.0.2",
            "number_of_runs_total": 3,
            "distribution_summary_statistics": {
              "min": 1.23,
              "max": 1.23,
              "count": 3,
              "mean": 1.23,
              "std": 0.0,
              "shifted_geometric_mean": 1.2300000000000022,
              "shift_parameter": 10.0
            },
            "distribution_percentiles": {
              "p01": 1.23,
              "p05": 1.23,
              "p10": 1.23,
              "p25": 1.23,
              "p50": 1.23,
              "p75": 1.23,
              "p90": 1.23,
              "p95": 1.23,
              "p99": 1.23
            }
          },
          "candidate": {
            "instance_id": "staging",
            "version_id": "v0.0.2",
            "number_of_runs_total": 3,
            "distribution_summary_statistics": {
              "min": 1.23,
              "max": 1.23,
              "count": 3,
              "mean": 1.23,
              "std": 0.0,
              "shifted_geometric_mean": 1.2300000000000022,
              "shift_parameter": 10.0
            },
            "distribution_percentiles": {
              "p01": 1.23,
              "p05": 1.23,
              "p10": 1.23,
              "p25": 1.23,
              "p50": 1.23,
              "p75": 1.23,
              "p90": 1.23,
              "p95": 1.23,
              "p99": 1.23
            }
          }
        },
        "passed": true
      }
    ]
  }
}

You can list all acceptance tests in the application using the nextmv cloud acceptance list command.

BashOutput

nextmv cloud acceptance list --app-id uncanny-rodent

⏳ Listing acceptance tests...
[
  {
    "id": "fluffy-acceptance-test",
    "name": "Acceptance test for a fluffy bunny",
    "description": "",
    "created_at": "2026-07-26T19:31:49.403359Z",
    "updated_at": "2026-07-26T19:31:49.403359Z",
    "app_id": "uncanny-rodent",
    "experiment_id": "fluffy-acceptance-test",
    "control": {
      "instance_id": "production",
      "version_id": "v0.0.2"
    },
    "candidate": {
      "instance_id": "staging",
      "version_id": "v0.0.2"
    },
    "metrics": [
      {
        "field": "value",
        "metric_type": "direct-comparison",
        "params": {
          "operator": "eq",
          "tolerance": {
            "type": "absolute",
            "value": 0.0
          }
        },
        "statistic": "mean"
      }
    ],
    "status": "unknown"
  },
  ...
  {
    "id": "acceptance-2as29cvk",
    "name": "acceptance-2as29cvk",
    "description": "",
    "created_at": "2026-07-26T19:18:15.331334Z",
    "updated_at": "2026-07-26T19:18:15.331334Z",
    "app_id": "uncanny-rodent",
    "experiment_id": "acceptance-2as29cvk",
    "control": {
      "instance_id": "production",
      "version_id": "v0.0.2"
    },
    "candidate": {
      "instance_id": "staging",
      "version_id": "v0.0.2"
    },
    "metrics": [
      {
        "field": "value",
        "metric_type": "direct-comparison",
        "params": {
          "operator": "eq",
          "tolerance": {
            "type": "absolute",
            "value": 0.0
          }
        },
        "statistic": "mean"
      }
    ],
    "status": "unknown"
  }
]

Update an acceptance test¶

You can update attributes of an acceptance test with the nextmv cloud acceptance update command, such as its:

Name
Description

Please use the --help option on the nextmv cloud acceptance update command to see all the available options for updating an acceptance test.

You cannot update the ID of an acceptance test.

BashOutput

nextmv cloud acceptance update \
    --app-id uncanny-rodent \
    --acceptance-test-id fluffy-acceptance-test \
    --name "Updated Acceptance Test Name" \
    --description "Updated description for the acceptance test"

⏳ Updating acceptance test...
✅ Acceptance test fluffy-acceptance-test updated successfully in application uncanny-rodent.
{
  "id": "fluffy-acceptance-test",
  "name": "Updated Acceptance Test Name",
  "description": "Updated description for the acceptance test",
  "created_at": "2026-07-26T19:31:49.403359Z",
  "updated_at": "2026-07-26T19:37:15.350565Z",
  "status": "unknown"
}

Delete an acceptance test¶

Warning

Deleting an acceptance test is irreversible. All the data associated with the acceptance test will be permanently deleted.

Delete an acceptance test using the nextmv cloud acceptance delete command. The CLI will prompt you to confirm deletion but you can override it with the --yes option.

BashOutput

nextmv cloud acceptance delete --app-id uncanny-rodent --acceptance-test-id fluffy-acceptance-test

Are you sure you want to delete acceptance test fluffy-acceptance-test from application uncanny-rodent? This action cannot be undone.
💡 Confirm Yes
✅ Acceptance test fluffy-acceptance-test deleted successfully from application uncanny-rodent.