Running acceptance tests¶

Info

Learn the concepts and fundamentals of acceptance tests in the Explanation page.

Acceptance tests are formal tests that verify if a system satisfies business requirements.

Included in this how-to guide are instructions on how to interact with acceptance tests using the /v1/applications/{application_id}/experiments/acceptance endpoints. Go the reference section to see all the available parameters for each endpoint.

Create an acceptance test¶

An acceptance test requires metrics to determine the pass/fail criteria. Pass the metrics field in the request payload as a list of metric objects.

Each metric has the following attributes:

field: Field of the metric to measure (e.g., "result.custom.unassigned").
metric_type: Type of metric comparison. Allowed values: direct-comparison.
params: A metric params object, with:
- operator: Comparison operator. Allowed values: eq, gt, ge, lt, le, and ne.
- tolerance: Tolerance for the comparison, with:
  - type: Type of tolerance. Allowed values: absolute, and relative.
  - value: Tolerance value (numeric).
statistic: Statistical method used to summarize the metric. Allowed values: min, max, mean, std, shifted_geometric_mean, p01, p05, p10, p25, p50, p75, p90, p95, and p99.

Consider the following example where a metric is defined stating that the mean of the value must be equal between two instances, without any tolerance.

{
  "field": "value",
  "metric_type": "direct-comparison",
  "params": {
    "operator": "eq",
    "tolerance": {"type": "absolute", "value": 0}
  },
  "statistic": "mean"
}

An acceptance test relies on an underlying batch experiment that shares the same ID. Use the POST /v1/applications/{application_id}/experiments/batch endpoint to create it, specifying the instance_ids to compare and the input set to test against. Afterwards, use the POST /v1/applications/{application_id}/experiments/acceptance endpoint, referencing that batch experiment's ID in the experiment_id field, to create the acceptance test itself.

Here is an example that uses the metric defined above to create a new acceptance test for an application, comparing the production and staging instances. The data is loaded from an input set.

BashOutput

TEST_ID="acceptance-$(openssl rand -hex 4)"

curl -s -X POST "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/batch" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  -H "Content-Type: application/json" \
  -d "{\"id\": \"${TEST_ID}\", \"name\": \"${TEST_ID}\", \"input_set_id\": \"leaping-hares\", \"instance_ids\": [\"staging\", \"production\"]}" \
  | jq '.'

curl -s -X POST "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/acceptance" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  -H "Content-Type: application/json" \
  -d "{\"id\": \"${TEST_ID}\", \"name\": \"${TEST_ID}\", \"experiment_id\": \"${TEST_ID}\", \"candidate\": {\"instance_id\": \"staging\"}, \"control\": {\"instance_id\": \"production\"}, \"metrics\": [{\"field\": \"value\", \"metric_type\": \"direct-comparison\", \"params\": {\"operator\": \"eq\", \"tolerance\": {\"type\": \"absolute\", \"value\": 0}}, \"statistic\": \"mean\"}]}" \
  | jq '.'

{
  "id": "acceptance-fe717ff5"
}
{
  "id": "acceptance-fe717ff5",
  "name": "acceptance-fe717ff5",
  "description": "",
  "app_id": "uncanny-rodent",
  "experiment_id": "acceptance-fe717ff5",
  "control": {
    "instance_id": "production",
    "version_id": "v0.0.2"
  },
  "candidate": {
    "instance_id": "staging",
    "version_id": "v0.0.2"
  },
  "metrics": [
    {
      "field": "value",
      "statistic": "mean",
      "metric_type": "direct-comparison",
      "params": {
        "operator": "eq",
        "tolerance": {
          "type": "absolute",
          "value": 0
        }
      }
    }
  ],
  "results_status": "",
  "created_at": "2026-07-29T14:35:08.895232419Z",
  "updated_at": "2026-07-29T14:35:08.895232419Z"
}

The call above will create a random ID, and use the same identifier for the acceptance test's name. The name of the test is used as a human-readable label. You can pass different values for the id and name fields in the payload to specify a custom ID and name for the acceptance test. For example:

BashOutput

TEST_ID="dusty-acceptance-test"

curl -s -X POST "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/batch" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  -H "Content-Type: application/json" \
  -d "{\"id\": \"${TEST_ID}\", \"name\": \"Acceptance test for a dusty bunny\", \"input_set_id\": \"leaping-hares\", \"instance_ids\": [\"staging\", \"production\"]}" \
  | jq '.'

curl -s -X POST "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/acceptance" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  -H "Content-Type: application/json" \
  -d "{\"id\": \"${TEST_ID}\", \"name\": \"Acceptance test for a dusty bunny\", \"experiment_id\": \"${TEST_ID}\", \"candidate\": {\"instance_id\": \"staging\"}, \"control\": {\"instance_id\": \"production\"}, \"metrics\": [{\"field\": \"value\", \"metric_type\": \"direct-comparison\", \"params\": {\"operator\": \"eq\", \"tolerance\": {\"type\": \"absolute\", \"value\": 0}}, \"statistic\": \"mean\"}]}" \
  | jq '.'

{
  "id": "dusty-acceptance-test"
}
{
  "id": "dusty-acceptance-test",
  "name": "Acceptance test for a dusty bunny",
  "description": "",
  "app_id": "uncanny-rodent",
  "experiment_id": "dusty-acceptance-test",
  "control": {
    "instance_id": "production",
    "version_id": "v0.0.2"
  },
  "candidate": {
    "instance_id": "staging",
    "version_id": "v0.0.2"
  },
  "metrics": [
    {
      "field": "value",
      "statistic": "mean",
      "metric_type": "direct-comparison",
      "params": {
        "operator": "eq",
        "tolerance": {
          "type": "absolute",
          "value": 0
        }
      }
    }
  ],
  "results_status": "",
  "created_at": "2026-07-29T14:35:51.560363497Z",
  "updated_at": "2026-07-29T14:35:51.560363497Z"
}

Get an acceptance test¶

Info

The best way to view and interact with acceptance test results is in the Nextmv Console.

You can get the results using the GET /v1/applications/{application_id}/experiments/acceptance/{acceptance_id} endpoint. The response includes the results for the evaluated metrics once the underlying batch experiment finishes running.

BashOutput

curl -s -X GET "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/acceptance/dusty-acceptance-test" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  | jq '.'

{
  "id": "dusty-acceptance-test",
  "name": "Acceptance test for a dusty bunny",
  "description": "",
  "app_id": "uncanny-rodent",
  "experiment_id": "dusty-acceptance-test",
  "control": {
    "instance_id": "production",
    "version_id": "v0.0.2"
  },
  "candidate": {
    "instance_id": "staging",
    "version_id": "v0.0.2"
  },
  "metrics": [
    {
      "field": "value",
      "statistic": "mean",
      "metric_type": "direct-comparison",
      "params": {
        "operator": "eq",
        "tolerance": {
          "type": "absolute",
          "value": 0
        }
      }
    }
  ],
  "created_at": "2026-07-29T14:35:51.560363497Z",
  "updated_at": "2026-07-29T14:36:23.246843508Z",
  "status": "completed",
  "results": {
    "passed": true,
    "metric_results": [
      {
        "metric": {
          "field": "value",
          "statistic": "mean",
          "metric_type": "direct-comparison",
          "params": {
            "operator": "eq",
            "tolerance": {
              "type": "absolute",
              "value": 0
            }
          }
        },
        "statistics": {
          "control": {
            "instance_id": "production",
            "version_id": "v0.0.2",
            "number_of_runs_total": 3,
            "distribution_summary_statistics": {
              "min": 1.23,
              "max": 1.23,
              "count": 3,
              "mean": 1.23,
              "std": 0,
              "shifted_geometric_mean": 1.2300000000000022,
              "shift_parameter": 10
            },
            "distribution_percentiles": {
              "p01": 1.23,
              "p05": 1.23,
              "p10": 1.23,
              "p25": 1.23,
              "p50": 1.23,
              "p75": 1.23,
              "p90": 1.23,
              "p95": 1.23,
              "p99": 1.23
            }
          },
          "candidate": {
            "instance_id": "staging",
            "version_id": "v0.0.2",
            "number_of_runs_total": 3,
            "distribution_summary_statistics": {
              "min": 1.23,
              "max": 1.23,
              "count": 3,
              "mean": 1.23,
              "std": 0,
              "shifted_geometric_mean": 1.2300000000000022,
              "shift_parameter": 10
            },
            "distribution_percentiles": {
              "p01": 1.23,
              "p05": 1.23,
              "p10": 1.23,
              "p25": 1.23,
              "p50": 1.23,
              "p75": 1.23,
              "p90": 1.23,
              "p95": 1.23,
              "p99": 1.23
            }
          }
        },
        "metric_pairs": {
          "latest-RUM_rQEDg": [1.23, 1.23],
          "latest-bdIl9QEDR": [1.23, 1.23],
          "latest-pmN_9wEvg": [1.23, 1.23]
        },
        "passed": true
      }
    ]
  }
}

You can list all acceptance tests in the application using the GET /v1/applications/{application_id}/experiments/acceptance endpoint. This endpoint is paginated, so the snippet below always uses pagination: it passes pagereturn=true to receive a next_page_token in the response, and keeps requesting pages by passing that token back as pagetoken until no token is returned.

BashOutput

ITEMS="[]"
PAGE_TOKEN=""

while :; do
  RESPONSE=$(curl -s -G "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/acceptance" \
    -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
    --data-urlencode "pagereturn=true" \
    --data-urlencode "pagetoken=${PAGE_TOKEN}")

  ITEMS=$(jq -n --argjson existing "$ITEMS" --argjson page "$(echo "$RESPONSE" | jq '.items')" '$existing + $page')
  PAGE_TOKEN=$(echo "$RESPONSE" | jq -r '.next_page_token // empty')

  [ -z "$PAGE_TOKEN" ] && break
done

echo "$ITEMS" | jq '.[:2]'

[
  {
    "id": "dusty-acceptance-test",
    "name": "Acceptance test for a dusty bunny",
    "description": "",
    "app_id": "uncanny-rodent",
    "experiment_id": "dusty-acceptance-test",
    "control": {
      "instance_id": "production",
      "version_id": "v0.0.2"
    },
    "candidate": {
      "instance_id": "staging",
      "version_id": "v0.0.2"
    },
    "metrics": [
      {
        "field": "value",
        "statistic": "mean",
        "metric_type": "direct-comparison",
        "params": {
          "operator": "eq",
          "tolerance": {
            "type": "absolute",
            "value": 0
          }
        }
      }
    ],
    "results_status": "",
    "created_at": "2026-07-29T14:35:51.560363497Z",
    "updated_at": "2026-07-29T14:35:51.560363497Z"
  },
  {
    "id": "acceptance-fe717ff5",
    "name": "acceptance-fe717ff5",
    "description": "",
    "app_id": "uncanny-rodent",
    "experiment_id": "acceptance-fe717ff5",
    "control": {
      "instance_id": "production",
      "version_id": "v0.0.2"
    },
    "candidate": {
      "instance_id": "staging",
      "version_id": "v0.0.2"
    },
    "metrics": [
      {
        "field": "value",
        "statistic": "mean",
        "metric_type": "direct-comparison",
        "params": {
          "operator": "eq",
          "tolerance": {
            "type": "absolute",
            "value": 0
          }
        }
      }
    ],
    "results_status": "",
    "created_at": "2026-07-29T14:35:08.895232419Z",
    "updated_at": "2026-07-29T14:35:08.895232419Z"
  }
]

Update an acceptance test¶

You can update attributes of an acceptance test with the PATCH /v1/applications/{application_id}/experiments/acceptance/{acceptance_id} endpoint, such as its:

Name
Description

The endpoint returns the updated acceptance test. You cannot update the ID of an acceptance test.

BashOutput

curl -s -X PATCH "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/acceptance/dusty-acceptance-test" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"name": "Updated Acceptance Test Name", "description": "Updated description for the acceptance test"}' \
  | jq '.'

{
  "id": "dusty-acceptance-test",
  "name": "Updated Acceptance Test Name",
  "description": "Updated description for the acceptance test",
  "created_at": "2026-07-29T14:35:51.560363497Z",
  "updated_at": "2026-07-29T14:36:23.246843508Z"
}

Delete an acceptance test¶

Warning

Deleting an acceptance test is irreversible. All the data associated with the acceptance test will be permanently deleted.

Delete an acceptance test using the DELETE /v1/applications/{application_id}/experiments/acceptance/{acceptance_id} endpoint.

Bash

curl -s -X DELETE "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/acceptance/dusty-acceptance-test" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}"