Running shadow tests¶

Info

Learn the concepts and fundamentals of shadow tests in the Explanation page.

A shadow test is an experiment that runs in the background and compares the results of two instances: baseline vs. candidate. When the shadow test has started, any run made on the baseline instance will trigger a run on the candidate instance using the same input and options. The results of the shadow test are often used to determine if a new version of a model is ready to be promoted to production.

This how-to guide explains how to interact with shadow tests using the /v1/applications/{application_id}/experiments/shadow endpoints. Go to the reference section to see all the available parameters for each endpoint.

Create a shadow test¶

Use the POST /v1/applications/{application_id}/experiments/shadow endpoint to create a new shadow test for an application. A shadow test requires comparisons to be defined, via the comparisons field in the request payload. Additionally, you must specify the maximum number of runs that the test will run for, using the termination_events field with the maximum_runs attribute.

Comparisons are defined as an object. The keys of the comparisons object are the baseline instance IDs, and the values are the candidate lists of instance IDs to compare against the respective baseline. Consider the following example for doing a simple comparison of the staging vs. production instances:

{"production": ["staging"]}

Here is an example that uses the comparison defined above to create a new shadow test for an application. The endpoint requires an id in the request payload, so generate one yourself if you don't want to choose a custom one:

BashOutput

SHADOW_TEST_ID="shadow-$(openssl rand -hex 4)"

curl -s -X POST "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/shadow" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  -H "Content-Type: application/json" \
  -d "{\"id\": \"${SHADOW_TEST_ID}\", \"name\": \"${SHADOW_TEST_ID}\", \"comparisons\": {\"production\": [\"staging\"]}, \"termination_events\": {\"maximum_runs\": 10}}" \
  | jq '.'

{
  "id": "shadow-1f91988a",
  "name": "shadow-1f91988a",
  "description": "",
  "app_id": "uncanny-rodent",
  "created_at": "2026-07-29T06:46:12.86899398Z",
  "updated_at": "2026-07-29T06:46:12.86899398Z",
  "status": "draft",
  "comparisons": [
    {
      "baseline_instance_id": "production",
      "candidate_instance_ids": [
        "staging"
      ]
    }
  ],
  "start_events": {},
  "termination_events": {
    "maximum_runs": 10
  }
}

The call above uses a randomly generated ID, and the same identifier for the shadow test's name. The name of the test is used as a human-readable label. You can pass different values for the id and name fields in the payload to specify a custom ID and name for the shadow test. For example:

BashOutput

curl -s -X POST "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/shadow" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"id": "giant-shadow-test", "name": "Giant Shadow Test", "comparisons": {"production": ["staging"]}, "termination_events": {"maximum_runs": 10}}' \
  | jq '.'

{
  "id": "giant-shadow-test",
  "name": "Giant Shadow Test",
  "description": "",
  "app_id": "uncanny-rodent",
  "created_at": "2026-07-29T06:46:16.694167016Z",
  "updated_at": "2026-07-29T06:46:16.694167016Z",
  "status": "draft",
  "comparisons": [
    {
      "baseline_instance_id": "production",
      "candidate_instance_ids": [
        "staging"
      ]
    }
  ],
  "start_events": {},
  "termination_events": {
    "maximum_runs": 10
  }
}

The POST /v1/applications/{application_id}/experiments/shadow endpoint creates a shadow test in draft mode. Once the test is created, you must start it.

Start a shadow test¶

As mentioned above, when a shadow test is created, it is in draft mode. A shadow test must be started so that it can begin executing runs and collecting data. There are two ways to start a shadow test:

Using the start_events field in the request payload of the POST /v1/applications/{application_id}/experiments/shadow endpoint. This will start the shadow test at the scheduled time.
Using the PUT /v1/applications/{application_id}/experiments/shadow/{shadow_id}/start endpoint. This will start the shadow test as soon as the endpoint is called.

As an example, here is how to start a shadow test using the PUT /v1/applications/{application_id}/experiments/shadow/{shadow_id}/start endpoint:

Bash

curl -s -X PUT "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/shadow/giant-shadow-test/start" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}"

Once a shadow test has started, you can stop it at any time or wait for it to complete based on the termination events defined when the test was created.

Get a shadow test¶

Info

The best way to view and interact with shadow test results is in the Nextmv Console.

Use the GET /v1/applications/{application_id}/experiments/shadow/{shadow_id}/metadata endpoint to retrieve the metadata for a shadow test, using the shadow test ID.

BashOutput

curl -s -X GET "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/shadow/giant-shadow-test/metadata" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  | jq '.'

{
  "id": "giant-shadow-test",
  "name": "Giant Shadow Test",
  "description": "",
  "app_id": "uncanny-rodent",
  "created_at": "2026-07-29T06:46:16Z",
  "updated_at": "2026-07-29T06:46:19Z",
  "status": "started"
}

Once the status of the shadow test is completed, you can get the results using the GET /v1/applications/{application_id}/experiments/shadow/{shadow_id} endpoint together with the GET /v1/applications/{application_id}/experiments/shadow/{shadow_id}/runs endpoint, whose output includes the runs that were made for the test. If the test hasn't completed, you can still get partial results.

A shadow test that has already started can be stopped at any time, or it will stop automatically when the termination events are fulfilled. Once a shadow test is stopped, then its status will move to completed.

The snippet below fetches the shadow test, fetches its runs, and uses jq to merge the runs into the shadow test object under the runs key.

BashOutput

curl -s -X GET "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/shadow/giant-shadow-test" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  | jq --argjson runs "$(curl -s -X GET "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/shadow/giant-shadow-test/runs" -H "Authorization: Bearer ${NEXTMV_API_KEY}" | jq '.runs')" \
    '. + {runs: $runs}'

{
  "id": "giant-shadow-test",
  "name": "Giant Shadow Test",
  "description": "",
  "app_id": "uncanny-rodent",
  "created_at": "2026-07-29T06:46:16.694167016Z",
  "updated_at": "2026-07-29T06:46:27.738235339Z",
  "status": "started",
  "comparisons": [
    {
      "baseline_instance_id": "production",
      "candidate_instance_ids": [
        "staging"
      ]
    }
  ],
  "start_events": {},
  "termination_events": {
    "maximum_runs": 10
  },
  "grouped_distributional_summaries": [
    {
      "group_keys": [
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "staging",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 2,
          "mean": 1.23,
          "std": 0,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 2
    },
    {
      "group_keys": [
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "production",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 2,
          "mean": 1.23,
          "std": 0,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 2
    }
  ],
  "runs": [
    {
      "id": "production-BwgNmQEDg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-29T06:46:26.356589483Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "type": "",
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "queuing_priority": 6,
      "queuing_disabled": true,
      "runtime": "python-3_11",
      "status": "succeeded",
      "status_v2": "succeeded",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.516
          }
        ]
      },
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "version"
        }
      ]
    },
    {
      "id": "staging-SZzNiwEDR",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-29T06:46:26.663568349Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "staging",
      "application_version_id": "v0.0.2",
      "run_type": {
        "type": "",
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "runtime": "python-3_11",
      "status": "succeeded",
      "status_v2": "succeeded",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 3.738
          }
        ]
      },
      "options": {
        "details": "true"
      },
      "request_options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ]
    },
    {
      "id": "production-eqzNmwEvg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-29T06:46:27.551161905Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "type": "",
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "queuing_priority": 6,
      "queuing_disabled": true,
      "runtime": "python-3_11",
      "status": "succeeded",
      "status_v2": "succeeded",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 3.81
          }
        ]
      },
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "version"
        }
      ]
    },
    {
      "id": "staging-s7mNiQPDR",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-29T06:46:27.767034805Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "staging",
      "application_version_id": "v0.0.2",
      "run_type": {
        "type": "",
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "runtime": "python-3_11",
      "status": "succeeded",
      "status_v2": "succeeded",
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.414
          }
        ]
      },
      "options": {
        "details": "true"
      },
      "request_options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ]
    }
  ]
}

You can list all shadow tests in the application using the GET /v1/applications/{application_id}/experiments/shadow endpoint. This endpoint is paginated, so the snippet below always uses pagination: it passes pagereturn=true to receive a next_page_token in the response, and keeps requesting pages by passing that token back as pagetoken until no token is returned.

BashOutput

ITEMS="[]"
PAGE_TOKEN=""

while :; do
  RESPONSE=$(curl -s -G "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/shadow" \
    -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
    --data-urlencode "pagereturn=true" \
    --data-urlencode "pagetoken=${PAGE_TOKEN}")

  ITEMS=$(jq -n --argjson existing "$ITEMS" --argjson page "$(echo "$RESPONSE" | jq '.items')" '$existing + $page')
  PAGE_TOKEN=$(echo "$RESPONSE" | jq -r '.next_page_token // empty')

  [ -z "$PAGE_TOKEN" ] && break
done

echo "$ITEMS" | jq '.'

[
  {
    "id": "giant-shadow-test",
    "name": "Giant Shadow Test",
    "description": "",
    "app_id": "uncanny-rodent",
    "created_at": "2026-07-29T06:46:16.694167016Z",
    "updated_at": "2026-07-29T06:46:27.738235339Z",
    "status": "started",
    "comparisons": [
      {
        "baseline_instance_id": "production",
        "candidate_instance_ids": [
          "staging"
        ]
      }
    ],
    "start_events": {},
    "termination_events": {
      "maximum_runs": 10
    }
  },
  ...
  {
    "id": "shadow-1f91988a",
    "name": "shadow-1f91988a",
    "description": "",
    "app_id": "uncanny-rodent",
    "created_at": "2026-07-29T06:46:12.86899398Z",
    "updated_at": "2026-07-29T06:46:12.86899398Z",
    "status": "draft",
    "comparisons": [
      {
        "baseline_instance_id": "production",
        "candidate_instance_ids": [
          "staging"
        ]
      }
    ],
    "start_events": {},
    "termination_events": {
      "maximum_runs": 10
    }
  }
]

Stop a shadow test¶

A shadow test will continue to execute until it stops, which is equivalent to it being completed. There are two ways to stop a shadow test:

Using the termination_events field in the request payload of the POST /v1/applications/{application_id}/experiments/shadow endpoint. Termination events will cause the shadow test to stop.
Using the PUT /v1/applications/{application_id}/experiments/shadow/{shadow_id}/stop endpoint. This will stop the shadow test as soon as the endpoint is called.

When using the PUT /v1/applications/{application_id}/experiments/shadow/{shadow_id}/stop endpoint, you must pass the intent field in the request payload (either complete or cancel) to track if you want to complete or cancel the shadow test. As an example, here is how to stop a shadow test and mark it as completed:

Bash

curl -s -X PUT "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/shadow/giant-shadow-test/stop" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"intent": "complete"}'

Update a shadow test¶

You can update attributes of a shadow test with the PATCH /v1/applications/{application_id}/experiments/shadow/{shadow_id} endpoint, such as its:

Name
Description

You cannot update the ID of a shadow test.

BashOutput

curl -s -X PATCH "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/shadow/giant-shadow-test" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"name": "Updated Shadow Test Name", "description": "Updated description for the shadow test"}' \
  | jq '.'

{
  "id": "giant-shadow-test",
  "name": "Updated Shadow Test Name",
  "description": "Updated description for the shadow test",
  "created_at": "2026-07-29T06:46:16.694167016Z",
  "updated_at": "2026-07-29T06:47:25.442016639Z"
}

Delete a shadow test¶

Warning

Deleting a shadow test is irreversible. All the data associated with the shadow test will be permanently deleted.

Delete a shadow test using the DELETE /v1/applications/{application_id}/experiments/shadow/{shadow_id} endpoint.

Bash

curl -s -X DELETE "https://api.cloud.nextmv.io/v1/applications/uncanny-rodent/experiments/shadow/giant-shadow-test" \
  -H "Authorization: Bearer ${NEXTMV_API_KEY}"