Skip to content

Running shadow tests

Info

Learn the concepts and fundamentals of shadow tests in the Explanation page.

A shadow test is an experiment that runs in the background and compares the results of two instances: baseline vs. candidate. When the shadow test has started, any run made on the baseline instance will trigger a run on the candidate instance using the same input and options. The results of the shadow test are often used to determine if a new version of a model is ready to be promoted to production.

Included in this how-to guide are instructions on how to interact with shadow tests using methods on the Application class. Go the reference section to see all the available parameters for each method.

Create a shadow test

Use the Application.new_shadow_test method to create a new shadow test for an application. A shadow test requires comparisons to be defined, via the comparisons keyword argument. Additionally, you must specify the maximum number of runs that the test will run for, using the termination_events keyword argument as a TerminationEvents object with the maximum_runs attribute.

Comparisons are defined as a dictionary. The keys of the comparisons dictionary are the baseline instance IDs, and the values are the candidate lists of instance IDs to compare against the respective baseline. Consider the following example for doing a simple comparison of the staging vs. production instances:

comparisons = {"production": ["staging"]}

Here is an example that uses the comparison defined above to create a new shadow test for an application.

import os

import nextmv
from nextmv import cloud
from nextmv.cloud import TerminationEvents

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
shadow_test = app.new_shadow_test(
    comparisons={"production": ["staging"]},
    termination_events=TerminationEvents(maximum_runs=10),
)

nextmv.write(shadow_test.to_dict())

uv run main.py

{
  "id": "shadow-dm77fy97",
  "name": "shadow-dm77fy97",
  "description": "",
  "app_id": "uncanny-rodent",
  "created_at": "2026-07-29T04:03:41.591708Z",
  "updated_at": "2026-07-29T04:03:41.591708Z",
  "status": "draft",
  "comparisons": [
    {
      "baseline_instance_id": "production",
      "candidate_instance_ids": [
        "staging"
      ]
    }
  ],
  "start_events": {},
  "termination_events": {
    "maximum_runs": 10
  }
}

The call above will create a random ID, and use the same identifier for the shadow test's name. The name of the test is used as a human-readable label. You can use the shadow_test_id and/or name keyword arguments to specify a custom ID and name for the shadow test. For example:

import os

import nextmv
from nextmv import cloud
from nextmv.cloud import TerminationEvents

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
shadow_test = app.new_shadow_test(
    comparisons={"production": ["staging"]},
    termination_events=TerminationEvents(maximum_runs=10),
    shadow_test_id="giant-shadow-test",
    name="Giant Shadow Test",
)

nextmv.write(shadow_test.to_dict())

uv run main.py

{
  "id": "giant-shadow-test",
  "name": "Giant Shadow Test",
  "description": "",
  "app_id": "uncanny-rodent",
  "created_at": "2026-07-29T04:03:46.649096Z",
  "updated_at": "2026-07-29T04:03:46.649096Z",
  "status": "draft",
  "comparisons": [
    {
      "baseline_instance_id": "production",
      "candidate_instance_ids": [
        "staging"
      ]
    }
  ],
  "start_events": {},
  "termination_events": {
    "maximum_runs": 10
  }
}

The Application.new_shadow_test method creates a shadow test in draft mode. Once the test is created, you must start it.

Start a shadow test

As mentioned above, when a shadow test is created, it is in draft mode. A shadow test must be started so that it can begin executing runs and collecting data. There are two ways to start a shadow test:

As an example, here is how to start a shadow test using the Application.start_shadow_test method:

import os

from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
app.start_shadow_test(shadow_test_id="giant-shadow-test")

uv run main.py

Once a shadow test has started, you can stop it at any time or wait for it to complete based on the termination events defined when the test was created.

Get a shadow test

Info

The best way to view and interact with shadow test results is in the Nextmv Console.

Use the Application.shadow_test_metadata method to retrieve the metadata for a shadow test, using the shadow test ID. The method returns a ShadowTestMetadata object.

import os

import nextmv
from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
metadata = app.shadow_test_metadata(shadow_test_id="giant-shadow-test")

nextmv.write(metadata.to_dict())

uv run main.py

{
  "id": "giant-shadow-test",
  "name": "Giant Shadow Test",
  "description": "",
  "app_id": "uncanny-rodent",
  "created_at": "2026-07-29T04:03:46Z",
  "updated_at": "2026-07-29T04:04:00Z",
  "status": "started"
}

Once the status of the shadow test is completed, you can get the results using the Application.shadow_test method. The method returns a ShadowTest object, whose output includes the runs that were made for the test. If the test hasn't completed, you can still get partial results.

A shadow test that has already started can be stopped at any time, or it will stop automatically when the termination events are fulfilled. Once a shadow test is stopped, then its status will move to completed.

import os

import nextmv
from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
shadow_test = app.shadow_test(shadow_test_id="giant-shadow-test")

nextmv.write(shadow_test.to_dict())

uv run main.py

{
  "id": "giant-shadow-test",
  "name": "Giant Shadow Test",
  "description": "",
  "app_id": "uncanny-rodent",
  "created_at": "2026-07-29T04:03:46.649096Z",
  "updated_at": "2026-07-29T04:04:21.873630Z",
  "status": "started",
  "comparisons": [
    {
      "baseline_instance_id": "production",
      "candidate_instance_ids": [
        "staging"
      ]
    }
  ],
  "start_events": {},
  "termination_events": {
    "maximum_runs": 10
  },
  "grouped_distributional_summaries": [
    {
      "group_keys": [
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "staging",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 2,
          "mean": 1.23,
          "std": 0,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 2
    },
    {
      "group_keys": [
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "production",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 2,
          "mean": 1.23,
          "std": 0,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 2
    }
  ],
  "runs": [
    {
      "id": "production-8jV6UUPDg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-29T04:04:20.536496Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": true,
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.377
          }
        ]
      },
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "version"
        }
      ]
    },
    {
      "id": "staging-YMSe8UPvR",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-29T04:04:20.757240Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "staging",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.373
          }
        ]
      },
      "options": {
        "details": "true"
      },
      "request_options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ]
    },
    {
      "id": "production-imHe88EvR",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-29T04:04:21.698655Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": true,
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.456
          }
        ]
      },
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "version"
        }
      ]
    },
    {
      "id": "staging-9IHeU8Pvg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-29T04:04:21.884426Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "staging",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 5.81
          }
        ]
      },
      "options": {
        "details": "true"
      },
      "request_options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ]
    }
  ]
}

You can list all shadow tests in the application using the Application.list_shadow_tests method.

import json
import os

from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
shadow_tests = app.list_shadow_tests()

print(json.dumps([shadow_test.to_dict() for shadow_test in shadow_tests[:2]], indent=2))

uv run main.py

[
  {
    "id": "giant-shadow-test",
    "name": "Giant Shadow Test",
    "description": "",
    "app_id": "uncanny-rodent",
    "created_at": "2026-07-29T04:03:46.649096Z",
    "updated_at": "2026-07-29T04:04:21.873630Z",
    "status": "started",
    "comparisons": [
      {
        "baseline_instance_id": "production",
        "candidate_instance_ids": [
          "staging"
        ]
      }
    ],
    "start_events": {},
    "termination_events": {
      "maximum_runs": 10
    }
  },
  {
    "id": "shadow-dm77fy97",
    "name": "shadow-dm77fy97",
    "description": "",
    "app_id": "uncanny-rodent",
    "created_at": "2026-07-29T04:03:41.591708Z",
    "updated_at": "2026-07-29T04:03:41.591708Z",
    "status": "draft",
    "comparisons": [
      {
        "baseline_instance_id": "production",
        "candidate_instance_ids": [
          "staging"
        ]
      }
    ],
    "start_events": {},
    "termination_events": {
      "maximum_runs": 10
    }
  }
]

Stop a shadow test

A shadow test will continue to execute until it stops, which is equivalent to it being completed. There are two ways to stop a shadow test:

When using the Application.stop_shadow_test method, you must use the intent keyword argument (a StopIntent value) to track if you want to complete or cancel the shadow test. As an example, here is how to stop a shadow test and mark it as completed:

import os

from nextmv import cloud
from nextmv.cloud import StopIntent

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
app.stop_shadow_test(shadow_test_id="giant-shadow-test", intent=StopIntent.COMPLETE)

uv run main.py

Update a shadow test

You can update attributes of a shadow test with the Application.update_shadow_test method, such as its:

  • Name
  • Description

The method returns a ShadowTest object. You cannot update the ID of a shadow test.

import os

import nextmv
from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
shadow_test = app.update_shadow_test(
    shadow_test_id="giant-shadow-test",
    name="Updated Shadow Test Name",
    description="Updated description for the shadow test",
)

nextmv.write(shadow_test.to_dict())

uv run main.py

{
  "id": "giant-shadow-test",
  "name": "Updated Shadow Test Name",
  "description": "Updated description for the shadow test",
  "created_at": "2026-07-29T04:03:46.649096Z",
  "updated_at": "2026-07-29T04:04:59.822417Z"
}

Delete a shadow test

Warning

Deleting a shadow test is irreversible. All the data associated with the shadow test will be permanently deleted.

Delete a shadow test using the Application.delete_shadow_test method.

import os

from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
app.delete_shadow_test(shadow_test_id="giant-shadow-test")

uv run main.py