Skip to content

Running shadow tests

Info

Learn the concepts and fundamentals of shadow tests in the Explanation page.

A shadow test is an experiment that runs in the background and compares the results of two instances: baseline vs. candidate. When the shadow test has started, any run made on the baseline instance will trigger a run on the candidate instance using the same input and options. The results of the shadow test are often used to determine if a new version of a model is ready to be promoted to production.

Included in this how-to guide are instructions on how to interact with the actions under the nextmv cloud shadow command tree. Go the reference section or use the --help option to see all the available options for each command.

Create a shadow test

Use the nextmv cloud shadow create command to create a new shadow test for an application. A shadow test requires comparisons to be defined, via the --comparisons option. Additionally, you must specify the maximum number of runs that the test will run for, using the --termination-maximum-runs flag.

Comparisons are defined as a JSON object. The keys of the comparisons object are the baseline instance IDs, and the values are the candidate lists of instance IDs to compare against the respective baseline. Consider the following example for doing a simple comparison of the staging vs. production instances:

{
  "production": ["staging"]
}

Here is an example command that uses the comparison defined above to create a new shadow test for an application.

COMPARISON='{
  "production": ["staging"]
}'
nextmv cloud shadow create \
    --app-id uncanny-rodent \
    --comparisons "$COMPARISON" \
    --termination-maximum-runs 10

⏳ Creating shadow test in draft mode...
{
  "id": "shadow-7olanfsl",
  "name": "shadow-7olanfsl",
  "description": "",
  "app_id": "uncanny-rodent",
  "created_at": "2026-07-26T20:35:53.150310Z",
  "updated_at": "2026-07-26T20:35:53.150310Z",
  "status": "draft",
  "comparisons": [
    {
      "baseline_instance_id": "production",
      "candidate_instance_ids": [
        "staging"
      ]
    }
  ],
  "start_events": {},
  "termination_events": {
    "maximum_runs": 10
  }
}

The command above will create a random ID, and use the same identifier for the shadow test's name. The name of the test is used as a human-readable You can use the --shadow-test-id and/or --name options to specify a custom ID and name for the shadow test. For example:

nextmv cloud shadow create \
    --app-id uncanny-rodent \
    --comparisons "$COMPARISON" \
    --termination-maximum-runs 10 \
    --shadow-test-id fluffy-shadow-test \
    --name "Fluffy Shadow Test"

⏳ Creating shadow test in draft mode...
{
  "id": "fluffy-shadow-test",
  "name": "Fluffy Shadow Test",
  "description": "",
  "app_id": "uncanny-rodent",
  "created_at": "2026-07-26T20:37:02.568386Z",
  "updated_at": "2026-07-26T20:37:02.568386Z",
  "status": "draft",
  "comparisons": [
    {
      "baseline_instance_id": "production",
      "candidate_instance_ids": [
        "staging"
      ]
    }
  ],
  "start_events": {},
  "termination_events": {
    "maximum_runs": 10
  }
}

The nextmv cloud shadow create command creates a shadow test in draft mode. Once the test is created, you must start it.

Start a shadow test

As mentioned above, when a shadow test is created, it is in draft mode. A shadow test must be started so that it can begin executing runs and collecting data. There are two ways to start a shadow test:

As an example, here is how to start a shadow test using the nextmv cloud shadow start command:

nextmv cloud shadow start --app-id uncanny-rodent --shadow-test-id fluffy-shadow-test

⏳ Starting shadow test...
✅ Shadow test fluffy-shadow-test started successfully in application uncanny-rodent.

Once a shadow test has started, you can stop it at any time or wait for it to complete based on the termination events defined when the test was created.

Get a shadow test

Info

The best way to view and interact with shadow test results is in the Nextmv Console.

Use the nextmv cloud shadow metadata command to retrieve the metadata for a shadow test, using the shadow test ID.

nextmv cloud shadow metadata --app-id uncanny-rodent --shadow-test-id fluffy-shadow-test

⏳ Getting shadow test metadata...
{
  "id": "fluffy-shadow-test",
  "name": "Fluffy Shadow Test",
  "description": "",
  "app_id": "uncanny-rodent",
  "created_at": "2026-07-26T20:37:02Z",
  "updated_at": "2026-07-26T20:45:55Z",
  "status": "started"
}

Once the status of the shadow test is completed, you can get the results using the nextmv cloud shadow get command. The output includes the runs that were made for the test. If the test hasn't completed, you can still get partial results.

A shadow test that has already started can be stopped at any time, or it will stop automatically when the termination events are fulfilled. Once a shadow test is stopped, then its status will move to completed.

nextmv cloud shadow get --app-id uncanny-rodent --shadow-test-id fluffy-shadow-test

⏳ Getting shadow test...
{
  "id": "fluffy-shadow-test",
  "name": "Fluffy Shadow Test",
  "description": "",
  "app_id": "uncanny-rodent",
  "created_at": "2026-07-26T20:37:02.568386Z",
  "updated_at": "2026-07-26T20:49:09.223748Z",
  "status": "started",
  "comparisons": [
    {
      "baseline_instance_id": "production",
      "candidate_instance_ids": [
        "staging"
      ]
    }
  ],
  "start_events": {},
  "termination_events": {
    "maximum_runs": 10
  },
  "grouped_distributional_summaries": [
    {
      "group_keys": [
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "staging",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 2,
          "mean": 1.23,
          "std": 0,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 2
    },
    {
      "group_keys": [
        "instanceID",
        "versionID"
      ],
      "group_values": [
        "production",
        "v0.0.2"
      ],
      "indicator_keys": [
        "value"
      ],
      "indicator_distributions": {
        "value": {
          "min": 1.23,
          "max": 1.23,
          "count": 2,
          "mean": 1.23,
          "std": 0,
          "shifted_geometric_mean": {
            "value": 1.2300000000000022,
            "shift": 10
          },
          "percentiles": {
            "p01": 1.23,
            "p05": 1.23,
            "p10": 1.23,
            "p25": 1.23,
            "p50": 1.23,
            "p75": 1.23,
            "p90": 1.23,
            "p95": 1.23,
            "p99": 1.23
          }
        }
      },
      "number_of_runs_total": 2
    }
  ],
  "runs": [
    {
      "id": "production-02ENubPDR",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-26T20:49:01.904853Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 6.175
          }
        ]
      },
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "version"
        }
      ]
    },
    {
      "id": "staging-S-ENubPDg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-26T20:49:02.055645Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "staging",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 6.183
          }
        ]
      },
      "options": {
        "details": "true"
      },
      "request_options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ]
    },
    {
      "id": "production-19QNXxPDg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-26T20:49:05.425988Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 7.363
          }
        ]
      },
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "version"
        }
      ]
    },
    {
      "id": "staging-eklNXxPvR",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-26T20:49:05.599041Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "staging",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 7.575
          }
        ]
      },
      "options": {
        "details": "true"
      },
      "request_options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ]
    },
    {
      "id": "production-o5jHuxEvg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-26T20:49:09.007326Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "production",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.773
          }
        ]
      },
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "version"
        }
      ]
    },
    {
      "id": "staging-a--CNubEvg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-26T20:49:09.235929Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "staging",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "running",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "statistics": {
        "status": "unknown",
        "indicators": [
          {
            "name": "metadata.duration",
            "value": 0
          }
        ]
      },
      "options": {
        "details": "true"
      },
      "request_options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "run"
        }
      ]
    }
  ]
}

You can list all shadow tests in the application using the nextmv cloud shadow list command.

nextmv cloud shadow list --app-id uncanny-rodent

⏳ Listing shadow tests...
[
  {
    "id": "fluffy-shadow-test",
    "name": "Fluffy Shadow Test",
    "description": "",
    "app_id": "uncanny-rodent",
    "created_at": "2026-07-26T20:37:02.568386Z",
    "updated_at": "2026-07-26T20:49:09.223748Z",
    "status": "started",
    "comparisons": [
      {
        "baseline_instance_id": "production",
        "candidate_instance_ids": [
          "staging"
        ]
      }
    ],
    "start_events": {},
    "termination_events": {
      "maximum_runs": 10
    }
  },
  {
    "id": "shadow-7olanfsl",
    "name": "shadow-7olanfsl",
    "description": "",
    "app_id": "uncanny-rodent",
    "created_at": "2026-07-26T20:35:53.150310Z",
    "updated_at": "2026-07-26T20:35:53.150310Z",
    "status": "draft",
    "comparisons": [
      {
        "baseline_instance_id": "production",
        "candidate_instance_ids": [
          "staging"
        ]
      }
    ],
    "start_events": {},
    "termination_events": {
      "maximum_runs": 10
    }
  }
]

Stop a shadow test

A shadow test will continue to execute until it stops, which is equivalent to it being completed. There are two ways to stop a shadow test:

  • Using the --termination-maximum-runs and/or --termination-time options with the nextmv cloud shadow create command. Termination events will cause the shadow test to stop.
  • Using the nextmv cloud shadow stop command. This will stop the shadow test as soon as the command is executed.

When using the nextmv cloud shadow stop command, you must use the --intent flag to track if you want to complete or cancel the shadow test. As an example, here is how to stop a shadow test and mark it as completed:

nextmv cloud shadow stop --app-id uncanny-rodent --shadow-test-id fluffy-shadow-test --intent complete

⏳ Stopping shadow test...
✅ Shadow test fluffy-shadow-test stopped successfully in application uncanny-rodent.

Update a shadow test

You can update attributes of a shadow test with the nextmv cloud shadow update command, such as its:

  • Name
  • Description

Please use the --help option on the nextmv cloud shadow update command to see all the available options for updating a shadow test.

You cannot update the ID of a shadow test.

nextmv cloud shadow update \
    --app-id uncanny-rodent \
    --shadow-test-id fluffy-shadow-test \
    --name "Updated Shadow Test Name" \
    --description "Updated description for the shadow test"

⏳ Updating shadow test...
✅ Shadow test fluffy-shadow-test updated successfully in application uncanny-rodent.
{
  "id": "fluffy-shadow-test",
  "name": "Updated Shadow Test Name",
  "description": "Updated description for the shadow test",
  "created_at": "2026-07-26T20:37:02.568386Z",
  "updated_at": "2026-07-26T22:34:21.663911Z"
}

Delete a shadow test

Warning

Deleting a shadow test is irreversible. All the data associated with the shadow test will be permanently deleted.

Delete a shadow test using the nextmv cloud shadow delete command. The CLI will prompt you to confirm deletion but you can override it with the --yes option.

nextmv cloud shadow delete --app-id uncanny-rodent --shadow-test-id fluffy-shadow-test

Are you sure you want to delete shadow test fluffy-shadow-test from application uncanny-rodent? This action cannot be undone.
💡 Confirm Yes
✅ Shadow test fluffy-shadow-test deleted successfully from application uncanny-rodent.