Skip to content

Shadow tests

Learn how to run shadow tests with your tool of choice.

  • CLI: the Nextmv Command Line Interface (CLI).
  • Python SDK: a Software Development Kit (SDK) for Python developers.
  • Cloud API: an HTTP API for developers to integrate with the Nextmv platform.

A shadow test is an experiment that runs in the background and compares the results of two instances: baseline vs. candidate. When the shadow test has started, any run made on the baseline instance will trigger a run on the candidate instance using the same input and options. The results of the shadow test are often used to determine if a new version of a model is ready to be promoted to production.

To be concise, the shadow test deploys the candidate instance as a shadow. Any time a run is made to the app, using the baseline instance, the input is also sent to the candidate instance. Once the test is completed, the results are compared with the same framework as in a batch experiment.

A shadow test is defined by the following parameters:

  • start_events (optional): a list of events that will trigger the start of the shadow test data collection. Supported start events are:
    • time: the shadow test will start at a specific time. The test can be started manually before the start time, but it will not begin collecting data until this start time is reached. If the start time is omitted, the test will be executed immediately when it is started.
  • termination_events (required): a list of events that will trigger the termination of the shadow test. One or all termination events may be defined, and the first condition met will cause the shadow test to stop. Supported termination events are:
    • maximum_runs (required): the shadow test will terminate after a specific number of runs.
    • time: the shadow test will terminate at a specific time.

These are the steps to start a shadow test:

  1. Create the shadow test. This step is like creating a draft of the test.
  2. Start the shadow test. A test does not start automatically even if a criteria part of start_events is met. You must start the test manually.

Info

There is a maximum run limit of 300 runs for shadow tests.

A shadow test may be manually completed or canceled at any time, if you don't want to wait until the termination_events are fulfilled. Once a shadow test starts, you can view partial results without having to wait for the test to complete. Results will be updated as the test progresses and finishes.

Shadow tests are designed to be visualized in the Nextmv Console. Go to the app, Experiments > Shadow tab.

Shadow test