Skip to content

Running switchback tests

Info

Learn the concepts and fundamentals of switchback tests in the Explanation page.

A switchback test is an experiment that runs in the background and compares the results of two instances: baseline vs. candidate. The experiment switches back and forth between the two instances, when a run is made to the app, hence the name.

Included in this how-to guide are instructions on how to interact with switchback tests using methods on the Application class. Go the reference section to see all the available parameters for each method.

Create a switchback test

Use the Application.new_switchback_test method to create a new switchback test for an application. A switchback test requires you to define the number of units that it is going to run for, via the units keyword argument. Each unit is a time duration in minutes, defined by the unit_duration_minutes keyword argument. Lastly, you must define two instances to compare against each other, using the comparison keyword argument as a TestComparisonSingle object: a baseline and a candidate.

Here is an example that creates a new switchback test for an application.

import os

import nextmv
from nextmv import cloud
from nextmv.cloud import TestComparisonSingle

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
switchback_test = app.new_switchback_test(
    comparison=TestComparisonSingle(
        baseline_instance_id="production",
        candidate_instance_id="staging",
    ),
    unit_duration_minutes=2,
    units=5,
)

nextmv.write(switchback_test.to_dict())

uv run main.py

{
  "id": "switchback-ds8itzqx",
  "name": "switchback-ds8itzqx",
  "description": "",
  "created_at": "2026-07-29T04:09:16.269205Z",
  "updated_at": "2026-07-29T04:09:16.269205Z",
  "status": "draft",
  "comparison": {
    "baseline_instance_id": "production",
    "candidate_instance_id": "staging"
  },
  "plan": {
    "units": [
      {
        "duration_minutes": 2.0,
        "instance_id": "production",
        "index": 0
      },
      {
        "duration_minutes": 2.0,
        "instance_id": "staging",
        "index": 1
      },
      {
        "duration_minutes": 2.0,
        "instance_id": "staging",
        "index": 2
      },
      {
        "duration_minutes": 2.0,
        "instance_id": "staging",
        "index": 3
      },
      {
        "duration_minutes": 2.0,
        "instance_id": "production",
        "index": 4
      }
    ]
  }
}

The call above will create a random ID, and use the same identifier for the switchback test’s name. The name of the test is used as a human-readable label. You can use the switchback_test_id and/or name keyword arguments to specify a custom ID and name for the switchback test. For example:

import os

import nextmv
from nextmv import cloud
from nextmv.cloud import TestComparisonSingle

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
switchback_test = app.new_switchback_test(
    comparison=TestComparisonSingle(
        baseline_instance_id="production",
        candidate_instance_id="staging",
    ),
    unit_duration_minutes=2,
    units=5,
    switchback_test_id="fluffy-switchback-test",
    name="Fluffy Switchback Test",
)

nextmv.write(switchback_test.to_dict())

uv run main.py

{
  "id": "fluffy-switchback-test",
  "name": "Fluffy Switchback Test",
  "description": "",
  "created_at": "2026-07-29T04:09:21.149085Z",
  "updated_at": "2026-07-29T04:09:21.149085Z",
  "status": "draft",
  "comparison": {
    "baseline_instance_id": "production",
    "candidate_instance_id": "staging"
  },
  "plan": {
    "units": [
      {
        "duration_minutes": 2.0,
        "instance_id": "staging",
        "index": 0
      },
      {
        "duration_minutes": 2.0,
        "instance_id": "production",
        "index": 1
      },
      {
        "duration_minutes": 2.0,
        "instance_id": "production",
        "index": 2
      },
      {
        "duration_minutes": 2.0,
        "instance_id": "staging",
        "index": 3
      },
      {
        "duration_minutes": 2.0,
        "instance_id": "staging",
        "index": 4
      }
    ]
  }
}

The Application.new_switchback_test method creates a switchback test in draft mode. Once the test is created, you must start it.

Start a switchback test

As mentioned above, when a switchback test is created, it is in draft mode. A switchback test must be started so that it can begin executing runs and collecting data. There are two ways to start a switchback test:

As an example, here is how to start a switchback test using the Application.start_switchback_test method:

import os

from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
app.start_switchback_test(switchback_test_id="fluffy-switchback-test")

uv run main.py

Once a switchback test has started, you can stop it at any time or wait for it to complete based on the termination events defined when the test was created.

Get a switchback test

Info

The best way to view and interact with switchback test results is in the Nextmv Console.

Use the Application.switchback_test_metadata method to retrieve the metadata for a switchback test, using the switchback test ID. The method returns a SwitchbackTestMetadata object.

import os

import nextmv
from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
metadata = app.switchback_test_metadata(switchback_test_id="fluffy-switchback-test")

nextmv.write(metadata.to_dict())

uv run main.py

{
  "id": "fluffy-switchback-test",
  "name": "Fluffy Switchback Test",
  "description": "",
  "app_id": "uncanny-rodent",
  "created_at": "2026-07-29T04:09:21Z",
  "updated_at": "2026-07-29T04:09:25Z",
  "status": "started"
}

Once the status of the switchback test is completed, you can get the results using the Application.switchback_test method. The method returns a SwitchbackTest object, whose output includes the runs that were made for the test. If the test hasn't completed, you can still get partial results.

A switchback test that has already started can be stopped at any time, or it will stop automatically when the termination events are fulfilled. Once a switchback test is stopped, then its status will move to completed.

import os

import nextmv
from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
switchback_test = app.switchback_test(switchback_test_id="fluffy-switchback-test")

nextmv.write(switchback_test.to_dict())

uv run main.py

{
  "id": "fluffy-switchback-test",
  "name": "Fluffy Switchback Test",
  "description": "",
  "created_at": "2026-07-29T04:09:21.149085Z",
  "updated_at": "2026-07-29T04:09:25.335058Z",
  "status": "started",
  "started_at": "2026-07-29T04:09:25.332407Z",
  "comparison": {
    "baseline_instance_id": "production",
    "candidate_instance_id": "staging"
  },
  "plan": {
    "start": "2026-07-29T04:09:25.332407Z",
    "units": [
      {
        "duration_minutes": 2.0,
        "instance_id": "staging",
        "index": 0
      },
      {
        "duration_minutes": 2.0,
        "instance_id": "production",
        "index": 1
      },
      {
        "duration_minutes": 2.0,
        "instance_id": "production",
        "index": 2
      },
      {
        "duration_minutes": 2.0,
        "instance_id": "staging",
        "index": 3
      },
      {
        "duration_minutes": 2.0,
        "instance_id": "staging",
        "index": 4
      }
    ]
  },
  "runs": [
    {
      "id": "staging-0NLvQUPDg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-29T04:10:02.896050Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "staging",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.322
          }
        ]
      },
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "version"
        }
      ]
    },
    {
      "id": "staging-FtxDQUEDR",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-29T04:09:59.979877Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "staging",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.402
          }
        ]
      },
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "version"
        }
      ]
    },
    {
      "id": "staging-8-ovQ8Evg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-29T04:09:57.112480Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "staging",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.527
          }
        ]
      },
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "version"
        }
      ]
    },
    {
      "id": "staging-z_5vQUEvg",
      "user_email": "sebastian@nextmv.io",
      "name": "",
      "description": "",
      "created_at": "2026-07-29T04:09:54.241438Z",
      "application_id": "uncanny-rodent",
      "application_instance_id": "staging",
      "application_version_id": "v0.0.2",
      "run_type": {
        "definition_id": "",
        "reference_id": ""
      },
      "execution_class": "6c9500mb870s",
      "runtime": "python-3_11",
      "status_v2": "succeeded",
      "queuing_priority": 6,
      "queuing_disabled": false,
      "metrics": {
        "status": "succeeded",
        "indicators": [
          {
            "name": "value",
            "value": 1.23
          },
          {
            "name": "metadata.duration",
            "value": 4.609
          }
        ]
      },
      "options": {
        "details": "true"
      },
      "options_summary": [
        {
          "name": "details",
          "value": "true",
          "source": "version"
        }
      ]
    }
  ]
}

You can list all switchback tests in the application using the Application.list_switchback_tests method.

import json
import os

from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
switchback_tests = app.list_switchback_tests()

print(json.dumps([switchback_test.to_dict() for switchback_test in switchback_tests[:2]], indent=2))

uv run main.py

[
  {
    "id": "fluffy-switchback-test",
    "name": "Fluffy Switchback Test",
    "description": "",
    "created_at": "2026-07-29T04:09:21.149085Z",
    "updated_at": "2026-07-29T04:09:25.335058Z",
    "status": "started",
    "started_at": "2026-07-29T04:09:25.332407Z",
    "comparison": {
      "baseline_instance_id": "production",
      "candidate_instance_id": "staging"
    }
  },
  {
    "id": "switchback-ds8itzqx",
    "name": "switchback-ds8itzqx",
    "description": "",
    "created_at": "2026-07-29T04:09:16.269205Z",
    "updated_at": "2026-07-29T04:09:16.269205Z",
    "status": "draft",
    "comparison": {
      "baseline_instance_id": "production",
      "candidate_instance_id": "staging"
    }
  }
]

Stop a switchback test

A switchback test will continue to execute until it stops, which is equivalent to it being completed. There are two ways to stop a switchback test:

When using the Application.stop_switchback_test method, you must use the intent keyword argument (a StopIntent value) to track if you want to complete or cancel the switchback test. As an example, here is how to stop a switchback test and mark it as completed:

import os

from nextmv import cloud
from nextmv.cloud import StopIntent

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
app.stop_switchback_test(switchback_test_id="fluffy-switchback-test", intent=StopIntent.COMPLETE)

uv run main.py

Update a switchback test

You can update attributes of a switchback test with the Application.update_switchback_test method, such as its:

  • Name
  • Description

The method returns a SwitchbackTest object. You cannot update the ID of a switchback test.

import os

import nextmv
from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
switchback_test = app.update_switchback_test(
    switchback_test_id="fluffy-switchback-test",
    name="Updated Switchback Test Name",
    description="Updated description for the switchback test",
)

nextmv.write(switchback_test.to_dict())

uv run main.py

{
  "id": "fluffy-switchback-test",
  "name": "Updated Switchback Test Name",
  "description": "Updated description for the switchback test",
  "created_at": "2026-07-29T04:09:21.149085Z",
  "updated_at": "2026-07-29T04:10:34.226095Z"
}

Delete a switchback test

Warning

Deleting a switchback test is irreversible. All the data associated with the switchback test will be permanently deleted.

Delete a switchback test using the Application.delete_switchback_test method.

import os

from nextmv import cloud

client = cloud.Client(api_key=os.getenv("NEXTMV_API_KEY"))
app = cloud.Application.get(client=client, id="uncanny-rodent")
app.delete_switchback_test(switchback_test_id="fluffy-switchback-test")

uv run main.py