Experiments & testing overview¶
As a DecisionOps platform, Nextmv provides first-class support for different types of testing that can be done on decision models. Generally speaking, these can be categorized into two types:
offline: experiments that run on stored data, which does not involve the production system.online: experiments that run on live data, which involves the production system.
Please find a brief overview of the different types of experiments below. For more information, go to each of the individual sections.
| Name | Type | Description |
|---|---|---|
| Batch experiments | offline |
Analyze the output from one or more decision models on a fixed input set. |
| Scenario tests | offline |
Compare the output from one or more scenarios. |
| Acceptance tests | offline |
Verify that a decision model meets the acceptance criteria. |
| Shadow tests | online |
Run a decision model in parallel with the production model to compare results. |
| Switchback tests | online |
Evaluate the impact of changes by switching between different versions of a decision model. |
Data for experiments can generally be provided by two mechanisms:
- Input sets: a collection of input data. Consists of one or more inputs, where an input can be obtained from a run or a managed input.
- Managed inputs: data that can be uploaded directly to Nextmv.