Experiments Tests

Input sets

A tutorial for creating and managing input sets on Nextmv Cloud.

Input sets are defined sets of input files to use for offline tests, such as batch experiments and acceptance tests. You can visualize the input sets of an app in the Console web interface. Go to the app, Experiments > Input Sets tab.

Input set

There are several interfaces for creating input sets:

At the moment, inputs for the input sets can only be retrieved from prior runs. The basic steps for creating an input set are:

  1. Make runs using the desired inputs that will make part of the input set.
  2. Create the input set by:
    • Specifying the run IDs of the runs that will make part of the input set.
    • Specifying a date range and an instance ID where runs were made to gather inputs for an input set.

Note that the maximum number of inputs allowed in an input set is 20.

Console

Go to the Console web interface, and open your app. Go to the Experiments > Input Sets tab. Click on New Input Set. Fill in the fields.

Input sets

Nextmv CLI

Define the desired input set ID and name. After, create the input set by:

  • Referencing the run IDs.

    nextmv experiment input-set create \
       --app-id $APP_ID \
       --input-set-id $INPUT_SET_ID \
       --name $INPUT_SET_ID \
       --run-ids "latest-RNBs7AKSg,latest-UlulZAKSR,latest-fK_wZ0FSg" \
       --description "An optional description"
    
    Copy
  • Referencing a date range and an instance ID.

    nextmv experiment input-set create \
       --app-id $APP_ID \
       --input-set-id $INPUT_SET_ID \
       --name $INPUT_SET_ID \
       --instance-id $INSTANCE_ID \
       --start-time "2024-01-05T00:00:00Z" \
       --end-time "2024-01-05T23:59:00Z" \
       --description "An optional description"
    
    Copy

Python SDK

Define the desired input set ID and name. After, create the input set by:

  • Referencing the run IDs.

    import json
    import os
    
    from nextmv.cloud import Application, Client
    
    client = Client(api_key=os.getenv("NEXTMV_API_KEY"))
    app = Application(client=client, id=os.getenv("APP_ID"))
    input_set = app.new_input_set(
       id=os.getenv("INPUT_SET_ID"),
       name=os.getenv("INPUT_SET_ID"),
       description="An optional description",
       run_ids=["latest-RNBs7AKSg", "latest-UlulZAKSR", "latest-fK_wZ0FSg"],
    )
    print(json.dumps(input_set.to_dict(), indent=2))  # Pretty print.
    
    Copy
  • Referencing a date range and an instance ID.

    import json
    import os
    from datetime import datetime, timezone  
    
    from nextmv.cloud import Application, Client
    
    client = Client(api_key=os.getenv("NEXTMV_API_KEY"))
    app = Application(client=client, id=os.getenv("APP_ID"))
    input_set = app.new_input_set(
       id=os.getenv("INPUT_SET_ID"),
       name=os.getenv("INPUT_SET_ID"),
       description="An optional description",
       instance_id=os.getenv("INSTANCE_ID"),
       start_time=datetime(2024, 1, 5, 0, 0, 0, tzinfo=timezone.utc),
       end_time=datetime(2024, 1, 5, 23, 59, 59, tzinfo=timezone.utc),
    )
    print(json.dumps(input_set.to_dict(), indent=2))  # Pretty print.
    
    Copy

Cloud API

POSThttps://api.cloud.nextmv.io/v1/applications/{application_id}/experiments/inputsets

Create new input set.

Create new application input set for experiments.

Define the desired input set ID and name. After, create the input set by:

  • Referencing the run IDs.
curl -sS -L -X POST \
    "https://api.cloud.nextmv.io/v1/applications/$APP_ID/experiments/inputsets" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $NEXTMV_API_KEY" \
    -d "{
      \"id\": \"$INPUT_SET_ID\",
      \"name\": \"$INPUT_SET_ID\",
      \"run_ids\": [\"$RUN_ID\"],
      \"description\": \"An optional description\"
    }" | jq
Copy
  • Referencing a date range and an instance ID.

    curl -L -X POST \
       "https://api.cloud.nextmv.io/v1/applications/$APP_ID/experiments/inputsets" \
       -H "Content-Type: application/json" \
       -H "Authorization: Bearer $NEXTMV_API_KEY" \
       -d "{
          \"app_id\": \"$APP_ID\",
          \"id\": \"$INPUT_SET_ID\",
          \"name\": \"$INPUT_SET_ID\",
          \"instance_id\": \"$INSTANCE_ID\",
          \"start_time\": \"2024-01-05T00:00:00Z\",
          \"end_time\": \"2024-01-05T23:59:00Z\",
          \"description\": \"An optional description\"
       }"
    
    Copy

Update an input set

Update an input set (change its name, description, etc...) using the following interfaces:

  • Console. Go to the Console web interface, and open your app. Go to the Experiments > Input Sets tab. Click on the input set you want to update. Click on Edit. Fill in the fields.

  • Nextmv CLI. Use the nextmv experiment input-set update subcommand.

  • Cloud API. Make sure your NEXTMV_API_KEY is exported as an environment variable. The API uses Bearer Authentication.

    PUThttps://api.cloud.nextmv.io/v1/applications/{application_id}/experiments/inputsets/{input_set_id}

    Update input set metadata.

    Update input set metadata with defined data, specified by application and input set ID.

curl -sS -L -X PUT \
      "https://api.cloud.nextmv.io/v1/applications/$APP_ID/experiments/inputsets/$INPUT_SET_ID" \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $NEXTMV_API_KEY" \
      -d "{
         \"name\": \"Another name\",
         \"description\": \"Another description\"
      }" | jq
Copy

Page last updated

Go to on-page nav menu