Large payloads on remote runs

When submitting a new run and retrieving the run's results, there are size limits to consider for the payload: 5MB for the input and output of a run.

You can use this large payloads workflow to handle bigger sizes. When using this alternative, note that:

• There is a maximum limit of 500MB when uploading a file.
• There is a maximum limit of 100MB when downloading a file.

To achieve this, you can use one of these interfaces:

• Cloud API
• Python SDK

To work with large payloads, follow these steps:

1. Request a presigned URL

Request a presigned URL to upload your large input file to.

curl -sS -L -X POST \
    "https://api.cloud.nextmv.io/v1/applications/$APP_ID/runs/uploadurl" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $NEXTMV_API_KEY" | jq

The result will contain an upload_id and upload_url.

{
  "upload_id": "00000000-0000-0000-0000-000000000000",
  "upload_url": "https://the-upload-url.com/some-hashes"
}

You can export the upload_id and upload_url to variables:

export UPLOAD_ID="ae7dce..."
export UPLOAD_URL="https://production-nextmv-tmp..."

2. Upload the large input

Upload your input using the upload_url that was obtained before. We use --data-binary here to make sure curl sends the content of the file unaltered.

curl -sS -L -X PUT \
  $(echo -n $UPLOAD_URL) \
  -H "Content-Type: application/json" \
  --compressed \
  --data-binary @$INPUT_FILE

An empty response means the upload was successful. Note that this may take a moment depending on the size of the input file.

3. Start the run

Once the upload has completed, you can start a new run with the large input using the upload_id from a previous step. This upload_id is sent in the body of the run request in place of the input. The result contains the run_id to retrieve the run results.

curl -sS -L -X POST \
    "https://api.cloud.nextmv.io/v1/applications/$APP_ID/runs?instance_id=$INSTANCE_ID" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $NEXTMV_API_KEY" \
    -d "{
      \"upload_id\": \"$UPLOAD_ID\", 
      \"options\": {
        \"duration\": \"2\"
      }
    }" | jq

You should receive a run_id like normal as a response.

{
  "run_id": "instance1-a_USaanxSR"
}

4. Get the run result

To get the result of your run, you use the same endpoint as with a standard run but with the appended query parameter ?format=url. This will return a standard run result payload but instead of the run output included in the returned JSON, in its place will be a URL to download the output file.

curl -sS -L -X GET \
    "https://api.cloud.nextmv.io/v1/applications/$APP_ID/runs/$RUN_ID?format=url" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $NEXTMV_API_KEY" | jq

The output.url key contains the presigned URL to download the output.

curl -sS -L -X GET \
    "https://api.cloud.nextmv.io/v1/applications/$APP_ID/runs/$RUN_ID?format=url" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $NEXTMV_API_KEY" | jq

Finally, you can download the output using the presigned URL:

curl -L -X GET \
  $(echo -n $OUTPUT_URL) \
  -H "Content-Type: application/json"