Skip to content

Handling step failures

Reference

Learn more about the on_failure decorator and the AppRunError class in the technical reference (on_failure, AppRunError).

By default, a failing step fails the whole workflow: the steps that are still running are cancelled and flow.run() raises a RuntimeError. Often, a workflow can still produce something useful, though. For example, it can fall back to a different solver or write a report that explains what went wrong.

Failed app runs

An app step fails if the run of its application does not succeed, e.g., because the application crashed without writing any output. The step then raises an AppRunError that describes the run:

  • app_id: the ID of the application.
  • run_id: the ID of the run.
  • status: the final status of the run, e.g., failed or canceled.
  • error: the error message of the run, if any.
  • console_url: the URL of the run in Nextmv Console.

If the workflow fails because of it, the error message contains the same information:

RuntimeError: Flow failed: Step solve failed: run latest-Q_kk_83DR of app solver failed: exit status 2 (see https://cloud.nextmv.io/app/solver/run/latest-Q_kk_83DR?view=details)

Continuing with a fallback

Use the @on_failure decorator to let a step fail without failing the workflow. The step is marked as failed, the workflow continues, and the successors of the step receive the fallback as its result.

The fallback can be a value, or a function that receives the error and returns the result to use. If no fallback is given, the successors receive None.

In the following example, the workflow runs a primary and a backup solver. The report step uses the result of the first solver that succeeded and always writes a status.json file:

import json
import os

from nextpipe import AppRunError, FlowSpec, app, needs, on_failure, step


def describe(error: Exception) -> dict:
    """Turn the error of a failed step into a result."""
    if isinstance(error, AppRunError):
        return {"failed": True, "run_id": error.run_id, "error": error.error}
    return {"failed": True, "error": str(error)}


class Flow(FlowSpec):
    @step
    def prepare(data: dict) -> dict:
        """Prepare the data."""
        return data

    @on_failure(fallback=describe)
    @app(app_id="primary-solver")
    @needs(predecessors=[prepare])
    @step
    def solve_primary():
        """Run the primary solver."""
        pass

    @on_failure(fallback=describe)
    @app(app_id="backup-solver")
    @needs(predecessors=[prepare])
    @step
    def solve_backup():
        """Run the backup solver."""
        pass

    @needs(predecessors=[solve_primary, solve_backup])
    @step
    def report(primary: dict, backup: dict) -> dict:
        """Use the first successful result and write a status file."""
        results = {"primary": primary, "backup": backup}
        used = next((name for name, r in results.items() if not r.get("failed")), None)
        status = {
            "ok": used is not None,
            "used": used,
            "failures": {name: r for name, r in results.items() if r.get("failed")},
        }
        os.makedirs("outputs", exist_ok=True)
        with open(os.path.join("outputs", "status.json"), "w") as f:
            json.dump(status, f, indent=2)
        return results[used] if used else status

If a step has multiple nodes, e.g., because a predecessor uses the foreach decorator, the fallback is applied to each failing node separately. The other nodes keep their results.

Note

The fallback is not used if the workflow is aborted, e.g., because another step failed or the run was cancelled. If the fallback function raises an exception itself, the workflow fails.

Writing an output when the workflow fails

Steps without @on_failure still fail the whole workflow. To write an output anyway, catch the RuntimeError raised by flow.run(). The results of the steps that completed are still available via flow.get_result():

def main():
    flow = Flow("DecisionFlow", input.data)
    try:
        flow.run()
    except RuntimeError as e:
        os.makedirs("outputs", exist_ok=True)
        with open(os.path.join("outputs", "status.json"), "w") as f:
            json.dump({"ok": False, "error": str(e), "prepared": flow.get_result(flow.prepare)}, f)
        raise  # Re-raise to mark the run as failed