Handling step failures¶
Reference
Learn more about the on_failure decorator and the AppRunError class in the
technical reference (on_failure, AppRunError).
By default, a failing step fails the whole workflow: the steps that are still running
are cancelled and flow.run() raises a RuntimeError. Often, a workflow can still
produce something useful, though. For example, it can fall back to a different solver
or write a report that explains what went wrong.
Failed app runs¶
An app step fails if the run of its application does not succeed, e.g., because
the application crashed without writing any output. The step then raises an
AppRunError that describes the run:
app_id: the ID of the application.run_id: the ID of the run.status: the final status of the run, e.g.,failedorcanceled.error: the error message of the run, if any.console_url: the URL of the run in Nextmv Console.
If the workflow fails because of it, the error message contains the same information:
RuntimeError: Flow failed: Step solve failed: run latest-Q_kk_83DR of app solver failed: exit status 2 (see https://cloud.nextmv.io/app/solver/run/latest-Q_kk_83DR?view=details)
Continuing with a fallback¶
Use the @on_failure decorator to let a step fail without failing the workflow. The
step is marked as failed, the workflow continues, and the successors of the step
receive the fallback as its result.
The fallback can be a value, or a function that receives the error and returns the
result to use. If no fallback is given, the successors receive None.
In the following example, the workflow runs a primary and a backup solver. The report
step uses the result of the first solver that succeeded and always writes a
status.json file:
import json
import os
from nextpipe import AppRunError, FlowSpec, app, needs, on_failure, step
def describe(error: Exception) -> dict:
"""Turn the error of a failed step into a result."""
if isinstance(error, AppRunError):
return {"failed": True, "run_id": error.run_id, "error": error.error}
return {"failed": True, "error": str(error)}
class Flow(FlowSpec):
@step
def prepare(data: dict) -> dict:
"""Prepare the data."""
return data
@on_failure(fallback=describe)
@app(app_id="primary-solver")
@needs(predecessors=[prepare])
@step
def solve_primary():
"""Run the primary solver."""
pass
@on_failure(fallback=describe)
@app(app_id="backup-solver")
@needs(predecessors=[prepare])
@step
def solve_backup():
"""Run the backup solver."""
pass
@needs(predecessors=[solve_primary, solve_backup])
@step
def report(primary: dict, backup: dict) -> dict:
"""Use the first successful result and write a status file."""
results = {"primary": primary, "backup": backup}
used = next((name for name, r in results.items() if not r.get("failed")), None)
status = {
"ok": used is not None,
"used": used,
"failures": {name: r for name, r in results.items() if r.get("failed")},
}
os.makedirs("outputs", exist_ok=True)
with open(os.path.join("outputs", "status.json"), "w") as f:
json.dump(status, f, indent=2)
return results[used] if used else status
If a step has multiple nodes, e.g., because a predecessor uses the
foreach decorator, the fallback is applied to each failing node separately.
The other nodes keep their results.
Note
The fallback is not used if the workflow is aborted, e.g., because another step
failed or the run was cancelled. If the fallback function raises an exception
itself, the workflow fails.
Writing an output when the workflow fails¶
Steps without @on_failure still fail the whole workflow. To write an output anyway,
catch the RuntimeError raised by flow.run(). The results of the steps that
completed are still available via flow.get_result():
def main():
flow = Flow("DecisionFlow", input.data)
try:
flow.run()
except RuntimeError as e:
os.makedirs("outputs", exist_ok=True)
with open(os.path.join("outputs", "status.json"), "w") as f:
json.dump({"ok": False, "error": str(e), "prepared": flow.get_result(flow.prepare)}, f)
raise # Re-raise to mark the run as failed