Flow module¶
Reference to the flow.py module.
flow
¶
Flow module for defining and executing decision pipelines.
This module provides classes and functions for creating, configuring, and executing decision pipelines.
| CLASS | DESCRIPTION |
|---|---|
FlowStep |
Represents a step in a workflow pipeline. |
FlowNode |
Represents a specific instance of a step in execution. |
FlowSpec |
Defines a complete workflow specification. |
FlowGraph |
Represents the directed acyclic graph (DAG) of steps in a flow. |
StepVisitor |
AST visitor for finding step functions in a flow class. |
Runner |
Handles the execution of a flow graph. |
AppRunError |
Raised when the run of an app step does not succeed. |
The module also defines status constants used throughout pipeline execution. |
|
STATUS_PENDING
module-attribute
¶
Status constant indicating a pending step or node.
STATUS_RUNNING
module-attribute
¶
Status constant indicating a running step or node.
STATUS_SUCCEEDED
module-attribute
¶
Status constant indicating a successfully completed step or node.
STATUS_FAILED
module-attribute
¶
Status constant indicating a failed step or node.
STATUS_SKIPPED
module-attribute
¶
Status constant indicating a skipped step or node.
SUB_RUN_CANCEL_TIMEOUT
module-attribute
¶
Maximum time in seconds to wait for sub-run cancel requests when the flow is aborted.
When a run is canceled on the Nextmv platform, the workflow process is interrupted and killed after a grace period of 10 seconds. This timeout, plus the final flow update sent afterwards, must fit within that grace period.
AppRunError
¶
Bases: Exception
Raised when the run of an app step does not succeed.
You can import the AppRunError class directly from nextpipe:
| ATTRIBUTE | DESCRIPTION |
|---|---|
step_id |
The ID of the node of the app step.
TYPE:
|
app_id |
The ID of the application that was run.
TYPE:
|
run_id |
The ID of the run.
TYPE:
|
status |
The final status of the run (e.g.,
TYPE:
|
error |
The error message of the run, if any.
TYPE:
|
console_url |
The URL of the run in the Nextmv Console.
TYPE:
|
Source code in nextpipe/nextpipe/flow.py
FlowStep
¶
FlowStep(
step_function: callable,
step_definition: Step,
docstring: str,
)
Represents a step in a workflow pipeline.
A FlowStep is created from a function decorated with @step and maintains
information about its position in the flow graph, including predecessors,
successors, and execution nodes.
| PARAMETER | DESCRIPTION |
|---|---|
|
The AST function node representing the step function.
TYPE:
|
|
The step decorator instance that contains the step's configuration.
TYPE:
|
|
The docstring of the step function.
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
step_function |
The AST function node representing the step function.
TYPE:
|
definition |
The step decorator instance.
TYPE:
|
docstring |
The docstring of the step function.
TYPE:
|
lock |
Thread lock for this step.
TYPE:
|
done |
Whether this step has completed execution.
TYPE:
|
skipped |
Whether this step was skipped.
TYPE:
|
skip_downstream |
Whether the skip of this step propagates to its successors.
TYPE:
|
successors |
Steps that depend on this step.
TYPE:
|
predecessors |
Steps that this step depends on.
TYPE:
|
nodes |
Execution nodes for this step.
TYPE:
|
Initialize a FlowStep.
| PARAMETER | DESCRIPTION |
|---|---|
|
The AST function node representing the step function.
TYPE:
|
|
The step decorator instance that contains the step's configuration.
TYPE:
|
|
The docstring of the step function.
TYPE:
|
Source code in nextpipe/nextpipe/flow.py
FlowNode
¶
Represents a specific instance of a step in execution.
A FlowNode is created for each execution instance of a FlowStep. For example, when using foreach or repeat decorators, multiple nodes will be created for a single step.
| PARAMETER | DESCRIPTION |
|---|---|
|
The parent step this node belongs to.
TYPE:
|
|
The index of this node within the parent step.
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
parent |
The parent step this node belongs to.
TYPE:
|
index |
The index of this node within the parent step.
TYPE:
|
id |
Unique identifier for this node.
TYPE:
|
status |
Current execution status (pending, running, succeeded, failed, skipped).
TYPE:
|
error |
Error message if the node execution failed.
TYPE:
|
predecessors |
Nodes that this node depends on.
TYPE:
|
run_id |
ID of the application run if this is an app step.
TYPE:
|
result |
Result data from this node's execution.
TYPE:
|
done |
Whether this node has completed execution.
TYPE:
|
cancel |
Flag to indicate if this node's execution should be cancelled.
TYPE:
|
Initialize a FlowNode.
| PARAMETER | DESCRIPTION |
|---|---|
|
The parent step this node belongs to.
TYPE:
|
|
The index of this node within the parent step.
TYPE:
|
Source code in nextpipe/nextpipe/flow.py
FlowSpec
¶
FlowSpec(
name: str,
input: dict,
conf: Configuration | None = None,
client: Client | None = None,
uplink_config: UplinkConfig | None = None,
)
Defines a complete workflow specification.
You can import the FlowSpec class directly from nextpipe:
FlowSpec is the main class to define a workflow. Users typically inherit from this class and decorate methods with @step to define the workflow steps.
| PARAMETER | DESCRIPTION |
|---|---|
|
Name of the flow specification.
TYPE:
|
|
Input data for the flow.
TYPE:
|
|
Configuration for the flow, by default None.
TYPE:
|
|
Nextmv client for API access, by default None.
TYPE:
|
|
Configuration for uplink, by default None.
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
name |
Name of the flow specification.
TYPE:
|
config |
Configuration for the flow.
TYPE:
|
client |
Nextmv client for API access.
TYPE:
|
uplink |
Client for communicating with the Nextmv platform.
TYPE:
|
graph |
Graph representing the workflow.
TYPE:
|
input |
Input data for the flow.
TYPE:
|
runner |
Runner responsible for executing the flow.
TYPE:
|
Examples:
from nextpipe import FlowSpec, step
class MyFlow(FlowSpec):
@step
def step1(self, input_data):
# Process input data
return {"processed": input_data}
@step
def step2(self, input_data):
# Further processing
return {"result": input_data["processed"] * 2}
# Create and run the flow
flow = MyFlow("my-flow", {"value": 5})
flow.run()
Initialize a FlowSpec.
| PARAMETER | DESCRIPTION |
|---|---|
|
Name of the flow specification.
TYPE:
|
|
Input data for the flow.
TYPE:
|
|
Configuration for the flow, by default None.
TYPE:
|
|
Nextmv client for API access, by default None.
TYPE:
|
|
Configuration for uplink, by default None.
TYPE:
|
Source code in nextpipe/nextpipe/flow.py
run
¶
Run the flow.
This method starts the flow execution and blocks until it completes or fails with an exception.
get_result
¶
get_result(step: callable) -> object | None
Get the result of a step.
| PARAMETER | DESCRIPTION |
|---|---|
|
The step function to get the result for.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
Union[object, None]
|
The result of the step, or None if the step is not done. If the step has multiple nodes, a list of results is returned. |
| RAISES | DESCRIPTION |
|---|---|
Exception
|
If the provided function does not have a step decorator. |
Examples:
Source code in nextpipe/nextpipe/flow.py
FlowGraph
¶
Represents the directed acyclic graph (DAG) of steps in a flow.
You can import the FlowGraph class directly from nextpipe:
A FlowGraph contains all steps of a flow and their connections, forming a DAG. It provides methods for converting the graph to different representations.
| PARAMETER | DESCRIPTION |
|---|---|
|
The flow specification this graph belongs to.
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
flow_spec |
The flow specification this graph belongs to.
TYPE:
|
steps |
List of all steps in the graph.
TYPE:
|
steps_by_definition |
Dictionary mapping step definitions to FlowStep objects.
TYPE:
|
start_steps |
List of steps with no predecessors (starting points).
TYPE:
|
Initialize a FlowGraph.
| PARAMETER | DESCRIPTION |
|---|---|
|
The flow specification this graph belongs to.
TYPE:
|
Source code in nextpipe/nextpipe/flow.py
StepVisitor
¶
StepVisitor(steps: list[FlowStep], flow_class: type)
Bases: NodeVisitor
AST visitor for finding step functions in a flow class.
This visitor traverses the abstract syntax tree of a flow class and identifies all methods decorated with @step.
| PARAMETER | DESCRIPTION |
|---|---|
|
List to collect found steps.
TYPE:
|
|
The flow class to visit.
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
steps |
List to collect found steps.
TYPE:
|
flow_class |
The flow class being visited.
TYPE:
|
Initialize a StepVisitor.
| PARAMETER | DESCRIPTION |
|---|---|
|
List to collect found steps.
TYPE:
|
|
The flow class to visit.
TYPE:
|
Source code in nextpipe/nextpipe/flow.py
visit_FunctionDef
¶
visit_FunctionDef(step_function)
Visit a function definition node in the AST.
If the function has a step decorator, it will be added to the steps list.
| PARAMETER | DESCRIPTION |
|---|---|
|
The function definition node.
TYPE:
|
Source code in nextpipe/nextpipe/flow.py
Runner
¶
Handles the execution of a flow graph.
This class is responsible for preparing inputs for steps, creating execution nodes, dispatching jobs to the thread pool, and monitoring execution progress.
| PARAMETER | DESCRIPTION |
|---|---|
|
The flow specification being executed.
TYPE:
|
|
The flow graph to execute.
TYPE:
|
|
Configuration for the runner.
TYPE:
|
|
Client for communicating with the Nextmv platform.
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
spec |
The flow specification being executed.
TYPE:
|
graph |
The flow graph to execute.
TYPE:
|
uplink |
Client for communicating with the Nextmv platform.
TYPE:
|
pool |
Thread pool for executing steps.
TYPE:
|
jobs |
List of jobs.
TYPE:
|
node_idxs |
Dictionary of node indices.
TYPE:
|
fail |
Whether the flow has failed.
TYPE:
|
fail_reason |
Reason for the flow failure, if any.
TYPE:
|
fail_error |
Error that made the flow fail, if any.
TYPE:
|
lock_fail |
Lock for updating fail state.
TYPE:
|
lock_running |
Lock for updating running steps.
TYPE:
|
Initialize a Runner.
| PARAMETER | DESCRIPTION |
|---|---|
|
The flow specification being executed.
TYPE:
|
|
The flow graph to execute.
TYPE:
|
|
Configuration for the runner.
TYPE:
|
|
Client for communicating with the Nextmv platform.
TYPE:
|
Source code in nextpipe/nextpipe/flow.py
run
¶
Run the flow.
This method starts the uplink communication, executes the flow steps in the correct order based on dependencies, and handles failures.
| RAISES | DESCRIPTION |
|---|---|
RuntimeError
|
If the flow execution fails. |