Decorators module¶
Reference to the decorators.py module.
decorators
¶
Decorators for defining pipeline steps and workflows.
This module provides decorators and helper classes for defining pipeline steps and their relationships in a workflow. These decorators are used to annotate functions that represent steps in a pipeline, and to define the order in which they are executed.
| CLASS | DESCRIPTION |
|---|---|
InputType |
Enumeration of input types for application steps. |
StepType |
Enumeration of step types. |
Step |
Represents a step in a pipeline. |
Needs |
Represents dependencies between steps. |
Optional |
Represents an optional step condition. |
OnFailure |
Represents the failure handling of a step. |
Repeat |
Represents a repeating step. |
Foreach |
Represents a step that fans out its output. |
Join |
Represents a step that joins multiple inputs. |
App |
Represents an external application step. |
| FUNCTION | DESCRIPTION |
|---|---|
step |
Decorator to mark a function as a step in the pipeline. |
needs |
Decorator to mark the predecessors of a step. |
optional |
Decorator to mark a step as optional. |
on_failure |
Decorator to let a step fail without failing the flow. |
repeat |
Decorator to make a step be repeated a number of times. |
foreach |
Decorator to perform a "fanout" operation. |
join |
Decorator to perform a "join" operation. |
app |
Decorator to mark a step as a Nextmv Application. |
InputType
¶
Bases: Enum
Enumeration of input types for application steps.
This enum defines the possible input types when using the app decorator.
| ATTRIBUTE | DESCRIPTION |
|---|---|
JSON |
Indicates that the input to the application is in JSON format.
TYPE:
|
FILES |
Indicates that the input to the application consists of files.
TYPE:
|
StepType
¶
Step
¶
Step(function: callable)
Represents a step in a pipeline.
A step is a function that has been decorated with the @step decorator.
It can have additional properties set by other decorators like @needs,
@optional, @repeat, @foreach, @join, or @app.
| ATTRIBUTE | DESCRIPTION |
|---|---|
function |
The function that has been decorated as a step.
TYPE:
|
type |
The type of step (DEFAULT or APP).
TYPE:
|
run_ids |
The IDs of the runs associated with this step.
TYPE:
|
_inputs |
The inputs to the step.
TYPE:
|
_output |
The output of the step.
TYPE:
|
Initialize a Step object.
| PARAMETER | DESCRIPTION |
|---|---|
|
The function that has been decorated as a step.
TYPE:
|
Source code in nextpipe/nextpipe/decorators.py
get_id
¶
Get the ID of the step.
| RETURNS | DESCRIPTION |
|---|---|
str
|
The name of the function that has been decorated as a step. |
is_needs
¶
Check if the step has predecessors.
| RETURNS | DESCRIPTION |
|---|---|
bool
|
True if the step has predecessors, False otherwise. |
is_optional
¶
Check if the step is optional.
| RETURNS | DESCRIPTION |
|---|---|
bool
|
True if the step is optional, False otherwise. |
is_on_failure
¶
Check if the step may fail without failing the flow.
| RETURNS | DESCRIPTION |
|---|---|
bool
|
True if the step has an |
skip
¶
Check if the step should be skipped.
Deprecated: skipping is decided by the runner, which also passes the
step's inputs to the condition and propagates skips. Use
is_optional() to check whether a step has a condition.
| RETURNS | DESCRIPTION |
|---|---|
bool
|
True if the step should be skipped, False otherwise. |
Source code in nextpipe/nextpipe/decorators.py
is_repeat
¶
Check if the step should be repeated.
| RETURNS | DESCRIPTION |
|---|---|
bool
|
True if the step should be repeated, False otherwise. |
get_repetitions
¶
Get the number of times the step should be repeated.
| RETURNS | DESCRIPTION |
|---|---|
int
|
The number of times the step should be repeated, or 1 if the step should not be repeated. |
Source code in nextpipe/nextpipe/decorators.py
is_app
¶
Check if the step is a Nextmv Application step.
| RETURNS | DESCRIPTION |
|---|---|
bool
|
True if the step is a Nextmv Application step, False otherwise. |
get_app_id
¶
Get the ID of the Nextmv Application.
| RETURNS | DESCRIPTION |
|---|---|
str or None
|
The ID of the Nextmv Application, or None if the step is not a Nextmv Application step. |
Source code in nextpipe/nextpipe/decorators.py
set_run_ids
¶
set_run_ids(run_ids: list[str])
get_run_ids
¶
Get the run IDs for this step.
| RETURNS | DESCRIPTION |
|---|---|
list[str]
|
The run IDs for this step. |
is_foreach
¶
Check if the step is a foreach step.
| RETURNS | DESCRIPTION |
|---|---|
bool
|
True if the step is a foreach step, False otherwise. |
is_join
¶
Check if the step is a join step.
| RETURNS | DESCRIPTION |
|---|---|
bool
|
True if the step is a join step, False otherwise. |
step
¶
Decorator to mark a function as a step in the pipeline.
You can import the step decorator directly from nextpipe:
This is the most basic decorator. This decorator doesn't require any parameters or the use of parentheses.
Example
A simple example shows that a step is executed.
Source code in nextpipe/nextpipe/decorators.py
Needs
¶
Needs(predecessors: list[Callable])
Represents dependencies between steps.
This class is used by the needs decorator to specify which steps
must be executed before a specific step.
| ATTRIBUTE | DESCRIPTION |
|---|---|
predecessors |
The steps that must be executed before the decorated step.
TYPE:
|
Initialize a Needs object.
| PARAMETER | DESCRIPTION |
|---|---|
|
The steps that must be executed before the decorated step.
TYPE:
|
Source code in nextpipe/nextpipe/decorators.py
needs
¶
needs(predecessors: list[Callable])
Decorator to mark the predecessors of a step.
You can import the needs decorator directly from nextpipe:
This is used to determine the order in which the steps are executed. The predecessors are the steps that need to be executed before this actual step can be run.
| PARAMETER | DESCRIPTION |
|---|---|
|
The list of predecessors
TYPE:
|
Example
In this example steps step1 and step2 are executed before step3.
from nextpipe import FlowSpec, log, needs, step
class Flow(FlowSpec):
@step
def step1() -> None:
log("Execute step 1")
@step
def step2() -> None:
log("Execute step 2")
@needs(predecessors=[step1, step2])
@step
def step3() -> None:
log("Execute step 3 after steps 1 and 2")
flow = Flow("DecisionFlow", None)
flow.run()
Source code in nextpipe/nextpipe/decorators.py
Optional
¶
Optional(
condition: callable, skip_downstream: bool = False
)
Represents an optional step condition.
This class is used by the optional decorator to specify a condition
under which a step should be executed.
| ATTRIBUTE | DESCRIPTION |
|---|---|
condition |
A function that takes a step and returns a boolean indicating whether the step should be executed or not.
TYPE:
|
skip_downstream |
Whether skipping this step also skips the steps that depend only on it.
TYPE:
|
Initialize an Optional object.
| PARAMETER | DESCRIPTION |
|---|---|
|
A function that takes a step and returns a boolean indicating whether the step should be executed or not.
TYPE:
|
|
Whether skipping this step also skips the steps that depend only on it, by default False.
TYPE:
|
Source code in nextpipe/nextpipe/decorators.py
evaluate
¶
Evaluate the condition.
Conditions with a single required parameter only receive the step.
Conditions with more required parameters (or *args) additionally
receive the inputs of the step (the results of its predecessors, or the
flow input if it has none).
| PARAMETER | DESCRIPTION |
|---|---|
|
The step the condition belongs to.
TYPE:
|
|
The inputs of the step.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
bool
|
True if the step should be executed, False otherwise. |
Source code in nextpipe/nextpipe/decorators.py
optional
¶
optional(
condition: Callable[..., bool],
skip_downstream: bool = False,
)
Decorator to mark a step as optional.
You can import the optional decorator directly from nextpipe:
This is used to determine whether the step should be executed or not. The condition is evaluated at runtime, right before the step would run. It receives the step and, if it accepts more arguments, the inputs of the step (the results of its predecessors, or the flow input if it has none).
A skipped step has the status skipped and its result is None. By
default, only the step itself is skipped: its successors still run and
receive None as its result.
With skip_downstream=True, the skip propagates: a step whose
predecessors were all skipped this way is skipped as well (and so on). A
step with at least one other predecessor still runs and receives None
for each skipped one. This allows modeling if/else branches that are
merged again by a common successor.
| PARAMETER | DESCRIPTION |
|---|---|
|
The condition to evaluate. This is a callable that takes the step (and optionally the inputs of the step) as arguments and returns a boolean indicating whether the step should be executed or not.
TYPE:
|
|
Whether skipping this step also skips the steps that depend only on it, by default False.
TYPE:
|
Example
In this example either solve_small or solve_large (and its successor
polish) is executed, depending on the input. report merges both branches.
from nextpipe import FlowSpec, log, needs, optional, step
class Flow(FlowSpec):
@step
def prepare(data: dict) -> dict:
return data
@optional(condition=lambda step, data: len(data["orders"]) <= 10, skip_downstream=True)
@needs(predecessors=[prepare])
@step
def solve_small(data: dict) -> dict:
return {"solver": "small"}
@optional(condition=lambda step, data: len(data["orders"]) > 10, skip_downstream=True)
@needs(predecessors=[prepare])
@step
def solve_large(data: dict) -> dict:
return {"solver": "large"}
@needs(predecessors=[solve_large])
@step
def polish(result: dict) -> dict: # skipped if solve_large is skipped
return result | {"polished": True}
@needs(predecessors=[solve_small, polish])
@step
def report(small: dict | None, large: dict | None) -> dict:
return small or large
flow = Flow("DecisionFlow", {"orders": [1, 2, 3]})
flow.run()
log(flow.get_result(flow.report)) # {'solver': 'small'}
Source code in nextpipe/nextpipe/decorators.py
542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 | |
OnFailure
¶
OnFailure(fallback: Any = None)
Represents the failure handling of a step.
This class is used by the on_failure decorator to specify the result that
replaces the result of a step if it fails.
| ATTRIBUTE | DESCRIPTION |
|---|---|
fallback |
The result to use instead, or a callable that receives the error and returns it.
TYPE:
|
Initialize an OnFailure object.
| PARAMETER | DESCRIPTION |
|---|---|
|
The result to use instead, or a callable that receives the error and returns it, by default None.
TYPE:
|
Source code in nextpipe/nextpipe/decorators.py
on_failure
¶
on_failure(fallback: Any = None)
Decorator to let a step fail without failing the flow.
You can import the on_failure decorator directly from nextpipe:
By default, a failing step fails the whole flow. With this decorator, a
failure (e.g., an exception or a failed app run) only fails the step
itself: it has the status failed, the flow continues and its successors
receive the fallback as its result. If the step has multiple nodes (e.g.,
foreach), this applies to each node separately.
| PARAMETER | DESCRIPTION |
|---|---|
|
The result to use instead of the result of the failed step. If it is
callable, it is called with the error (e.g., an
TYPE:
|
Example
In this example the flow still produces an output if the solver app fails.
from nextpipe import FlowSpec, app, log, needs, on_failure, step
class Flow(FlowSpec):
@step
def prepare(data: dict) -> dict:
return data
@on_failure(fallback=lambda error: {"error": str(error)})
@app(app_id="solver")
@needs(predecessors=[prepare])
@step
def solve() -> None:
pass
@needs(predecessors=[solve])
@step
def report(result: dict) -> dict:
if "error" in result:
return {"status": "failed", "reason": result["error"]}
return {"status": "ok", "solution": result["solution"]}
flow = Flow("DecisionFlow", {"orders": [1, 2, 3]})
flow.run()
log(flow.get_result(flow.report))
Source code in nextpipe/nextpipe/decorators.py
Repeat
¶
Repeat(repetitions: int)
Represents a repeating step.
This class is used by the repeat decorator to specify how many times
a step should be repeated.
| ATTRIBUTE | DESCRIPTION |
|---|---|
repetitions |
The number of times to repeat the step.
TYPE:
|
Initialize a Repeat object.
| PARAMETER | DESCRIPTION |
|---|---|
|
The number of times to repeat the step.
TYPE:
|
Source code in nextpipe/nextpipe/decorators.py
repeat
¶
repeat(repetitions: int)
Decorator to make a step be repeated a number of times. The number of repetitions determines how many times the step will be run.
You can import the repeat decorator directly from nextpipe:
| PARAMETER | DESCRIPTION |
|---|---|
|
The number of times to repeat the step.
TYPE:
|
Example
In this example the step step1 is repeated 3 times.
Source code in nextpipe/nextpipe/decorators.py
Foreach
¶
Represents a step that fans out its output.
This class is used by the foreach decorator to indicate that a step's
output should be spread across multiple instances of the successor step.
Initialize a Foreach object.
Source code in nextpipe/nextpipe/decorators.py
foreach
¶
Decorator to perform a "fanout", which means creating multiple parallel steps out of a single step.
You can import the foreach decorator directly from nextpipe:
The function that is decorated should return a list of some sort. Each element of the list is consumed as an input by the successor step. When using this decorator, use parentheses without any parameters.
Example
In this example the step step2 is executed for each element in the list
returned by step1. The input to step2 is the element of the list.
from nextpipe import FlowSpec, foreach, log, needs, step
class Flow(FlowSpec):
@foreach()
@step
def step1() -> list[dict[str, Any]]:
return [{"input": 1}, {"input": 2}, {"input": 3}]
@needs(predecessors=[step1])
@step
def step2(data: dict) -> None:
log(data)
flow = Flow("DecisionFlow", None)
flow.run()
Source code in nextpipe/nextpipe/decorators.py
Join
¶
Represents a step that joins multiple inputs.
This class is used by the join decorator to indicate that a step
should receive the outputs of multiple predecessor steps as a list.
Initialize a Join object.
Source code in nextpipe/nextpipe/decorators.py
join
¶
Decorator to perform a "join", which means collecting the results of multiple parallel predecessor steps into a single step.
You can import the join decorator directly from nextpipe:
The outputs of the predecessor steps should be received as a list. The order of the elements in the list is the same as the order of the predecessor steps. Unpack the list to obtain the results and perform processing on them as needed. When using this decorator, use parentheses without any parameters.
Example
In this example the step step3 is executed after step1 and step2.
The input to step3 is a list containing the outputs of step1 and
step2.
from nextpipe import FlowSpec, join, log, needs, step
class Flow(FlowSpec):
@step
def step1() -> dict[str, Any]:
return {"input": 1}
@step
def step2() -> dict[str, Any]:
return {"input": 2}
@join()
@needs(predecessors=[step1, step2])
@step
def step3(data: list[dict[str, Any]]) -> None:
log(data)
flow = Flow("DecisionFlow", None)
flow.run()
Source code in nextpipe/nextpipe/decorators.py
App
¶
App(
app_id: str,
instance_id: str = "",
input_type: InputType = None,
parameters: dict[str, Any] = None,
options: dict[str, Any] = None,
full_result: bool = False,
run_configuration: RunConfiguration = None,
polling_options: PollingOptions
| None = DEFAULT_POLLING_OPTIONS,
bypass_result: Any = None,
emit_live_logs: bool = False,
)
Represents an external application step.
This class is used by the app decorator to specify an external
Nextmv Application to run as part of the pipeline.
| ATTRIBUTE | DESCRIPTION |
|---|---|
app_id |
The ID of the Nextmv Application to run.
TYPE:
|
instance_id |
The ID of the instance to run.
TYPE:
|
options |
The options to pass to the application.
TYPE:
|
full_result |
Whether to return the full result including metadata.
TYPE:
|
run_configuration |
The configuration to apply when running the app.
TYPE:
|
polling_options |
Options for polling for the results of the app run.
TYPE:
|
bypass_result |
A value that can be provided to bypass the actual execution of the app.
TYPE:
|
emit_live_logs |
Whether to emit live logs from the application run. If set to
TYPE:
|
Initialize an App object.
| PARAMETER | DESCRIPTION |
|---|---|
|
The ID of the Nextmv Application to run.
TYPE:
|
|
The ID of the instance to run. Default is defined by the app on Platform.
A predecessor step may override this per run by returning an
TYPE:
|
|
The options to pass to the application, by default None.
TYPE:
|
|
Whether to return the full result including metadata, by default False.
TYPE:
|
|
The configuration to apply when running the app, by default None. A
predecessor step may override this per run by returning an
TYPE:
|
|
Options for polling for the results of the app run, by default _DEFAULT_POLLING_OPTIONS.
TYPE:
|
|
A value that can be provided to bypass the actual execution of the app. This is mainly used for testing purposes, by default None.
TYPE:
|
|
Whether to emit live logs from the application run. If set to
TYPE:
|
Source code in nextpipe/nextpipe/decorators.py
1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 | |
app
¶
app(
app_id: str,
instance_id: str = "",
parameters: dict[str, Any] = None,
options: dict[str, Any] = None,
input_type: InputType = None,
full_result: bool = False,
run_configuration: RunConfiguration = None,
polling_options: PollingOptions
| None = DEFAULT_POLLING_OPTIONS,
bypass_result: Any = None,
emit_live_logs: bool = False,
)
Decorator to mark a step as a Nextmv Application (external application) step.
You can import the app decorator directly from nextpipe:
If this decorator is used, an external application will be run, using the
specified options. You need to have a valid Nextmv account and
Application before you can use this decorator. Make sure the
NEXTMV_API_KEY environment variable is set as well.
| PARAMETER | DESCRIPTION |
|---|---|
|
The ID of the application to run.
TYPE:
|
|
The ID of the instance to run. Default is defined by the app on Platform. A
predecessor step may override this per run by returning an
TYPE:
|
|
The options to pass to the application. This is a dictionary of parameter names and values. The values must be JSON serializable.
TYPE:
|
|
Whether to return the full result of the application run. If this is
set to
TYPE:
|
|
The configuration to apply when running the app. A predecessor step may
override this per run by returning an
TYPE:
|
|
Options for polling for the results of the app run. This is used to
configure the polling behavior, such as the timeout and backoff
options. Default (or when undefined) is the predefined options in the
class itself. Please note that the
TYPE:
|
|
A value that can be provided to bypass the actual execution of the app. This is mainly used for testing purposes, by default None.
TYPE:
|
|
Whether to emit live logs from the application run. If set to
TYPE:
|
Example
In this example the step pre_process is executed first. After
pre-processing is completed, the result is passed to the solve step. This
step runs a Nextmv Application with the ID echo. The result of the
application run is passed to the final step post_process, which
post-processes the result.
from nextpipe import FlowSpec, app, log, needs, step
class Flow(FlowSpec):
@step
def pre_process(input: dict[str, Any]) -> dict[str, Any]:
log("You can pre-process your data here.")
return input
@app(app_id="echo")
@needs(predecessors=[pre_process])
@step
def solve():
pass
@needs(predecessors=[solve])
@step
def post_process(result: dict[str, Any]) -> dict[str, Any]:
log("You can post-process your data here.")
return result
data = {"foo": "bar"}
flow = Flow("DecisionFlow", data)
flow.run()
log(flow.get_result(flow.post_process))
Source code in nextpipe/nextpipe/decorators.py
1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 | |