ralsei.graph.pipeline¶

Module Contents¶

Classes¶

Aliases¶

API¶

class ralsei.graph.pipeline.Pipeline¶

Bases: abc.ABC

This is where you declare your tasks, that later get compiled into a ralsei.graph.DAG

abstract create_tasks() → ralsei.graph.pipeline.Tasks¶

Returns:¶

A dictionary with task name to value pairs, where the value can be:

• A task definition (ralsei.task.TaskDef)

• A nested ralsei.graph.Pipeline

• A nested dictionary

outputof(*task_paths: str | ralsei.graph.path.TreePath) → ralsei.graph.outputof.OutputOf¶

Refer to the output of another task from this pipeline, that will later be resolved.

Dependencies are taken into account when deciding the order of task execution.

Parameters:¶

*task_paths: str | ralsei.graph.path.TreePath¶

path from the root of the pipeline, either a string separated with . or a TreePath object

Multiple paths are allowed, but all tasks must have the same output. This is useful when depending on multiple AddColumnsSql tasks if both sets of columns are required

build_dag(env: ralsei.jinja.SqlEnvironment) → ralsei.graph.dag.DAG¶

Resolve dependencies and generate a graph of tasks

class ralsei.graph.pipeline.Tasks¶

type Tasks = collections.abc.Mapping[str, ralsei.task.TaskDef | ralsei.graph.pipeline.Pipeline | ralsei.graph.pipeline.Tasks]

A dictionary with task name to value pairs, used to define a Pipeline

Acceptable values:

• A task definition (ralsei.task.TaskDef)

• A nested Pipeline

• A nested dictionary