Task Graph¶

Namespace: dftracer::utils::task_graph

For usage guide and examples, see Task Graph API.

class TaskGraph¶

Builder for constructing task DAGs.

Features:

Fluent API for building task graphs
Optional auto-registration with Pipeline
Support for fan-in, fan-out, transform, reduce patterns
Bi-directional connectivity with external tasks via wrap()
Configurable max_concurrency to limit in-flight parallel tasks

Usage:

Pipeline pipeline(config);
auto graph = TaskGraph::builder({.name = "MyGraph",
                                 .max_concurrency = 128});
auto readers = graph.parallel<int>(8, reader_fn,
                                   {.name = "Reader"});
auto reduced = graph.reduce<int>(readers, split_every{2}, reducer,
                                 {.name = "Merge"});
pipeline.set_source(readers.tasks()[0]);
pipeline.execute();

Public Functions

template<typename T> inline TaskGroup<T> wrap(std::shared_ptr<Task> external_task)¶

Wrap an external task as entry point to the graph

Use this to connect TaskGraph operations to existing barebone tasks.

inline void add(std::shared_ptr<Task> task)¶

Add an external task to the graph

Use this to add utility-adapter tasks or other externally created tasks that should be tracked by the graph but weren’t created via graph methods.

template<typename T, typename Func> inline TaskGroup<T> source(Func &&func, TaskGraphSourceConfig opts = {})¶: Create a single source task

template<typename T, typename Func> inline TaskGroup<T> parallel(std::size_t count, Func &&func, TaskGraphParallelConfig opts = {})¶

Create N parallel tasks (no dependencies between them)

Each task receives its index (0 to count-1) as parameter.

When max_concurrency is set (via options or graph-level default), a sliding window dependency chain limits in-flight tasks: task[i] depends on task[i - max_concurrency], so at most max_concurrency tasks execute simultaneously.

template<typename U, typename T, typename Func> inline TaskGroup fan_out(const TaskGroup<T> &source, num_outputs count, Func &&mapper, TaskGraphFanOutConfig opts = {})¶

Fan-out: 1 -> N (one input produces N outputs)

Each output task receives the source output and its index.

When max_concurrency is set, a sliding window dependency chain limits in-flight fan-out tasks.

template<typename U, typename T, typename Combiner> inline TaskGroup fan_in(const TaskGroup<T> &group, Combiner &&combiner, TaskGraphFanInConfig opts = {})¶: Fan-in: M -> 1 (all inputs combine to single output)

template<typename U, typename T, typename Combiner> inline TaskGroup fan_in(const TaskGroup<T> &group, split_every count, Combiner &&combiner, TaskGraphFanInConfig opts = {})¶: Fan-in with grouping: M -> ceil(M/N) (group every N inputs)

template<typename U, typename T, typename Mapper> inline TaskGroup map(const TaskGroup<T> &group, Mapper &&mapper, TaskGraphMapConfig opts = {})¶

Map: 1-to-1 mapping

When max_concurrency is set, a sliding window dependency chain limits in-flight map tasks.

template<typename U, typename T, typename Reducer> inline TaskGroup reduce(const TaskGroup<T> &group, split_every count, Reducer &&reducer, TaskGraphReduceConfig opts = {})¶: Tree reduce: M -> 1 with O(log N) depth

template<typename T, typename BinaryOp> inline TaskGroup<T> fold(const TaskGroup<T> &group, T init, split_every count, BinaryOp &&op, TaskGraphFoldConfig opts = {})¶

Fold: tree reduction with initial value and binary operation

Each node folds its inputs: op(op(op(init, a), b), c)

template<typename U, typename Intermediate, typename T, typename MapFn, typename ReduceFn> inline TaskGroup aggregate(const TaskGroup<T> &group, MapFn &&map_fn, split_every count, ReduceFn &&reduce_fn, TaskGraphAggregateConfig opts = {})¶

Aggregate: map-reduce pattern

Applies mapper to each task (1:1), then reduces the results.

template<typename T> inline TaskGroup<std::vector<T>> partition(const std::vector<T> &data, num_partitions count, TaskGraphPartitionConfig opts = {})¶

Partition: split data into N contiguous chunks

Each task produces its chunk as std::vector<T>.

template<typename T> inline TaskGroup<std::vector<T>> concat_partitions(const TaskGroup<std::vector<T>> &group, split_every count = split_every{2}, TaskGraphConcatConfig opts = {})¶

Concat partitions: combine vector partitions into a single vector

Uses tree reduction to efficiently concatenate.

inline std::shared_ptr<Task> first_task() const¶: Get first task in the graph (convenience for setting Pipeline source)

inline std::shared_ptr<Task> last_task() const¶: Get last task in the graph (convenience for setting Pipeline destination)

inline const std::vector<std::shared_ptr<Task>> &tasks() const¶: Get all tasks in the graph

inline const std::string &name() const¶: Get graph name

Public Static Functions

static inline TaskGraph builder(TaskGraphConfig opts = {})¶: Create a TaskGraph builder with options