daggy/README.md

Daggy: Ya like dags?

Description

Daggy is a work orchestration framework for running workflows modeled as directed, acyclic graphs (DAGs). These are quite useful when modeling data ingestion / processing pipelines.

Building

git clone https://gitlab.com/iroddis/daggy
cd daggy
mkdir build
cd build
cmake ..
make

Architecture

• Server
  • Interactive endpoint
  • DAGs
    • CRUDs DAG definitions
    • Kick off runs
    • Return status
• Scheduler
  • Accepts Executors
  • Accepts task lists and parameters
  • Runs and monitors DAGs

Flow

1. DAG Generated from JSON spec
2. Execution Pool Selected
3. DAGRun created from storage
4. DAGExecutor runs with

• Logger object
• Execution Pool
• DAG definition