Daggy: Ya like dags?
Description
Daggy is a work orchestration framework for running workflows modeled as directed, acyclic graphs (DAGs). These are quite useful when modeling data ingestion / processing pipelines.
Building
git clone https://gitlab.com/iroddis/daggy
cd daggy
mkdir build
cd build
cmake ..
make
Architecture
- Server
- Interactive endpoint
- DAGs
- CRUDs DAG definitions
- Kick off runs
- Return status
- Scheduler
- Accepts Executors
- Accepts task lists and parameters
- Runs and monitors DAGs
Flow
- DAG Generated from JSON spec
- Execution Pool Selected
- DAGRun created from storage
- DAGExecutor runs with
- Logger object
- Execution Pool
- DAG definition