45 lines
737 B
Markdown
45 lines
737 B
Markdown
Daggy: Ya like dags?
|
|
|
|
Description
|
|
==
|
|
|
|
Daggy is a work orchestration framework for running workflows modeled as
|
|
directed, acyclic graphs (DAGs). These are quite useful when modeling
|
|
data ingestion / processing pipelines.
|
|
|
|
Building
|
|
==
|
|
|
|
```
|
|
git clone https://gitlab.com/iroddis/daggy
|
|
cd daggy
|
|
mkdir build
|
|
cd build
|
|
cmake ..
|
|
make
|
|
```
|
|
|
|
Architecture
|
|
==
|
|
|
|
- Server
|
|
- Interactive endpoint
|
|
- DAGs
|
|
- CRUDs DAG definitions
|
|
- Kick off runs
|
|
- Return status
|
|
- Scheduler
|
|
- Accepts Executors
|
|
- Accepts task lists and parameters
|
|
- Runs and monitors DAGs
|
|
|
|
Flow
|
|
==
|
|
|
|
1. DAG Generated from JSON spec
|
|
2. Execution Pool Selected
|
|
3. DAGRun created from storage
|
|
4. DAGExecutor runs with
|
|
- Logger object
|
|
- Execution Pool
|
|
- DAG definition |