Go to file

Ian Roddis 28c5b3eea3 - Adding StdOutLogger and adding tests for DAG execution to verify it works.

- Roughing in FileSystemLogger
- Deleting Scheduler code and associated unit tests as being too complicated for maintenance.
- Refactoring namespaces for loggers and executors.

2021-08-09 14:59:54 -03:00

cmake

- Adding StdOutLogger and adding tests for DAG execution to verify it works.

2021-08-09 14:59:23 -03:00

daggy

- Adding StdOutLogger and adding tests for DAG execution to verify it works.

2021-08-09 14:59:54 -03:00

examples

Checkpointing progress on task builder.

2021-08-02 11:59:43 -03:00

tests

- Adding StdOutLogger and adding tests for DAG execution to verify it works.

2021-08-09 14:59:54 -03:00

.gitignore

- Adding TODO list to keep track of things.

2021-08-05 15:24:17 -03:00

CMakeLists.txt

- Adding StdOutLogger and adding tests for DAG execution to verify it works.

2021-08-09 14:59:23 -03:00

README.md

Adding JSON parsing helpers to Utilities.

2021-08-02 11:31:46 -03:00

TODO.md

- Roughing out the DAGLogger interface

2021-08-09 10:39:46 -03:00

README.md

Daggy: Ya like dags?

Description

Daggy is a work orchestration framework for running workflows modeled as directed, acyclic graphs (DAGs). These are quite useful when modeling data ingestion / processing pipelines.

Building

git clone https://gitlab.com/iroddis/daggy
cd daggy
mkdir build
cd build
cmake ..
make

Architecture

Server
- Interactive endpoint
- DAGs
  - CRUDs DAG definitions
  - Kick off runs
  - Return status
Scheduler
- Accepts Executors
- Accepts task lists and parameters
- Runs and monitors DAGs

Flow

DAG Generated from JSON spec
Execution Pool Selected
DAGRun created from storage
DAGExecutor runs with

Logger object
Execution Pool
DAG definition

Languages

C++ 88.7%

Vue 6.3%

CMake 3.6%

CSS 0.9%

JavaScript 0.3%

Other 0.2%