Commit Graph

20 Commits

Author SHA1 Message Date
Ian Roddis
288ce28d29 Adding clang-format, and reformating all sourcecode 2021-09-21 09:41:11 -03:00
Ian Roddis
39d5ae08be Adding a No-op task executor for testing
Fixing DFS implementation of DAG validation to be much faster
Adding in additional tests to ensure the run order of expanded tasks is preserved
Adding additional compile-time checks, resolving issues that came up as a result
2021-09-20 19:05:56 -03:00
Ian Roddis
2daaa83d82 Small changes for quality of life 2021-09-17 16:36:59 -03:00
Ian Roddis
f83a17d610 Enhancing the performance of dag building by delaying validation until the end 2021-09-16 17:13:40 -03:00
Ian Roddis
0c1a634ffa Adding in and updating previously commented-out tests 2021-09-15 14:18:44 -03:00
Ian Roddis
d4ec744773 Fixing race condition issue with references in forking executor 2021-09-15 14:12:55 -03:00
Ian Roddis
d731f9f5b1 Adding support for execution on slurm grids
- Adding support for SlurmTaskExecutor in `daggyd` if DAGGY_ENABLE_SLURM is defined.
- Renaming some test cases
- Enabling compile-time slurm support
- Adding slurm documentation
2021-09-10 10:53:58 -03:00
Ian Roddis
d15580f47f Massive re-org to allow per-executor job specification formats and executor-specific task validation and expansion.
A few different renames to try and keep things more consistent.
2021-09-03 09:10:38 -03:00
Ian Roddis
2c00001e0b Adding support for isGenerator tasks
- Changing how DAG is represented, both in code and how DAGs are defined
  in JSON.
- Removing std::vector<Task> representation in favour of a map that will
  enforce unique task names
- Task names now have a name (generated), and a definedName.
- Adding support to loggers to add tasks after a DAGRun has been
  initialized.
2021-08-30 22:05:37 -03:00
Ian Roddis
d4c9b3fcee - Adjusting tests so they run without referencing /tmp 2021-08-23 17:37:33 -03:00
Ian Roddis
2083b1c3f1 Refactoring RunState, fixing logic error in when to end errored DAG runs, added convenience functions to retry failed DAGs. 2021-08-20 21:44:12 -03:00
Ian Roddis
dc8ea4c369 - Adding support for state to OStreamLogger to make it more useful for test cases
- Making runDAG return the end DAG
- Adding much more robust test for DAG execution for basic tests.
2021-08-20 12:43:01 -03:00
Ian Roddis
de8025e0f9 - Fixing bug that yielded empty expanded command sets. 2021-08-20 10:58:10 -03:00
Ian Roddis
1668117566 - Checkpointing work on expanding commands. 2021-08-20 10:39:39 -03:00
Ian Roddis
a97c3ff783 - Removing Catch2 code from codebase, will pull it via FetchContent instead.
- Changing StdOutLogger to OStreamLogger, so that test cases output can be silenced.
2021-08-09 15:48:53 -03:00
Ian Roddis
a8e85f8feb - Adding StdOutLogger and adding tests for DAG execution to verify it works.
- Roughing in FileSystemLogger
- Deleting Scheduler code and associated unit tests as being too complicated for maintenance.
- Refactoring namespaces for loggers and executors.
2021-08-09 14:59:23 -03:00
Ian Roddis
0516bde09f - Refactoring struct (de)serialization 2021-08-09 10:33:49 -03:00
Ian Roddis
1849a2fee4 - Adding TODO list to keep track of things.
- Adding .idea to gitignore
- Adding DAG execution to utilities, redefining the logger
2021-08-05 15:24:17 -03:00
Ian Roddis
afcb917cc9 Completing task building. 2021-08-02 15:44:41 -03:00
Ian Roddis
54e8170c68 Adding JSON parsing helpers to Utilities. 2021-08-02 11:31:46 -03:00