Commit Graph

23 Commits

Author SHA1 Message Date
Ian Roddis
65ab439848 Squashed commit of the following:
commit b06b11cbb5d09c6d091551e39767cd3316f88376
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Oct 5 11:57:37 2021 -0300

    Fixing failing unit test

commit fe2a43a19b2a16a9aedd9e95e71e672935ecaeb1
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Oct 5 11:54:01 2021 -0300

    Adding endpoints and updating documentation

commit 46e0deeefb8b06291ae5e2d6b8ec4749c5b0ea6f
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Oct 5 11:49:43 2021 -0300

    Completing unit tests and relevant fixes

commit e0569f370624844feee6aae4708bfe683f4156cf
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Oct 4 17:30:59 2021 -0300

    Adding in gcc tsan for debug builds to help with race conditions, fixing many of those, and fixing really crummy assumption about how futures worked that will speed up task execution by a ton.

commit c748a4f592e1ada5546908be5281d04f4749539d
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Oct 4 10:14:43 2021 -0300

    Checkpointing work that seems to have resolved the race condition

commit 7a79f2943e0d50545d976a28b4b379340a90dded
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Sep 29 09:27:07 2021 -0300

    Completing the rough-in for DAG killing / pausing / resuming

commit 4cf8d81d5f6fcf4a7dd83d8fca3e23f153aa8acb
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Sep 28 14:53:50 2021 -0300

    Adding dagrunner unit tests, adding a resetRunning method to resume

commit 54e2c1f9f5e7d5b339d71be024e0e390c4d2bf61
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Sep 28 14:45:57 2021 -0300

    Refactoring runDAG into DAGRunner

commit 682be7a11e2fae850e1bc3e207628d2335768c2b
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Sep 28 14:34:43 2021 -0300

    Adding DAGRunner class to replace Utilities::runDAG, making Slurm cancellation rc agree with SIGKILL

commit 4171b3a6998791abfc71b04f8de1ae93c4f90a78
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Sep 28 14:14:17 2021 -0300

    Adding unit tests for stopping jobs to slurm

commit dc0b1ff26a5d98471164132d35bb8a552cc75ff8
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Sep 28 14:04:15 2021 -0300

    Adding in stop method for task executors

commit e752b44f55113be54392bcbb5c3d6f251d673cfa
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Sep 28 12:32:06 2021 -0300

    Adding additional tests for loggers

commit f0773d5a84a422738fc17c9277a2b735a21a3d04
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Sep 28 12:29:21 2021 -0300

    Unit tests pass

commit 993ff2810de2d53dc6a59ab53d620fecf152d4a0
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Sep 28 12:24:34 2021 -0300

    Adding handling for new routes, still need to add tests for new routes

commit 676623b14e45759872a2dbcbc98f6a744e022a71
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Sep 28 12:12:43 2021 -0300

    Adding handling for new routes, still need to add tests for new routes

commit b9edb6ba291eb064f4c459a308ea6912fba9fa02
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Sep 27 11:59:14 2021 -0300

    Defining new endpoints, fixing dag resumption code, adding PAUSED state, refactoring DAGSpec and adding deserializer
2021-10-05 11:57:55 -03:00
Ian Roddis
510aa236c4 Converting underlying DAG implementation to use vectors for storage
Reduces hashing required and speeds up traversals
2021-09-22 14:19:20 -03:00
Ian Roddis
e7b15b3847 Implement a bunch of clang-tidy suggested changes. Remove FilesystemLogger 2021-09-22 10:30:27 -03:00
Ian Roddis
288ce28d29 Adding clang-format, and reformating all sourcecode 2021-09-21 09:41:11 -03:00
Ian Roddis
39d5ae08be Adding a No-op task executor for testing
Fixing DFS implementation of DAG validation to be much faster
Adding in additional tests to ensure the run order of expanded tasks is preserved
Adding additional compile-time checks, resolving issues that came up as a result
2021-09-20 19:05:56 -03:00
Ian Roddis
2daaa83d82 Small changes for quality of life 2021-09-17 16:36:59 -03:00
Ian Roddis
f83a17d610 Enhancing the performance of dag building by delaying validation until the end 2021-09-16 17:13:40 -03:00
Ian Roddis
0c1a634ffa Adding in and updating previously commented-out tests 2021-09-15 14:18:44 -03:00
Ian Roddis
d4ec744773 Fixing race condition issue with references in forking executor 2021-09-15 14:12:55 -03:00
Ian Roddis
d731f9f5b1 Adding support for execution on slurm grids
- Adding support for SlurmTaskExecutor in `daggyd` if DAGGY_ENABLE_SLURM is defined.
- Renaming some test cases
- Enabling compile-time slurm support
- Adding slurm documentation
2021-09-10 10:53:58 -03:00
Ian Roddis
d15580f47f Massive re-org to allow per-executor job specification formats and executor-specific task validation and expansion.
A few different renames to try and keep things more consistent.
2021-09-03 09:10:38 -03:00
Ian Roddis
2c00001e0b Adding support for isGenerator tasks
- Changing how DAG is represented, both in code and how DAGs are defined
  in JSON.
- Removing std::vector<Task> representation in favour of a map that will
  enforce unique task names
- Task names now have a name (generated), and a definedName.
- Adding support to loggers to add tasks after a DAGRun has been
  initialized.
2021-08-30 22:05:37 -03:00
Ian Roddis
d4c9b3fcee - Adjusting tests so they run without referencing /tmp 2021-08-23 17:37:33 -03:00
Ian Roddis
2083b1c3f1 Refactoring RunState, fixing logic error in when to end errored DAG runs, added convenience functions to retry failed DAGs. 2021-08-20 21:44:12 -03:00
Ian Roddis
dc8ea4c369 - Adding support for state to OStreamLogger to make it more useful for test cases
- Making runDAG return the end DAG
- Adding much more robust test for DAG execution for basic tests.
2021-08-20 12:43:01 -03:00
Ian Roddis
de8025e0f9 - Fixing bug that yielded empty expanded command sets. 2021-08-20 10:58:10 -03:00
Ian Roddis
1668117566 - Checkpointing work on expanding commands. 2021-08-20 10:39:39 -03:00
Ian Roddis
a97c3ff783 - Removing Catch2 code from codebase, will pull it via FetchContent instead.
- Changing StdOutLogger to OStreamLogger, so that test cases output can be silenced.
2021-08-09 15:48:53 -03:00
Ian Roddis
a8e85f8feb - Adding StdOutLogger and adding tests for DAG execution to verify it works.
- Roughing in FileSystemLogger
- Deleting Scheduler code and associated unit tests as being too complicated for maintenance.
- Refactoring namespaces for loggers and executors.
2021-08-09 14:59:23 -03:00
Ian Roddis
0516bde09f - Refactoring struct (de)serialization 2021-08-09 10:33:49 -03:00
Ian Roddis
1849a2fee4 - Adding TODO list to keep track of things.
- Adding .idea to gitignore
- Adding DAG execution to utilities, redefining the logger
2021-08-05 15:24:17 -03:00
Ian Roddis
afcb917cc9 Completing task building. 2021-08-02 15:44:41 -03:00
Ian Roddis
54e8170c68 Adding JSON parsing helpers to Utilities. 2021-08-02 11:31:46 -03:00