Commit Graph

142 Commits

Author SHA1 Message Date
Ian Roddis
65ab439848 Squashed commit of the following:
commit b06b11cbb5d09c6d091551e39767cd3316f88376
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Oct 5 11:57:37 2021 -0300

    Fixing failing unit test

commit fe2a43a19b2a16a9aedd9e95e71e672935ecaeb1
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Oct 5 11:54:01 2021 -0300

    Adding endpoints and updating documentation

commit 46e0deeefb8b06291ae5e2d6b8ec4749c5b0ea6f
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Oct 5 11:49:43 2021 -0300

    Completing unit tests and relevant fixes

commit e0569f370624844feee6aae4708bfe683f4156cf
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Oct 4 17:30:59 2021 -0300

    Adding in gcc tsan for debug builds to help with race conditions, fixing many of those, and fixing really crummy assumption about how futures worked that will speed up task execution by a ton.

commit c748a4f592e1ada5546908be5281d04f4749539d
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Oct 4 10:14:43 2021 -0300

    Checkpointing work that seems to have resolved the race condition

commit 7a79f2943e0d50545d976a28b4b379340a90dded
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Sep 29 09:27:07 2021 -0300

    Completing the rough-in for DAG killing / pausing / resuming

commit 4cf8d81d5f6fcf4a7dd83d8fca3e23f153aa8acb
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Sep 28 14:53:50 2021 -0300

    Adding dagrunner unit tests, adding a resetRunning method to resume

commit 54e2c1f9f5e7d5b339d71be024e0e390c4d2bf61
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Sep 28 14:45:57 2021 -0300

    Refactoring runDAG into DAGRunner

commit 682be7a11e2fae850e1bc3e207628d2335768c2b
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Sep 28 14:34:43 2021 -0300

    Adding DAGRunner class to replace Utilities::runDAG, making Slurm cancellation rc agree with SIGKILL

commit 4171b3a6998791abfc71b04f8de1ae93c4f90a78
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Sep 28 14:14:17 2021 -0300

    Adding unit tests for stopping jobs to slurm

commit dc0b1ff26a5d98471164132d35bb8a552cc75ff8
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Sep 28 14:04:15 2021 -0300

    Adding in stop method for task executors

commit e752b44f55113be54392bcbb5c3d6f251d673cfa
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Sep 28 12:32:06 2021 -0300

    Adding additional tests for loggers

commit f0773d5a84a422738fc17c9277a2b735a21a3d04
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Sep 28 12:29:21 2021 -0300

    Unit tests pass

commit 993ff2810de2d53dc6a59ab53d620fecf152d4a0
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Sep 28 12:24:34 2021 -0300

    Adding handling for new routes, still need to add tests for new routes

commit 676623b14e45759872a2dbcbc98f6a744e022a71
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Sep 28 12:12:43 2021 -0300

    Adding handling for new routes, still need to add tests for new routes

commit b9edb6ba291eb064f4c459a308ea6912fba9fa02
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Sep 27 11:59:14 2021 -0300

    Defining new endpoints, fixing dag resumption code, adding PAUSED state, refactoring DAGSpec and adding deserializer
2021-10-05 11:57:55 -03:00
Ian Roddis
dded91220f Updating todo 2021-09-23 09:32:07 -03:00
Ian Roddis
d19648fa84 Refactoring CMake configs, adding Catch2 benchmark support 2021-09-22 16:33:50 -03:00
Ian Roddis
510aa236c4 Converting underlying DAG implementation to use vectors for storage
Reduces hashing required and speeds up traversals
2021-09-22 14:19:20 -03:00
Ian Roddis
33942e2c63 Update TODO with current goals. 2021-09-22 10:43:03 -03:00
Ian Roddis
e7b15b3847 Implement a bunch of clang-tidy suggested changes. Remove FilesystemLogger 2021-09-22 10:30:27 -03:00
Ian Roddis
288ce28d29 Adding clang-format, and reformating all sourcecode 2021-09-21 09:41:11 -03:00
Ian Roddis
39d5ae08be Adding a No-op task executor for testing
Fixing DFS implementation of DAG validation to be much faster
Adding in additional tests to ensure the run order of expanded tasks is preserved
Adding additional compile-time checks, resolving issues that came up as a result
2021-09-20 19:05:56 -03:00
Ian Roddis
2daaa83d82 Small changes for quality of life 2021-09-17 16:36:59 -03:00
Ian Roddis
f83a17d610 Enhancing the performance of dag building by delaying validation until the end 2021-09-16 17:13:40 -03:00
Ian Roddis
fac741742a Adding check for non-existent dagid 2021-09-16 11:03:47 -03:00
Ian Roddis
e895103b1c Increasing max send/receive sizes for REST server to 4G 2021-09-16 10:07:06 -03:00
Ian Roddis
0c1a634ffa Adding in and updating previously commented-out tests 2021-09-15 14:18:44 -03:00
Ian Roddis
d4ec744773 Fixing race condition issue with references in forking executor 2021-09-15 14:12:55 -03:00
Ian Roddis
a6a7501d12 Changing how execution parallelism is handled, so that different
executors can implement their own idea of parallelism.
2021-09-15 13:05:04 -03:00
Ian Roddis
4562ac755e Fixing argparse issue and populating stdin file 2021-09-14 15:04:23 -03:00
Ian Roddis
96c1928daf Adding fixes for slurm linking and uncommon slurm node attributes 2021-09-11 00:00:21 -03:00
Ian Roddis
ff5a75abb0 Removing gitlab CI for now 2021-09-10 13:36:14 -03:00
Ian Roddis
abdbf9f2bf Adding missing CMakeLists.txt 2021-09-10 13:34:56 -03:00
Ian Roddis
d519b11247 Fixing slurm detection in cmake 2021-09-10 13:33:53 -03:00
Ian Roddis
ea53735069 Adjusting path after argparse changed its structure out from underneath me 2021-09-10 12:16:25 -03:00
Ian Roddis
f8cf2d48a5 Adding find_library for slurm 2021-09-10 11:10:20 -03:00
Ian Roddis
2f0548cb0a Disabling gitlab CI for now 2021-09-10 10:55:22 -03:00
Ian Roddis
d731f9f5b1 Adding support for execution on slurm grids
- Adding support for SlurmTaskExecutor in `daggyd` if DAGGY_ENABLE_SLURM is defined.
- Renaming some test cases
- Enabling compile-time slurm support
- Adding slurm documentation
2021-09-10 10:53:58 -03:00
Ian Roddis
d15580f47f Massive re-org to allow per-executor job specification formats and executor-specific task validation and expansion.
A few different renames to try and keep things more consistent.
2021-09-03 09:10:38 -03:00
Ian Roddis
e746f8c163 - Adding daemonization for daggyd
- Changing how parameter keys are stored in the map to allow for future key handling.
2021-09-01 13:32:16 -03:00
Ian Roddis
4e71bf5917 Adding daggyc rough-in 2021-08-31 19:37:23 -03:00
Ian Roddis
77f7819584 Fixing test where return by reference would cause unexpected results when compiled with release flags 2021-08-31 19:14:34 -03:00
Ian Roddis
441cdb5b2e Renaming rest_server to daggyd, and daggy to libdaggy 2021-08-31 17:04:03 -03:00
Ian Roddis
7b07380e16 - Removing duplicate information (taskName stored in 3 places) 2021-08-31 12:21:34 -03:00
Ian Roddis
2c00001e0b Adding support for isGenerator tasks
- Changing how DAG is represented, both in code and how DAGs are defined
  in JSON.
- Removing std::vector<Task> representation in favour of a map that will
  enforce unique task names
- Task names now have a name (generated), and a definedName.
- Adding support to loggers to add tasks after a DAGRun has been
  initialized.
2021-08-30 22:05:37 -03:00
Ian Roddis
dd6159dda8 Adding simple defaults for rest_server for testing 2021-08-30 12:47:06 -03:00
Ian Roddis
307e0c0559 Converting to better_enums to support gcc 8 2021-08-27 12:06:36 -03:00
Ian Roddis
7fa47c1c4a More CI stupidity 2021-08-25 16:16:58 -03:00
Ian Roddis
fcdc225e28 Adjusting test case to eventually exit with an error after a reasonable time. 2021-08-25 15:20:08 -03:00
Ian Roddis
212bd80df2 Squashed commit of the following:
commit 73994327de890590eede353c8131f3f7c1e8aaa3
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Aug 25 13:38:29 2021 -0300

    - Fixing up checks for individual dag runs

commit f20e3a3dec8c063111cf60f2bec2b8f84c8a4100
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Aug 25 10:49:43 2021 -0300

    - Finishing serialization of DAGRun
    - Checkpointing work.

commit b490abadf93e3085e4204003de7eaa8183b4e1d5
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Aug 25 10:34:08 2021 -0300

    - Consolidating struct definitions into Defines.hpp
    - Renaming DAGRunRecord member runStates to taskRunStates

commit 050346ec1fd10d1091f261905c6175ffe0bcf001
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Aug 25 09:27:05 2021 -0300

    - Adding additional tests for server endpoints
2021-08-25 13:40:05 -03:00
Ian Roddis
6ed57806d0 - More fixes for CI failing tests 2021-08-24 07:58:58 -03:00
Ian Roddis
39fe716968 CI: Adding missing dependencies 2021-08-23 21:05:27 +00:00
Ian Roddis
307cff73f8 - More fixes for CI failing tests 2021-08-23 17:58:43 -03:00
Ian Roddis
7016cb37ba - More fixes for CI failing tests 2021-08-23 17:53:47 -03:00
Ian Roddis
cf37f9a6fc - More fixes for CI failing tests 2021-08-23 17:49:04 -03:00
Ian Roddis
1bdc5dafd5 - More fixes for CI failing tests 2021-08-23 17:42:43 -03:00
Ian Roddis
d4c9b3fcee - Adjusting tests so they run without referencing /tmp 2021-08-23 17:37:33 -03:00
Ian Roddis
8eb966377f CI: Adding some debugs and assuming test script picks up where build left off. 2021-08-23 20:26:34 +00:00
Ian Roddis
2460b24b04 CI: Adding some debugs and assuming test script picks up where build left off. 2021-08-23 20:21:14 +00:00
Ian Roddis
461900b18d CI: Adding some debugs and assuming test script picks up where build left off. 2021-08-23 20:16:34 +00:00
Ian Roddis
22accec2a9 Adding missing packages 2021-08-23 20:11:06 +00:00
Ian Roddis
fb3c450bfc Setting image to gcc for CI Pipeline 2021-08-23 20:09:09 +00:00
Ian Roddis
574f6c1fb1 Update .gitlab-ci.yml file 2021-08-23 20:07:00 +00:00
Ian Roddis
be20bb5665 Adding MIT License 2021-08-23 16:51:27 -03:00