Kinesin/daggy
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
241 Commits 1 Branch 0 Tags
master
Commit Graph

241 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ian Roddis
dded91220f Updating todo 2021-09-23 09:32:07 -03:00
Ian Roddis
d19648fa84 Refactoring CMake configs, adding Catch2 benchmark support 2021-09-22 16:33:50 -03:00
Ian Roddis
510aa236c4 Converting underlying DAG implementation to use vectors for storage 
Reduces hashing required and speeds up traversals
 2021-09-22 14:19:20 -03:00
Ian Roddis
33942e2c63 Update TODO with current goals. 2021-09-22 10:43:03 -03:00
Ian Roddis
e7b15b3847 Implement a bunch of clang-tidy suggested changes. Remove FilesystemLogger 2021-09-22 10:30:27 -03:00
Ian Roddis
288ce28d29 Adding clang-format, and reformating all sourcecode 2021-09-21 09:41:11 -03:00
Ian Roddis
39d5ae08be Adding a No-op task executor for testing 
Fixing DFS implementation of DAG validation to be much faster
Adding in additional tests to ensure the run order of expanded tasks is preserved
Adding additional compile-time checks, resolving issues that came up as a result
 2021-09-20 19:05:56 -03:00
Ian Roddis
2daaa83d82 Small changes for quality of life 2021-09-17 16:36:59 -03:00
Ian Roddis
f83a17d610 Enhancing the performance of dag building by delaying validation until the end 2021-09-16 17:13:40 -03:00
Ian Roddis
fac741742a Adding check for non-existent dagid 2021-09-16 11:03:47 -03:00
Ian Roddis
e895103b1c Increasing max send/receive sizes for REST server to 4G 2021-09-16 10:07:06 -03:00
Ian Roddis
0c1a634ffa Adding in and updating previously commented-out tests 2021-09-15 14:18:44 -03:00
Ian Roddis
d4ec744773 Fixing race condition issue with references in forking executor 2021-09-15 14:12:55 -03:00
Ian Roddis
a6a7501d12 Changing how execution parallelism is handled, so that different 
executors can implement their own idea of parallelism.
 2021-09-15 13:05:04 -03:00
Ian Roddis
4562ac755e Fixing argparse issue and populating stdin file 2021-09-14 15:04:23 -03:00
Ian Roddis
96c1928daf Adding fixes for slurm linking and uncommon slurm node attributes 2021-09-11 00:00:21 -03:00
Ian Roddis
ff5a75abb0 Removing gitlab CI for now 2021-09-10 13:36:14 -03:00
Ian Roddis
abdbf9f2bf Adding missing CMakeLists.txt 2021-09-10 13:34:56 -03:00
Ian Roddis
d519b11247 Fixing slurm detection in cmake 2021-09-10 13:33:53 -03:00
Ian Roddis
ea53735069 Adjusting path after argparse changed its structure out from underneath me 2021-09-10 12:16:25 -03:00
Ian Roddis
f8cf2d48a5 Adding find_library for slurm 2021-09-10 11:10:20 -03:00
Ian Roddis
2f0548cb0a Disabling gitlab CI for now 2021-09-10 10:55:22 -03:00
Ian Roddis
d731f9f5b1 Adding support for execution on slurm grids 
- Adding support for SlurmTaskExecutor in `daggyd` if DAGGY_ENABLE_SLURM is defined.
- Renaming some test cases
- Enabling compile-time slurm support
- Adding slurm documentation
 2021-09-10 10:53:58 -03:00
Ian Roddis
d15580f47f Massive re-org to allow per-executor job specification formats and executor-specific task validation and expansion. 
A few different renames to try and keep things more consistent.
 2021-09-03 09:10:38 -03:00
Ian Roddis
e746f8c163 - Adding daemonization for daggyd 
- Changing how parameter keys are stored in the map to allow for future key handling.
 2021-09-01 13:32:16 -03:00
Ian Roddis
4e71bf5917 Adding daggyc rough-in 2021-08-31 19:37:23 -03:00
Ian Roddis
77f7819584 Fixing test where return by reference would cause unexpected results when compiled with release flags 2021-08-31 19:14:34 -03:00
Ian Roddis
441cdb5b2e Renaming rest_server to daggyd, and daggy to libdaggy 2021-08-31 17:04:03 -03:00
Ian Roddis
7b07380e16 - Removing duplicate information (taskName stored in 3 places) 2021-08-31 12:21:34 -03:00
Ian Roddis
2c00001e0b Adding support for isGenerator tasks 
- Changing how DAG is represented, both in code and how DAGs are defined
  in JSON.
- Removing std::vector<Task> representation in favour of a map that will
  enforce unique task names
- Task names now have a name (generated), and a definedName.
- Adding support to loggers to add tasks after a DAGRun has been
  initialized.
 2021-08-30 22:05:37 -03:00
Ian Roddis
dd6159dda8 Adding simple defaults for rest_server for testing 2021-08-30 12:47:06 -03:00
Ian Roddis
307e0c0559 Converting to better_enums to support gcc 8 2021-08-27 12:06:36 -03:00
Ian Roddis
7fa47c1c4a More CI stupidity 2021-08-25 16:16:58 -03:00
Ian Roddis
fcdc225e28 Adjusting test case to eventually exit with an error after a reasonable time. 2021-08-25 15:20:08 -03:00
Ian Roddis
212bd80df2 Squashed commit of the following: 
commit 73994327de890590eede353c8131f3f7c1e8aaa3
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Aug 25 13:38:29 2021 -0300

    - Fixing up checks for individual dag runs

commit f20e3a3dec8c063111cf60f2bec2b8f84c8a4100
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Aug 25 10:49:43 2021 -0300

    - Finishing serialization of DAGRun
    - Checkpointing work.

commit b490abadf93e3085e4204003de7eaa8183b4e1d5
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Aug 25 10:34:08 2021 -0300

    - Consolidating struct definitions into Defines.hpp
    - Renaming DAGRunRecord member runStates to taskRunStates

commit 050346ec1fd10d1091f261905c6175ffe0bcf001
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Aug 25 09:27:05 2021 -0300

    - Adding additional tests for server endpoints
 2021-08-25 13:40:05 -03:00
Ian Roddis
6ed57806d0 - More fixes for CI failing tests 2021-08-24 07:58:58 -03:00
Ian Roddis
39fe716968 CI: Adding missing dependencies 2021-08-23 21:05:27 +00:00
Ian Roddis
307cff73f8 - More fixes for CI failing tests 2021-08-23 17:58:43 -03:00
Ian Roddis
7016cb37ba - More fixes for CI failing tests 2021-08-23 17:53:47 -03:00
Ian Roddis
cf37f9a6fc - More fixes for CI failing tests 2021-08-23 17:49:04 -03:00
Ian Roddis
1bdc5dafd5 - More fixes for CI failing tests 2021-08-23 17:42:43 -03:00
Ian Roddis
d4c9b3fcee - Adjusting tests so they run without referencing /tmp 2021-08-23 17:37:33 -03:00
Ian Roddis
8eb966377f CI: Adding some debugs and assuming test script picks up where build left off. 2021-08-23 20:26:34 +00:00
Ian Roddis
2460b24b04 CI: Adding some debugs and assuming test script picks up where build left off. 2021-08-23 20:21:14 +00:00
Ian Roddis
461900b18d CI: Adding some debugs and assuming test script picks up where build left off. 2021-08-23 20:16:34 +00:00
Ian Roddis
22accec2a9 Adding missing packages 2021-08-23 20:11:06 +00:00
Ian Roddis
fb3c450bfc Setting image to gcc for CI Pipeline 2021-08-23 20:09:09 +00:00
Ian Roddis
574f6c1fb1 Update .gitlab-ci.yml file 2021-08-23 20:07:00 +00:00
Ian Roddis
be20bb5665 Adding MIT License 2021-08-23 16:51:27 -03:00
Ian Roddis
c12309c3f6 Adding MIT License 2021-08-23 16:51:20 -03:00