Kinesin/daggy
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
141 Commits 1 Branch 0 Tags
dded91220fe3c1aee108ce3277e733f16ca482b8
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Ian Roddis dded91220f Updating todo 2021-09-23 09:32:07 -03:00
Ian Roddis d19648fa84 Refactoring CMake configs, adding Catch2 benchmark support 2021-09-22 16:33:50 -03:00
Ian Roddis 510aa236c4 Converting underlying DAG implementation to use vectors for storage 
Reduces hashing required and speeds up traversals
 2021-09-22 14:19:20 -03:00
Ian Roddis 33942e2c63 Update TODO with current goals. 2021-09-22 10:43:03 -03:00
Ian Roddis e7b15b3847 Implement a bunch of clang-tidy suggested changes. Remove FilesystemLogger 2021-09-22 10:30:27 -03:00
Ian Roddis 288ce28d29 Adding clang-format, and reformating all sourcecode 2021-09-21 09:41:11 -03:00
Ian Roddis 39d5ae08be Adding a No-op task executor for testing 
Fixing DFS implementation of DAG validation to be much faster
Adding in additional tests to ensure the run order of expanded tasks is preserved
Adding additional compile-time checks, resolving issues that came up as a result
 2021-09-20 19:05:56 -03:00
Ian Roddis 2daaa83d82 Small changes for quality of life 2021-09-17 16:36:59 -03:00
Ian Roddis f83a17d610 Enhancing the performance of dag building by delaying validation until the end 2021-09-16 17:13:40 -03:00
Ian Roddis fac741742a Adding check for non-existent dagid 2021-09-16 11:03:47 -03:00
Ian Roddis e895103b1c Increasing max send/receive sizes for REST server to 4G 2021-09-16 10:07:06 -03:00
Ian Roddis 0c1a634ffa Adding in and updating previously commented-out tests 2021-09-15 14:18:44 -03:00
Ian Roddis d4ec744773 Fixing race condition issue with references in forking executor 2021-09-15 14:12:55 -03:00
Ian Roddis a6a7501d12 Changing how execution parallelism is handled, so that different 
executors can implement their own idea of parallelism.
 2021-09-15 13:05:04 -03:00
Ian Roddis 4562ac755e Fixing argparse issue and populating stdin file 2021-09-14 15:04:23 -03:00
Ian Roddis 96c1928daf Adding fixes for slurm linking and uncommon slurm node attributes 2021-09-11 00:00:21 -03:00
Ian Roddis ff5a75abb0 Removing gitlab CI for now 2021-09-10 13:36:14 -03:00
Ian Roddis abdbf9f2bf Adding missing CMakeLists.txt 2021-09-10 13:34:56 -03:00
Ian Roddis d519b11247 Fixing slurm detection in cmake 2021-09-10 13:33:53 -03:00
Ian Roddis ea53735069 Adjusting path after argparse changed its structure out from underneath me 2021-09-10 12:16:25 -03:00
Ian Roddis f8cf2d48a5 Adding find_library for slurm 2021-09-10 11:10:20 -03:00
Ian Roddis 2f0548cb0a Disabling gitlab CI for now 2021-09-10 10:55:22 -03:00
Ian Roddis d731f9f5b1 Adding support for execution on slurm grids 
- Adding support for SlurmTaskExecutor in `daggyd` if DAGGY_ENABLE_SLURM is defined.
- Renaming some test cases
- Enabling compile-time slurm support
- Adding slurm documentation
 2021-09-10 10:53:58 -03:00
Ian Roddis d15580f47f Massive re-org to allow per-executor job specification formats and executor-specific task validation and expansion. 
A few different renames to try and keep things more consistent.
 2021-09-03 09:10:38 -03:00
Ian Roddis e746f8c163 - Adding daemonization for daggyd 
- Changing how parameter keys are stored in the map to allow for future key handling.
 2021-09-01 13:32:16 -03:00
Ian Roddis 4e71bf5917 Adding daggyc rough-in 2021-08-31 19:37:23 -03:00
Ian Roddis 77f7819584 Fixing test where return by reference would cause unexpected results when compiled with release flags 2021-08-31 19:14:34 -03:00
Ian Roddis 441cdb5b2e Renaming rest_server to daggyd, and daggy to libdaggy 2021-08-31 17:04:03 -03:00
Ian Roddis 7b07380e16 - Removing duplicate information (taskName stored in 3 places) 2021-08-31 12:21:34 -03:00
Ian Roddis 2c00001e0b Adding support for isGenerator tasks 
- Changing how DAG is represented, both in code and how DAGs are defined
  in JSON.
- Removing std::vector<Task> representation in favour of a map that will
  enforce unique task names
- Task names now have a name (generated), and a definedName.
- Adding support to loggers to add tasks after a DAGRun has been
  initialized.
 2021-08-30 22:05:37 -03:00
Ian Roddis dd6159dda8 Adding simple defaults for rest_server for testing 2021-08-30 12:47:06 -03:00
Ian Roddis 307e0c0559 Converting to better_enums to support gcc 8 2021-08-27 12:06:36 -03:00
Ian Roddis 7fa47c1c4a More CI stupidity 2021-08-25 16:16:58 -03:00
Ian Roddis fcdc225e28 Adjusting test case to eventually exit with an error after a reasonable time. 2021-08-25 15:20:08 -03:00
Ian Roddis 212bd80df2 Squashed commit of the following: 
commit 73994327de890590eede353c8131f3f7c1e8aaa3
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Aug 25 13:38:29 2021 -0300

    - Fixing up checks for individual dag runs

commit f20e3a3dec8c063111cf60f2bec2b8f84c8a4100
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Aug 25 10:49:43 2021 -0300

    - Finishing serialization of DAGRun
    - Checkpointing work.

commit b490abadf93e3085e4204003de7eaa8183b4e1d5
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Aug 25 10:34:08 2021 -0300

    - Consolidating struct definitions into Defines.hpp
    - Renaming DAGRunRecord member runStates to taskRunStates

commit 050346ec1fd10d1091f261905c6175ffe0bcf001
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Aug 25 09:27:05 2021 -0300

    - Adding additional tests for server endpoints
 2021-08-25 13:40:05 -03:00
Ian Roddis 6ed57806d0 - More fixes for CI failing tests 2021-08-24 07:58:58 -03:00
Ian Roddis 39fe716968 CI: Adding missing dependencies 2021-08-23 21:05:27 +00:00
Ian Roddis 307cff73f8 - More fixes for CI failing tests 2021-08-23 17:58:43 -03:00
Ian Roddis 7016cb37ba - More fixes for CI failing tests 2021-08-23 17:53:47 -03:00
Ian Roddis cf37f9a6fc - More fixes for CI failing tests 2021-08-23 17:49:04 -03:00
Ian Roddis 1bdc5dafd5 - More fixes for CI failing tests 2021-08-23 17:42:43 -03:00
Ian Roddis d4c9b3fcee - Adjusting tests so they run without referencing /tmp 2021-08-23 17:37:33 -03:00
Ian Roddis 8eb966377f CI: Adding some debugs and assuming test script picks up where build left off. 2021-08-23 20:26:34 +00:00
Ian Roddis 2460b24b04 CI: Adding some debugs and assuming test script picks up where build left off. 2021-08-23 20:21:14 +00:00
Ian Roddis 461900b18d CI: Adding some debugs and assuming test script picks up where build left off. 2021-08-23 20:16:34 +00:00
Ian Roddis 22accec2a9 Adding missing packages 2021-08-23 20:11:06 +00:00
Ian Roddis fb3c450bfc Setting image to gcc for CI Pipeline 2021-08-23 20:09:09 +00:00
Ian Roddis 574f6c1fb1 Update .gitlab-ci.yml file 2021-08-23 20:07:00 +00:00
Ian Roddis be20bb5665 Adding MIT License 2021-08-23 16:51:27 -03:00
Ian Roddis c12309c3f6 Adding MIT License 2021-08-23 16:51:20 -03:00
Ian Roddis 5b8d5fabe8 - Fixing errors in JSON 2021-08-23 14:18:16 -03:00
Ian Roddis 808a7f9af4 - Adding more documentation 
- Renaming "parameters" to "taskParameters" for clarity, and to distinguish from "executionParameters" that will be implemented down the road.
 2021-08-23 14:17:22 -03:00
Ian Roddis 6b9baffe27 Changing Executor interface to allow for more flexible tasks down the road 2021-08-23 12:03:16 -03:00
Ian Roddis a83da567d0 - Updating TODOs 
- Adding jwt-cpp dependency (not really related to dag_execution)
 2021-08-21 12:18:01 -03:00
Ian Roddis 2083b1c3f1 Refactoring RunState, fixing logic error in when to end errored DAG runs, added convenience functions to retry failed DAGs. 2021-08-20 21:44:12 -03:00
Ian Roddis dc8ea4c369 - Adding support for state to OStreamLogger to make it more useful for test cases 
- Making runDAG return the end DAG
- Adding much more robust test for DAG execution for basic tests.
 2021-08-20 12:43:01 -03:00
Ian Roddis 0f1f00362c - rest_server is now a complete binary, just needs more config options boilerplate. 2021-08-20 11:18:26 -03:00
Ian Roddis 1f2712b090 - Fixing issue with parameter expansion on DAG submission to server 
- Adding sections to unit_server tests
- Adding cleanup
 2021-08-20 11:11:12 -03:00
Ian Roddis 8fa9af95af Merge remote-tracking branch 'origin/rest_driver' into rest_driver 2021-08-20 10:59:52 -03:00
Ian Roddis 791e7a15a7 - Fixing request logic to avoid hangs when REQUIREs fail before the client has been shutdown. 2021-08-20 10:58:57 -03:00
Ian Roddis db47bc1593 - Adding unit tests for Server 2021-08-20 10:58:57 -03:00
Ian Roddis 0a2a66bc59 - Adding dagRun REST method 2021-08-20 10:58:57 -03:00
Ian Roddis ce2c0dd30f - Checkpointing work so I can switch computers. 2021-08-20 10:58:57 -03:00
Ian Roddis c1489ce967 - Checkpointing work so I can switch computers. 2021-08-20 10:58:57 -03:00
Ian Roddis 3fde95339e - Checkpointing work before I restart the IDE. 2021-08-20 10:58:57 -03:00
Ian Roddis 35857f9513 - More cmake cleanup 2021-08-20 10:58:57 -03:00
Ian Roddis 71344b3a2a - Sidequest: Clean up header-only library dependencies with cmake. 2021-08-20 10:58:57 -03:00
Ian Roddis de8025e0f9 - Fixing bug that yielded empty expanded command sets. 2021-08-20 10:58:10 -03:00
Ian Roddis 1668117566 - Checkpointing work on expanding commands. 2021-08-20 10:39:39 -03:00
Ian Roddis 6bf376984b - Fixing request logic to avoid hangs when REQUIREs fail before the client has been shutdown. 2021-08-19 14:46:57 -03:00
Ian Roddis 90f34c21b9 - Adding unit tests for Server 2021-08-19 14:23:40 -03:00
Ian Roddis 4c6bc2a540 - Adding dagRun REST method 2021-08-14 11:11:12 -03:00
Ian Roddis 2525731f5a - Checkpointing work so I can switch computers. 2021-08-14 08:36:55 -03:00
Ian Roddis a668bf81f1 - Checkpointing work so I can switch computers. 2021-08-14 08:36:49 -03:00
Ian Roddis eb8e530f9a - Checkpointing work before I restart the IDE. 2021-08-13 13:28:47 -03:00
Ian Roddis d731f8b6dc - More cmake cleanup 2021-08-13 13:21:55 -03:00
Ian Roddis dfb71b63b8 - Sidequest: Clean up header-only library dependencies with cmake. 2021-08-13 12:43:39 -03:00
Ian Roddis 9f90f54b67 - More work on DAGLoggers 
- Still need unit tests for the FilesystemLogger
 2021-08-13 10:23:55 -03:00
Ian Roddis 4d519cc596 - Adding default pretty-printer for Task 
- Adding equivalency check for Task
- Adding test to ensure serialization / deserialization is reproducible.
 2021-08-11 10:16:38 -03:00
Ian Roddis a152588368 - Making task children unordered_set to avoid multiple edges between parent and child. 2021-08-10 12:42:54 -03:00
Ian Roddis 621467dd5a - Adding task serialization 
- Renaming DAGLogger to DAGRunLogger
- Adding more functionality to FileSystemLogger
 2021-08-10 11:44:56 -03:00
Ian Roddis 1af9ba2124 Adding more TODOs 2021-08-09 16:32:14 -03:00
Ian Roddis a97c3ff783 - Removing Catch2 code from codebase, will pull it via FetchContent instead. 
- Changing StdOutLogger to OStreamLogger, so that test cases output can be silenced.
 2021-08-09 15:48:53 -03:00
Ian Roddis 9a3671aba4 - More refactoring 
- Fixing cmake source discovery with GLOB_RECURSE
 2021-08-09 15:24:28 -03:00
Ian Roddis 30aea0818c - Running project through clang-tidy formatter. 2021-08-09 15:07:16 -03:00
Ian Roddis 1fcdf66829 - Adjusting file layouts to match between include and src 
- Updating TODO items.
 2021-08-09 15:05:25 -03:00
Ian Roddis 28c5b3eea3 - Adding StdOutLogger and adding tests for DAG execution to verify it works. 
- Roughing in FileSystemLogger
- Deleting Scheduler code and associated unit tests as being too complicated for maintenance.
- Refactoring namespaces for loggers and executors.
 2021-08-09 14:59:54 -03:00
Ian Roddis a8e85f8feb - Adding StdOutLogger and adding tests for DAG execution to verify it works. 
- Roughing in FileSystemLogger
- Deleting Scheduler code and associated unit tests as being too complicated for maintenance.
- Refactoring namespaces for loggers and executors.
 2021-08-09 14:59:23 -03:00
Ian Roddis 6a2c2ddb9a Merge remote-tracking branch 'origin/daglogger' into daglogger 
# Conflicts:
#	daggy/include/daggy/AttemptRecord.hpp
#	daggy/include/daggy/Utilities.hpp
 2021-08-09 11:15:28 -03:00
Ian Roddis c3b88d24ac - Rebasing on master. 
- Fixes so all tests pass.
 2021-08-09 11:15:04 -03:00
Ian Roddis df63d944c1 Checkpointing work, while serialization / deserialization is figured out. 2021-08-09 10:39:54 -03:00
Ian Roddis 1a467428a2 - Roughing out the DAGLogger interface 
- Adding spec for FileSystem Logger
 2021-08-09 10:39:46 -03:00
Ian Roddis 0516bde09f - Refactoring struct (de)serialization 2021-08-09 10:33:49 -03:00
Ian Roddis 7cd9fc5e6e - Refactoring Defines 
- Moving de/serialization routines to a separate header
 2021-08-08 13:13:32 -03:00
Ian Roddis bea21d912f Checkpointing work, while serialization / deserialization is figured out. 2021-08-08 13:05:13 -03:00
Ian Roddis 5a3796a8a1 - Roughing out the DAGLogger interface 
- Adding spec for FileSystem Logger
 2021-08-08 12:22:39 -03:00
Ian Roddis 1849a2fee4 - Adding TODO list to keep track of things. 
- Adding .idea to gitignore
- Adding DAG execution to utilities, redefining the logger
 2021-08-05 15:24:17 -03:00
Ian Roddis 745c950280 - Adding TODO list to keep track of things. 
- Adding .idea to gitignore
 2021-08-02 15:50:20 -03:00
Ian Roddis afcb917cc9 Completing task building. 2021-08-02 15:44:41 -03:00
Ian Roddis a5b4e6ce42 Checkpointing progress on task builder. 2021-08-02 11:59:43 -03:00