Kinesin/daggy
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
127 Commits 1 Branch 0 Tags
4562ac755e3094ffe6e52b6645558cb17455d6a7
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Ian Roddis 4562ac755e Fixing argparse issue and populating stdin file 2021-09-14 15:04:23 -03:00
Ian Roddis 96c1928daf Adding fixes for slurm linking and uncommon slurm node attributes 2021-09-11 00:00:21 -03:00
Ian Roddis ff5a75abb0 Removing gitlab CI for now 2021-09-10 13:36:14 -03:00
Ian Roddis abdbf9f2bf Adding missing CMakeLists.txt 2021-09-10 13:34:56 -03:00
Ian Roddis d519b11247 Fixing slurm detection in cmake 2021-09-10 13:33:53 -03:00
Ian Roddis ea53735069 Adjusting path after argparse changed its structure out from underneath me 2021-09-10 12:16:25 -03:00
Ian Roddis f8cf2d48a5 Adding find_library for slurm 2021-09-10 11:10:20 -03:00
Ian Roddis 2f0548cb0a Disabling gitlab CI for now 2021-09-10 10:55:22 -03:00
Ian Roddis d731f9f5b1 Adding support for execution on slurm grids 
- Adding support for SlurmTaskExecutor in `daggyd` if DAGGY_ENABLE_SLURM is defined.
- Renaming some test cases
- Enabling compile-time slurm support
- Adding slurm documentation
 2021-09-10 10:53:58 -03:00
Ian Roddis d15580f47f Massive re-org to allow per-executor job specification formats and executor-specific task validation and expansion. 
A few different renames to try and keep things more consistent.
 2021-09-03 09:10:38 -03:00
Ian Roddis e746f8c163 - Adding daemonization for daggyd 
- Changing how parameter keys are stored in the map to allow for future key handling.
 2021-09-01 13:32:16 -03:00
Ian Roddis 4e71bf5917 Adding daggyc rough-in 2021-08-31 19:37:23 -03:00
Ian Roddis 77f7819584 Fixing test where return by reference would cause unexpected results when compiled with release flags 2021-08-31 19:14:34 -03:00
Ian Roddis 441cdb5b2e Renaming rest_server to daggyd, and daggy to libdaggy 2021-08-31 17:04:03 -03:00
Ian Roddis 7b07380e16 - Removing duplicate information (taskName stored in 3 places) 2021-08-31 12:21:34 -03:00
Ian Roddis 2c00001e0b Adding support for isGenerator tasks 
- Changing how DAG is represented, both in code and how DAGs are defined
  in JSON.
- Removing std::vector<Task> representation in favour of a map that will
  enforce unique task names
- Task names now have a name (generated), and a definedName.
- Adding support to loggers to add tasks after a DAGRun has been
  initialized.
 2021-08-30 22:05:37 -03:00
Ian Roddis dd6159dda8 Adding simple defaults for rest_server for testing 2021-08-30 12:47:06 -03:00
Ian Roddis 307e0c0559 Converting to better_enums to support gcc 8 2021-08-27 12:06:36 -03:00
Ian Roddis 7fa47c1c4a More CI stupidity 2021-08-25 16:16:58 -03:00
Ian Roddis fcdc225e28 Adjusting test case to eventually exit with an error after a reasonable time. 2021-08-25 15:20:08 -03:00
Ian Roddis 212bd80df2 Squashed commit of the following: 
commit 73994327de890590eede353c8131f3f7c1e8aaa3
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Aug 25 13:38:29 2021 -0300

    - Fixing up checks for individual dag runs

commit f20e3a3dec8c063111cf60f2bec2b8f84c8a4100
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Aug 25 10:49:43 2021 -0300

    - Finishing serialization of DAGRun
    - Checkpointing work.

commit b490abadf93e3085e4204003de7eaa8183b4e1d5
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Aug 25 10:34:08 2021 -0300

    - Consolidating struct definitions into Defines.hpp
    - Renaming DAGRunRecord member runStates to taskRunStates

commit 050346ec1fd10d1091f261905c6175ffe0bcf001
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Aug 25 09:27:05 2021 -0300

    - Adding additional tests for server endpoints
 2021-08-25 13:40:05 -03:00
Ian Roddis 6ed57806d0 - More fixes for CI failing tests 2021-08-24 07:58:58 -03:00
Ian Roddis 39fe716968 CI: Adding missing dependencies 2021-08-23 21:05:27 +00:00
Ian Roddis 307cff73f8 - More fixes for CI failing tests 2021-08-23 17:58:43 -03:00
Ian Roddis 7016cb37ba - More fixes for CI failing tests 2021-08-23 17:53:47 -03:00
Ian Roddis cf37f9a6fc - More fixes for CI failing tests 2021-08-23 17:49:04 -03:00
Ian Roddis 1bdc5dafd5 - More fixes for CI failing tests 2021-08-23 17:42:43 -03:00
Ian Roddis d4c9b3fcee - Adjusting tests so they run without referencing /tmp 2021-08-23 17:37:33 -03:00
Ian Roddis 8eb966377f CI: Adding some debugs and assuming test script picks up where build left off. 2021-08-23 20:26:34 +00:00
Ian Roddis 2460b24b04 CI: Adding some debugs and assuming test script picks up where build left off. 2021-08-23 20:21:14 +00:00
Ian Roddis 461900b18d CI: Adding some debugs and assuming test script picks up where build left off. 2021-08-23 20:16:34 +00:00
Ian Roddis 22accec2a9 Adding missing packages 2021-08-23 20:11:06 +00:00
Ian Roddis fb3c450bfc Setting image to gcc for CI Pipeline 2021-08-23 20:09:09 +00:00
Ian Roddis 574f6c1fb1 Update .gitlab-ci.yml file 2021-08-23 20:07:00 +00:00
Ian Roddis be20bb5665 Adding MIT License 2021-08-23 16:51:27 -03:00
Ian Roddis c12309c3f6 Adding MIT License 2021-08-23 16:51:20 -03:00
Ian Roddis 5b8d5fabe8 - Fixing errors in JSON 2021-08-23 14:18:16 -03:00
Ian Roddis 808a7f9af4 - Adding more documentation 
- Renaming "parameters" to "taskParameters" for clarity, and to distinguish from "executionParameters" that will be implemented down the road.
 2021-08-23 14:17:22 -03:00
Ian Roddis 6b9baffe27 Changing Executor interface to allow for more flexible tasks down the road 2021-08-23 12:03:16 -03:00
Ian Roddis a83da567d0 - Updating TODOs 
- Adding jwt-cpp dependency (not really related to dag_execution)
 2021-08-21 12:18:01 -03:00
Ian Roddis 2083b1c3f1 Refactoring RunState, fixing logic error in when to end errored DAG runs, added convenience functions to retry failed DAGs. 2021-08-20 21:44:12 -03:00
Ian Roddis dc8ea4c369 - Adding support for state to OStreamLogger to make it more useful for test cases 
- Making runDAG return the end DAG
- Adding much more robust test for DAG execution for basic tests.
 2021-08-20 12:43:01 -03:00
Ian Roddis 0f1f00362c - rest_server is now a complete binary, just needs more config options boilerplate. 2021-08-20 11:18:26 -03:00
Ian Roddis 1f2712b090 - Fixing issue with parameter expansion on DAG submission to server 
- Adding sections to unit_server tests
- Adding cleanup
 2021-08-20 11:11:12 -03:00
Ian Roddis 8fa9af95af Merge remote-tracking branch 'origin/rest_driver' into rest_driver 2021-08-20 10:59:52 -03:00
Ian Roddis 791e7a15a7 - Fixing request logic to avoid hangs when REQUIREs fail before the client has been shutdown. 2021-08-20 10:58:57 -03:00
Ian Roddis db47bc1593 - Adding unit tests for Server 2021-08-20 10:58:57 -03:00
Ian Roddis 0a2a66bc59 - Adding dagRun REST method 2021-08-20 10:58:57 -03:00
Ian Roddis ce2c0dd30f - Checkpointing work so I can switch computers. 2021-08-20 10:58:57 -03:00
Ian Roddis c1489ce967 - Checkpointing work so I can switch computers. 2021-08-20 10:58:57 -03:00
Ian Roddis 3fde95339e - Checkpointing work before I restart the IDE. 2021-08-20 10:58:57 -03:00
Ian Roddis 35857f9513 - More cmake cleanup 2021-08-20 10:58:57 -03:00
Ian Roddis 71344b3a2a - Sidequest: Clean up header-only library dependencies with cmake. 2021-08-20 10:58:57 -03:00
Ian Roddis de8025e0f9 - Fixing bug that yielded empty expanded command sets. 2021-08-20 10:58:10 -03:00
Ian Roddis 1668117566 - Checkpointing work on expanding commands. 2021-08-20 10:39:39 -03:00
Ian Roddis 6bf376984b - Fixing request logic to avoid hangs when REQUIREs fail before the client has been shutdown. 2021-08-19 14:46:57 -03:00
Ian Roddis 90f34c21b9 - Adding unit tests for Server 2021-08-19 14:23:40 -03:00
Ian Roddis 4c6bc2a540 - Adding dagRun REST method 2021-08-14 11:11:12 -03:00
Ian Roddis 2525731f5a - Checkpointing work so I can switch computers. 2021-08-14 08:36:55 -03:00
Ian Roddis a668bf81f1 - Checkpointing work so I can switch computers. 2021-08-14 08:36:49 -03:00
Ian Roddis eb8e530f9a - Checkpointing work before I restart the IDE. 2021-08-13 13:28:47 -03:00
Ian Roddis d731f8b6dc - More cmake cleanup 2021-08-13 13:21:55 -03:00
Ian Roddis dfb71b63b8 - Sidequest: Clean up header-only library dependencies with cmake. 2021-08-13 12:43:39 -03:00
Ian Roddis 9f90f54b67 - More work on DAGLoggers 
- Still need unit tests for the FilesystemLogger
 2021-08-13 10:23:55 -03:00
Ian Roddis 4d519cc596 - Adding default pretty-printer for Task 
- Adding equivalency check for Task
- Adding test to ensure serialization / deserialization is reproducible.
 2021-08-11 10:16:38 -03:00
Ian Roddis a152588368 - Making task children unordered_set to avoid multiple edges between parent and child. 2021-08-10 12:42:54 -03:00
Ian Roddis 621467dd5a - Adding task serialization 
- Renaming DAGLogger to DAGRunLogger
- Adding more functionality to FileSystemLogger
 2021-08-10 11:44:56 -03:00
Ian Roddis 1af9ba2124 Adding more TODOs 2021-08-09 16:32:14 -03:00
Ian Roddis a97c3ff783 - Removing Catch2 code from codebase, will pull it via FetchContent instead. 
- Changing StdOutLogger to OStreamLogger, so that test cases output can be silenced.
 2021-08-09 15:48:53 -03:00
Ian Roddis 9a3671aba4 - More refactoring 
- Fixing cmake source discovery with GLOB_RECURSE
 2021-08-09 15:24:28 -03:00
Ian Roddis 30aea0818c - Running project through clang-tidy formatter. 2021-08-09 15:07:16 -03:00
Ian Roddis 1fcdf66829 - Adjusting file layouts to match between include and src 
- Updating TODO items.
 2021-08-09 15:05:25 -03:00
Ian Roddis 28c5b3eea3 - Adding StdOutLogger and adding tests for DAG execution to verify it works. 
- Roughing in FileSystemLogger
- Deleting Scheduler code and associated unit tests as being too complicated for maintenance.
- Refactoring namespaces for loggers and executors.
 2021-08-09 14:59:54 -03:00
Ian Roddis a8e85f8feb - Adding StdOutLogger and adding tests for DAG execution to verify it works. 
- Roughing in FileSystemLogger
- Deleting Scheduler code and associated unit tests as being too complicated for maintenance.
- Refactoring namespaces for loggers and executors.
 2021-08-09 14:59:23 -03:00
Ian Roddis 6a2c2ddb9a Merge remote-tracking branch 'origin/daglogger' into daglogger 
# Conflicts:
#	daggy/include/daggy/AttemptRecord.hpp
#	daggy/include/daggy/Utilities.hpp
 2021-08-09 11:15:28 -03:00
Ian Roddis c3b88d24ac - Rebasing on master. 
- Fixes so all tests pass.
 2021-08-09 11:15:04 -03:00
Ian Roddis df63d944c1 Checkpointing work, while serialization / deserialization is figured out. 2021-08-09 10:39:54 -03:00
Ian Roddis 1a467428a2 - Roughing out the DAGLogger interface 
- Adding spec for FileSystem Logger
 2021-08-09 10:39:46 -03:00
Ian Roddis 0516bde09f - Refactoring struct (de)serialization 2021-08-09 10:33:49 -03:00
Ian Roddis 7cd9fc5e6e - Refactoring Defines 
- Moving de/serialization routines to a separate header
 2021-08-08 13:13:32 -03:00
Ian Roddis bea21d912f Checkpointing work, while serialization / deserialization is figured out. 2021-08-08 13:05:13 -03:00
Ian Roddis 5a3796a8a1 - Roughing out the DAGLogger interface 
- Adding spec for FileSystem Logger
 2021-08-08 12:22:39 -03:00
Ian Roddis 1849a2fee4 - Adding TODO list to keep track of things. 
- Adding .idea to gitignore
- Adding DAG execution to utilities, redefining the logger
 2021-08-05 15:24:17 -03:00
Ian Roddis 745c950280 - Adding TODO list to keep track of things. 
- Adding .idea to gitignore
 2021-08-02 15:50:20 -03:00
Ian Roddis afcb917cc9 Completing task building. 2021-08-02 15:44:41 -03:00
Ian Roddis a5b4e6ce42 Checkpointing progress on task builder. 2021-08-02 11:59:43 -03:00
Ian Roddis 54e8170c68 Adding JSON parsing helpers to Utilities. 2021-08-02 11:31:46 -03:00
Ian Roddis f2f3d22574 Renaming Executor to TaskEecutor 2021-08-01 11:28:30 -03:00
Ian Roddis 0349a5109b * Formatting code with clang-tidy 
* Roughing in more metastore work
 2021-07-22 12:57:51 -03:00
Ian Roddis 987718334f Adding missing stdc++fs link for std::filesystem 2021-07-09 15:51:27 -03:00
Ian Roddis a603ebead5 Scheduler now completing work 2021-07-06 11:35:30 -03:00
Ian Roddis 9b9409d504 Things mostly work, just a strange hang when executing code with forking executor 2021-07-05 15:37:29 -03:00
Ian Roddis 468993edb5 Checkpointing work 2021-07-05 11:57:38 -03:00
Ian Roddis b7b8d5b6a1 Checkpointing work 2021-06-20 12:05:13 -03:00
Ian Roddis 60cab3d5e0 Adding better implementation note 2021-06-20 10:34:48 -03:00
Ian Roddis 209ec6f380 Fixing up threadpool 2021-06-20 10:33:35 -03:00
Ian Roddis 1003e88303 adding unit test 2021-06-16 13:55:27 -03:00
Ian Roddis 5e3098c5f9 Checkpoint 2021-06-16 13:55:17 -03:00
Ian Roddis 01517f3421 adding futures 2021-06-16 10:51:38 -03:00
Ian Roddis d871572d5e Checkpointing work on first-pass implementation 2021-06-16 10:48:31 -03:00