dded91220f
Updating todo
Ian Roddis
2021-09-23 09:32:07 -03:00
d19648fa84
Refactoring CMake configs, adding Catch2 benchmark support
Ian Roddis
2021-09-22 16:33:50 -03:00
510aa236c4
Converting underlying DAG implementation to use vectors for storage Reduces hashing required and speeds up traversals
Ian Roddis
2021-09-22 14:19:20 -03:00
33942e2c63
Update TODO with current goals.
Ian Roddis
2021-09-22 10:43:03 -03:00
e7b15b3847
Implement a bunch of clang-tidy suggested changes. Remove FilesystemLogger
Ian Roddis
2021-09-22 10:30:27 -03:00
288ce28d29
Adding clang-format, and reformating all sourcecode
Ian Roddis
2021-09-21 09:41:11 -03:00
39d5ae08be
Adding a No-op task executor for testing Fixing DFS implementation of DAG validation to be much faster Adding in additional tests to ensure the run order of expanded tasks is preserved Adding additional compile-time checks, resolving issues that came up as a result
Ian Roddis
2021-09-20 19:05:56 -03:00
2daaa83d82
Small changes for quality of life
Ian Roddis
2021-09-17 16:36:59 -03:00
f83a17d610
Enhancing the performance of dag building by delaying validation until the end
Ian Roddis
2021-09-16 17:13:40 -03:00
fac741742a
Adding check for non-existent dagid
Ian Roddis
2021-09-16 11:03:47 -03:00
e895103b1c
Increasing max send/receive sizes for REST server to 4G
Ian Roddis
2021-09-16 10:07:06 -03:00
0c1a634ffa
Adding in and updating previously commented-out tests
Ian Roddis
2021-09-15 14:18:44 -03:00
d4ec744773
Fixing race condition issue with references in forking executor
Ian Roddis
2021-09-15 14:12:55 -03:00
a6a7501d12
Changing how execution parallelism is handled, so that different executors can implement their own idea of parallelism.
Ian Roddis
2021-09-15 13:05:04 -03:00
4562ac755e
Fixing argparse issue and populating stdin file
Ian Roddis
2021-09-14 15:04:23 -03:00
96c1928daf
Adding fixes for slurm linking and uncommon slurm node attributes
Ian Roddis
2021-09-11 00:00:21 -03:00
ff5a75abb0
Removing gitlab CI for now
Ian Roddis
2021-09-10 13:36:14 -03:00
abdbf9f2bf
Adding missing CMakeLists.txt
Ian Roddis
2021-09-10 13:34:56 -03:00
d519b11247
Fixing slurm detection in cmake
Ian Roddis
2021-09-10 13:33:53 -03:00
ea53735069
Adjusting path after argparse changed its structure out from underneath me
Ian Roddis
2021-09-10 12:16:25 -03:00
f8cf2d48a5
Adding find_library for slurm
Ian Roddis
2021-09-10 11:10:20 -03:00
2f0548cb0a
Disabling gitlab CI for now
Ian Roddis
2021-09-10 10:55:22 -03:00
d731f9f5b1
Adding support for execution on slurm grids - Adding support for SlurmTaskExecutor in daggyd if DAGGY_ENABLE_SLURM is defined. - Renaming some test cases - Enabling compile-time slurm support - Adding slurm documentation
Ian Roddis
2021-09-10 10:53:58 -03:00
d15580f47f
Massive re-org to allow per-executor job specification formats and executor-specific task validation and expansion. A few different renames to try and keep things more consistent.
Ian Roddis
2021-09-03 09:10:38 -03:00
e746f8c163
- Adding daemonization for daggyd - Changing how parameter keys are stored in the map to allow for future key handling.
Ian Roddis
2021-09-01 13:32:16 -03:00
4e71bf5917
Adding daggyc rough-in
Ian Roddis
2021-08-31 19:37:23 -03:00
77f7819584
Fixing test where return by reference would cause unexpected results when compiled with release flags
Ian Roddis
2021-08-31 19:14:34 -03:00
441cdb5b2e
Renaming rest_server to daggyd, and daggy to libdaggy
Ian Roddis
2021-08-31 17:04:03 -03:00
7b07380e16
- Removing duplicate information (taskName stored in 3 places)
Ian Roddis
2021-08-31 12:21:34 -03:00
2c00001e0b
Adding support for isGenerator tasks - Changing how DAG is represented, both in code and how DAGs are defined in JSON. - Removing std::vector<Task> representation in favour of a map that will enforce unique task names - Task names now have a name (generated), and a definedName. - Adding support to loggers to add tasks after a DAGRun has been initialized.
Ian Roddis
2021-08-30 22:05:37 -03:00
dd6159dda8
Adding simple defaults for rest_server for testing
Ian Roddis
2021-08-30 12:47:06 -03:00
307e0c0559
Converting to better_enums to support gcc 8
Ian Roddis
2021-08-27 12:06:36 -03:00
7fa47c1c4a
More CI stupidity
Ian Roddis
2021-08-25 16:16:58 -03:00
fcdc225e28
Adjusting test case to eventually exit with an error after a reasonable time.
Ian Roddis
2021-08-25 15:20:08 -03:00
212bd80df2
Squashed commit of the following:
Ian Roddis
2021-08-25 13:40:05 -03:00
6ed57806d0
- More fixes for CI failing tests
Ian Roddis
2021-08-24 07:58:30 -03:00
39fe716968
CI: Adding missing dependencies
Ian Roddis
2021-08-23 21:05:27 +00:00
307cff73f8
- More fixes for CI failing tests
Ian Roddis
2021-08-23 17:58:43 -03:00
7016cb37ba
- More fixes for CI failing tests
Ian Roddis
2021-08-23 17:53:47 -03:00
cf37f9a6fc
- More fixes for CI failing tests
Ian Roddis
2021-08-23 17:49:04 -03:00
1bdc5dafd5
- More fixes for CI failing tests
Ian Roddis
2021-08-23 17:42:43 -03:00
d4c9b3fcee
- Adjusting tests so they run without referencing /tmp
Ian Roddis
2021-08-23 17:37:33 -03:00
8eb966377f
CI: Adding some debugs and assuming test script picks up where build left off.
Ian Roddis
2021-08-23 20:26:34 +00:00
2460b24b04
CI: Adding some debugs and assuming test script picks up where build left off.
Ian Roddis
2021-08-23 20:21:14 +00:00
461900b18d
CI: Adding some debugs and assuming test script picks up where build left off.
Ian Roddis
2021-08-23 20:16:34 +00:00
22accec2a9
Adding missing packages
Ian Roddis
2021-08-23 20:11:06 +00:00
fb3c450bfc
Setting image to gcc for CI Pipeline
Ian Roddis
2021-08-23 20:09:09 +00:00
574f6c1fb1
Update .gitlab-ci.yml file
Ian Roddis
2021-08-23 20:07:00 +00:00
be20bb5665
Adding MIT License
Ian Roddis
2021-08-23 16:51:27 -03:00
c12309c3f6
Adding MIT License
Ian Roddis
2021-08-23 16:51:20 -03:00
5b8d5fabe8
- Fixing errors in JSON
Ian Roddis
2021-08-23 14:18:16 -03:00
808a7f9af4
- Adding more documentation - Renaming "parameters" to "taskParameters" for clarity, and to distinguish from "executionParameters" that will be implemented down the road.
Ian Roddis
2021-08-23 14:17:22 -03:00
6b9baffe27
Changing Executor interface to allow for more flexible tasks down the road
Ian Roddis
2021-08-23 12:03:16 -03:00
a83da567d0
- Updating TODOs - Adding jwt-cpp dependency (not really related to dag_execution)
Ian Roddis
2021-08-21 12:18:01 -03:00
2083b1c3f1
Refactoring RunState, fixing logic error in when to end errored DAG runs, added convenience functions to retry failed DAGs.
Ian Roddis
2021-08-20 21:44:12 -03:00
dc8ea4c369
- Adding support for state to OStreamLogger to make it more useful for test cases - Making runDAG return the end DAG - Adding much more robust test for DAG execution for basic tests.
Ian Roddis
2021-08-20 12:43:01 -03:00
0f1f00362c
- rest_server is now a complete binary, just needs more config options boilerplate.
Ian Roddis
2021-08-20 11:18:26 -03:00
1f2712b090
- Fixing issue with parameter expansion on DAG submission to server - Adding sections to unit_server tests - Adding cleanup
Ian Roddis
2021-08-20 11:11:12 -03:00
8fa9af95af
Merge remote-tracking branch 'origin/rest_driver' into rest_driver
Ian Roddis
2021-08-20 10:59:52 -03:00
791e7a15a7
- Fixing request logic to avoid hangs when REQUIREs fail before the client has been shutdown.
Ian Roddis
2021-08-19 14:46:57 -03:00
db47bc1593
- Adding unit tests for Server
Ian Roddis
2021-08-19 14:23:40 -03:00
2525731f5a
- Checkpointing work so I can switch computers.
Ian Roddis
2021-08-14 08:36:55 -03:00
a668bf81f1
- Checkpointing work so I can switch computers.
Ian Roddis
2021-08-14 08:36:49 -03:00
eb8e530f9a
- Checkpointing work before I restart the IDE.
Ian Roddis
2021-08-13 13:28:47 -03:00
d731f8b6dc
- More cmake cleanup
Ian Roddis
2021-08-13 13:21:55 -03:00
dfb71b63b8
- Sidequest: Clean up header-only library dependencies with cmake.
Ian Roddis
2021-08-13 12:43:39 -03:00
9f90f54b67
- More work on DAGLoggers - Still need unit tests for the FilesystemLogger
Ian Roddis
2021-08-13 10:23:55 -03:00
4d519cc596
- Adding default pretty-printer for Task - Adding equivalency check for Task - Adding test to ensure serialization / deserialization is reproducible.
Ian Roddis
2021-08-11 10:16:38 -03:00
a152588368
- Making task children unordered_set to avoid multiple edges between parent and child.
Ian Roddis
2021-08-10 12:42:54 -03:00
621467dd5a
- Adding task serialization - Renaming DAGLogger to DAGRunLogger - Adding more functionality to FileSystemLogger
Ian Roddis
2021-08-10 11:44:56 -03:00
1af9ba2124
Adding more TODOs
Ian Roddis
2021-08-09 16:32:14 -03:00
a97c3ff783
- Removing Catch2 code from codebase, will pull it via FetchContent instead. - Changing StdOutLogger to OStreamLogger, so that test cases output can be silenced.
Ian Roddis
2021-08-09 15:48:53 -03:00
9a3671aba4
- More refactoring - Fixing cmake source discovery with GLOB_RECURSE
Ian Roddis
2021-08-09 15:24:28 -03:00
30aea0818c
- Running project through clang-tidy formatter.
Ian Roddis
2021-08-09 15:07:16 -03:00
1fcdf66829
- Adjusting file layouts to match between include and src - Updating TODO items.
Ian Roddis
2021-08-09 15:05:25 -03:00
28c5b3eea3
- Adding StdOutLogger and adding tests for DAG execution to verify it works. - Roughing in FileSystemLogger - Deleting Scheduler code and associated unit tests as being too complicated for maintenance. - Refactoring namespaces for loggers and executors.
Ian Roddis
2021-08-09 14:59:54 -03:00
a8e85f8feb
- Adding StdOutLogger and adding tests for DAG execution to verify it works. - Roughing in FileSystemLogger - Deleting Scheduler code and associated unit tests as being too complicated for maintenance. - Refactoring namespaces for loggers and executors.
Ian Roddis
2021-08-09 14:59:23 -03:00
6a2c2ddb9a
Merge remote-tracking branch 'origin/daglogger' into daglogger
Ian Roddis
2021-08-09 11:15:28 -03:00
c3b88d24ac
- Rebasing on master. - Fixes so all tests pass.
Ian Roddis
2021-08-09 11:15:04 -03:00
df63d944c1
Checkpointing work, while serialization / deserialization is figured out.
Ian Roddis
2021-08-08 13:05:13 -03:00
1a467428a2
- Roughing out the DAGLogger interface - Adding spec for FileSystem Logger
Ian Roddis
2021-08-08 12:22:39 -03:00
0516bde09f
- Refactoring struct (de)serialization
Ian Roddis
2021-08-09 10:33:49 -03:00
7cd9fc5e6e
- Refactoring Defines - Moving de/serialization routines to a separate header
Ian Roddis
2021-08-08 13:13:32 -03:00
bea21d912f
Checkpointing work, while serialization / deserialization is figured out.
Ian Roddis
2021-08-08 13:05:13 -03:00
5a3796a8a1
- Roughing out the DAGLogger interface - Adding spec for FileSystem Logger
Ian Roddis
2021-08-08 12:22:39 -03:00
1849a2fee4
- Adding TODO list to keep track of things. - Adding .idea to gitignore - Adding DAG execution to utilities, redefining the logger
Ian Roddis
2021-08-05 15:24:17 -03:00
745c950280
- Adding TODO list to keep track of things. - Adding .idea to gitignore
Ian Roddis
2021-08-02 15:50:20 -03:00
afcb917cc9
Completing task building.
Ian Roddis
2021-08-02 15:44:41 -03:00
a5b4e6ce42
Checkpointing progress on task builder.
Ian Roddis
2021-08-02 11:59:43 -03:00
Ian Roddis