17 Commits

Author SHA1 Message Date
Ian Roddis
e4e50fc219 Migrating to vanilla redis from rejson, using a threadsafe RedisHelper
Squashed commit of the following:

commit f3549005c0192fd77bf47d208b74a11dd21380fa
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Sun Oct 17 10:52:12 2021 -0300

    Fixing issues with serialization of attempt records that included newlines

commit ab387d62850428e320a05cdf54fd2026369d0bb6
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Fri Oct 15 12:57:14 2021 -0300

    Migrating to vanilla redis from rejson

commit f648cf065dea2d0a7e30aaec17441e9a37531ba0
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Fri Oct 15 10:23:34 2021 -0300

    Making rediscontext threadsafe

commit ca51d95c5c4a5f4aaa13c5abe32da161e919d66c
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Fri Oct 15 10:01:33 2021 -0300

    updating todo
2021-10-17 10:52:27 -03:00
Ian Roddis
9a0d2bb145 Adding support for RedisJSON logger
Squashed commit of the following:

commit dc3a1bf07b5e7afdfd45e56f34596300dab6fd70
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 13 15:15:28 2021 -0300

    Updating documentation a bit

commit 8ec9c8c74f587368b32d034d3240a5537a69d4b1
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 13 15:11:23 2021 -0300

    Completing tests for redis

commit a6308dfa35b40b5a147394af8e3322ada871eb92
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 13 14:56:22 2021 -0300

    Resolving some errors with forking environment

commit 34691b6f85abae67001f4a4c234a4f7314407331
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 13 10:53:55 2021 -0300

    Checkpointing work on unit tests

commit 44c2b50fde30348938d901703ead9e279c3cd237
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 13 09:09:58 2021 -0300

    Checkpointing work on redis

commit a8051b725257087e25bc452673633ba6b40e3985
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Fri Oct 8 13:31:41 2021 -0300

    Checkpointing progress, changing state updates to a single record type

commit 456b84ad8c7dee0ff0dd39d5a7caead1ccd1126c
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Thu Oct 7 16:43:48 2021 -0300

    Checkpointing progress

commit f19dcaa4e417c3f2f6e527c288fe51401c9fe1d7
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Thu Oct 7 11:53:35 2021 -0300

    Moving back to hiredis to avoid boost dependency

commit e4bea6c589e82c82fd41476f164d946d77677193
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 6 10:41:16 2021 -0300

    fixing comments

commit 807a73c2a406817001eec048483938545a60194c
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 6 10:40:38 2021 -0300

    Switching to redis-cpp

commit d060c008d4d96bf3a81a19d35067f95f3638b8ca
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Oct 5 17:54:06 2021 -0300

    Adding hiredis dep
2021-10-13 15:18:01 -03:00
Ian Roddis
dded91220f Updating todo 2021-09-23 09:32:07 -03:00
Ian Roddis
33942e2c63 Update TODO with current goals. 2021-09-22 10:43:03 -03:00
Ian Roddis
d731f9f5b1 Adding support for execution on slurm grids
- Adding support for SlurmTaskExecutor in `daggyd` if DAGGY_ENABLE_SLURM is defined.
- Renaming some test cases
- Enabling compile-time slurm support
- Adding slurm documentation
2021-09-10 10:53:58 -03:00
Ian Roddis
d15580f47f Massive re-org to allow per-executor job specification formats and executor-specific task validation and expansion.
A few different renames to try and keep things more consistent.
2021-09-03 09:10:38 -03:00
Ian Roddis
e746f8c163 - Adding daemonization for daggyd
- Changing how parameter keys are stored in the map to allow for future key handling.
2021-09-01 13:32:16 -03:00
Ian Roddis
7b07380e16 - Removing duplicate information (taskName stored in 3 places) 2021-08-31 12:21:34 -03:00
Ian Roddis
2c00001e0b Adding support for isGenerator tasks
- Changing how DAG is represented, both in code and how DAGs are defined
  in JSON.
- Removing std::vector<Task> representation in favour of a map that will
  enforce unique task names
- Task names now have a name (generated), and a definedName.
- Adding support to loggers to add tasks after a DAGRun has been
  initialized.
2021-08-30 22:05:37 -03:00
Ian Roddis
212bd80df2 Squashed commit of the following:
commit 73994327de890590eede353c8131f3f7c1e8aaa3
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Aug 25 13:38:29 2021 -0300

    - Fixing up checks for individual dag runs

commit f20e3a3dec8c063111cf60f2bec2b8f84c8a4100
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Aug 25 10:49:43 2021 -0300

    - Finishing serialization of DAGRun
    - Checkpointing work.

commit b490abadf93e3085e4204003de7eaa8183b4e1d5
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Aug 25 10:34:08 2021 -0300

    - Consolidating struct definitions into Defines.hpp
    - Renaming DAGRunRecord member runStates to taskRunStates

commit 050346ec1fd10d1091f261905c6175ffe0bcf001
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Aug 25 09:27:05 2021 -0300

    - Adding additional tests for server endpoints
2021-08-25 13:40:05 -03:00
Ian Roddis
a83da567d0 - Updating TODOs
- Adding jwt-cpp dependency (not really related to dag_execution)
2021-08-21 12:18:01 -03:00
Ian Roddis
71344b3a2a - Sidequest: Clean up header-only library dependencies with cmake. 2021-08-20 10:58:57 -03:00
Ian Roddis
1af9ba2124 Adding more TODOs 2021-08-09 16:32:14 -03:00
Ian Roddis
1fcdf66829 - Adjusting file layouts to match between include and src
- Updating TODO items.
2021-08-09 15:05:25 -03:00
Ian Roddis
1a467428a2 - Roughing out the DAGLogger interface
- Adding spec for FileSystem Logger
2021-08-09 10:39:46 -03:00
Ian Roddis
1849a2fee4 - Adding TODO list to keep track of things.
- Adding .idea to gitignore
- Adding DAG execution to utilities, redefining the logger
2021-08-05 15:24:17 -03:00
Ian Roddis
745c950280 - Adding TODO list to keep track of things.
- Adding .idea to gitignore
2021-08-02 15:50:20 -03:00