Kinesin/daggy
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Adding support for RedisJSON logger

Browse Source 
Squashed commit of the following:

commit dc3a1bf07b5e7afdfd45e56f34596300dab6fd70
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 13 15:15:28 2021 -0300

    Updating documentation a bit

commit 8ec9c8c74f587368b32d034d3240a5537a69d4b1
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 13 15:11:23 2021 -0300

    Completing tests for redis

commit a6308dfa35b40b5a147394af8e3322ada871eb92
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 13 14:56:22 2021 -0300

    Resolving some errors with forking environment

commit 34691b6f85abae67001f4a4c234a4f7314407331
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 13 10:53:55 2021 -0300

    Checkpointing work on unit tests

commit 44c2b50fde30348938d901703ead9e279c3cd237
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 13 09:09:58 2021 -0300

    Checkpointing work on redis

commit a8051b725257087e25bc452673633ba6b40e3985
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Fri Oct 8 13:31:41 2021 -0300

    Checkpointing progress, changing state updates to a single record type

commit 456b84ad8c7dee0ff0dd39d5a7caead1ccd1126c
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Thu Oct 7 16:43:48 2021 -0300

    Checkpointing progress

commit f19dcaa4e417c3f2f6e527c288fe51401c9fe1d7
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Thu Oct 7 11:53:35 2021 -0300

    Moving back to hiredis to avoid boost dependency

commit e4bea6c589e82c82fd41476f164d946d77677193
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 6 10:41:16 2021 -0300

    fixing comments

commit 807a73c2a406817001eec048483938545a60194c
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 6 10:40:38 2021 -0300

    Switching to redis-cpp

commit d060c008d4d96bf3a81a19d35067f95f3638b8ca
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Oct 5 17:54:06 2021 -0300

    Adding hiredis dep
This commit is contained in:
Ian Roddis
2021-10-13 15:18:01 -03:00
parent 0d4c45f5fc
commit 9a0d2bb145
25 changed files with 909 additions and 137 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
README.md
View File

@@ -30,8 +30,10 @@ Individual tasks (vertices) are run via a task executor. Daggy supports multiple
fork), to distributed work managers like [slurm](https://slurm.schedmd.com/overview.html)
or [kubernetes](https://kubernetes.io/) (planned).
State is maintained via state loggers. Currently daggy supports an in-memory state manager (OStreamLogger).
Future plans include supporting [redis](https://redis.io) and [postgres](https://postgresql.org).
State is maintained via state loggers. Currently daggy supports an in-memory state manager (OStreamLogger), and
[RedisJSON](https://oss.redis.com/redisjson/).
Future plans include supporting [postgres](https://postgresql.org).
Building
==
@@ -377,3 +379,9 @@ files will then be read after the task has completed, and stored in the AttemptR
For this reason, it's important that the `tmpDir` directory **be readable by the daggy engine**. i.e in a distributed
environment, it should be a shared filesystem. If this isn't the case, the job output will not be captured by daggy,
although it will still be available wherever it was written by slurm.
Loggers
=======
RedisJSON
---------