Kinesin/daggy
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
192 Commits 1 Branch 0 Tags
9e7d78788b9a9166ebee4c3e6ae547db7203853a
Commit Graph

192 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ian Roddis
9e7d78788b Fixing issue in dagrunner where tasks were never removed from running list 2022-01-13 12:52:00 -04:00
Ian Roddis
854ca5be22 Reducing the scope of a lock 2022-01-13 10:49:09 -04:00
Ian Roddis
3f5d31e036 Removing needless second map lookup 2022-01-12 19:59:10 -04:00
Ian Roddis
c143034c00 Implementing a redirect for a GET on / to list of dagruns 2022-01-12 15:17:00 -04:00
Ian Roddis
868e74fbeb Seems to work now 2022-01-12 13:11:55 -04:00
Ian Roddis
9a5a247f15 Checkpointing work 2022-01-12 12:50:46 -04:00
Ian Roddis
04e95cfcf3 Making fork process more descriptive in the case of failure 2022-01-10 13:12:19 -04:00
Ian Roddis
7312776e39 Fixing issue with logic for daggyr submission 2022-01-10 13:09:11 -04:00
Ian Roddis
53308c063d Fixing a number of scaling issues: 
- Missed closing of file descriptor made ForkingTaskExecutor
  silently die after running out of FDs
- Tightened up scope for locks to prevent http timeout
- Simplified threadpool
 2022-01-10 13:02:10 -04:00
Ian Roddis
efd4078f70 Adding executor log to output 2022-01-07 17:40:07 -04:00
Ian Roddis
db6fd64601 Fixing dagrun counts 2022-01-07 17:38:29 -04:00
Ian Roddis
d58dc2fc35 Removing ability for multiple servers to listen to the same address 2022-01-07 16:31:07 -04:00
Ian Roddis
58604d98cd Removing extra prints 2022-01-07 16:14:39 -04:00
Ian Roddis
6ed15b17b2 Fixing issue with querying dag as json 2022-01-07 16:13:12 -04:00
Ian Roddis
f9076be081 Adding fix for race condition in task resource management 2022-01-07 16:00:04 -04:00
Ian Roddis
4da78c9dd6 Adding executor description for verbose output, adding output directory for build artifacts 2022-01-07 14:37:06 -04:00
Ian Roddis
856e5bd2f4 Adding in task attempts drilldown 2022-01-06 15:20:06 -04:00
Ian Roddis
1786b53f7b Fixing sample dag 2022-01-04 17:05:45 -04:00
Ian Roddis
64a85cd996 Updating example dag 2022-01-04 17:03:42 -04:00
Ian Roddis
f1479a72d9 Fixing test cases on daggyr for new polling 2022-01-04 17:03:11 -04:00
Ian Roddis
5a4c6d0756 Fixing invalid URL 2021-12-30 14:32:43 -04:00
Ian Roddis
a4b26bce04 Handling http errors a bit more gracefully 2021-12-30 14:29:56 -04:00
Ian Roddis
d9479f14e0 Adding in check for no such file 2021-12-30 14:06:28 -04:00
Ian Roddis
ac490b6f3e Adding html output 2021-12-30 13:44:17 -04:00
Ian Roddis
509a1fcd23 Fixing argument parsing for integers 2021-12-27 22:15:16 -04:00
Ian Roddis
3c6966a9ac Adding discovery of capacities after the fact. 2021-12-24 10:52:06 -04:00
Ian Roddis
8ca5cdafe2 Fixing issue with resolving 2021-12-24 10:44:19 -04:00
Ian Roddis
d832b935d7 Moving to a poll method for workers, and daggyd-preserved capacities 2021-12-24 10:22:58 -04:00
Ian Roddis
779d6adaea Moving to a poll method for workers, and daggyd-preserved capacities 2021-12-24 10:21:19 -04:00
Ian Roddis
0914ede8fb Fixing test, since executor now autovivifies certain job attributes 2021-12-23 11:50:16 -04:00
Ian Roddis
dd473ab8f0 Adjusting capacity impact calculation to yield a more even distribution of jobs 2021-12-23 11:47:55 -04:00
Ian Roddis
85d252f43c Small bug fixes 2021-12-21 17:23:58 -04:00
Ian Roddis
d90f49b2bb Lots of fixes to poor daggyr implementation, added debugging messages 2021-12-21 16:49:45 -04:00
Ian Roddis
505ffb02bd Incorporating ifdefs for features in daggyd 2021-12-16 15:18:29 -04:00
Ian Roddis
2d35d6ed4a Making features opt-in 2021-12-16 14:17:44 -04:00
Ian Roddis
a035010669 Fixing feature checking 2021-12-16 14:17:03 -04:00
Ian Roddis
8d00621908 Adding support for remote execution daemons. 
Squashed commit of the following:

commit 69d5ef7a256b86a86d46e5ae374c00fded1497ea
Author: Ian Roddis <tech@kinesin.ca>
Date:   Thu Dec 16 12:15:55 2021 -0400

    Updating readme

commit 94a9f676d0f9cc0b55cdc18c4927eaea40d82c77
Author: Ian Roddis <tech@kinesin.ca>
Date:   Thu Dec 16 12:05:36 2021 -0400

    Fixing serialization of attempt records when querying entire dag

commit 945e5f90b24abf07c9af1bc4c6bbcb33e93b8069
Author: Ian Roddis <tech@kinesin.ca>
Date:   Thu Dec 16 11:37:59 2021 -0400

    Compiles cleanly...

commit 8b23e46081d47fb80dc1a2d998fc6dc4bbf301a8
Author: Ian Roddis <tech@kinesin.ca>
Date:   Thu Dec 16 10:43:03 2021 -0400

    Adding in missing source file to cmake build list

commit 6d10d9791206e2bc15788beadeea580b8e43a853
Author: Ian Roddis <tech@kinesin.ca>
Date:   Thu Dec 16 10:41:43 2021 -0400

    Adding new executors

commit 42a2c67f4d6ae99df95d917c8621d78cd99837a1
Author: Ian Roddis <tech@kinesin.ca>
Date:   Thu Dec 16 10:27:14 2021 -0400

    Fixing missing curl cmake dependency

commit 394bc4c5d51ecee7bf14712f719c8bf7e97fb0fa
Author: Ian Roddis <tech@kinesin.ca>
Date:   Thu Dec 16 10:21:58 2021 -0400

    Fixing missing curl cmake dependency

commit dd9efc8e7e7770ea1bcbccb70a1af9cfcff0414c
Author: Ian Roddis <tech@kinesin.ca>
Date:   Wed Dec 15 17:15:38 2021 -0400

    Checkpointing progress

commit 3b3b55d6037bb96e46de6763f486f4ecb92fe6a0
Author: Ian Roddis <tech@kinesin.ca>
Date:   Wed Dec 15 14:21:18 2021 -0400

    updating readme

commit 303027c11452941b2a0c0d1b04ac5942e79efd74
Author: Ian Roddis <tech@kinesin.ca>
Date:   Wed Dec 15 14:17:16 2021 -0400

    Namespacing daggyd
    Adding more error checking around deserialization of parameters
    Adding tests for runner agent

commit c592eaeba12e2a449bae401e8c1d9ed236416d52
Author: Ian Roddis <tech@kinesin.ca>
Date:   Wed Dec 15 11:20:21 2021 -0400

    Checkpointing work

commit fb1862d1cefe2b53a98659cce3c8c73d88bf5d84
Author: Ian Roddis <tech@kinesin.ca>
Date:   Wed Dec 15 09:52:29 2021 -0400

    Copying daggyd for daggyr template, adding in basic routes
 2021-12-16 12:16:12 -04:00
Ian Roddis
14d0ef4a3f Adding environment interpolation for noop, forking, and slurm executors 2021-11-13 12:09:51 -04:00
Ian Roddis
c0315b4f0b Changing up variable interpolation in preparation of supporting interpolation for environments 2021-11-12 16:08:57 -04:00
Ian Roddis
ac3928c7f6 fixing error messages 2021-11-11 12:02:37 -04:00
Ian Roddis
e7980d666b Adding in missing stdc++fs linkage 2021-11-11 10:24:56 -04:00
Ian Roddis
470a6f2bb7 Large re-organization to split daggyd away from the core libdaggy. 
This paves the way for implementing daggys and other utilities.

Squashed commit of the following:

commit 1f77239ab3c9e44d190eef94531a39501c8c4dfe
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Oct 18 16:25:02 2021 -0300

    Adding README, stdout support for daggyd logging

commit c2c237224e84a3be68aaa597ce98af1365e74a13
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Oct 18 16:10:29 2021 -0300

    removing old daggyd

commit cfea2baf61ca10c535801c5a391d2d525a1a2d04
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Oct 18 16:10:09 2021 -0300

    Moving tests into their sub-project folders

commit e41ca42069bea1db16dd76b6684a3f692fef6b15
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Oct 18 15:57:40 2021 -0300

    Splitting out daggyd from libdaggy

commit be97b146c1d2446f5c03cb78707e921f18c60bd8
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Oct 18 15:56:55 2021 -0300

    Splitting out daggyd from libdaggy

commit cb61e140e9d6d8832d61fb7037fd4c0ff6edad00
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Oct 18 15:49:47 2021 -0300

    moving daggy to libdaggy
 2021-10-18 16:28:40 -03:00
Ian Roddis
612bc8af8a Adding config file supoprt for daggyd 2021-10-18 12:12:04 -03:00
Ian Roddis
ff4b9ff08b Applying changes to fix build issues 2021-10-18 11:14:48 -03:00
Ian Roddis
860cea6004 Adding support for cmake test, breaking tests apart a bit for easier testing 2021-10-17 17:25:33 -03:00
Ian Roddis
e4e50fc219 Migrating to vanilla redis from rejson, using a threadsafe RedisHelper 
Squashed commit of the following:

commit f3549005c0192fd77bf47d208b74a11dd21380fa
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Sun Oct 17 10:52:12 2021 -0300

    Fixing issues with serialization of attempt records that included newlines

commit ab387d62850428e320a05cdf54fd2026369d0bb6
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Fri Oct 15 12:57:14 2021 -0300

    Migrating to vanilla redis from rejson

commit f648cf065dea2d0a7e30aaec17441e9a37531ba0
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Fri Oct 15 10:23:34 2021 -0300

    Making rediscontext threadsafe

commit ca51d95c5c4a5f4aaa13c5abe32da161e919d66c
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Fri Oct 15 10:01:33 2021 -0300

    updating todo
 2021-10-17 10:52:27 -03:00
Ian Roddis
08260043d0 Adding RedisLogger as a default for daggyd 2021-10-13 15:30:13 -03:00
Ian Roddis
9a0d2bb145 Adding support for RedisJSON logger 
Squashed commit of the following:

commit dc3a1bf07b5e7afdfd45e56f34596300dab6fd70
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 13 15:15:28 2021 -0300

    Updating documentation a bit

commit 8ec9c8c74f587368b32d034d3240a5537a69d4b1
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 13 15:11:23 2021 -0300

    Completing tests for redis

commit a6308dfa35b40b5a147394af8e3322ada871eb92
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 13 14:56:22 2021 -0300

    Resolving some errors with forking environment

commit 34691b6f85abae67001f4a4c234a4f7314407331
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 13 10:53:55 2021 -0300

    Checkpointing work on unit tests

commit 44c2b50fde30348938d901703ead9e279c3cd237
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 13 09:09:58 2021 -0300

    Checkpointing work on redis

commit a8051b725257087e25bc452673633ba6b40e3985
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Fri Oct 8 13:31:41 2021 -0300

    Checkpointing progress, changing state updates to a single record type

commit 456b84ad8c7dee0ff0dd39d5a7caead1ccd1126c
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Thu Oct 7 16:43:48 2021 -0300

    Checkpointing progress

commit f19dcaa4e417c3f2f6e527c288fe51401c9fe1d7
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Thu Oct 7 11:53:35 2021 -0300

    Moving back to hiredis to avoid boost dependency

commit e4bea6c589e82c82fd41476f164d946d77677193
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 6 10:41:16 2021 -0300

    fixing comments

commit 807a73c2a406817001eec048483938545a60194c
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 6 10:40:38 2021 -0300

    Switching to redis-cpp

commit d060c008d4d96bf3a81a19d35067f95f3638b8ca
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Oct 5 17:54:06 2021 -0300

    Adding hiredis dep
 2021-10-13 15:18:01 -03:00
Ian Roddis
0d4c45f5fc Adding missing header to SlurmExecutor 2021-10-07 10:26:39 -03:00
Ian Roddis
cfefdae4f3 Squashed commit of the following: 
commit 8a4e0fb24d191bf1c1009bd4c8800b4adab21f81
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Oct 5 17:23:21 2021 -0300

    Adding support for commandString

commit 9055cbde34d2489065b03c25c02a8bea56e42d54
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Oct 5 17:10:01 2021 -0300

    Completing support for environment variables

commit 989adef378724bbc9451c5048ea9d1285eebe2f9
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Oct 5 12:29:31 2021 -0300

    Adding environment support to ForkingTaskExecutor
 2021-10-05 17:26:30 -03:00