Kinesin/daggy
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
194 Commits 1 Branch 0 Tags
2784aeee28ece4e600e2666d561f7eb0459a652c
Commit Graph

194 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ian Roddis
2784aeee28 Adding in check for cores and memoryMB 2022-01-14 15:48:06 -04:00
Ian Roddis
07646de2cd Adding SSH executor 2022-01-14 15:39:57 -04:00
Ian Roddis
9e7d78788b Fixing issue in dagrunner where tasks were never removed from running list 2022-01-13 12:52:00 -04:00
Ian Roddis
854ca5be22 Reducing the scope of a lock 2022-01-13 10:49:09 -04:00
Ian Roddis
3f5d31e036 Removing needless second map lookup 2022-01-12 19:59:10 -04:00
Ian Roddis
c143034c00 Implementing a redirect for a GET on / to list of dagruns 2022-01-12 15:17:00 -04:00
Ian Roddis
868e74fbeb Seems to work now 2022-01-12 13:11:55 -04:00
Ian Roddis
9a5a247f15 Checkpointing work 2022-01-12 12:50:46 -04:00
Ian Roddis
04e95cfcf3 Making fork process more descriptive in the case of failure 2022-01-10 13:12:19 -04:00
Ian Roddis
7312776e39 Fixing issue with logic for daggyr submission 2022-01-10 13:09:11 -04:00
Ian Roddis
53308c063d Fixing a number of scaling issues: 
- Missed closing of file descriptor made ForkingTaskExecutor
  silently die after running out of FDs
- Tightened up scope for locks to prevent http timeout
- Simplified threadpool
 2022-01-10 13:02:10 -04:00
Ian Roddis
efd4078f70 Adding executor log to output 2022-01-07 17:40:07 -04:00
Ian Roddis
db6fd64601 Fixing dagrun counts 2022-01-07 17:38:29 -04:00
Ian Roddis
d58dc2fc35 Removing ability for multiple servers to listen to the same address 2022-01-07 16:31:07 -04:00
Ian Roddis
58604d98cd Removing extra prints 2022-01-07 16:14:39 -04:00
Ian Roddis
6ed15b17b2 Fixing issue with querying dag as json 2022-01-07 16:13:12 -04:00
Ian Roddis
f9076be081 Adding fix for race condition in task resource management 2022-01-07 16:00:04 -04:00
Ian Roddis
4da78c9dd6 Adding executor description for verbose output, adding output directory for build artifacts 2022-01-07 14:37:06 -04:00
Ian Roddis
856e5bd2f4 Adding in task attempts drilldown 2022-01-06 15:20:06 -04:00
Ian Roddis
1786b53f7b Fixing sample dag 2022-01-04 17:05:45 -04:00
Ian Roddis
64a85cd996 Updating example dag 2022-01-04 17:03:42 -04:00
Ian Roddis
f1479a72d9 Fixing test cases on daggyr for new polling 2022-01-04 17:03:11 -04:00
Ian Roddis
5a4c6d0756 Fixing invalid URL 2021-12-30 14:32:43 -04:00
Ian Roddis
a4b26bce04 Handling http errors a bit more gracefully 2021-12-30 14:29:56 -04:00
Ian Roddis
d9479f14e0 Adding in check for no such file 2021-12-30 14:06:28 -04:00
Ian Roddis
ac490b6f3e Adding html output 2021-12-30 13:44:17 -04:00
Ian Roddis
509a1fcd23 Fixing argument parsing for integers 2021-12-27 22:15:16 -04:00
Ian Roddis
3c6966a9ac Adding discovery of capacities after the fact. 2021-12-24 10:52:06 -04:00
Ian Roddis
8ca5cdafe2 Fixing issue with resolving 2021-12-24 10:44:19 -04:00
Ian Roddis
d832b935d7 Moving to a poll method for workers, and daggyd-preserved capacities 2021-12-24 10:22:58 -04:00
Ian Roddis
779d6adaea Moving to a poll method for workers, and daggyd-preserved capacities 2021-12-24 10:21:19 -04:00
Ian Roddis
0914ede8fb Fixing test, since executor now autovivifies certain job attributes 2021-12-23 11:50:16 -04:00
Ian Roddis
dd473ab8f0 Adjusting capacity impact calculation to yield a more even distribution of jobs 2021-12-23 11:47:55 -04:00
Ian Roddis
85d252f43c Small bug fixes 2021-12-21 17:23:58 -04:00
Ian Roddis
d90f49b2bb Lots of fixes to poor daggyr implementation, added debugging messages 2021-12-21 16:49:45 -04:00
Ian Roddis
505ffb02bd Incorporating ifdefs for features in daggyd 2021-12-16 15:18:29 -04:00
Ian Roddis
2d35d6ed4a Making features opt-in 2021-12-16 14:17:44 -04:00
Ian Roddis
a035010669 Fixing feature checking 2021-12-16 14:17:03 -04:00
Ian Roddis
8d00621908 Adding support for remote execution daemons. 
Squashed commit of the following:

commit 69d5ef7a256b86a86d46e5ae374c00fded1497ea
Author: Ian Roddis <tech@kinesin.ca>
Date:   Thu Dec 16 12:15:55 2021 -0400

    Updating readme

commit 94a9f676d0f9cc0b55cdc18c4927eaea40d82c77
Author: Ian Roddis <tech@kinesin.ca>
Date:   Thu Dec 16 12:05:36 2021 -0400

    Fixing serialization of attempt records when querying entire dag

commit 945e5f90b24abf07c9af1bc4c6bbcb33e93b8069
Author: Ian Roddis <tech@kinesin.ca>
Date:   Thu Dec 16 11:37:59 2021 -0400

    Compiles cleanly...

commit 8b23e46081d47fb80dc1a2d998fc6dc4bbf301a8
Author: Ian Roddis <tech@kinesin.ca>
Date:   Thu Dec 16 10:43:03 2021 -0400

    Adding in missing source file to cmake build list

commit 6d10d9791206e2bc15788beadeea580b8e43a853
Author: Ian Roddis <tech@kinesin.ca>
Date:   Thu Dec 16 10:41:43 2021 -0400

    Adding new executors

commit 42a2c67f4d6ae99df95d917c8621d78cd99837a1
Author: Ian Roddis <tech@kinesin.ca>
Date:   Thu Dec 16 10:27:14 2021 -0400

    Fixing missing curl cmake dependency

commit 394bc4c5d51ecee7bf14712f719c8bf7e97fb0fa
Author: Ian Roddis <tech@kinesin.ca>
Date:   Thu Dec 16 10:21:58 2021 -0400

    Fixing missing curl cmake dependency

commit dd9efc8e7e7770ea1bcbccb70a1af9cfcff0414c
Author: Ian Roddis <tech@kinesin.ca>
Date:   Wed Dec 15 17:15:38 2021 -0400

    Checkpointing progress

commit 3b3b55d6037bb96e46de6763f486f4ecb92fe6a0
Author: Ian Roddis <tech@kinesin.ca>
Date:   Wed Dec 15 14:21:18 2021 -0400

    updating readme

commit 303027c11452941b2a0c0d1b04ac5942e79efd74
Author: Ian Roddis <tech@kinesin.ca>
Date:   Wed Dec 15 14:17:16 2021 -0400

    Namespacing daggyd
    Adding more error checking around deserialization of parameters
    Adding tests for runner agent

commit c592eaeba12e2a449bae401e8c1d9ed236416d52
Author: Ian Roddis <tech@kinesin.ca>
Date:   Wed Dec 15 11:20:21 2021 -0400

    Checkpointing work

commit fb1862d1cefe2b53a98659cce3c8c73d88bf5d84
Author: Ian Roddis <tech@kinesin.ca>
Date:   Wed Dec 15 09:52:29 2021 -0400

    Copying daggyd for daggyr template, adding in basic routes
 2021-12-16 12:16:12 -04:00
Ian Roddis
14d0ef4a3f Adding environment interpolation for noop, forking, and slurm executors 2021-11-13 12:09:51 -04:00
Ian Roddis
c0315b4f0b Changing up variable interpolation in preparation of supporting interpolation for environments 2021-11-12 16:08:57 -04:00
Ian Roddis
ac3928c7f6 fixing error messages 2021-11-11 12:02:37 -04:00
Ian Roddis
e7980d666b Adding in missing stdc++fs linkage 2021-11-11 10:24:56 -04:00
Ian Roddis
470a6f2bb7 Large re-organization to split daggyd away from the core libdaggy. 
This paves the way for implementing daggys and other utilities.

Squashed commit of the following:

commit 1f77239ab3c9e44d190eef94531a39501c8c4dfe
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Oct 18 16:25:02 2021 -0300

    Adding README, stdout support for daggyd logging

commit c2c237224e84a3be68aaa597ce98af1365e74a13
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Oct 18 16:10:29 2021 -0300

    removing old daggyd

commit cfea2baf61ca10c535801c5a391d2d525a1a2d04
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Oct 18 16:10:09 2021 -0300

    Moving tests into their sub-project folders

commit e41ca42069bea1db16dd76b6684a3f692fef6b15
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Oct 18 15:57:40 2021 -0300

    Splitting out daggyd from libdaggy

commit be97b146c1d2446f5c03cb78707e921f18c60bd8
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Oct 18 15:56:55 2021 -0300

    Splitting out daggyd from libdaggy

commit cb61e140e9d6d8832d61fb7037fd4c0ff6edad00
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Oct 18 15:49:47 2021 -0300

    moving daggy to libdaggy
 2021-10-18 16:28:40 -03:00
Ian Roddis
612bc8af8a Adding config file supoprt for daggyd 2021-10-18 12:12:04 -03:00
Ian Roddis
ff4b9ff08b Applying changes to fix build issues 2021-10-18 11:14:48 -03:00
Ian Roddis
860cea6004 Adding support for cmake test, breaking tests apart a bit for easier testing 2021-10-17 17:25:33 -03:00
Ian Roddis
e4e50fc219 Migrating to vanilla redis from rejson, using a threadsafe RedisHelper 
Squashed commit of the following:

commit f3549005c0192fd77bf47d208b74a11dd21380fa
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Sun Oct 17 10:52:12 2021 -0300

    Fixing issues with serialization of attempt records that included newlines

commit ab387d62850428e320a05cdf54fd2026369d0bb6
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Fri Oct 15 12:57:14 2021 -0300

    Migrating to vanilla redis from rejson

commit f648cf065dea2d0a7e30aaec17441e9a37531ba0
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Fri Oct 15 10:23:34 2021 -0300

    Making rediscontext threadsafe

commit ca51d95c5c4a5f4aaa13c5abe32da161e919d66c
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Fri Oct 15 10:01:33 2021 -0300

    updating todo
 2021-10-17 10:52:27 -03:00
Ian Roddis
08260043d0 Adding RedisLogger as a default for daggyd 2021-10-13 15:30:13 -03:00
Ian Roddis
9a0d2bb145 Adding support for RedisJSON logger 
Squashed commit of the following:

commit dc3a1bf07b5e7afdfd45e56f34596300dab6fd70
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 13 15:15:28 2021 -0300

    Updating documentation a bit

commit 8ec9c8c74f587368b32d034d3240a5537a69d4b1
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 13 15:11:23 2021 -0300

    Completing tests for redis

commit a6308dfa35b40b5a147394af8e3322ada871eb92
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 13 14:56:22 2021 -0300

    Resolving some errors with forking environment

commit 34691b6f85abae67001f4a4c234a4f7314407331
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 13 10:53:55 2021 -0300

    Checkpointing work on unit tests

commit 44c2b50fde30348938d901703ead9e279c3cd237
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 13 09:09:58 2021 -0300

    Checkpointing work on redis

commit a8051b725257087e25bc452673633ba6b40e3985
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Fri Oct 8 13:31:41 2021 -0300

    Checkpointing progress, changing state updates to a single record type

commit 456b84ad8c7dee0ff0dd39d5a7caead1ccd1126c
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Thu Oct 7 16:43:48 2021 -0300

    Checkpointing progress

commit f19dcaa4e417c3f2f6e527c288fe51401c9fe1d7
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Thu Oct 7 11:53:35 2021 -0300

    Moving back to hiredis to avoid boost dependency

commit e4bea6c589e82c82fd41476f164d946d77677193
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 6 10:41:16 2021 -0300

    fixing comments

commit 807a73c2a406817001eec048483938545a60194c
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Wed Oct 6 10:40:38 2021 -0300

    Switching to redis-cpp

commit d060c008d4d96bf3a81a19d35067f95f3638b8ca
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Tue Oct 5 17:54:06 2021 -0300

    Adding hiredis dep
 2021-10-13 15:18:01 -03:00