Ian Roddis
b3397381dc
Fixing logic error in submission batches
2022-02-01 13:34:19 -04:00
Ian Roddis
df108dfd28
Increasing curl timeout
2022-02-01 11:02:33 -04:00
Ian Roddis
cc7646f386
Adding delete method
2022-02-01 10:33:58 -04:00
Ian Roddis
3fdd568b00
Adding samples
2022-01-29 13:25:40 -04:00
Ian Roddis
e3a6c9086a
Removing verbose logging
2022-01-28 14:36:33 -04:00
Ian Roddis
782e6a8b97
Changing behaviour of runner to submit at most MAX_SUBMITS for execution before checking for completed tasks
2022-01-28 14:28:46 -04:00
Ian Roddis
5af8deabec
Making DaggyTaskRunner block until a runner is available
2022-01-28 14:18:01 -04:00
Ian Roddis
cbd493ec38
Fixing issue with daggyd config parsing
2022-01-28 13:28:29 -04:00
Ian Roddis
c9820c19a0
Small fixes to test cases
2022-01-28 10:23:21 -04:00
Ian Roddis
c09d5b7d8b
Adding in queuing in the daggyd process for more even distribution
2022-01-28 10:09:55 -04:00
Ian Roddis
070d2ceddf
Adding support for environment variables
2022-01-18 10:54:25 -04:00
Ian Roddis
2784aeee28
Adding in check for cores and memoryMB
2022-01-14 15:48:06 -04:00
Ian Roddis
07646de2cd
Adding SSH executor
2022-01-14 15:39:57 -04:00
Ian Roddis
9e7d78788b
Fixing issue in dagrunner where tasks were never removed from running list
2022-01-13 12:52:00 -04:00
Ian Roddis
854ca5be22
Reducing the scope of a lock
2022-01-13 10:49:09 -04:00
Ian Roddis
3f5d31e036
Removing needless second map lookup
2022-01-12 19:59:10 -04:00
Ian Roddis
c143034c00
Implementing a redirect for a GET on / to list of dagruns
2022-01-12 15:17:00 -04:00
Ian Roddis
868e74fbeb
Seems to work now
2022-01-12 13:11:55 -04:00
Ian Roddis
9a5a247f15
Checkpointing work
2022-01-12 12:50:46 -04:00
Ian Roddis
04e95cfcf3
Making fork process more descriptive in the case of failure
2022-01-10 13:12:19 -04:00
Ian Roddis
7312776e39
Fixing issue with logic for daggyr submission
2022-01-10 13:09:11 -04:00
Ian Roddis
53308c063d
Fixing a number of scaling issues:
...
- Missed closing of file descriptor made ForkingTaskExecutor
silently die after running out of FDs
- Tightened up scope for locks to prevent http timeout
- Simplified threadpool
2022-01-10 13:02:10 -04:00
Ian Roddis
efd4078f70
Adding executor log to output
2022-01-07 17:40:07 -04:00
Ian Roddis
db6fd64601
Fixing dagrun counts
2022-01-07 17:38:29 -04:00
Ian Roddis
d58dc2fc35
Removing ability for multiple servers to listen to the same address
2022-01-07 16:31:07 -04:00
Ian Roddis
58604d98cd
Removing extra prints
2022-01-07 16:14:39 -04:00
Ian Roddis
6ed15b17b2
Fixing issue with querying dag as json
2022-01-07 16:13:12 -04:00
Ian Roddis
f9076be081
Adding fix for race condition in task resource management
2022-01-07 16:00:04 -04:00
Ian Roddis
4da78c9dd6
Adding executor description for verbose output, adding output directory for build artifacts
2022-01-07 14:37:06 -04:00
Ian Roddis
856e5bd2f4
Adding in task attempts drilldown
2022-01-06 15:20:06 -04:00
Ian Roddis
1786b53f7b
Fixing sample dag
2022-01-04 17:05:45 -04:00
Ian Roddis
64a85cd996
Updating example dag
2022-01-04 17:03:42 -04:00
Ian Roddis
f1479a72d9
Fixing test cases on daggyr for new polling
2022-01-04 17:03:11 -04:00
Ian Roddis
5a4c6d0756
Fixing invalid URL
2021-12-30 14:32:43 -04:00
Ian Roddis
a4b26bce04
Handling http errors a bit more gracefully
2021-12-30 14:29:56 -04:00
Ian Roddis
d9479f14e0
Adding in check for no such file
2021-12-30 14:06:28 -04:00
Ian Roddis
ac490b6f3e
Adding html output
2021-12-30 13:44:17 -04:00
Ian Roddis
509a1fcd23
Fixing argument parsing for integers
2021-12-27 22:15:16 -04:00
Ian Roddis
3c6966a9ac
Adding discovery of capacities after the fact.
2021-12-24 10:52:06 -04:00
Ian Roddis
8ca5cdafe2
Fixing issue with resolving
2021-12-24 10:44:19 -04:00
Ian Roddis
d832b935d7
Moving to a poll method for workers, and daggyd-preserved capacities
2021-12-24 10:22:58 -04:00
Ian Roddis
779d6adaea
Moving to a poll method for workers, and daggyd-preserved capacities
2021-12-24 10:21:19 -04:00
Ian Roddis
0914ede8fb
Fixing test, since executor now autovivifies certain job attributes
2021-12-23 11:50:16 -04:00
Ian Roddis
dd473ab8f0
Adjusting capacity impact calculation to yield a more even distribution of jobs
2021-12-23 11:47:55 -04:00
Ian Roddis
85d252f43c
Small bug fixes
2021-12-21 17:23:58 -04:00
Ian Roddis
d90f49b2bb
Lots of fixes to poor daggyr implementation, added debugging messages
2021-12-21 16:49:45 -04:00
Ian Roddis
505ffb02bd
Incorporating ifdefs for features in daggyd
2021-12-16 15:18:29 -04:00
Ian Roddis
2d35d6ed4a
Making features opt-in
2021-12-16 14:17:44 -04:00
Ian Roddis
a035010669
Fixing feature checking
2021-12-16 14:17:03 -04:00
Ian Roddis
8d00621908
Adding support for remote execution daemons.
...
Squashed commit of the following:
commit 69d5ef7a256b86a86d46e5ae374c00fded1497ea
Author: Ian Roddis <tech@kinesin.ca >
Date: Thu Dec 16 12:15:55 2021 -0400
Updating readme
commit 94a9f676d0f9cc0b55cdc18c4927eaea40d82c77
Author: Ian Roddis <tech@kinesin.ca >
Date: Thu Dec 16 12:05:36 2021 -0400
Fixing serialization of attempt records when querying entire dag
commit 945e5f90b24abf07c9af1bc4c6bbcb33e93b8069
Author: Ian Roddis <tech@kinesin.ca >
Date: Thu Dec 16 11:37:59 2021 -0400
Compiles cleanly...
commit 8b23e46081d47fb80dc1a2d998fc6dc4bbf301a8
Author: Ian Roddis <tech@kinesin.ca >
Date: Thu Dec 16 10:43:03 2021 -0400
Adding in missing source file to cmake build list
commit 6d10d9791206e2bc15788beadeea580b8e43a853
Author: Ian Roddis <tech@kinesin.ca >
Date: Thu Dec 16 10:41:43 2021 -0400
Adding new executors
commit 42a2c67f4d6ae99df95d917c8621d78cd99837a1
Author: Ian Roddis <tech@kinesin.ca >
Date: Thu Dec 16 10:27:14 2021 -0400
Fixing missing curl cmake dependency
commit 394bc4c5d51ecee7bf14712f719c8bf7e97fb0fa
Author: Ian Roddis <tech@kinesin.ca >
Date: Thu Dec 16 10:21:58 2021 -0400
Fixing missing curl cmake dependency
commit dd9efc8e7e7770ea1bcbccb70a1af9cfcff0414c
Author: Ian Roddis <tech@kinesin.ca >
Date: Wed Dec 15 17:15:38 2021 -0400
Checkpointing progress
commit 3b3b55d6037bb96e46de6763f486f4ecb92fe6a0
Author: Ian Roddis <tech@kinesin.ca >
Date: Wed Dec 15 14:21:18 2021 -0400
updating readme
commit 303027c11452941b2a0c0d1b04ac5942e79efd74
Author: Ian Roddis <tech@kinesin.ca >
Date: Wed Dec 15 14:17:16 2021 -0400
Namespacing daggyd
Adding more error checking around deserialization of parameters
Adding tests for runner agent
commit c592eaeba12e2a449bae401e8c1d9ed236416d52
Author: Ian Roddis <tech@kinesin.ca >
Date: Wed Dec 15 11:20:21 2021 -0400
Checkpointing work
commit fb1862d1cefe2b53a98659cce3c8c73d88bf5d84
Author: Ian Roddis <tech@kinesin.ca >
Date: Wed Dec 15 09:52:29 2021 -0400
Copying daggyd for daggyr template, adding in basic routes
2021-12-16 12:16:12 -04:00
