Ian Roddis
015c3a3e04
Adding in minimal js to resubmit jobs, still need to get functionality working
2022-02-10 17:15:11 -04:00
Ian Roddis
0fba67adce
Making logging to stdout the default. To disable logging use --log-level NONE
2022-02-04 12:47:01 -04:00
Ian Roddis
61e392cba3
Fixing unit tests, fixing cmake so make test works properly
2022-02-04 12:36:15 -04:00
Ian Roddis
ea3f67f226
Adding a general logger and integrating it with daggyr
2022-02-04 11:58:45 -04:00
Ian Roddis
57e93b5045
Simplifying daggyr server, and returning to a
...
task submit / task poll model.
Squashed commit of the following:
commit 0ef57f095d15f0402915de54f83c1671120bd228
Author: Ian Roddis <tech@kinesin.ca >
Date: Wed Feb 2 08:18:03 2022 -0400
Simplifying task polling and reducing lock scopes
commit d77ef02021cc728849c7d1fb0185dd1a861b4a3d
Author: Ian Roddis <tech@kinesin.ca >
Date: Wed Feb 2 08:02:47 2022 -0400
Simplifying check
commit c1acf34440162abb890a959f3685c2d184242ed5
Author: Ian Roddis <tech@kinesin.ca >
Date: Wed Feb 2 08:01:13 2022 -0400
Removing capacity tracking from runner, since it is maintained in daggyd
commit 9401246f92113ab140143c1895978b9de8bd9972
Author: Ian Roddis <tech@kinesin.ca >
Date: Wed Feb 2 07:47:28 2022 -0400
Adding retry for submission
commit 398aa04a320347bb35f23f3f101d91ab4df25652
Author: Ian Roddis <tech@kinesin.ca >
Date: Tue Feb 1 14:54:20 2022 -0400
Adding in execution note, as well as requeuing the result if the peer disconnects
commit 637b14af6d5b53f25b9c38d4c8a7ed8532af5599
Author: Ian Roddis <tech@kinesin.ca >
Date: Tue Feb 1 14:13:59 2022 -0400
Fixing locking issues
commit 4d6716dfda8aa7f51e0abbdab833aff618915ba0
Author: Ian Roddis <tech@kinesin.ca >
Date: Tue Feb 1 13:33:14 2022 -0400
Single task daggyr working
commit bd48a5452a92817faf25ee44a6115aaa2f6c30d1
Author: Ian Roddis <tech@kinesin.ca >
Date: Tue Feb 1 12:22:04 2022 -0400
Checkpointing work
2022-02-02 21:12:05 -04:00
Ian Roddis
c9bfce31e6
I cannot type today
2022-02-01 13:38:08 -04:00
Ian Roddis
b3397381dc
Fixing logic error in submission batches
2022-02-01 13:34:19 -04:00
Ian Roddis
df108dfd28
Increasing curl timeout
2022-02-01 11:02:33 -04:00
Ian Roddis
cc7646f386
Adding delete method
2022-02-01 10:33:58 -04:00
Ian Roddis
3fdd568b00
Adding samples
2022-01-29 13:25:40 -04:00
Ian Roddis
e3a6c9086a
Removing verbose logging
2022-01-28 14:36:33 -04:00
Ian Roddis
782e6a8b97
Changing behaviour of runner to submit at most MAX_SUBMITS for execution before checking for completed tasks
2022-01-28 14:28:46 -04:00
Ian Roddis
5af8deabec
Making DaggyTaskRunner block until a runner is available
2022-01-28 14:18:01 -04:00
Ian Roddis
cbd493ec38
Fixing issue with daggyd config parsing
2022-01-28 13:28:29 -04:00
Ian Roddis
c9820c19a0
Small fixes to test cases
2022-01-28 10:23:21 -04:00
Ian Roddis
c09d5b7d8b
Adding in queuing in the daggyd process for more even distribution
2022-01-28 10:09:55 -04:00
Ian Roddis
070d2ceddf
Adding support for environment variables
2022-01-18 10:54:25 -04:00
Ian Roddis
2784aeee28
Adding in check for cores and memoryMB
2022-01-14 15:48:06 -04:00
Ian Roddis
07646de2cd
Adding SSH executor
2022-01-14 15:39:57 -04:00
Ian Roddis
9e7d78788b
Fixing issue in dagrunner where tasks were never removed from running list
2022-01-13 12:52:00 -04:00
Ian Roddis
854ca5be22
Reducing the scope of a lock
2022-01-13 10:49:09 -04:00
Ian Roddis
3f5d31e036
Removing needless second map lookup
2022-01-12 19:59:10 -04:00
Ian Roddis
c143034c00
Implementing a redirect for a GET on / to list of dagruns
2022-01-12 15:17:00 -04:00
Ian Roddis
868e74fbeb
Seems to work now
2022-01-12 13:11:55 -04:00
Ian Roddis
9a5a247f15
Checkpointing work
2022-01-12 12:50:46 -04:00
Ian Roddis
04e95cfcf3
Making fork process more descriptive in the case of failure
2022-01-10 13:12:19 -04:00
Ian Roddis
7312776e39
Fixing issue with logic for daggyr submission
2022-01-10 13:09:11 -04:00
Ian Roddis
53308c063d
Fixing a number of scaling issues:
...
- Missed closing of file descriptor made ForkingTaskExecutor
silently die after running out of FDs
- Tightened up scope for locks to prevent http timeout
- Simplified threadpool
2022-01-10 13:02:10 -04:00
Ian Roddis
efd4078f70
Adding executor log to output
2022-01-07 17:40:07 -04:00
Ian Roddis
db6fd64601
Fixing dagrun counts
2022-01-07 17:38:29 -04:00
Ian Roddis
d58dc2fc35
Removing ability for multiple servers to listen to the same address
2022-01-07 16:31:07 -04:00
Ian Roddis
58604d98cd
Removing extra prints
2022-01-07 16:14:39 -04:00
Ian Roddis
6ed15b17b2
Fixing issue with querying dag as json
2022-01-07 16:13:12 -04:00
Ian Roddis
f9076be081
Adding fix for race condition in task resource management
2022-01-07 16:00:04 -04:00
Ian Roddis
4da78c9dd6
Adding executor description for verbose output, adding output directory for build artifacts
2022-01-07 14:37:06 -04:00
Ian Roddis
856e5bd2f4
Adding in task attempts drilldown
2022-01-06 15:20:06 -04:00
Ian Roddis
1786b53f7b
Fixing sample dag
2022-01-04 17:05:45 -04:00
Ian Roddis
64a85cd996
Updating example dag
2022-01-04 17:03:42 -04:00
Ian Roddis
f1479a72d9
Fixing test cases on daggyr for new polling
2022-01-04 17:03:11 -04:00
Ian Roddis
5a4c6d0756
Fixing invalid URL
2021-12-30 14:32:43 -04:00
Ian Roddis
a4b26bce04
Handling http errors a bit more gracefully
2021-12-30 14:29:56 -04:00
Ian Roddis
d9479f14e0
Adding in check for no such file
2021-12-30 14:06:28 -04:00
Ian Roddis
ac490b6f3e
Adding html output
2021-12-30 13:44:17 -04:00
Ian Roddis
509a1fcd23
Fixing argument parsing for integers
2021-12-27 22:15:16 -04:00
Ian Roddis
3c6966a9ac
Adding discovery of capacities after the fact.
2021-12-24 10:52:06 -04:00
Ian Roddis
8ca5cdafe2
Fixing issue with resolving
2021-12-24 10:44:19 -04:00
Ian Roddis
d832b935d7
Moving to a poll method for workers, and daggyd-preserved capacities
2021-12-24 10:22:58 -04:00
Ian Roddis
779d6adaea
Moving to a poll method for workers, and daggyd-preserved capacities
2021-12-24 10:21:19 -04:00
Ian Roddis
0914ede8fb
Fixing test, since executor now autovivifies certain job attributes
2021-12-23 11:50:16 -04:00
Ian Roddis
dd473ab8f0
Adjusting capacity impact calculation to yield a more even distribution of jobs
2021-12-23 11:47:55 -04:00
