Commit Graph

222 Commits

Author SHA1 Message Date
Ian Roddis
3822426446 Adding check for unsatisfyable requirements in a remote task 2022-02-23 13:34:28 -04:00
Ian Roddis
6389f1a508 fixing up rst 2022-02-22 10:36:34 -04:00
Ian Roddis
b63739c0a9 Adding StopTask endpoint and endpoint documentation 2022-02-22 10:32:31 -04:00
Ian Roddis
0ee198e3e7 Fixing issue building dag with states where states were not properly transfered 2022-02-16 14:33:04 -04:00
Ian Roddis
3866332e77 Reverting to rapidjson for attempt record serialization 2022-02-16 10:06:48 -04:00
Ian Roddis
71756d9ec2 Fixing daggyr issues when reporting on tasks with very large outputs
(>10kb).

Squashed commit of the following:

commit b87fa418b4aca78928186a8fa992bef701e044a4
Author: Ian Roddis <tech@kinesin.ca>
Date:   Mon Feb 14 12:55:34 2022 -0400

    removing memory leak

commit 5e284ab92dbea991262a08c0cd50d6fc2f912e3b
Author: Ian Roddis <tech@kinesin.ca>
Date:   Mon Feb 14 11:58:57 2022 -0400

    Speeding up serialization, fixing payload sizing issue on daggyr

commit e5e358820da4c2587741abdc3b6b103e5a4d4dd3
Author: Ian Roddis <tech@kinesin.ca>
Date:   Sun Feb 13 22:24:04 2022 -0400

    changing newlines to std::endl for flush goodness

commit 705ec86b75be947e64f4124ec8017cba2c8465e6
Author: Ian Roddis <tech@kinesin.ca>
Date:   Sun Feb 13 22:16:56 2022 -0400

    adding more logging

commit aa3db9c23e55da7a0523dc57e268b605ce8faac3
Author: Ian Roddis <tech@kinesin.ca>
Date:   Sun Feb 13 22:13:56 2022 -0400

    Adding threadid

commit 3b1a0f1333b2d43bc5ecad0746435504babbaa61
Author: Ian Roddis <tech@kinesin.ca>
Date:   Sun Feb 13 22:13:24 2022 -0400

    Adding some debugging

commit 804507e65251858fa597b7c27bcece8d8dfd589d
Author: Ian Roddis <tech@kinesin.ca>
Date:   Sun Feb 13 21:52:53 2022 -0400

    Removing curl global cleanup
2022-02-15 11:22:21 -04:00
Ian Roddis
0d365f3a7b Increasing curl timeout 2022-02-11 22:29:27 -04:00
Ian Roddis
80cca0d6bf removing cout statement 2022-02-11 13:12:08 -04:00
Ian Roddis
f08b17317c Simplifying code 2022-02-11 13:09:01 -04:00
Ian Roddis
443bc789cf Unlocking unique lock to prevent one bad agent stalling the rest 2022-02-11 13:06:20 -04:00
Ian Roddis
6aab7be0a1 Same without buttons 2022-02-10 17:18:27 -04:00
Ian Roddis
015c3a3e04 Adding in minimal js to resubmit jobs, still need to get functionality working 2022-02-10 17:15:11 -04:00
Ian Roddis
0fba67adce Making logging to stdout the default. To disable logging use --log-level NONE 2022-02-04 12:47:01 -04:00
Ian Roddis
61e392cba3 Fixing unit tests, fixing cmake so make test works properly 2022-02-04 12:36:15 -04:00
Ian Roddis
ea3f67f226 Adding a general logger and integrating it with daggyr 2022-02-04 11:58:45 -04:00
Ian Roddis
57e93b5045 Simplifying daggyr server, and returning to a
task submit / task poll model.

Squashed commit of the following:

commit 0ef57f095d15f0402915de54f83c1671120bd228
Author: Ian Roddis <tech@kinesin.ca>
Date:   Wed Feb 2 08:18:03 2022 -0400

    Simplifying task polling and reducing lock scopes

commit d77ef02021cc728849c7d1fb0185dd1a861b4a3d
Author: Ian Roddis <tech@kinesin.ca>
Date:   Wed Feb 2 08:02:47 2022 -0400

    Simplifying check

commit c1acf34440162abb890a959f3685c2d184242ed5
Author: Ian Roddis <tech@kinesin.ca>
Date:   Wed Feb 2 08:01:13 2022 -0400

    Removing capacity tracking from runner, since it is maintained in daggyd

commit 9401246f92113ab140143c1895978b9de8bd9972
Author: Ian Roddis <tech@kinesin.ca>
Date:   Wed Feb 2 07:47:28 2022 -0400

    Adding retry for submission

commit 398aa04a320347bb35f23f3f101d91ab4df25652
Author: Ian Roddis <tech@kinesin.ca>
Date:   Tue Feb 1 14:54:20 2022 -0400

    Adding in execution note, as well as requeuing the result if the peer disconnects

commit 637b14af6d5b53f25b9c38d4c8a7ed8532af5599
Author: Ian Roddis <tech@kinesin.ca>
Date:   Tue Feb 1 14:13:59 2022 -0400

    Fixing locking issues

commit 4d6716dfda8aa7f51e0abbdab833aff618915ba0
Author: Ian Roddis <tech@kinesin.ca>
Date:   Tue Feb 1 13:33:14 2022 -0400

    Single task daggyr working

commit bd48a5452a92817faf25ee44a6115aaa2f6c30d1
Author: Ian Roddis <tech@kinesin.ca>
Date:   Tue Feb 1 12:22:04 2022 -0400

    Checkpointing work
2022-02-02 21:12:05 -04:00
Ian Roddis
c9bfce31e6 I cannot type today 2022-02-01 13:38:08 -04:00
Ian Roddis
b3397381dc Fixing logic error in submission batches 2022-02-01 13:34:19 -04:00
Ian Roddis
df108dfd28 Increasing curl timeout 2022-02-01 11:02:33 -04:00
Ian Roddis
cc7646f386 Adding delete method 2022-02-01 10:33:58 -04:00
Ian Roddis
3fdd568b00 Adding samples 2022-01-29 13:25:40 -04:00
Ian Roddis
e3a6c9086a Removing verbose logging 2022-01-28 14:36:33 -04:00
Ian Roddis
782e6a8b97 Changing behaviour of runner to submit at most MAX_SUBMITS for execution before checking for completed tasks 2022-01-28 14:28:46 -04:00
Ian Roddis
5af8deabec Making DaggyTaskRunner block until a runner is available 2022-01-28 14:18:01 -04:00
Ian Roddis
cbd493ec38 Fixing issue with daggyd config parsing 2022-01-28 13:28:29 -04:00
Ian Roddis
c9820c19a0 Small fixes to test cases 2022-01-28 10:23:21 -04:00
Ian Roddis
c09d5b7d8b Adding in queuing in the daggyd process for more even distribution 2022-01-28 10:09:55 -04:00
Ian Roddis
070d2ceddf Adding support for environment variables 2022-01-18 10:54:25 -04:00
Ian Roddis
2784aeee28 Adding in check for cores and memoryMB 2022-01-14 15:48:06 -04:00
Ian Roddis
07646de2cd Adding SSH executor 2022-01-14 15:39:57 -04:00
Ian Roddis
9e7d78788b Fixing issue in dagrunner where tasks were never removed from running list 2022-01-13 12:52:00 -04:00
Ian Roddis
854ca5be22 Reducing the scope of a lock 2022-01-13 10:49:09 -04:00
Ian Roddis
3f5d31e036 Removing needless second map lookup 2022-01-12 19:59:10 -04:00
Ian Roddis
c143034c00 Implementing a redirect for a GET on / to list of dagruns 2022-01-12 15:17:00 -04:00
Ian Roddis
868e74fbeb Seems to work now 2022-01-12 13:11:55 -04:00
Ian Roddis
9a5a247f15 Checkpointing work 2022-01-12 12:50:46 -04:00
Ian Roddis
04e95cfcf3 Making fork process more descriptive in the case of failure 2022-01-10 13:12:19 -04:00
Ian Roddis
7312776e39 Fixing issue with logic for daggyr submission 2022-01-10 13:09:11 -04:00
Ian Roddis
53308c063d Fixing a number of scaling issues:
- Missed closing of file descriptor made ForkingTaskExecutor
  silently die after running out of FDs
- Tightened up scope for locks to prevent http timeout
- Simplified threadpool
2022-01-10 13:02:10 -04:00
Ian Roddis
efd4078f70 Adding executor log to output 2022-01-07 17:40:07 -04:00
Ian Roddis
db6fd64601 Fixing dagrun counts 2022-01-07 17:38:29 -04:00
Ian Roddis
d58dc2fc35 Removing ability for multiple servers to listen to the same address 2022-01-07 16:31:07 -04:00
Ian Roddis
58604d98cd Removing extra prints 2022-01-07 16:14:39 -04:00
Ian Roddis
6ed15b17b2 Fixing issue with querying dag as json 2022-01-07 16:13:12 -04:00
Ian Roddis
f9076be081 Adding fix for race condition in task resource management 2022-01-07 16:00:04 -04:00
Ian Roddis
4da78c9dd6 Adding executor description for verbose output, adding output directory for build artifacts 2022-01-07 14:37:06 -04:00
Ian Roddis
856e5bd2f4 Adding in task attempts drilldown 2022-01-06 15:20:06 -04:00
Ian Roddis
1786b53f7b Fixing sample dag 2022-01-04 17:05:45 -04:00
Ian Roddis
64a85cd996 Updating example dag 2022-01-04 17:03:42 -04:00
Ian Roddis
f1479a72d9 Fixing test cases on daggyr for new polling 2022-01-04 17:03:11 -04:00