53308c063d
- Missed closing of file descriptor made ForkingTaskExecutor silently die after running out of FDs - Tightened up scope for locks to prevent http timeout - Simplified threadpool
Daggy Runner
daggyr is a REST server process that acts as a remote task executor.
Running it
daggyr # That's it, will listen on 127.0.0.1:2504 , and run with a local executor
daggyr -d # Daemonize
daggyr --config FILE # Run with a config file
Capacity and Allocation
On startup, a server's capacity is determined automatically. The capacities are:
| Capacity | Determined by | Default | Notes |
|---|---|---|---|
| cores | std::thread::hardware_concurrency() |
max(1, max - 2) |
A value of 0 will mean all cores |
| memory_mb | sysinfo.h |
max(100, totalram * 0.75) |
totalram is converted to MB |
When a daggyd process is selecting a runner to send a task to, it will
query the current capacities, and choose the runner that:
- Can satisfy the requirements of the task
- Has the lowest impact, which is the largest relative drop in available capacity across all capacities.
For instance, if a job were submitted that requires 2 cores and 5g of memory, and three runners reported the following capacities:
| Runner | free_cores | impact_cores | free_memory | impact_memory | max_impact |
|---|---|---|---|---|---|
| 1 | 70 | 2.8% | 20g | 25.00% | 25% |
| 2 | 4 | 50.0% | 80g | 6.25% | 50% |
| 3 | 10 | 20.0% | 30g | 16.67% | 20% |
Runner 3 would be selected. Even though it doesn't have the most memory or CPU capacity, allocating the job to it minimizes the impact to the overall availability.
Submission and Execution
Tasks submitted to the runner will be executed with cgroups to enforce limits.
Jobs are submitted asynchronously, and rely on the client to poll for
results using the GET /api/v1/task/:task_id to get the resulting
TaskAttempt.
Runners are stateless, meaning that killing one will kill any running tasks and any stored results will be lost.
Config Files
{
"web-threads": 50,
"port": 2504,
"ip": "localhost",
"capacity_overrides": {
"cores": 10,
"memory_mb": 100
}
}
Capacities can be overriden from the auto-discovered results.