daggy/daggyr

Daggy Runner

daggyr is a REST server process that acts as a remote task executor.

Running it

daggyr    # That's it, will listen on 127.0.0.1:2504 , and run with a local executor
daggyr -d # Daemonize

daggyr --config FILE # Run with a config file

Capacity and Allocation

On startup, a server's capacity is determined automatically. The capacities are:

Capacity	Determined by	Default	Notes
cores	std::thread::hardware_concurrency()	max(1, max - 2)	A value of 0 will mean all cores
memory_mb	sysinfo.h	max(100, totalram * 0.75)	totalram is converted to MB

When a daggyd process is selecting a runner to send a task to, it will query the current capacities, and choose the runner that:

• Can satisfy the requirements of the task
• Has the lowest impact, which is the largest relative drop in available capacity across all capacities.

For instance, if a job were submitted that requires 2 cores and 5g of memory, and three runners reported the following capacities:

Runner	free_cores	impact_cores	free_memory	impact_memory	max_impact
1	70	2.8%	20g	25.00%	25%
2	4	50.0%	80g	6.25%	50%
3	10	20.0%	30g	16.67%	20%

Runner 3 would be selected. Even though it doesn't have the most memory or CPU capacity, allocating the job to it minimizes the impact to the overall availability.

Submission and Execution

Tasks submitted to the runner will be executed with cgroups to enforce limits.

Jobs are submitted asynchronously, and rely on the client to poll for results using the GET /api/v1/task/:task_id to get the resulting TaskAttempt.

Runners are stateless, meaning that killing one will kill any running tasks and any stored results will be lost.

Config Files

{
  "web-threads": 50,
  "port":  2504,
  "ip": "localhost",
  "capacity_overrides": {
    "cores": 10,
    "memory_mb": 100
  }
}

Capacities can be overriden from the auto-discovered results.