71756d9ec2
(>10kb). Squashed commit of the following: commit b87fa418b4aca78928186a8fa992bef701e044a4 Author: Ian Roddis <tech@kinesin.ca> Date: Mon Feb 14 12:55:34 2022 -0400 removing memory leak commit 5e284ab92dbea991262a08c0cd50d6fc2f912e3b Author: Ian Roddis <tech@kinesin.ca> Date: Mon Feb 14 11:58:57 2022 -0400 Speeding up serialization, fixing payload sizing issue on daggyr commit e5e358820da4c2587741abdc3b6b103e5a4d4dd3 Author: Ian Roddis <tech@kinesin.ca> Date: Sun Feb 13 22:24:04 2022 -0400 changing newlines to std::endl for flush goodness commit 705ec86b75be947e64f4124ec8017cba2c8465e6 Author: Ian Roddis <tech@kinesin.ca> Date: Sun Feb 13 22:16:56 2022 -0400 adding more logging commit aa3db9c23e55da7a0523dc57e268b605ce8faac3 Author: Ian Roddis <tech@kinesin.ca> Date: Sun Feb 13 22:13:56 2022 -0400 Adding threadid commit 3b1a0f1333b2d43bc5ecad0746435504babbaa61 Author: Ian Roddis <tech@kinesin.ca> Date: Sun Feb 13 22:13:24 2022 -0400 Adding some debugging commit 804507e65251858fa597b7c27bcece8d8dfd589d Author: Ian Roddis <tech@kinesin.ca> Date: Sun Feb 13 21:52:53 2022 -0400 Removing curl global cleanup
Daggy Daemon
daggyd is the REST server process that handles receiving and running DAG specs.
Running it
daggyd # That's it, will listen on 127.0.0.1:2503 , and run with a local executor
daggyd -d # Daemonize
daggyd --config FILE # Run with a config file
Config Files
{
"web-threads": 50,
"dag-threads": 50,
"port": 2503,
"ip": "localhost",
"logger": {
"name": "LoggerName",
"config": {
...
}
},
"executor": {
"name": "ExecutorName"
"config": {
...
}
}
}
Loggers
OStreamLogger
OStreamLogger doesn't persist data, but can write even updates to a file or stdout.
The config for OStreamLogger looks like this:
{
...
"logger": {
"name": "OStreamLogger",
"config": {
"file": "/path/to/file"
}
}
...
}
If file is equal to "-", then the logger will print events to stdout. This configuration
is the default if no logger is specified at all.
RedisLogger
RedisLogger stores state in a Redis instance.
The config for OStreamLogger looks like this (along with default values):
{
...
"logger": {
"name": "RedisLogger",
"config": {
"prefix": "daggy",
"host": "localhost",
"port": 6379
}
}
...
}
The prefix attribute is used to distinguish daggy instances. All keys will be prefixed with
the value of prefix.
Executors
ForkingTaskExecutor
ForkingTaskExecutor does pretty much what the name implies: it will execute tasks by forking on the local machine.
It's config with default values looks like:
{
...
"executor": {
"name": "ForkingTaskExecutor",
"config": {
"threads": 10
}
}
...
}
If no executor is sepcified in the config, this is the executor used.
SlurmTaskExecutor
The SlurmTaskExecutor will execute tasks on a slurm cluster. It relies on the slurm config to manage any parallelism limits and quotas.
It's config with default values looks like:
{
...
"executor": {
"name": "ForkingTaskExecutor",
"config": { }
}
...
}