470a6f2bb7
This paves the way for implementing daggys and other utilities. Squashed commit of the following: commit 1f77239ab3c9e44d190eef94531a39501c8c4dfe Author: Ian Roddis <gitlab@ie2r.com> Date: Mon Oct 18 16:25:02 2021 -0300 Adding README, stdout support for daggyd logging commit c2c237224e84a3be68aaa597ce98af1365e74a13 Author: Ian Roddis <gitlab@ie2r.com> Date: Mon Oct 18 16:10:29 2021 -0300 removing old daggyd commit cfea2baf61ca10c535801c5a391d2d525a1a2d04 Author: Ian Roddis <gitlab@ie2r.com> Date: Mon Oct 18 16:10:09 2021 -0300 Moving tests into their sub-project folders commit e41ca42069bea1db16dd76b6684a3f692fef6b15 Author: Ian Roddis <gitlab@ie2r.com> Date: Mon Oct 18 15:57:40 2021 -0300 Splitting out daggyd from libdaggy commit be97b146c1d2446f5c03cb78707e921f18c60bd8 Author: Ian Roddis <gitlab@ie2r.com> Date: Mon Oct 18 15:56:55 2021 -0300 Splitting out daggyd from libdaggy commit cb61e140e9d6d8832d61fb7037fd4c0ff6edad00 Author: Ian Roddis <gitlab@ie2r.com> Date: Mon Oct 18 15:49:47 2021 -0300 moving daggy to libdaggy
128 lines
2.1 KiB
Markdown
128 lines
2.1 KiB
Markdown
# Daggy Daemon
|
|
|
|
`daggyd` is the REST server process that handles receiving and running DAG specs.
|
|
|
|
# Running it
|
|
|
|
```bash
|
|
daggyd # That's it, will listen on 127.0.0.1:2503 , and run with a local executor
|
|
daggyd -d # Daemonize
|
|
|
|
daggyd --config FILE # Run with a config file
|
|
```
|
|
|
|
# Config Files
|
|
|
|
```json
|
|
{
|
|
"web-threads": 50,
|
|
"dag-threads": 50,
|
|
"port": 2503,
|
|
"ip": "localhost",
|
|
"logger": {
|
|
"name": "LoggerName",
|
|
"config": {
|
|
...
|
|
}
|
|
},
|
|
"executor": {
|
|
"name": "ExecutorName"
|
|
"config": {
|
|
...
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
## Loggers
|
|
|
|
### OStreamLogger
|
|
|
|
OStreamLogger doesn't persist data, but can write even updates to a file or
|
|
stdout.
|
|
|
|
The config for OStreamLogger looks like this:
|
|
|
|
```
|
|
{
|
|
...
|
|
"logger": {
|
|
"name": "OStreamLogger",
|
|
"config": {
|
|
"file": "/path/to/file"
|
|
}
|
|
}
|
|
...
|
|
}
|
|
```
|
|
|
|
If `file` is equal to `"-"`, then the logger will print events to stdout. This configuration
|
|
is the default if no logger is specified at all.
|
|
|
|
### RedisLogger
|
|
|
|
RedisLogger stores state in a [Redis](https://redis.io) instance.
|
|
|
|
The config for OStreamLogger looks like this (along with default values):
|
|
|
|
```
|
|
{
|
|
...
|
|
"logger": {
|
|
"name": "RedisLogger",
|
|
"config": {
|
|
"prefix": "daggy",
|
|
"host": "localhost",
|
|
"port": 6379
|
|
}
|
|
}
|
|
...
|
|
}
|
|
```
|
|
|
|
The `prefix` attribute is used to distinguish daggy instances. All keys will be prefixed with
|
|
the value of `prefix`.
|
|
|
|
|
|
## Executors
|
|
|
|
### ForkingTaskExecutor
|
|
|
|
ForkingTaskExecutor does pretty much what the name implies: it will execute tasks by
|
|
forking on the local machine.
|
|
|
|
It's config with default values looks like:
|
|
|
|
```
|
|
{
|
|
...
|
|
"executor": {
|
|
"name": "ForkingTaskExecutor",
|
|
"config": {
|
|
"threads": 10
|
|
}
|
|
}
|
|
...
|
|
}
|
|
```
|
|
|
|
If no executor is sepcified in the config, this is the executor used.
|
|
|
|
### SlurmTaskExecutor
|
|
|
|
The SlurmTaskExecutor will execute tasks on a [slurm](https://slurm.schedmd.com) cluster. It relies
|
|
on the slurm config to manage any parallelism limits and quotas.
|
|
|
|
It's config with default values looks like:
|
|
|
|
```
|
|
{
|
|
...
|
|
"executor": {
|
|
"name": "ForkingTaskExecutor",
|
|
"config": { }
|
|
}
|
|
...
|
|
}
|
|
```
|