Files

Ian Roddis 470a6f2bb7 Large re-organization to split daggyd away from the core libdaggy.

This paves the way for implementing daggys and other utilities.

Squashed commit of the following:

commit 1f77239ab3c9e44d190eef94531a39501c8c4dfe
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Oct 18 16:25:02 2021 -0300

    Adding README, stdout support for daggyd logging

commit c2c237224e84a3be68aaa597ce98af1365e74a13
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Oct 18 16:10:29 2021 -0300

    removing old daggyd

commit cfea2baf61ca10c535801c5a391d2d525a1a2d04
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Oct 18 16:10:09 2021 -0300

    Moving tests into their sub-project folders

commit e41ca42069bea1db16dd76b6684a3f692fef6b15
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Oct 18 15:57:40 2021 -0300

    Splitting out daggyd from libdaggy

commit be97b146c1d2446f5c03cb78707e921f18c60bd8
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Oct 18 15:56:55 2021 -0300

    Splitting out daggyd from libdaggy

commit cb61e140e9d6d8832d61fb7037fd4c0ff6edad00
Author: Ian Roddis <gitlab@ie2r.com>
Date:   Mon Oct 18 15:49:47 2021 -0300

    moving daggy to libdaggy

2021-10-18 16:28:40 -03:00

2.1 KiB

Raw Blame History

Daggy Daemon

daggyd is the REST server process that handles receiving and running DAG specs.

Running it

daggyd    # That's it, will listen on 127.0.0.1:2503 , and run with a local executor
daggyd -d # Daemonize

daggyd --config FILE # Run with a config file

Config Files

{
  "web-threads": 50,
    "dag-threads": 50,
    "port":  2503,
    "ip": "localhost",
    "logger": {
      "name": "LoggerName",
      "config": {
        ...
      }
    },
    "executor": {
      "name": "ExecutorName"
        "config": {
          ...
        }
    }
}

Loggers

OStreamLogger

OStreamLogger doesn't persist data, but can write even updates to a file or stdout.

The config for OStreamLogger looks like this:

{
  ...
  "logger": {
    "name": "OStreamLogger",
    "config": {
      "file": "/path/to/file"
    }
  }
  ...
}

If file is equal to "-", then the logger will print events to stdout. This configuration is the default if no logger is specified at all.

RedisLogger

RedisLogger stores state in a Redis instance.

The config for OStreamLogger looks like this (along with default values):

{
  ...
  "logger": {
    "name": "RedisLogger",
    "config": {
      "prefix": "daggy",
      "host": "localhost",
      "port": 6379
    }
  }
  ...
}

The prefix attribute is used to distinguish daggy instances. All keys will be prefixed with the value of prefix.

Executors

ForkingTaskExecutor

ForkingTaskExecutor does pretty much what the name implies: it will execute tasks by forking on the local machine.

It's config with default values looks like:

{
  ...
  "executor": {
    "name": "ForkingTaskExecutor",
    "config": {
      "threads": 10
    }
  }
  ...
}

If no executor is sepcified in the config, this is the executor used.

SlurmTaskExecutor

The SlurmTaskExecutor will execute tasks on a slurm cluster. It relies on the slurm config to manage any parallelism limits and quotas.