daggy/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

Daggy is a work orchestration framework for running workflows modeled as directed acyclic graphs (DAGs). It's designed for data processing pipelines where tasks have dependencies on each other.

The system consists of several components:

• libdaggy: Core library with DAG implementation, thread pool, task executors, and loggers
• daggyd: REST server that receives and runs DAG specifications
• daggyr: Remote worker service that runs as a task executor
• utils/daggyc: Command-line client
• webui: Web interface for monitoring DAG execution

Build Commands

Prerequisites

• cmake >= 3.14
• gcc >= 8
• npm (for the webui)
• libslurm (optional)

Basic Build

# Create build directory
mkdir -p build
cd build

# Configure with CMake
cmake ..

# Build
make -j$(nproc)

Build with Optional Features

# Enable Slurm support
cmake -DDAGGY_ENABLE_SLURM=ON ..

# Enable Redis support
cmake -DDAGGY_ENABLE_REDIS=ON ..

# Enable both features
cmake -DDAGGY_ENABLE_SLURM=ON -DDAGGY_ENABLE_REDIS=ON ..

Web UI

cd webui
npm install
npm run build

Running Tests

# Run all tests
cd build
tests/tests

# Run specific test case
cd build
tests/tests "[specific_test_tag]"

Running the Application

Running daggyd with Web UI

build/bin/daggyd -v --assets-dir webui/dist

Architecture

Daggy uses a modular architecture consisting of:

1. DAG Core: Implementation of directed acyclic graph data structure with task dependencies

2. Task Executors: Different executors for running tasks:

   • ForkingTaskExecutor (local execution)
   • SlurmTaskExecutor (for HPC environments)
   • DaggyRunnerTaskExecutor (for remote execution)
   • SSHTaskExecutor (for SSH-based remote execution)
   • NoopTaskExecutor (for testing)

3. State Loggers: Track execution state

   • OStreamLogger (in-memory state management)
   • RedisLogger (persisting state to Redis)

4. Dynamic Task Features:

   • Task parameterization
   • Runtime task generation

Tasks are defined in JSON with dependencies specified via "parents" or "children" attributes, allowing complex workflow patterns.