94 lines
2.2 KiB
Markdown
94 lines
2.2 KiB
Markdown
# CLAUDE.md
|
|
|
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
|
|
## Project Overview
|
|
|
|
Daggy is a work orchestration framework for running workflows modeled as directed acyclic graphs (DAGs). It's designed for data processing pipelines where tasks have dependencies on each other.
|
|
|
|
The system consists of several components:
|
|
- **libdaggy**: Core library with DAG implementation, thread pool, task executors, and loggers
|
|
- **daggyd**: REST server that receives and runs DAG specifications
|
|
- **daggyr**: Remote worker service that runs as a task executor
|
|
- **utils/daggyc**: Command-line client
|
|
- **webui**: Web interface for monitoring DAG execution
|
|
|
|
## Build Commands
|
|
|
|
### Prerequisites
|
|
- cmake >= 3.14
|
|
- gcc >= 8
|
|
- npm (for the webui)
|
|
- libslurm (optional)
|
|
|
|
### Basic Build
|
|
```sh
|
|
# Create build directory
|
|
mkdir -p build
|
|
cd build
|
|
|
|
# Configure with CMake
|
|
cmake ..
|
|
|
|
# Build
|
|
make -j$(nproc)
|
|
```
|
|
|
|
### Build with Optional Features
|
|
```sh
|
|
# Enable Slurm support
|
|
cmake -DDAGGY_ENABLE_SLURM=ON ..
|
|
|
|
# Enable Redis support
|
|
cmake -DDAGGY_ENABLE_REDIS=ON ..
|
|
|
|
# Enable both features
|
|
cmake -DDAGGY_ENABLE_SLURM=ON -DDAGGY_ENABLE_REDIS=ON ..
|
|
```
|
|
|
|
### Web UI
|
|
```sh
|
|
cd webui
|
|
npm install
|
|
npm run build
|
|
```
|
|
|
|
### Running Tests
|
|
```sh
|
|
# Run all tests
|
|
cd build
|
|
tests/tests
|
|
|
|
# Run specific test case
|
|
cd build
|
|
tests/tests "[specific_test_tag]"
|
|
```
|
|
|
|
## Running the Application
|
|
|
|
### Running daggyd with Web UI
|
|
```sh
|
|
build/bin/daggyd -v --assets-dir webui/dist
|
|
```
|
|
|
|
## Architecture
|
|
|
|
Daggy uses a modular architecture consisting of:
|
|
|
|
1. **DAG Core**: Implementation of directed acyclic graph data structure with task dependencies
|
|
2. **Task Executors**: Different executors for running tasks:
|
|
- ForkingTaskExecutor (local execution)
|
|
- SlurmTaskExecutor (for HPC environments)
|
|
- DaggyRunnerTaskExecutor (for remote execution)
|
|
- SSHTaskExecutor (for SSH-based remote execution)
|
|
- NoopTaskExecutor (for testing)
|
|
|
|
3. **State Loggers**: Track execution state
|
|
- OStreamLogger (in-memory state management)
|
|
- RedisLogger (persisting state to Redis)
|
|
|
|
4. **Dynamic Task Features**:
|
|
- Task parameterization
|
|
- Runtime task generation
|
|
|
|
Tasks are defined in JSON with dependencies specified via "parents" or "children" attributes, allowing complex workflow patterns. |