sovereign/README.md

# Sovereign

Sovereign is an Ansible project that deploys a complete self-hosted infrastructure stack for small businesses using Docker and Docker Compose on a single Linux host. Every service is behind Traefik (TLS via Let's Encrypt), authenticated through Authentik (OIDC/OAuth2), and ships logs to Graylog (GELF UDP).

## Table of Contents

- [Services](#services)
- [Requirements](#requirements)
- [Quick Start](#quick-start)
- [New Tenant Setup](#new-tenant-setup)
- [Configuration Reference](#configuration-reference)
- [Deployment](#deployment)
- [Testing](#testing)
- [Maintenance](#maintenance)
- [Architecture Notes](#architecture-notes)

---

## Services

| Role | Service | URL |
|------|---------|-----|
| `common` | Traefik (reverse proxy + TLS) | `traefik.<domain>` |
| `graylog` | Graylog + OpenSearch + MongoDB | `logs.<domain>` |
| `authentik` | Authentik (identity provider) | `auth.<domain>` |
| `minio` | MinIO (object storage) | `s3.<domain>`, `minio.<domain>` |
| `nextcloud` | Nextcloud + MariaDB + Redis | `cloud.<domain>` |
| `stalwart` | Stalwart Mail (SMTP/IMAP) | `mail.<domain>` |
| `roundcube` | Roundcube (webmail) | `webmail.<domain>` |
| `matrix` | Synapse + Element | `matrix.<domain>`, `chat.<domain>` |
| `jitsi` | Jitsi Meet | `meet.<domain>` |
| `headscale` | Headscale (WireGuard mesh VPN) | `headscale.<domain>` |
| `wazuh` | Wazuh Manager + Indexer + Dashboard | `wazuh.<domain>` |
| `vaultwarden` | Vaultwarden + PostgreSQL | `vault.<domain>` |
| `forgejo` | Forgejo + PostgreSQL | `git.<domain>` |
| `website` | Nginx (static website) | `<domain>` |

---

## Requirements

**Control machine** (where you run Ansible):

- Python 3.9+
- Ansible 8+ — installed via `pip install -r requirements.txt` (see [Installing Dependencies](#installing-dependencies))
- Ansible collections (see [Installing Dependencies](#installing-dependencies))

**Target host**:

- Ubuntu 22.04 or 24.04 (amd64)
- Root or sudo access
- Ports 80, 443, and 51820/UDP open
- DNS A records pointing `<domain>` and `*.<domain>` to the host IP

### Installing Dependencies

Install Python packages (Ansible, Molecule, and linting tools) and Ansible collections:

```bash
pip install -r requirements.txt
ansible-galaxy collection install -r requirements.yml
```

Python packages (`requirements.txt`): `ansible`, `molecule`, `ansible-lint`, `yamllint`.

Ansible collections (`requirements.yml`): `community.docker >=3.0.0`, `community.general >=8.0.0`, `ansible.posix >=1.5.0`.

---

## Quick Start

```bash
# 1. Clone the repo
git clone <repo-url> sovereign && cd sovereign

# 2. Install Python packages and Ansible collections
pip install -r requirements.txt
ansible-galaxy collection install -r requirements.yml

# 3. Configure the target host
export SOVEREIGN_HOST=203.0.113.10
export SOVEREIGN_USER=ubuntu
export SOVEREIGN_SSH_KEY=~/.ssh/id_rsa

# 4. Edit the tenant config
cp inventories/production/group_vars/all.yml inventories/production/group_vars/all.yml.bak
$EDITOR inventories/production/group_vars/all.yml

# 5. Deploy
ansible-playbook playbooks/site.yml
```

---

## New Tenant Setup

Each deployment is controlled entirely by `inventories/production/group_vars/all.yml`. Follow these steps for every new tenant.

### 1. Set the base domain

```yaml
base_domain: "example.com"
```

All service subdomains are derived from this value automatically. DNS must have an A record for `example.com` and a wildcard `*.example.com` pointing to the server IP before deployment.

### 2. Replace all secrets

Search the config file for every `changeme_*` placeholder and replace with secure values. The sections below describe how to generate each one.

**General secrets** (use a password manager or `openssl rand -base64 32`):

```bash
# Generic random secret
openssl rand -base64 32

# Graylog password_secret (min 16 chars, recommend 64)
openssl rand -base64 48

# Graylog root password hash
echo -n 'yourpassword' | sha256sum | awk '{print $1}'

# Traefik dashboard password (htpasswd format)
htpasswd -nb admin yourpassword
```

**Authentik secret key** — must be exactly 50 characters:

```bash
openssl rand -base64 37 | head -c 50
```

**Roundcube DES key** — must be exactly 24 characters:

```bash
openssl rand -base64 18 | head -c 24
```

**Forgejo tokens** — generate three separate secrets:

```bash
# forgejo_secret_key, forgejo_internal_token, forgejo_lfs_jwt_secret
for i in 1 2 3; do openssl rand -hex 32; done
```

**Vaultwarden admin token** — hash a password with argon2:

```bash
# Requires the vaultwarden container to be running, or use the web tool at
# https://argon2.online — use the token output directly
echo -n 'yourpassword' | argon2 "$(openssl rand -base64 32)" -id -t 3 -m 16 -p 4 -l 32 -e
```

### 3. Configure SMTP

The `smtp_*` variables control outbound email for all services. The default routes through the bundled Stalwart mail server:

```yaml
smtp_host: "stalwart"
smtp_port: 587
smtp_from: "noreply@{{ base_domain }}"
smtp_user: "noreply@{{ base_domain }}"
smtp_password: "changeme_smtp"
smtp_tls: "starttls"
```

To use an external relay (SendGrid, Postmark, etc.), replace `smtp_host` with the relay hostname and update credentials accordingly.

### 4. Create Authentik OIDC applications

After the first deployment, log into Authentik at `https://auth.<domain>` and create an OAuth2/OIDC provider and application for each service that integrates with SSO. Then fill in the `changeme_*_oidc_secret` placeholders in the relevant compose templates under `roles/<service>/templates/`.

Services that require Authentik OIDC configuration:

| Service | Template variable |
|---------|------------------|
| MinIO | `changeme_minio_oidc_secret` |
| Headscale | `changeme_headscale_oidc_secret` |
| Vaultwarden | `changeme_vaultwarden_oidc_secret` |
| Forgejo | `changeme_forgejo_oidc_secret` |

### 5. Wazuh TLS certificates

Wazuh requires TLS certificates between its manager, indexer, and dashboard components before the first run. Generate them using the Wazuh certificate tool:

```bash
# Download the Wazuh certs generation tool
curl -sO https://packages.wazuh.com/4.9/wazuh-certs-tool.sh
curl -sO https://packages.wazuh.com/4.9/config.yml

# Edit config.yml with your node hostnames, then run:
bash wazuh-certs-tool.sh -A

# Copy the resulting certs into the wazuh data directory on the target host
# before running the wazuh role for the first time.
```

Refer to the [Wazuh Docker documentation](https://documentation.wazuh.com/current/deployment-options/docker/wazuh-container.html) for full details.

### 6. Static website content

Place your static HTML/CSS/JS files in `/opt/sovereign/website/html/` on the target host. Nginx serves this directory at `https://<domain>`. The directory is created by the `website` role on first deployment — you can populate it before or after running the playbook.

---

## Configuration Reference

All variables live in `inventories/production/group_vars/all.yml`.

### Global

| Variable | Default | Description |
|----------|---------|-------------|
| `base_domain` | `example.com` | Root domain. All subdomains are derived from this. |
| `sovereign_base_dir` | `/opt/sovereign` | Base path on the target host for all service data. |

### Branding

These variables apply consistent tenant branding across all services that support it. Services apply branding via environment variables, config file templates, or post-deploy API calls (e.g. Nextcloud `occ`, Authentik blueprints).

| Variable | Default | Description |
|----------|---------|-------------|
| `tenant_name` | `Example Corp` | Display name shown in service UIs, email subjects, and page titles. |
| `tenant_logo_local_path` | `""` | Path to a logo image on the Ansible control machine (PNG recommended). Leave empty to use each service's default logo. Example: `files/logo.png`. |
| `tenant_primary_color` | `#2563eb` | Primary brand colour (hex). Used for backgrounds, buttons, and highlights. |
| `tenant_accent_color` | `#1e40af` | Secondary/accent colour (hex). |

Services with branding support: Authentik (title, colour, logo via blueprint), Element/Matrix (brand name, theme), Forgejo (app name, logo), Nextcloud (name, colour, logo via `occ`), Jitsi (app name, watermark), Roundcube (product name), Wazuh dashboard (title).

### Traefik (`common` role)

| Variable | Default | Description |
|----------|---------|-------------|
| `traefik_acme_email` | `admin@<domain>` | Email used for Let's Encrypt certificate registration. |
| `traefik_domain` | `traefik.<domain>` | Traefik dashboard URL. |
| `traefik_dashboard_password` | — | htpasswd-formatted credential for dashboard basic auth. |
| `traefik_version` | `v3.1` | Traefik image tag. |

### Graylog

| Variable | Default | Description |
|----------|---------|-------------|
| `graylog_domain` | `logs.<domain>` | Graylog web UI URL. |
| `graylog_version` | `6.0` | Graylog image tag. |
| `graylog_password_secret` | — | Random secret, minimum 16 characters. |
| `graylog_root_password_sha2` | — | SHA-256 hash of the root (admin) password. |
| `graylog_host` | `127.0.0.1` | IP address reachable from Docker containers for GELF ingestion. Usually the host's Docker bridge IP or `127.0.0.1` when using host networking. |
| `graylog_gelf_port` | `12201` | UDP port for GELF log ingestion. |

### Authentik

| Variable | Default | Description |
|----------|---------|-------------|
| `authentik_domain` | `auth.<domain>` | Authentik URL. |
| `authentik_version` | `2024.10.5` | Authentik image tag. |
| `authentik_secret_key` | — | 50-character random string used for signing. |
| `authentik_db_password` | — | PostgreSQL password for Authentik's database. |
| `authentik_admin_email` | `admin@<domain>` | Initial admin account email. |
| `authentik_admin_password` | — | Initial admin account password. |

### Stalwart Mail

| Variable | Default | Description |
|----------|---------|-------------|
| `stalwart_domain` | `mail.<domain>` | Stalwart web admin URL. |
| `stalwart_version` | `latest` | Stalwart image tag. |
| `stalwart_admin_password` | — | Stalwart admin password. |

### Roundcube

| Variable | Default | Description |
|----------|---------|-------------|
| `roundcube_domain` | `webmail.<domain>` | Roundcube URL. |
| `roundcube_version` | `latest` | Roundcube image tag. |
| `roundcube_db_password` | — | MariaDB password for Roundcube's database. |
| `roundcube_des_key` | — | Exactly 24-character key for session encryption. |

### Wazuh

| Variable | Default | Description |
|----------|---------|-------------|
| `wazuh_domain` | `wazuh.<domain>` | Wazuh dashboard URL. |
| `wazuh_version` | `4.9.0` | Wazuh image tag. |
| `wazuh_admin_password` | — | Wazuh dashboard admin password. |
| `wazuh_api_password` | — | Wazuh REST API password. |

### Headscale / WireGuard

| Variable | Default | Description |
|----------|---------|-------------|
| `headscale_domain` | `headscale.<domain>` | Headscale control plane URL. |
| `headscale_version` | `0.23.0` | Headscale image tag. |
| `wireguard_port` | `51820` | UDP port for WireGuard traffic. Must be open in the firewall. |
| `headscale_noise_private_key` | `""` | Leave blank; generated automatically on first run. |

### Matrix / Element

| Variable | Default | Description |
|----------|---------|-------------|
| `matrix_domain` | `matrix.<domain>` | Synapse homeserver URL. |
| `element_domain` | `chat.<domain>` | Element web client URL. |
| `matrix_version` | `v1.118.0` | Synapse image tag. |
| `matrix_registration_secret` | — | Shared secret for server-side user registration. |
| `matrix_db_password` | — | PostgreSQL password for Synapse's database. |

### Jitsi

| Variable | Default | Description |
|----------|---------|-------------|
| `jitsi_domain` | `meet.<domain>` | Jitsi Meet URL. |
| `jitsi_version` | `stable-9753` | Jitsi image tag. |
| `jitsi_jicofo_auth_password` | — | Internal XMPP password for Jicofo. |
| `jitsi_jvb_auth_password` | — | Internal XMPP password for the video bridge. |
| `jitsi_jibri_recorder_password` | — | Internal XMPP password for Jibri (recording). |
| `jitsi_jibri_xmpp_password` | — | Internal XMPP password for Jibri XMPP. |
| `jitsi_turn_secret` | — | Shared secret for TURN server authentication. |

### MinIO

| Variable | Default | Description |
|----------|---------|-------------|
| `minio_domain` | `s3.<domain>` | MinIO S3 API endpoint. |
| `minio_console_domain` | `minio.<domain>` | MinIO web console URL. |
| `minio_version` | `latest` | MinIO image tag. |
| `minio_root_user` | `minioadmin` | MinIO root username. |
| `minio_root_password` | — | MinIO root password. |
| `minio_nextcloud_bucket` | `nextcloud` | Bucket name for Nextcloud primary storage. |
| `minio_nextcloud_access_key` | `nextcloud` | Access key for Nextcloud's MinIO credentials. |
| `minio_nextcloud_secret_key` | — | Secret key for Nextcloud's MinIO credentials. |

### Nextcloud

| Variable | Default | Description |
|----------|---------|-------------|
| `nextcloud_domain` | `cloud.<domain>` | Nextcloud URL. |
| `nextcloud_version` | `29` | Nextcloud image tag. |
| `nextcloud_admin_user` | `admin` | Initial Nextcloud admin username. |
| `nextcloud_admin_password` | — | Initial Nextcloud admin password. |
| `nextcloud_db_password` | — | MariaDB password for Nextcloud's database. |
| `nextcloud_db_root_password` | — | MariaDB root password. |

### Vaultwarden

| Variable | Default | Description |
|----------|---------|-------------|
| `vaultwarden_domain` | `vault.<domain>` | Vaultwarden URL. |
| `vaultwarden_version` | `latest` | Vaultwarden image tag. |
| `vaultwarden_admin_token` | — | Argon2-hashed token for the `/admin` panel. |
| `vaultwarden_db_password` | — | PostgreSQL password for Vaultwarden's database. |

### Forgejo

| Variable | Default | Description |
|----------|---------|-------------|
| `forgejo_domain` | `git.<domain>` | Forgejo URL. |
| `forgejo_version` | `latest` | Forgejo image tag. |
| `forgejo_db_password` | — | PostgreSQL password for Forgejo's database. |
| `forgejo_secret_key` | — | Random hex secret for internal signing. |
| `forgejo_internal_token` | — | Random hex token for internal API calls. |
| `forgejo_lfs_jwt_secret` | — | Random hex secret for Git LFS JWT tokens. |
| `forgejo_admin_user` | `admin` | Initial admin username. |
| `forgejo_admin_password` | — | Initial admin password. |
| `forgejo_admin_email` | `admin@<domain>` | Initial admin email. |
| `forgejo_ssh_port` | `2222` | Host port for Forgejo SSH access. Must be open in the firewall. |

### Website

| Variable | Default | Description |
|----------|---------|-------------|
| `website_nginx_version` | `alpine` | Nginx image tag used to serve the static site. |

### SMTP (shared)

These variables are consumed by every service that sends email.

| Variable | Default | Description |
|----------|---------|-------------|
| `smtp_host` | `stalwart` | SMTP relay hostname. Default routes through the bundled Stalwart container. |
| `smtp_port` | `587` | SMTP submission port. |
| `smtp_from` | `noreply@<domain>` | Default sender address. |
| `smtp_user` | `noreply@<domain>` | SMTP authentication username. |
| `smtp_password` | — | SMTP authentication password. |
| `smtp_tls` | `starttls` | TLS mode: `starttls` or `tls`. |

---

## Deployment

### Environment variables

The inventory reads connection details from environment variables:

```bash
export SOVEREIGN_HOST=203.0.113.10   # target host IP or hostname
export SOVEREIGN_USER=ubuntu          # SSH user with sudo privileges
export SOVEREIGN_SSH_KEY=~/.ssh/id_rsa
```

### Full deployment

```bash
ansible-playbook playbooks/site.yml
```

Services are deployed in dependency order: Graylog (logging) → Authentik (auth) → all other services.

### Deploy a single service

Use the role's tag to deploy only that service:

```bash
ansible-playbook playbooks/site.yml --tags authentik
ansible-playbook playbooks/site.yml --tags nextcloud
ansible-playbook playbooks/site.yml --tags website
```

Available tags: `common`, `graylog`, `authentik`, `minio`, `nextcloud`, `stalwart`, `roundcube`, `matrix`, `jitsi`, `headscale`, `wazuh`, `vaultwarden`, `forgejo`, `website`.

### Dry run

Preview changes without applying them:

```bash
ansible-playbook playbooks/site.yml --check --diff
```

### Syntax check / lint

```bash
ansible-playbook playbooks/site.yml --syntax-check
ansible-lint
```

---

## Testing

Each role has a [Molecule](https://ansible.readthedocs.io/projects/molecule/) test scenario under `roles/<role>/molecule/default/`. Tests run entirely on the local machine — no target host or Docker daemon required.

### What the tests cover

- **Directory creation** — all expected data directories are created with correct permissions.
- **Template rendering** — every Jinja2 template renders without errors and with all variables substituted (no unresolved `{{ }}` in output files).
- **Config file content** — role-specific config files (Element `config.json`, Headscale `config.yaml`, Authentik branding blueprint, Roundcube `custom.inc.php`, Jitsi interface config, Wazuh dashboard YAML) contain the expected values.
- **Docker Compose structure** — `docker-compose.yml` references the correct image, Traefik routing labels, GELF logging address, and external network declaration.
- **Idempotency** — Molecule re-runs each role after converge and asserts zero changed tasks.

Docker/OS tasks (container start, `apt`, `systemd`, `sysctl`, health checks) are skipped during tests via the `molecule_test_mode` variable, which defaults to `false` and has no effect on real deployments.

### Install test dependencies

```bash
pip install -r requirements.txt
ansible-galaxy collection install -r requirements.yml
```

### Run tests for a single role

```bash
cd roles/authentik
molecule test
```

`molecule test` runs the full lifecycle: dependency → converge → idempotency check → verify → cleanup.

For a faster iteration loop during development:

```bash
# Apply the role and run assertions (skip create/destroy lifecycle)
molecule converge && molecule verify

# Clean up temp files when done
molecule destroy
```

### Run tests for all roles

```bash
for role in roles/*/; do
  echo "=== Testing $role ==="
  (cd "$role" && molecule test)
done
```

### Lint

```bash
ansible-lint       # Ansible best-practice checks across all roles
yamllint .         # YAML formatting checks
```

Both tools are configured via `.ansible-lint` and `.yamllint` at the repo root. The ansible-lint config mocks Docker and system modules so linting works without a live environment.

### Adding tests for a new role

1. Create `roles/<service>/molecule/default/` with `molecule.yml`, `converge.yml`, and `verify.yml` following the pattern of an existing simple role (e.g. `roles/website/molecule/default/`).
2. Add the new role's variables to `molecule/shared/vars.yml`.
3. Add `when: not (molecule_test_mode | default(false))` to any tasks that call `community.docker.docker_compose_v2`, `ansible.builtin.uri` (health checks), or `ansible.builtin.command` (docker exec).
4. Add the same guard to the role's restart handler in `handlers/main.yml`.

---

## Maintenance

### Updating a service

Change the version variable in `all.yml` (e.g., `nextcloud_version: "30"`) and re-run the relevant tag:

```bash
ansible-playbook playbooks/site.yml --tags nextcloud
```

The handler will recreate the container with the new image.

### Restarting a service

SSH into the host and use Docker Compose directly:

```bash
cd /opt/sovereign/nextcloud
docker compose restart
```

Or pull and recreate:

```bash
docker compose pull
docker compose up -d --force-recreate
```

### Viewing logs

All containers ship logs to Graylog via GELF UDP. Use the Graylog web UI at `https://logs.<domain>` to search and filter.

To tail logs directly on the host:

```bash
docker logs -f nextcloud
docker logs -f authentik-server
```

### Backing up data

All persistent data is stored under `/opt/sovereign/` on the target host. A minimal backup strategy:

```bash
# Stop services, snapshot, restart
cd /opt/sovereign/vaultwarden && docker compose stop
tar czf /backup/vaultwarden-$(date +%F).tar.gz /opt/sovereign/vaultwarden
cd /opt/sovereign/vaultwarden && docker compose start
```

For databases, prefer native dumps over filesystem snapshots taken while the container is running:

```bash
# PostgreSQL (Vaultwarden, Forgejo, Matrix, Authentik)
docker exec vaultwarden-db pg_dump -U vaultwarden vaultwarden > vaultwarden-$(date +%F).sql

# MariaDB (Nextcloud, Roundcube)
docker exec nextcloud-db mysqldump -u root -p"$NEXTCLOUD_DB_ROOT_PASSWORD" nextcloud > nextcloud-$(date +%F).sql
```

### Rotating secrets

1. Update the value in `all.yml`.
2. Re-run the affected role: `ansible-playbook playbooks/site.yml --tags <service>`.
3. Some services (Authentik, Graylog) require a container restart to pick up new environment variables — this happens automatically via the role's handler.

### Adding a new service role

Follow the pattern used by existing roles:

1. Create `roles/<service>/{defaults,handlers,tasks,templates}/main.yml` and `docker-compose.yml.j2`.
2. Add service variables to `inventories/production/group_vars/all.yml`.
3. Add the role to `playbooks/site.yml` with an appropriate tag.
4. Attach the container to the `sovereign` Docker network and add Traefik labels for routing.
5. Add `logging: driver: gelf` with `gelf-address: "udp://{{ graylog_host }}:{{ graylog_gelf_port }}"` to ship logs.
6. Add a Molecule scenario — see [Adding tests for a new role](#adding-tests-for-a-new-role).

---

## Architecture Notes

- **Reverse proxy**: Traefik handles all inbound HTTPS traffic, terminates TLS using Let's Encrypt (TLS challenge), and routes to containers via Docker labels.
- **Authentication**: The `authentik` Traefik forward-auth middleware is defined in the `common` role and can be applied to any router label: `traefik.http.routers.<name>.middlewares=authentik`.
- **Networking**: All containers that need Traefik routing join the external `sovereign` Docker network. Services with databases also have a private `internal` network for backend isolation.
- **Logging**: Every container uses the `gelf` log driver pointed at `graylog_host:12201`. `graylog_host` should be an IP reachable from inside Docker containers (typically the host's IP on the Docker bridge, not `localhost`).
- **Data persistence**: Each service stores data under `{{ sovereign_base_dir }}/<service>/` (default `/opt/sovereign/<service>/`). This path is defined in each role's `defaults/main.yml` as `<service>_data_dir`.