Testing Hop3¶

Note: For comprehensive testing documentation, see Testing Strategy.

Overview¶

Hop3 uses a multi-layer testing approach across three complementary runners (pytest for platform code, hop3-test for applications, validoc for tutorials):

Test Layers (pytest-based)¶

Three layers live under each package's tests/. The layer is decided by what a test needs: Docker, root, or host-mutation. Duplication across layers is allowed (see ADR 043).

1. Unit Tests (tests/a_unit/): Individual components in isolation; no Docker; count toward coverage; tier fast
2. Integration Tests (tests/b_integration/): Multiple components within a subsystem, in-process against a real in-memory DB; no Docker; count toward coverage; tier check
3. E2E Tests (tests/c_e2e/): Full deployments with a real Docker target; no coverage; run in the check tier (Docker) and nightly

Application Testing (hop3-test)¶

Beyond pytest, two more runners cover the other domains (only pytest produces coverage):

• hop3-test: applications. Deploys real apps and demos to a DeploymentTarget (Docker, SSH, Hetzner) and verifies them.
• validoc: narratives. Runs the tutorials as doc-as-tests.

Quick Start¶

Run Unit + Integration Tests¶

# Fast lane — unit only, all packages, no Docker (< 1 min)
make test-fast

# Check tier — unit + integration, all packages, no Docker
make test

# Or using pytest directly
pytest packages/hop3-server/tests/a_unit/ packages/hop3-server/tests/b_integration/

Run E2E Tests (Docker-based)¶

# The Docker e2e layer (c_e2e): real deploys, backups, git-push
make test-e2e

# Or using pytest directly (Docker-only; pass --ssh-host to target a real box)
pytest packages/hop3-server/tests/c_e2e/

Which Docker daemon¶

Every Docker test uses the daemon the docker command would use: DOCKER_HOST, else DOCKER_CONTEXT, else the current context (docker context use). The harness resolves it once, prints it (Docker daemon: …), and exports it as DOCKER_HOST, so the docker command, docker-py and hop3-deploy-server all reach the same daemon. To run the tests on another machine:

docker context create builder --docker "host=ssh://you@builder.local"
docker context use builder        # or: export DOCKER_CONTEXT=builder

A container's ports are then published on that machine, each on a host port Docker picks (a fixed one collides with whatever else the machine runs), and the harness reaches them at the machine's hostname, which must resolve from here. They listen on all its interfaces, and the test container's API accepts a known fallback key, so keep that machine on a trusted network. Build contexts hold only what the test images copy (the root .dockerignore), so a build sends a few megabytes rather than the checkout. One run at a time per developer machine: the Docker target's container name is fixed, and hop3-test takes a lock (~/.hop3/docker-target.lock) before it starts.

Run Application Tests¶

# Deploy the real-app catalog on Docker via hop3-test
make test-apps

# Test a single app or path
make test-app APP=../hop3-catalog/apps/golden/gitea

Test Commands Reference¶

Makefile Targets¶

Hetzner Cloud Testing¶

hop3-test CLI¶

# System testing (deploys Hop3, then deploys + verifies apps)
hop3-test run --docker                       # Deploy + test defaults on Docker
hop3-test run --docker --clean --with all    # Clean install with all addons
hop3-test run --docker --status golden       # One maturity tier
hop3-test run --docker gitea                 # One app, by catalog id
hop3-test run --host server.example.com    # Remote via SSH (or set $HOP3_HOST)
hop3-test run --reuse --host server.example.com   # Skip deploy, test existing
hop3-test run --docker --from git --branch devel  # Deploy from git
hop3-test run --docker --mode nightly        # Wider matrix (smoke | ci | nightly | full | ...)

# Upgrade-chain: install a baseline release on a FRESH box, then upgrade
# in-place through a version chain — each version by its OWN installer.
hop3-test upgrade-chain --docker                    # 0.6.2 -> local tree
hop3-test upgrade-chain --docker --chain 0.6.2,local
hop3-test upgrade-chain --docker --chain local,local   # cheapest smoke (no old version)
hop3-test upgrade-chain --provider hetzner --image ubuntu-24.04  # fresh cloud VPS

# List / inspect
hop3-test list                      # List available app/demo/tutorial tests
hop3-test why <run-id>              # Show the diagnostic bundle for a failed run

Upgrade-chain testing¶

hop3-test upgrade-chain verifies that a running server survives a chain of in-place upgrades. Each hop is a git ref (a release tag, or local for the current tree), installed by that version's own hop3-deploy-server (checked out into a worktree and run via uv run) on a fresh box (Docker container or a rebuilt Hetzner VPS). Every hop after the first is an in-place update. Each one has to come back healthy with a readable schema (ADR 043 §10, Cross-version upgrade validation). 0.6.0 is not a viable baseline (its hop3-rootd can't start) and is excluded from the default chain.

Test Organization¶

A test's layer is decided by what it needs: if it needs Docker, root, or host-mutation it belongs in c_e2e; otherwise it stays in-process in a_unit or b_integration. Each layer gets its marker from the root conftest.py, so -m fast, -m integration, -m e2e, and -m "not needs_docker" select the right lane regardless of package.

pytest -m fast                 # a_unit (+ flat unit suites)
pytest -m "not needs_docker"   # everything except the Docker e2e layer

Layer 1: Unit Tests¶

Location: packages/hop3-server/tests/a_unit/ Speed: < 1 second Requirements: None; counts toward coverage Marker: fast

pytest packages/hop3-server/tests/a_unit/ -v

Layer 2: Integration Tests¶

Location: packages/hop3-server/tests/b_integration/ Speed: ~10 seconds Requirements: In-process, real in-memory DB; counts toward coverage Marker: integration

pytest packages/hop3-server/tests/b_integration/ -v

Layer 3: E2E Tests¶

Location: packages/hop3-server/tests/c_e2e/ Speed: 10-20 minutes Requirements: Docker (real deploy); no coverage Marker: e2e (+ needs_docker)

# Docker-only by default; the root conftest strips HOP3_DEV_HOST / HOP3_TEST_HOST
# (ADR 043), and --ssh-host is the only way to target a real box.
pytest packages/hop3-server/tests/c_e2e/ -v

Application Testing with hop3-test¶

Test Apps Directory¶

Test applications live under several apps/ directories (plus demos/):

apps/test-apps-procfile/   # Procfile-only fixtures (standalone test.toml)
apps/test-apps-nix/        # Nix fixtures
demos/                     # Demos (discovered via demo-script.py)
docs/tutorials/            # Tutorials (discovered by their validoc fences)

../hop3-catalog/apps/      # Real applications, in the sibling catalog checkout
  golden/ beta/            #   published, and scanned by default
  alpha/ broken/ retired/  #   kept as a record; reachable with --status

Test Configuration ([test] in hop3.toml)¶

Test configuration lives in the app's hop3.toml file under a [test] section. Most apps keep all their test config there, with no separate test.toml file.

Most fields are derived from the rest of hop3.toml: the test name from [metadata].id, category from [build].builder, required services from [[addons]], base healthcheck path from [healthcheck], and the need for an admin domain from [access] require = "auth": a gated app is refused at deploy on a server without one, so the runner asks the server (hop3-server admin:server-url) and leaves such a test out, with a warning, when it has none. The [test] section only declares what's test-framework-specific.

[metadata]
id = "flask-hello"

[build]
builder = "nix"

[healthcheck]
path = "/"

# Test-harness metadata
[test]
priority = "P0"                        # P0 (critical), P1, P2
tier = "fast"                          # fast | medium | slow | very-slow (report label only)
targets = ["docker", "remote"]
covers = ["python", "flask", "pip", "uwsgi"]

[[test.validations]]
path = "/"
status = 200
contains = "Hello"

Note: tier is only a report-grouping label. All builds share a single 30-minute budget (see config.md). The legacy [build].tier field no longer exists.

Standalone test.toml files still exist for Procfile-only test apps (apps/test-apps-procfile/*/), which don't pair with a hop3.toml. Two other kinds of test are configured differently:

• Negative-test cases are the recipes the catalog files as broken. They carry a normal hop3.toml, and the runner treats any deploy under broken/ as expected-to-fail (a failed deploy counts as PASS). An expects-failure = false in [test] opts one back out (a recipe dropped for business reasons that still deploys fine), and expects-failure = true opts any other app in.
• Demos (demos/*/) and tutorials (docs/tutorials/**/) are discovered structurally, without a test.toml: a demo by its demo-script.py, a tutorial by its bash exec/output/file markers.

Test Modes¶

When no apps are named, --mode filters the catalog by tier/priority (default: smoke):

The old names dev (→ smoke) and release (→ full) still work as aliases.

Test Targets¶

The run command picks a target via a flag (the DeploymentTarget ABC covers Docker, SSH, and cloud):

Test Output¶

Recap Summary¶

After tests complete, a recap shows what was tested:

============================================================
All 8 tests passed!
Total time: 148.55s
============================================================

Recap:
  ✓ deployment: 8/8 passed
  Tiers: fast=5, medium=3
  Covers: flask, go, golang, gunicorn, minimal, nginx, nodejs, pip, ...
  Avg time per test: 18.6s

Use -q/--quiet to suppress the recap.

Diagnostic Logs¶

On failure of a Docker e2e or app test, a diagnostic bundle is collected for the run. Replay it with the run id printed in the failure headline:

hop3-test why <run-id>            # Show the full bundle
hop3-test why <run-id> --list     # List available sections
hop3-test why <run-id> --section nginx   # Replay one section

Per-app logs are also written under test-logs/ (override with --logs-dir):

test-logs/
└── 20260110_155610/
    └── system-hop3-test-docker/
        ├── diagnostics.json
        ├── nginx-error.log
        ├── uwsgi.log
        └── hop3-server.log

Tests that read the catalog¶

The app recipes live in the separate hop3-catalog repository (ADR 059), so this repository's suites do not read it: a fresh checkout runs them with nothing beside it. A test that does read the catalog, through the catalog_apps or catalog_recipe fixture, is marked catalog and is deselected unless pytest is given --catalog=<checkout>; make test-catalog gives it. Write the option with =: as a separate argument, the path moves pytest's rootdir. A test that only needs a realistic recipe as input keeps a frozen copy beside it (tests/.../recipes/<app>.toml) instead.

Continuous Integration¶

CI runs on SourceHut (build manifests live in .builds/). A typical pipeline runs:

make lint             # Linting and type checking
make test             # Check tier: unit + integration, no Docker
make test-e2e         # The Docker e2e layer (c_e2e)

The wider app/demo/tutorial matrix runs nightly:

uv run hop3-test run --docker --mode nightly --report html

See: https://builds.sr.ht/~sfermigier/hop3/

Coverage¶

Coverage is measured on the in-process layers only (a_unit + b_integration). The Docker e2e layer runs out-of-process and contributes none.

# Coverage on the in-process layers
make test-cov

# Or directly (HTML report)
pytest --cov=hop3 --cov-report=html \
  packages/hop3-server/tests/a_unit packages/hop3-server/tests/b_integration

# Open report
open htmlcov/index.html

Cloud Testing¶

For E2E testing on real cloud infrastructure, run hop3-test run --provider hetzner. It requires the HETZNER_API_TOKEN environment variable (Hetzner provider).

# List available images
hop3-test run --list-images

# Test on a single distribution
hop3-test run --provider hetzner --image ubuntu-24.04

# Test across multiple distributions
hop3-test run --provider hetzner --images ubuntu-24.04,debian-13

# Choose which app directories to test (positional, like `run`)
hop3-test run --provider hetzner --status golden demos

# Use local code (the default; --from pypi installs from PyPI instead)
hop3-test run --provider hetzner --from local --images all

# Test against an already-provisioned server (no rebuild, no deploy)
hop3-test run --host <server> --reuse

Supported Images¶

• ubuntu-24.04: Ubuntu 24.04 LTS (default)
• debian-13: Debian 13 (trixie)
• debian-12: Debian 12 (bookworm)
• fedora-42: Fedora 42
• rocky-9: Rocky Linux 9
• alma-9: AlmaLinux 9

Troubleshooting¶

A Docker e2e or app test failed¶

Inspect the saved diagnostic bundle by run id (printed in the failure headline):

hop3-test why <run-id>

Tests Hang¶

• Check Docker daemon: docker ps
• Use verbose mode: -v -s
• Check container logs: docker logs hop3-app-test

Import Errors¶

uv sync

Docker Issues¶

# Clean up containers
docker rm -f hop3-app-test hop3-system-test

# Force a rebuild of the cached e2e image (the next c_e2e run rebuilds it)
docker rmi hop3-e2e:test

For detailed information, see Testing Strategy.