Hop3 Testing Cheat Sheet¶

Updated by ADR 043. The pytest pyramid is now three layers — a_unit (fast, no Docker) · b_integration (in-process, real in-memory DB, no Docker) · c_e2e (Docker/real-deploy, renamed from d_e2e). The old c_system layer is dissolved. A test's layer is decided by whether it needs Docker/root/host-mutation, not by complexity; coverage is measured on a_unit + b_integration only (e2e runs out-of-process). Markers (fast/integration/e2e/needs_docker) are stamped from the directory layer (root conftest.py), so pytest -m fast / -m "not needs_docker" work everywhere.

Quick reference for developers running tests.

Quick Commands¶

What	Command
Fast unit tests (< 1 min)	`make test-fast`
Check tier (unit + integration, no Docker)	`make test`
Docker e2e (real deploys, backups, git-push)	`make test-e2e`
App tests (Docker)	`make test-apps`
Lint & type check	`make lint`
System tests (Docker)	`hop3-test system --docker`
Hetzner Cloud test	`hop3-test hetzner --image ubuntu-24.04`
Multi-distro test	`hop3-test multi-distro`

hop3-test CLI¶

The unified test runner for Hop3 deployment testing.

System Testing (Testing Hop3 Itself)¶

Deploys Hop3 using hop3-deploy, then runs tests against it.

# Deploy local code to Docker and test
hop3-test system --docker

# Deploy from git branch
hop3-test system --docker --deploy-from git --branch main

# Deploy from PyPI
hop3-test system --docker --deploy-from pypi

# Clean install (remove existing)
hop3-test system --docker --clean

# Reuse existing deployment (skip deploy)
hop3-test system --docker --reuse
# Or equivalently:
hop3-test system --docker --deploy-from none

# Remote server via SSH
hop3-test system --ssh --host server.example.com

# SSH using HOP3_TEST_HOST env var
export HOP3_TEST_HOST=server.example.com
hop3-test system --ssh

# Test mode: dev (fast) or ci (more thorough)
hop3-test system --docker --mode ci

# Keep target running after tests
hop3-test system --docker --keep

# Generate HTML report
hop3-test system --docker --report html

Deploy-from Options¶

Option	Description
`--deploy-from local`	Upload and install local code (default)
`--deploy-from git`	Clone and install from git branch
`--deploy-from pypi`	Install from PyPI
`--deploy-from none`	Skip deployment, use existing
`--reuse`	Alias for `--deploy-from none`

App Testing (Testing Apps, Not Hop3)¶

Uses a pre-built Docker image with Hop3 already installed.

# First, build the ready image (one-time)
hop3-test build-ready-image

# Test all apps
hop3-test apps

# Test specific app
hop3-test apps 010-flask-pip-wsgi

# Test by category
hop3-test apps --category python

# Keep apps deployed after testing
hop3-test apps --keep

# Against remote server
hop3-test apps --target remote --host server.example.com

Listing and Inspecting Tests¶

# List all tests
hop3-test list

# Filter by category
hop3-test list --category deployment

# Filter by tier
hop3-test list --tier fast

# Show test details
hop3-test show 010-flask-pip-wsgi

Package Validation¶

For package authors testing their apps against stable Hop3.

# Validate an app package
hop3-test package /path/to/my-app

Building Docker Images¶

# Build ready image (pre-installed Hop3)
hop3-test build-ready-image
hop3-test build-ready-image --tag my-hop3:v1
hop3-test build-ready-image --no-cache

# Build test image (for system tests)
hop3-test build-test-image
hop3-test build-test-image --no-cache

Quick Modes¶

# Developer mode (fast P0 tests only)
hop3-test dev

# CI mode (fast + medium P0 tests)
hop3-test ci

# Nightly mode (all tiers, all priorities)
hop3-test nightly

Hetzner Cloud Testing¶

Run E2E tests on real Hetzner Cloud infrastructure. Requires HETZNER_API_TOKEN.

# List available images
hop3-test multi-distro --list-images

# Single distribution test
hop3-test hetzner --image ubuntu-24.04 --suites test-apps

# Multi-distribution test
hop3-test multi-distro --images ubuntu-24.04 debian-13 fedora-42

# Use local code instead of git
hop3-test hetzner --use-local-repo

# Skip phases for debugging
hop3-test hetzner --skip-reset    # Keep existing server state
hop3-test hetzner --skip-deploy   # Use existing Hop3 installation

Pytest Tests¶

Run by Layer¶

There are three layers under each package's tests/. A test's layer is decided by what it needs (Docker / root / host-mutation), not by complexity; duplication across layers is allowed.

# Unit tests — no Docker, counts toward coverage (tier: fast)
uv run pytest packages/hop3-server/tests/a_unit

# Integration tests — in-process, real in-memory DB, no Docker, counts toward
# coverage (tier: check)
uv run pytest packages/hop3-server/tests/b_integration

# E2E tests — real Docker deploy, NO coverage (check (Docker) + nightly)
uv run pytest packages/hop3-server/tests/c_e2e

# CLI tests
uv run pytest packages/hop3-cli/tests

Markers are stamped from the directory by the root conftest.py, so you can select layers anywhere:

uv run pytest -m fast                  # a_unit + flat unit suites
uv run pytest -m "not needs_docker"    # everything except the Docker e2e layer
uv run pytest packages/hop3-server/tests/c_e2e   # the Docker e2e layer

Run Specific Tests¶

# Single file
uv run pytest packages/hop3-server/tests/a_unit/test_app_config.py

# Single test
uv run pytest packages/hop3-server/tests/a_unit/test_app_config.py::test_function_name

# By keyword
uv run pytest -k "backup" packages/hop3-server/tests

# By marker (fast / integration / e2e / needs_docker)
uv run pytest -m "not needs_docker" packages/hop3-server/tests

Useful Flags¶

# Verbose output
uv run pytest -v

# Stop on first failure
uv run pytest -x

# Show print statements
uv run pytest -s

# Parallel execution (faster)
uv run pytest -n 4

# Show slowest tests
uv run pytest --durations=10

# Coverage report
uv run pytest --cov=hop3 --cov-report=term-missing

Common Workflows¶

Before Committing¶

make lint       # Check formatting and types
make test-fast  # Fast unit tests (the inner loop, < 1 min)
make test       # Check tier: unit + integration, all packages, no Docker

Quick Validation (Developer)¶

# Fast tests against Docker
hop3-test system --docker --mode dev

Full Validation¶

# Check tier (in-process, no Docker) plus the Docker e2e layer
make test
make test-e2e

# Deploy the real app catalog on Docker
make test-apps

# Or run the system suite manually
hop3-test system --docker --mode ci --report html

Debug a Failing Test¶

# Run with verbose output
uv run pytest -v -s path/to/test.py::test_name

# Keep target running for inspection
hop3-test apps --keep 010-flask-pip-wsgi

# Run system tests and keep target
hop3-test system --docker --keep

# Reuse container for fast iteration
hop3-test system --docker --reuse --keep

# Generate HTML report for analysis
hop3-test system --docker --report html

Test Coverage¶

make test-with-coverage

# HTML report
uv run pytest --cov=hop3 --cov-report=html
open htmlcov/index.html

Test Directory Structure¶

packages/hop3-server/tests/
├── a_unit/          # Unit; no Docker; counts toward coverage (tier: fast)
├── b_integration/   # In-process, real in-memory DB; no Docker; coverage (tier: check)
└── c_e2e/           # End-to-end; real Docker deploy; no coverage (check (Docker) + nightly)

packages/hop3-testing/    # Test framework
├── src/hop3_testing/
│   ├── catalog/         # Test catalog (reads [test] section from hop3.toml)
│   ├── cli/             # CLI commands
│   ├── runners/         # Test runners
│   ├── results/         # Result storage and reporting
│   ├── selector/        # Test selection logic
│   └── targets/         # Deployment targets

apps/test-apps/          # Test applications
├── 000-static/
├── 010-flask-pip-wsgi/
├── 020-nodejs-express/
├── 030-golang-gin/
├── 040-sinatra/
├── 100-flask-gunicorn-pip/
├── 110-flask-gunicorn-poetry/
└── 130-golang-minimal/

Test Configuration (`[test]` in hop3.toml)¶

Tests are configured via a [test] section in the app's hop3.toml. Most fields are derived from other sections (metadata, build, addons, healthcheck); the [test] block only holds the test-framework-specific bits:

[test]
priority = "P0"                    # P0 | P1 | P2
tier = "fast"                      # report label only — no longer drives timeouts
targets = ["docker", "remote"]
covers = ["python", "flask", "pip", "uwsgi"]

[[test.validations]]
path = "/"
status = 200

Legacy standalone test.toml files are still used by procfile-only test apps (apps/test-apps-procfile/*/), negative-test cases, demos, and tutorials — anywhere there's no sibling hop3.toml. See config.md for the full reference.

Environment Variables¶

Variable	Purpose
`HOP3_DEV_HOST`	SSH target for deployment
`HOP3_TEST_HOST`	SSH target for `--ssh` without `--host`
`HOP3_TEST_SSH_KEY`	SSH key for remote tests
`HOP3_UNSAFE=true`	Disable auth in Docker tests
`HETZNER_API_TOKEN`	Hetzner Cloud API token (for hetzner/multi-distro)

Troubleshooting¶

Docker Tests Fail¶

# Check if container is running
docker ps -a | grep hop3

# View container logs
docker logs hop3-test

# Rebuild test image
hop3-test build-test-image --no-cache

App Tests Fail (Ready Image)¶

# Rebuild ready image
hop3-test build-ready-image --no-cache

# Test with verbose output
hop3-test apps -v 010-flask-pip-wsgi

# Check HTML report
hop3-test apps --report html

System Tests Timeout¶

# Reuse existing container to debug
hop3-test system --docker --reuse --keep

# Inspect the diagnostic bundle collected on a failed Docker e2e/app run
hop3-test why <run-id>

Remote Tests Fail¶

# Verify SSH connection
ssh hop3@$HOP3_TEST_HOST "hop3 --version"

# Check server status
ssh root@$HOP3_TEST_HOST "systemctl status hop3-server"

Target Types¶

Target	Use Case	Speed
`--docker`	System tests with fresh deploy	Slow (~5 min startup)
`ready`	App tests with pre-built image	Fast (~30s startup)
`--ssh`	Tests against real servers	Variable

When to Use Each¶

hop3-test system --docker: Testing Hop3 changes (deploys Hop3 first)
hop3-test system --docker --reuse: Fast iteration on existing container
hop3-test system --ssh --host X: Testing against remote servers
hop3-test apps: Testing app configurations (uses pre-built image)