Administrator Manual¶

This guide covers server administration tasks for Hop3 operators, including installation, configuration, security, monitoring, and maintenance.

Server Requirements¶

Minimum Requirements¶

Supported Operating Systems¶

• Debian 12 (Bookworm) - Recommended
• Ubuntu 24.04 LTS, 26.04 LTS
• Rocky Linux 9
• NixOS (experimental — via the Nix flake / services.hop3 module)

On the distros above, hop3-server is installed by the standard installer; on NixOS it is deployed via the Nix flake and the services.hop3 NixOS module. (Separately, the Nix package manager is also available as an app builder on any supported host.)

Installation¶

Quick Install (Single Command)¶

curl -LsSf https://hop3.cloud/install-server.py | sudo python3 -

Manual Installation¶

# Download installer
curl -LsSf https://hop3.cloud/install-server.py -o install-server.py

# Review the script (recommended)
less install-server.py

# Run installation
sudo python3 install-server.py

Installation Options¶

# Install with optional services (mysql, redis, docker, or all)
sudo python3 install-server.py --with mysql,redis

# Install all optional features
sudo python3 install-server.py --with all

# Specify domain for Let's Encrypt certificate
sudo python3 install-server.py --domain hop3.example.com

See the Server Setup Guide for complete installation options.

Post-Installation Verification¶

# Check system health
hop3 system status

# View service status
systemctl status hop3-server
systemctl status nginx
systemctl status uwsgi-hop3

Upgrading Hop3¶

Upgrading the server (Hop3 itself) is done with the installer / deployer. Re-run the same tool you installed with:

# Production (single-file installer): re-run to pull the latest release
curl -LsSf https://hop3.cloud/install-server.py | sudo python3 -

# Development / self-hosted from source
hop3-deploy-server --host server.example.com            # git
hop3-deploy-server --host server.example.com --local    # local checkout

Each upgrade runs the same sequence: install the new code → hop3-server db:upgrade (migrations) → restart → verify the server answers.

What the upgrade guarantees. Migrations run between install and restart, and both ends are gated so a broken upgrade never leaves you silently down:

• A failed migration aborts before the restart — the old server keeps running on the old schema. Fix the migration and re-run.
• After the restart, the upgrade verifies the server actually answers. If the new code fails to start, the upgrade fails loudly (it does not report "complete") and prints the exact command to revert to the previous release. The schema has already migrated forward, so a revert may also need a pre-upgrade database restore.

Recovering from a bad upgrade. Follow the revert command the failed upgrade printed (it names the previous git commit or PyPI version). Then diagnose:

journalctl -u hop3-server -n 100 --no-pager     # why the new server didn't start
sudo -u hop3 /home/hop3/venv/bin/hop3-server db:current   # current schema revision

Because platform migrations are forward-only, if the previous code cannot run on the already-migrated schema, restore a database backup taken before the upgrade (see Backup & Restore).

Upgrading an application¶

Individual apps have their own safe-upgrade command that snapshots, redeploys, and rolls back automatically on failure — see Backup & Restore:

hop3 app upgrade --app myapp      # snapshot -> redeploy + migrate -> verify -> auto-rollback on failure
hop3 app rollback --app myapp     # restore the most recent backup (--to <backup-id> for a specific one)

Directory Structure¶

/home/hop3/                    # HOP3_ROOT
├── apps/                      # Application deployments
│   └── myapp/
│       ├── git/               # Bare git repository
│       ├── src/               # Checked-out source code
│       ├── data/              # Persistent data
│       ├── log/               # Log files
│       └── venv/              # Virtual environment (if applicable)
├── nginx/                     # Nginx configs and certs
├── uwsgi-available/           # uWSGI configs
├── uwsgi-enabled/             # Active uWSGI configs (symlinks)
├── backups/                   # Application backups
├── hop3-server.toml           # Server configuration
└── hop3.db                    # SQLite database

Configuration¶

Server Configuration¶

Location: /home/hop3/hop3-server.toml

The server reads top-level keys from this file (each key is also overridable by an environment variable of the same name). It is a flat key/value file — there are no [section] tables for these settings.

HOP3_SECRET_KEY = "your-secret-key-here"
HOP3_TOKEN_EXPIRY_HOURS = 24
HOP3_PROXY_TYPE = "nginx"            # nginx, caddy, or traefik
HOP3_LOG_LEVEL = "INFO"

# Addon superuser credentials
POSTGRES_SUPERUSER = "postgres"
POSTGRES_SUPERUSER_PASSWORD = "your-postgres-password"

MYSQL_SUPERUSER = "root"
MYSQL_SUPERUSER_PASSWORD = "your-mysql-password"

REDIS_HOST = "localhost"
REDIS_PORT = 6379

Nginx Configuration¶

Hop3 automatically manages Nginx configurations. Manual changes are not recommended.

Application configs: /home/hop3/nginx/<appname>.conf Main config: /etc/nginx/nginx.conf

To reload Nginx after manual changes:

sudo nginx -t && sudo systemctl reload nginx

User Management¶

Creating Admin Users¶

The recommended way to create an admin user is from your workstation:

hop3 init --ssh root@your-server.com

This connects via SSH, prompts for admin credentials, and saves your API token locally.

Alternative: Server-side creation

If you need to create users directly on the server:

ssh root@your-server.com
hop3-server admin:create admin admin@example.com
# Enter password when prompted

Then log into the server from your local CLI (the printed token is the API token):

hop3 auth login --token <paste-token-here> --url https://your-server.com

Generating API Tokens¶

# Generate a token for an existing user (run on server)
hop3-server admin:token admin

Listing Users¶

hop3 user list

Database Addon Management¶

Hop3 supports PostgreSQL, MySQL, and Redis as backing services. For complete addon command documentation, see the CLI Reference: Services (Addons).

Quick Reference¶

hop3 addon create postgres mydb        # Create PostgreSQL database
hop3 addon create mysql mydb           # Create MySQL database
hop3 addon create redis mycache        # Create Redis instance
hop3 addon attach mydb --app myapp     # Attach to app (sets DATABASE_URL)
hop3 addon show mydb                   # Get connection info
hop3 addon destroy mydb                # Delete (requires confirmation)

Server Configuration¶

Configure addon credentials as flat keys in /home/hop3/hop3-server.toml:

POSTGRES_SUPERUSER = "postgres"
POSTGRES_SUPERUSER_PASSWORD = "secure-password"

MYSQL_SUPERUSER = "root"
MYSQL_SUPERUSER_PASSWORD = "secure-password"

REDIS_HOST = "localhost"
REDIS_PORT = 6379

SSL/TLS Certificates¶

Automatic Certificates (Let's Encrypt)¶

Hop3 automatically provisions SSL certificates via Let's Encrypt when:

1. Application has HOST_NAME configured
2. Domain DNS points to server IP
3. Port 80 is accessible for ACME challenge

# Configure hostname
hop3 env set --app myapp HOST_NAME=myapp.example.com

# Redeploy to provision certificate
hop3 deploy --app myapp

Manual Certificate Installation¶

# Copy certificates to standard location
sudo cp fullchain.pem /etc/ssl/certs/myapp.example.com.crt
sudo cp privkey.pem /etc/ssl/private/myapp.example.com.key

# Set permissions
sudo chmod 644 /etc/ssl/certs/myapp.example.com.crt
sudo chmod 600 /etc/ssl/private/myapp.example.com.key

Certificate Renewal¶

Let's Encrypt certificates auto-renew via systemd timer:

# Check timer status
systemctl status certbot.timer

# Manual renewal test
sudo certbot renew --dry-run

Monitoring & Health Checks¶

System Health Check¶

# Comprehensive health report (services, addons, disk, certs)
hop3 system status

# One-line summary, suitable for scripting
hop3 system status --quiet

Checks performed: - Core services (hop3-server, nginx, uwsgi) - Database addons (PostgreSQL, MySQL, Redis) - Filesystem permissions - Disk space - SSL certificates

Application Status¶

# List all applications with status
hop3 app list

# Detailed app info
hop3 app status --app myapp

Log Monitoring¶

# View application logs
hop3 app logs --app myapp

# View last N lines
hop3 app logs --app myapp --lines 100

# Filter to lines matching a pattern
hop3 app logs --app myapp --grep error

# Only the logs since the last deployment
hop3 app logs --app myapp --since-deploy

# System-wide logs
hop3 system logs
hop3 system logs --lines 100

Process Monitoring¶

# Check uWSGI processes
systemctl status uwsgi-hop3

# View all app workers
ps aux | grep uwsgi

Backup & Restore¶

For application-level backups, see the Backup and Restore Guide.

Application Backups¶

hop3 backup create --app myapp   # Create app backup
hop3 backup list myapp           # List backups for an app
hop3 backup restore <id>         # Restore from backup

Full Server Backup¶

For disaster recovery, backup these directories:

Example server backup script:

#!/bin/bash
BACKUP_DIR="/backups/hop3/$(date +%Y-%m-%d)"
mkdir -p "$BACKUP_DIR"

tar -czf "$BACKUP_DIR/apps.tar.gz" /home/hop3/apps/
cp /home/hop3/hop3-server.toml /home/hop3/hop3.db "$BACKUP_DIR/"
sudo -u postgres pg_dumpall | gzip > "$BACKUP_DIR/postgres.sql.gz"

Disaster Recovery¶

1. Install Hop3 on new server
2. Restore configuration and database files
3. Restore application directories
4. Redeploy applications: hop3 deploy --app myapp

Security Hardening¶

Firewall Configuration¶

# Allow SSH
sudo ufw allow 22/tcp

# Allow HTTP/HTTPS
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

# Enable firewall
sudo ufw enable

Verifying a host key before the first deploy¶

hop3-deploy-server connects with StrictHostKeyChecking=accept-new, so the host key of an unknown server is accepted on first contact. That first connection is also the one that installs Hop3 as root, so on a host you did not just provision yourself, verify the key out-of-band first:

# On the target (via console, or your provider's control panel):
ssh-keyscan -t ed25519 localhost | ssh-keygen -lf -

# On your workstation, compare, then pin it before deploying:
ssh-keyscan -t ed25519 your-server.com >> ~/.ssh/known_hosts
ssh-keygen -lf ~/.ssh/known_hosts | grep your-server.com

Once the key is in known_hosts, a mismatch on any later connection is refused rather than accepted.

SSH Hardening¶

Edit /etc/ssh/sshd_config:

PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes

Reload SSH:

sudo systemctl reload sshd

Database Security¶

PostgreSQL - Edit /etc/postgresql/*/main/pg_hba.conf:

# Local connections
local   all   all                 peer
# Docker bridge network
host    all   all   172.16.0.0/12   md5
# Docker Compose networks
host    all   all   192.168.0.0/16  md5

MySQL - Ensure strong root password:

sudo mysql_secure_installation

Application Isolation¶

Each application runs: - In isolated directory (/home/hop3/apps/<name>/) - With dedicated virtual environment - Under the hop3 user - With separate uWSGI worker processes

Performance Tuning¶

uWSGI Workers¶

Edit application's uWSGI config or set via environment:

hop3 env set --app myapp UWSGI_WORKERS=4
hop3 env set --app myapp UWSGI_THREADS=2

Nginx Optimization¶

# /etc/nginx/nginx.conf
worker_processes auto;
worker_connections 1024;

# Enable gzip
gzip on;
gzip_types text/plain application/json application/javascript text/css;

# Enable caching
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=cache:10m;

Database Tuning¶

PostgreSQL - Edit /etc/postgresql/*/main/postgresql.conf:

shared_buffers = 256MB
effective_cache_size = 1GB
maintenance_work_mem = 64MB

Docker Network Configuration¶

When running many Docker-based applications (20+), you may encounter the error:

failed to create network: all predefined address pools have been fully subnetted

Cause: Docker's default configuration allocates /16 subnets for bridge networks, limiting you to approximately 16 networks from the default 172.17.0.0/12 pool.

Solution: Configure Docker to use smaller /24 subnets, allowing up to 4096 networks:

# Create or edit /etc/docker/daemon.json
sudo tee /etc/docker/daemon.json << EOF
{
  "default-address-pools": [
    {"base": "172.17.0.0/12", "size": 24}
  ]
}
EOF

# Restart Docker to apply changes
sudo systemctl restart docker

Note: Restarting Docker will stop all running containers. They will need to be restarted or redeployed.

Important: After changing Docker's network configuration, you must also update PostgreSQL to listen on the new network range:

# Update PostgreSQL to listen on all interfaces
sudo sed -i "s/listen_addresses = .*/listen_addresses = '*'/" /etc/postgresql/*/main/postgresql.conf

# Update pg_hba.conf to allow connections from Docker networks
# Add this line if not present:
echo "host    all    all    172.16.0.0/12    scram-sha-256" | sudo tee -a /etc/postgresql/*/main/pg_hba.conf

# Restart PostgreSQL
sudo systemctl restart postgresql

Verification:

# Check current networks
docker network ls

# After deploying many apps, verify no pool exhaustion
docker network create test-network && docker network rm test-network

Troubleshooting¶

Common Issues¶

Application Won't Start¶

# Check application logs
hop3 app logs --app myapp --lines 50

# Check uWSGI status
systemctl status uwsgi-hop3

# Verify environment variables
hop3 env show --app myapp

502 Bad Gateway¶

1. Check if application is running: hop3 app list
2. Check application logs: hop3 app logs --app myapp
3. Verify Nginx config: sudo nginx -t
4. Check uWSGI socket: ls -la /tmp/uwsgi-*.sock

Database Connection Failed¶

# Verify addon is attached
hop3 addon list

# Check DATABASE_URL is set
hop3 env show --app myapp | grep DATABASE

# Re-check server health (services and addons)
hop3 system status

SSL Certificate Issues¶

# Check certificate status
sudo certbot certificates

# Force renewal
sudo certbot renew --force-renewal

# Check Nginx SSL config
sudo nginx -t

Getting Help¶

# Built-in help
hop3 --help
hop3 <command> --help

# System information
hop3 system info

# Health report with diagnostics
hop3 system status

Maintenance Tasks¶

Regular Maintenance Checklist¶

Daily: - Monitor disk space: df -h - Check server health: hop3 system status

Weekly: - Review logs for errors - Verify backups are running - Check certificate expiry: sudo certbot certificates

Monthly: - Update system packages: sudo apt update && sudo apt upgrade - Review and rotate logs - Test backup restoration - Review security logs

Updating Hop3¶

# Check the running server version
hop3 system info

# Update Hop3 server by re-running the installer (idempotent: it preserves
# operator config and secrets, and restarts the services)
curl -LsSf https://hop3.cloud/install-server.py | sudo python3 -

Log Rotation¶

Hop3 logs are managed by systemd journal. Configure retention:

# Edit /etc/systemd/journald.conf
SystemMaxUse=500M
MaxRetentionSec=30day

Apply changes:

sudo systemctl restart systemd-journald

Reference¶

Service Management¶

Important File Locations¶

Environment Variables¶

Related Guides¶

• Server Setup - Initial server installation
• User Guide - Core concepts and daily operations
• Backup and Restore - Data protection and recovery
• Troubleshooting - Diagnose and fix common issues
• CLI Reference - Complete command documentation