Hop3 0.8 release notes¶

This page holds the full notes for each 0.8 release, newest first. The changelog gives each release's date and a summary.

0.8.1¶

Fixed¶

A deploy over HTTPS accepts an archive up to 200 MB. The admin domain's nginx vhost set no client_max_body_size, so nginx refused any archive over the 1 MB default with a 413, before it reached hop3-server, which accepts 200 MB. The CLI then reported a limit of 200 MB. The installer now sets 200 MB in /etc/nginx/conf.d/hop3-http.conf, a file each update reconciles. When nginx refuses an archive, the CLI now names nginx as the cause. Deploys through an SSH tunnel never passed through nginx.
hop3 app restart applies a changed environment. It touched the app's uWSGI configuration, so the app came back with the values of the last deploy, while hop3 env set promised that a restart would apply the new ones. A restart is now a stop and a fresh start from the current settings. A Docker app picks up a changed environment only at a deploy; env set now tells you so for one.
Upgrading with the install one-liner migrates the database. Only hop3-deploy-server ran hop3-server db:upgrade. The one-liner restarted hop3-server on the new code against the old schema. The installer now migrates before it restarts the server and aborts when the migration fails; --skip-migrations skips the step. A server upgraded to 0.8.0 with the one-liner is migrated by the next install.
hop3 backup restore points at the restored app's status. It told you to start the app with hop3 restart <app>, a spelling 0.8 retired, although the restore had already redeployed it.
A failed deploy under --json prints the error. The CLI exited before it wrote the JSON it had collected, so stdout was empty.
The installer's setup step aborts the install when it fails. The installer ignored the setup command's exit code.
The deployer and the server backup and restore scripts no longer call sudo. They run commands as hop3 or postgres with su or runuser, as the installer does.
The dashboard's live logs stream. The stream opened the app's log directory as a file and failed on every connection, while the page showed "Live". It now tails each worker's log file. Streamed lines are shown as text: they were inserted as HTML, so a log line an app wrote from a client's request could run script in the operator's signed-in session. The log search and level filter now work. Pause keeps new lines back until you resume.
The dashboard fixes the WCAG 2.2 AA failures an accessibility audit found. The menu opens on a phone or a zoomed screen, where the navigation and Sign out were unreachable. Every field and icon button has a name; text, field borders and app icons have enough contrast; status changes and copy results are announced; certificate expiry is no longer shown by colour only; breadcrumbs use the navigation's names; the mobile category picker no longer navigates on selection. The help text shows the current commands (hop3 env set --app, hop3 deploy --app), and the resource-tier legend shows the tiers' actual names and limits.
HOP3_UNSAFE in hop3-server.toml meets the same safety check as the environment variable. The check that requires HOP3_UNSAFE_ACK and forces the bypass off in production read only the environment, while the authentication guards also read the file: HOP3_UNSAFE = true there disabled authentication with neither safeguard. A value that is not a valid boolean (yes, on) is now refused at startup, where the check accepted it and the guards rejected it.
The installer and the deployer no longer act as root on files the hop3 user controls. The installer read and rewrote hop3's .bashrc and ~/.config/nix/nix.conf as root, following symlinks, so an app (which runs as hop3) could point that file at a system file and have root rewrite it and give it to hop3 on the next install. Those files are now read and written as hop3. hop3-deploy-server ran hop3's own venv Python as root to reload nginx and read versions. It also ran the venv's pip, and git in hop3's checkout, as root on every update. All of them now run as hop3, which hop3-rootd admits for the reload. The server backup and restore scripts no longer run the venv's Python as root.
hop3-rootd runs from a venv root owns. The daemon runs as root but ran from hop3's venv, which hop3, and so any app, can write: code put there ran as root at the next restart. It now lives in /opt/hop3-rootd, owned by root and made from a root-owned interpreter, and is installed there as root from PyPI, git, or an uploaded tree the installer first checks only root can change. The next install or hop3-deploy-server update moves an existing server over: it creates the venv, installs the daemon, points the systemd unit (or the supervisor program) at it and restarts it, and removes the copy left in hop3's venv. The installer also edits hop3-server.toml, the pip configuration and the build record as hop3.
A dashboard change that was not saved says so. A handler that answered without committing reported success while its database changes were rolled back after the response. The server now returns an error naming the problem. A handler that fails after writing rolls back cleanly. The Test Lab does the same.
Errors keep their meaning across the RPC boundary. The server re-wrapped every command failure as an invalid-parameter error (-32602); a full disk or a failed subprocess now arrives as an internal error (-32603) with its Diagnosis. The CLI exits 1 for these, refined by the message (an unknown app exits 3), where it used to exit 2 or 7.
hop3 addon destroy no longer reports success when the database server is down. It reported that the addon did not exist and left the database. It now aborts and destroys nothing. addon detach is refused when the addon's credentials cannot be read, instead of deleting the credential and leaving the variables behind, and a deploy aborts when an attached addon's credentials cannot be decrypted, instead of starting the app without them.
Two PostgreSQL addons can be created at the same time. Hop3's admin connections used template1, the database CREATE DATABASE copies, which PostgreSQL refuses to copy while any other session is connected to it; two apps provisioning a database at once (parallel builds) failed at random with source database "template1" is being accessed by other users. Admin connections now use the postgres maintenance database.
Two addons can no longer share a database or a user. PostgreSQL maps - to _ and MySQL cuts user names at 32 characters, so a-b and a_b, or two long MySQL names, landed on one database or one user, and creating the second took over the first. A new name that would collide is refused. PostgreSQL also refuses to adopt a leftover database that contains data, as MySQL does. Existing addons are not renamed.
Destroy reports what it could not remove. A Docker daemon that did not answer, a firewall rule, WAF rules or a cgroup left behind were reported as a clean teardown. hop3 app destroy now lists each leftover and exits non-zero.
Destroying an app no longer removes another app's containers. The orphan cleanup that runs on every destroy matched containers by name, a filter Docker applies to any part of a name: destroying app removed myapp-web-1. It now matches the app's Compose project exactly.
Destroy removes read-only directories. Go makes the module cache, which a build puts in the app's HOME, read-only, and destroy left it behind with Permission denied. It now makes each directory writable before emptying it.
A refused or unanswered deploy no longer marks a running app failed. A deploy refused because the build queue was full, or a status check that could not read the app's artifact, set the app to FAILED while it kept serving. The reason is now recorded and the state kept; a start that times out still fails, with the reason.
An open log page no longer occupies a database connection. Each live log stream kept one for as long as the page stayed open, so about fifteen open pages blocked every other request, sign-in included.
An unreadable build artifact is an error. A damaged BUILD_ARTIFACT.json was read as "never built". The app then ran with the platform's defaults instead of what the build declared. It is now refused with the file's name and a hint to redeploy.
hop3-test fails a test whose teardown left anything behind. The leftover was printed and the test reported PASS, though it blocks the next deploy of the name.
A backup restore deploys the whole app. Restore deployed without a database session, so it silently skipped addon provisioning, the generated admin account and the smoke test.
Apps with more than one language build. A Django, Rails or Laravel app with a package.json built fully and then found no deployer. The toolchains now produce one artifact: the toolchain that serves the app runs it, and the others contribute their build steps and environment.
An app with both a Dockerfile and a hop3.nix must name its builder. It built with Docker without saying so. The deploy now aborts and asks for [build].builder; the local builder still yields to Docker or Nix when they apply. An unknown [build].toolchain is an error listing the valid names. A build with uv.lock aborts instead of falling back to pip when uv is unavailable.
The installer provisions uv 0.13.4 for the hop3 user in every install mode, so a build with uv.lock no longer downloads uv at build time.
The install one-liner runs on Python 3.10 again. It used Python 3.11 features and crashed on Ubuntu 22.04's python3.
The installer aborts on a command that times out. A hung command was treated as a success. A redeploy reuses the server's secrets unless their files are absent; a permission error now aborts instead of generating new ones. Nix's configuration and daemon are reconciled on every install, the deployer reloads nginx through hop3-rootd's verified reload, and hop3-rootd's health report shows which startup reconciles were skipped or failed.
The dashboard no longer freezes the server. Opening a backup checksummed the archive on the server's only event loop. Every sign-in ran bcrypt there too. Database, file and subprocess work now runs in a thread. The application status poll no longer writes to the database, the dashboard refuses app names the CLI refuses, a catalog install that is refused shows the reason on the app's page, links can no longer put text of their choice on a dashboard page, and the backups page pages through every backup instead of showing the first 100.
A busy server says so. A long command (a backup, a restore) used to hold every other RPC, sign-in included, without a timeout. Waiting callers now get "server busy" naming the running command after 30 seconds; the CLI exits 6.
The CLI refuses what it cannot parse. A malformed hop3.toml is an error naming the file, where it used to be read as empty (dropping [build].ignore). Unknown flags to login and init are usage errors, and -d and -y are now passed to them. hop3 use takes --app and exits 2 on a usage error.
Nginx response caching works. The cache configuration failed to render on every server and the failure was logged and ignored, so caching never took effect. An invalid cache setting now fails the deploy with the value it rejected.
External plugins load as documented. The plugin guide named the entry-point group hop3.plugins while the server loads hop3; a plugin following it never loaded. The guide is fixed, a distribution that declares the old group stops the server with its name, and the guide no longer recommends an import that registered a plugin twice.
hop3 login trusts a certificate you pinned with hop3 settings set ssl_cert. Its check of the server's HTTPS address used only the system certificates, so a server with its own certificate kept the SSH tunnel at every login, and a login with --url refused it while telling you to pin the certificate and retry. Every later command already honoured the pin. A --url the CLI cannot connect to now fails the login instead of being saved with a warning.
Destroying a Docker Compose app removes its volumes and networks, or says it could not. When compose down failed or timed out, the app's volumes stayed on disk while destroy reported a clean teardown.
addon destroy finishes a create that failed halfway. A PostgreSQL role or MySQL user left without its database counted as no addon, so destroy reported nothing to do and left it. MySQL's destroy also reports a user it could not drop instead of ignoring the error.
An upgrade that fails partway leaves hop3-rootd able to restart. The installer removed the old daemon from hop3's venv before pointing its unit at /opt/hop3-rootd, so a failure in between left the unit naming a deleted binary. The old copy now goes once the new daemon is up. A daemon that cannot create its initial state now aborts the install, where the installer used to create one itself.
The installer checks that nginx took a reload on hosts without systemd, where a rejected configuration used to be reported as reloaded.
A deploy aborts when its hostname would serve nothing. An app with a hostname but no web or static worker was warned about, then published a 502.
The backup script reports hop3-rootd's version and gives a rollback that works with the daemon in /opt/hop3-rootd, and accesses hop3's database as hop3, not as root.
hop3 login refuses a token in the URL wherever it appears on the line, never printing it back, and the local commands refuse a flag given twice or a flag taken as another's value.
An oversized deploy refused by nginx gets advice that fits: update the server when the archive is under its 200 MB limit, shrink it otherwise.
The dashboard's log view reads at most 256 KiB of a new or rotated log file, and says how much it skipped.
The server install aborts when it can't install the hop3 client. A missing client source was only a warning: the server came up without the client that hop3 login relies on. A pre-release install now takes the pre-release client, as it does the server and hop3-rootd.
The CLI installer checks for venv and ensurepip before creating its environment. The bundled one-liner always passed the check, so a Debian system without python3-venv failed later, with Python's own error.

Changed¶

hop3 login --token reads the token from stdin (--token -), a file (--token @path) or a hidden prompt (--token). A token on the command line lands in shell history and in ps. The literal form still works, with a warning, and is no longer documented. hop3-tui --token, the admin commands and hop3-deploy-server accept the same forms.
The JSON-RPC response includes a Hop3-RPC-Protocol header. A client that sends params.protocol with another version is refused with an error saying which side to upgrade.
Apps may be named admin, api, app, apps, dashboard, hop3, static or www. The catalog install and the dashboard refused these names while hop3 app create, hop3 deploy and git push accepted them. Nothing in the platform collides with an app of one of these names (vhosts, files, hostnames and routes are all per app), so the list is gone and every path applies the same name rule.

0.8.0¶

Highlights¶

Accounts have roles. [access] puts an app behind a Hop3 sign-in (ADR 061, ADR 062).
An app keeps its state across a redeploy. Volumes are linked after the build and keep what the app wrote before declaring them; a scratch volume is for what must not be kept; each app has a HOME and TMPDIR of its own. All recipes in the published catalog pass a deploy and a redeploy, installed from the published catalog.
App code can no longer reach root through hop3-rootd. An app's secrets are also off the command line, where other users could read them.
Caddy and Traefik serve apps, PHP apps included where the proxy can.
A Nix-built PHP app runs under php-fpm, as a native one does since 0.7.5. A generated Nix app runs its setup once per deploy, before it starts.
The catalog has a versioned address. Each entry names the server it needs.
The installer aborts on a failure with the command's output, where it used to print a warning and report success.
Each checkout names the server it deploys to. Commands print where they go. A context is a server, and hop3 context use binds a checkout to one; a production server accepts changes only when named on the command line (ADR 042).

Upgrading from 0.7¶

Redeploy each native app once. Two fixes take effect only at the next deploy: an app's environment moves off the command line; a PHP app's hidden files are no longer served. hop3 app restart reuses the old configuration.
Accounts get roles. Admins stay admins and the other accounts become deployers, which is what they could already do. A new account is a member until an admin grants it more. A command above an account's role is refused with exit code 5.
The catalog moves to https://apps.hop3.cloud/catalog/v1/catalog.tar.gz. Move a CATALOG_SOURCE_URL set to the old path. Servers before 0.8 keep fetching the old path, which serves the same catalog, and are refused the entries that need 0.8, by name, before anything is created.
On a Docker install, the uWSGI emperor is uwsgi-hop3 under supervisor: supervisorctl restart uwsgi-hop3.
Generated Nix recipes build against nixos-26.05. Dependency hashes recorded against the old pin may need recording again; all node-deps-hash values do. A go-frontend-source-root that includes the archive's top-level directory is refused.
Some commands take different arguments: hop3 cert renew --app <app> --force-renewal, hop3 nix eject --app <app> > hop3.nix, and a command missing an argument now exits 2.
Take a new backup of each app with an S3 addon. An S3 backup taken before 0.8 contains no objects. Restoring one is now refused, where it used to report success and restore nothing.
Remove the hop3-harness and tutorial-runner accounts from any server hop3-test ran against: earlier versions created them, with their usernames as passwords.
Bind each project checkout to a context once: hop3 context use <name>. Inside a directory with a hop3.toml, a command that contacts a server now uses the checkout's binding, -c or HOP3_CONTEXT, and is refused without one; the global current context applies only outside a project. After a plain hop3 login, the context is named default. Move any server or bootstrap key out of a hop3.toml [contexts.*] block into hop3 context add <name> --server <address>: the block now contains only app, domains and env, and both the CLI and the server refuse the old keys. On a production server, set PROTECTED = true in hop3-server.toml.
Apps still share one Unix user. One app can read another's data, and the control plane's, until per-app users land in 0.9.

Security¶

An app can no longer call hop3-rootd. Native apps all run as hop3, the uid rootd admits, so any app's code could ask the privileged helper for anything: root, with no exploit. rootd now also refuses a caller running inside the app runtime: the uwsgi-hop3 service's cgroup or hop3.slice under systemd, and descendants of the uWSGI Emperor on a host without systemd (weaker: a process that double-forks escapes it). Apps still share one user, so one app can read another's data until per-app users land (planned for 0.9).
An app's environment is no longer visible to other users on the server. Background workers, php-fpm pools and apps started with a start command received their [env] as export statements on a command line that stayed visible to ps for as long as the app ran. Any local account could read the secrets of all apps that way, including the web server's and the database's. The environment now goes to a file readable only by the app's owner. The fix applies from the app's next deploy; hop3 app restart reuses the old configuration, so redeploy each native app, or stop and start it.
A PHP app's hidden files are no longer served. nginx served any file under a PHP app's document root with a name starting with a dot, such as .env or .htaccess, apart from .git. nginx and Caddy now answer them with 404, except /.well-known/. The fix applies from the app's next deploy.
Database passwords no longer appear on a command line. The installer set the PostgreSQL superuser password with psql -c "ALTER USER ... PASSWORD '...'" and created the MySQL hop3 user with mysql -e, both readable by any local user in /proc/<pid>/cmdline while they ran; the SQL now goes on stdin.
Scaling a process type down removes that worker's copy of the app's secrets. Each worker's environment is a 0600 file; a removed worker's file stayed on disk, stale after any later rotation.
A gated app's nginx include file cannot forward around the gate. The gate replaces the client's X-Hop3-User, X-Hop3-Role and Cookie in the locations Hop3 generates; a location in NGINX_INCLUDE_FILE with a proxy_pass of its own inherited none of that and forwarded whatever identity the client sent. Such an include is now refused on an app with [access].
A server installs no dependency released in the last seven days. A compromised package is usually caught and yanked within days. An install resolving the latest version would take it the hour it appears. The installer puts the cutoff in the server venv's pip.conf, so upgrades and developer deploys resolve under it too. Hop3's packages are exempt, so a release can be installed the day it ships. Builds of apps are unaffected.
hop3 and hop3-tui refuse a malformed API token without printing it. A token containing a line break (two tokens read from one file) made the HTTP library reject the Authorization header with an error that quoted it, and the client printed that error, token included.
The test harness no longer creates admin accounts. It created hop3-harness and tutorial-runner with their usernames as passwords, published in the repository, and on a --host server they outlived the run. It now obtains a token from admin:ssh-token, as hop3 login --ssh does. Servers the harness ran against should have those two accounts removed.
The test harness no longer replaces a server's mail setup. It switched any server to the catch backend for apps that declare email and left it there; it now does so only when no backend, or catch, is configured, and refuses otherwise.
The tutorial runner no longer puts the admin token on a command line. On a remote target it passed HOP3_API_TOKEN=<token> in the shell line, readable by any local user while validoc ran; the token now stays in a 0600 file on the server, shared with the harness's other admin steps.

Accounts and access¶

Accounts have roles: admin, deployer, viewer and member. hop3 user set-role alice viewer grants one; hop3 user add --role sets it at creation. A viewer reads apps, addons, backups and certificates but no secret. A deployer deploys and operates apps and addons. An admin also manages accounts and the server. A member signs in to the web interface and sees a profile page. A deployer's builds run with the control plane's privileges, so treat that role as an administrator credential. (ADR 061)
The server checks the account's role on each command and dashboard page. Until now any account could run everything but user management. A command above the account's role is refused with the role it needs and the hop3 user set-role that would grant it, and hop3 exits with code 5. The dashboard refuses a page the same way and sends a member to the profile page.
A new account is a member until an admin grants it more. Upgrading keeps admins as admins and makes the other accounts deployers, which is what they could already do. A member gets no API token.
[access] puts an app behind a Hop3 sign-in. With require = "auth", a visitor without a session is sent to the sign-in on the admin domain and back to the page they asked for. role sets the lowest role admitted; public-paths are open to anyone, for webhooks and health checks. The app learns who is calling from X-Hop3-User and X-Hop3-Role, which a client cannot forge, and never receives the gate's cookie. A sign-in link works only in the browser that started it, so no one can sign a visitor in to their own account. One sign-in covers all gated apps, and signing out of Hop3 signs out of all of them. The gate works on nginx, Caddy and Traefik, but the installer routes the admin domain, where the sign-in lives, on nginx only, so a server installed with Caddy or Traefik cannot use it yet. (ADR 062)
A token for a deleted or disabled account is rejected on its next call. hop3 and hop3-tui now show the server's reason, where they used to print only "authentication required" or an HTTP status.
hop3 login --token checks the token it was given. The check sent whatever the CLI already had stored for that server, or HOP3_API_TOKEN, so a new token for a server with an expired old one was refused with "Authentication required" without ever being sent. All hop3-deploy-server --docker --clean runs hit this, since the new container signs with a new key.
A 401 names the case in the error data (error.data.kind: unauthenticated, account_unusable). The CLI and TUI read that, where they used to read the message's wording.

Apps and deploys¶

[build].build runs once. The toolchain runs it in place of the toolchain's install. The deploy then ran it a second time. A Docker or Nix app declaring it is refused, since those builders run no toolchain; before-build and after-build run for all builders.
[[volumes]] has a scratch type, for directories an app writes that must not be kept: a framework's compiled templates or bootstrap cache. It lives outside src/, is reset to the build's output whenever a deploy replaces src/, and is excluded from backups.
The deploy that first declares a volume keeps what the app had already written there. Declaring [[volumes]] for data an app kept in src/ used to delete that data: the deploy replaced src/ before linking the new, empty volume. The old tree is now set aside until the volumes are linked. An empty volume is filled from it first, then from what the build produced.
Volumes are linked after the build, before the app starts. They used to be linked before the build, where a before-build that unpacks the application and moves directories into place collided with them.
Each app runs with a HOME and TMPDIR of its own, <app>/home/ and <app>/tmp/, where they were the server's /home/hop3 and /tmp. All apps shared one ~/.ssh, so Gitea and Forgejo could rewrite the authorized_keys containing Hop3's git-push keys, and app caches landed among the server's files. This covers the workers, before-run, the [admin] and [probe] create commands, hop3 run and hop3 app shell. The create commands also stop inheriting the server's whole environment: they get the workers' environment, including its precedence ([env] does not override a toolchain-owned variable). Build steps keep the server's HOME for now, where the toolchains and their caches live.
[proxy] deny lists paths the proxy blocks with a 404. deny = ["/data"] blocks /data and everything beneath it before the app or the disk sees the request. A PHP script there is refused without running. It is for data an app keeps inside the served tree, which upstream protects with an .htaccess that nginx and Caddy ignore. Traefik cannot block a path, so a deploy that declares one is refused there.
A git push shuts down the running app before replacing the source, as hop3 deploy already did. A process still holding files in src/ could race the rebuild.
A redeploy without new source builds from the source the app was delivered with. hop3 app upgrade, a backup restore and hop3 app scale ran the build again on the previous build's output, so a recipe that downloads and unpacks an application met the leftovers: LimeSurvey's upgrade failed with mv: cannot overwrite './vendor', and so did the rollback. Each deploy keeps the tree it delivered (src.delivered). Builds start from a copy of it, and backups include it. An app deployed by an earlier version keeps building in place, with a notice, until the next deploy.
A redeploy after a failed deploy works the first time. After a deploy failed while setting up the proxy, the next deploy of the same app failed once with Cannot transition app from STOPPED to RUNNING, then succeeded. A failed deploy now marks the app FAILED with the error behind it, where the app stayed STARTING. The request no longer races the build over the app's state.
A build command waiting for input fails at once, where it used to hang the deploy. Build steps inherited the server's stdin, which under supervisor is a pipe with no writer: unzip asking whether to replace a file held a Vikunja upgrade until the client gave up. Build, before-run, admin and probe commands now read from /dev/null.
Two deploy steps no longer go on past a failure. Both failures were logged while the deploy reported success. One was a served tree the web server could not be granted read on, so all assets answered 403. The other was a volume that could not be given to the app's user, so the app could not write to it. Both now fail the deploy, naming the path and the error.
An [env] value that is only $NAME is refused, like ${NAME} already was: [env] is exported literally, so NITRO_PORT = "$PORT" handed Nuxt the text $PORT and it died on startup. The message names the section that works ([run].start for the port, [env.computed] with ${NAME} otherwise). A $ inside a longer value stays legal.
The hop3.toml JSON schema lists the [access] role values, so an editor completes and checks them.
Ruby apps with a lockfile written by Bundler 4 build again. Hop3 installed gems with bundle install --frozen; Bundler switches to the version that wrote the lockfile. Bundler 4 removed --frozen, so all such apps failed to build. Frozen mode is now set through BUNDLE_FROZEN, which both read. An app without a Gemfile.lock is refused with the command that makes one.
hop3-fetch resumes a download cut short. A retry used to start again from the first byte, so a server that drops long transfers failed all attempts: releases.wikimedia.org drops a transfer after about 75 seconds, and cut mediawiki's 116 MB tarball six times running, each time between 47 and 80 MB. A retry now requests the rest with a Range header and checks the reassembled length against the total the server declares. Only an attempt that brings no new bytes counts against the six allowed, so a slow link that advances a few MB per attempt finishes. The resume names the version it continues (If-Range), so a file replaced upstream between attempts is downloaded again from zero. A connection reset mid-body is retried like a timeout, where it used to end the build with a traceback.
hop3 addon expose and an app's [[ports]] work on an upgraded server. The migrations that created the port, network and ban tables gave them a BIGINT id, which SQLite does not assign on insert, so inserts into them failed on a server that had run those migrations. addon expose reported each attempt as a port taken by a concurrent operation, and a deploy with [[ports]] blamed another app. A migration rebuilds the three tables and keeps their rows. Other database errors are now reported as they are.
An app with a runtime that has no deployer is reported as an error. Its status read as stopped: hop3 status called the running app CRASHED. An app being stopped was marked stopped while it still ran. The state sync now marks that app FAILED with the reason and keeps syncing the others.

PHP apps¶

A Nix-built PHP app is served by php-fpm, as a native one is. The php-app template served it through PHP's single-threaded built-in server, where an app that requests its own URL deadlocks against the only worker; serve-mode = "fpm" is now the default, and builtin and artisan remain for a recipe that requests them. A hand-written hop3.nix asks for the pool with a php_fpm entry in runtime.json: Hop3 generates the pool, runs it, and has the proxy speak FastCGI to it, refusing hidden files and the recipe's [proxy] deny paths.
A PHP document root must lie inside the application. [build].document-root = "..", or a Nix app's web-root, served the directory above it (or crashed the Nix build); both are refused with the key that set them.

Nix builds¶

A generated Nix recipe can fetch source from GitHub by repository: github = "OWNER/REPO", rev and hash in [nix]. The hash is of the unpacked tree, which GitHub keeps stable, where the bytes of GitHub's generated archives change without notice; a url pointing at a GitHub /archive/ link is now refused with the command that prints the hash.
ruby-bundler builds gems with a Rust extension. gem-cargo-hashes = { commonmarker = "sha256-…" } in [nix] vendors the gem's crates before the offline build, which Redmine 6 needs.
A generated Nix app writes config files and runs pre-exec once per deploy, before it starts. They used to run in the wrapper on each start: inside the start timeout, which BookStack's migrations outlasted, and as files the running app wrote into src/. They now run as the build's before-run step, ahead of [run] before-run, and a failure fails the deploy. A hop3.nix written by the recipe author gets the same with a before_run list in runtime.json. Likewise, writable-home-at-runtime now copies the package into src/ at deploy time, as the writable tree, where the wrapper used to copy it into src/.<name>-home on first start; $PWD, not $HOME_DIR, is where it lives.
Generated Nix recipes build against nixos-26.05, from nixos-24.11, with pnpm 10 as the default pnpm. Dependency hashes from the old pin may need recording again, from the got: line of a failed build; a recipe that pins a nixpkgs-rev keeps that pin.
go-frontend-source-root names a directory in the repository (frontend), found under whatever directory the source unpacks to. It used to include the release archive's top-level directory, which changes with the fetcher; that form is now refused with the fix in the message.
hop3 app upgrade works for an app built from a [nix] template. The generated expression was written into the source tree as hop3.nix, so a second deploy from that tree (an upgrade, and the restore that rolls a failed upgrade back) refused it as "both hop3.nix and a [nix].template". It is now .hop3-generated.nix; a hop3.nix an earlier deploy generated, identical to the template's output, is replaced.
A nix-gen app built from its own directory now builds. The local-source mode of three templates had only ever been parsed. python-venv emitted a wrapper Nix could not parse, node-pnpm-install wrote into a read-only copy of the app, and the pnpm dependency store hashed differently on each fetch because pnpm 10 records the build directory's path in it. Each template now has a fixture under apps/test-apps-nix-gen/ that builds and serves. The pnpm fix changes all node-deps-hash values: directus-nixgen's must be re-recorded.
A Nix recipe built with node-pnpm-install reports why the install failed, and installs under pnpm 10. The install ran with --reporter=silent, so a failure read "builder failed with exit code 243" and nothing else. It also installed from the read-only fetched store, which pnpm 10 modifies; it now copies the store first, as nixpkgs does. The rebuild of the native packages a recipe declares (node-native-packages) also runs under pnpm 10 now, which skipped them without a word, so the build succeeded and the app failed at start.
A Nix node-gyp addon that never included <cstdint> builds again. The libstdc++ in the nixos-26.05 pin no longer pulls <cstdint> in through <memory> or <functional>, so isolated-vm 5.0.3, which all Directus releases still pin, failed with "'uint32_t' has not been declared". The node-pnpm-install template now compiles each node-native-packages addon with -include cstdint.
A Nix app on a server without Nix names the cause. [build].builder = "nix" on a server installed without --with nix ended with bash: nix-build: command not found; it now names the missing Nix and the install command.
A Nix app on a server without Nix is refused however the builder is chosen. Auto-selection declined it and moved on, so an app with a Procfile next to its hop3.nix was built by another builder without a word.
A Nix app's runtime.json cannot declare a web worker alongside a php-fpm pool. The pool replaced it without a word; both together are refused, naming the keys.
nix-gen checks serve-mode, and strip-components/executable with github, when hop3.toml is read. A typo surfaced deep inside a deploy; the file-only keys were dropped silently.
A nix-gen template that installs one file refuses a GitHub source. java-war, prebuilt-binary and node-prebuilt failed deep inside the Nix build on a github = ... tree; they now ask for the release asset by name.
A generated hop3.nix no longer names the wrong app. The eject hint used [metadata].id, which is not the name an app is deployed under; it now tells you to run hop3 nix eject in the app's directory.

Proxies¶

Caddy serves apps. Each deploy on a Caddy server failed validation: Hop3 named each app's config <app>.caddy, which Caddy parses as JSON. With that fixed, an app that allowed plain HTTP still answered only on port 443, and the client address reached the app as ip:port. Caddy now also serves PHP apps through their php-fpm pool.
Traefik serves apps. Its routes named a certificate resolver nothing defined, sent plain HTTP nowhere, routed only an app's first hostname, and told the app all requests were HTTPS. They now use the certificate Hop3 issues, route all hostnames on both ports, and refuse /.git as the other proxies do.
A deploy the server's proxy cannot serve is refused before the build. Traefik proxied HTTP to php-fpm and sent static files to an address it cannot open. Caddy and Traefik also routed a [waf] app around the firewall. Each deploy reported success. They are now refused with the reason. Traefik serves no files and speaks no FastCGI. Only nginx fronts the WAF. The certbot engine is likewise refused on Caddy and Traefik, since Hop3 obtains and renews certificates through nginx; use the self-signed engine there.
A static directory requested without the trailing slash redirects to it. nginx and Caddy served /docs from docs/index.html in place, so the page's relative links resolved against / and pointed at the wrong files. Both now redirect /docs to /docs/, with a relative Location that keeps the query string.
nginx accepts any legal hostname. The default server_names_hash_bucket_size (64) rejected a long name (hop3-tuto-nix-gen-flask.135.181.203.156.sslip.io) depending on which other vhosts were present, so the deploy failed nginx -t. The installer sets 256 in /etc/nginx/conf.d/hop3-http.conf.
NGINX_INCLUDE_FILE and CADDY_INCLUDE_FILE name a file in the app's repository. They were resolved one directory above it, so a file the app shipped was reported missing unless the path started with src/. A missing file ended the deploy with a traceback. A path outside the repository is now refused, and so is a missing file, each with the variable's name.

Backups and restores¶

A PostgreSQL restore gives back the dump, or fails and changes nothing. The dump was replayed over the live database: CREATE statements failed on the tables already there, while psql continued and exited 0. The restored database mixed the backup with what it was meant to replace, which is the database an upgrade's rollback left. The dump now loads into a fresh database, in one transaction that ends at the first error, and replaces the live one only once it loaded.
An app with an email addon can be backed up, and so upgraded. The email addon refused to be backed up because it contains no data. That failed all backups of such an app, along with all upgrades, since an upgrade's snapshot is a backup. The backup records the addon by name, without its settings; a restore checks it is still configured, and names the command that re-creates it otherwise.
hop3 addon <type> import loads the dump. It uploaded the dump, staged it in /tmp, and handed it to a restore that reads only from /home/hop3/backups, so it failed for PostgreSQL, MySQL and Redis on each call since 0.7.0. The dump is now staged in the backup tree and removed after the restore.
A PostgreSQL dump without owners restores into a database the app can use. A dump taken with pg_dump --no-owner, or on another platform, loaded as the admin and left all tables, views, sequences, functions and types owned by the admin: the restore succeeded and the app got "permission denied" on its own data. Whatever the admin is left owning is now given to the app's user before the restored database replaces the live one.
An S3 addon's backup contains the bucket's objects. Restoring it puts them back. The backup was a JSON manifest of the bucket's metadata. The restore read it and discarded it, so hop3 addon s3 restore, clone, import and an app's restore all reported success and restored no object. The backup is now a tar of the bucket's objects with their content types. The restore reads it whole before touching the bucket, writes back all objects, then deletes the objects absent from the backup.
hop3 backup register refuses a directory outside the backup tree. A backup registered from elsewhere restored the app's source, data and environment, then halted at the first addon, since addon restores read only from /home/hop3/backups. The app was left half restored.
hop3-backup-server snapshots Garage, and hop3-restore-server puts it back. The server snapshot left out the S3 service: the config, the metadata and all stored objects. The snapshot now includes them, with a consistent copy of the metadata Garage modifies while running. --only garage restores them, keeping the replaced tree next to it, and succeeds only once Garage is back. A running but unresponsive Garage fails the backup.
hop3-backup-server copies a stopped Redis's data, and fails on an unresponsive running database. Any unresponsive Redis, PostgreSQL or MySQL was one line, "skip", and a snapshot without data. A stopped Redis's last save is now copied as is; a stopped PostgreSQL or MySQL is named as missing from the snapshot; a running but unresponsive one fails the backup, within a bounded wait. hop3-restore-server --only redis succeeds once Redis is back. On a host without systemd it used to leave Redis running, which then saved over the restored dump.
A MySQL backup no longer logs an access error. mysqldump dumps tablespaces by default, which needs the server-wide PROCESS privilege the app's user does not have; it complained on each backup, and on each upgrade's snapshot. Dumps now leave tablespaces out.

Catalog¶

The catalog is fetched from a path that names its format: https://apps.hop3.cloud/catalog/v1/catalog.tar.gz. The server reads the format the signed index declares and refuses one it does not know, naming it and keeping the catalog it already has; the publisher had written "format": 1 since the catalog shipped, and no server read it. A future format is published next to this one, under v2/, so a server limited to format 1 keeps receiving updates. Servers before 0.8 keep fetching the old path, which serves the same catalog, and are refused the entries that need 0.8. A CATALOG_SOURCE_URL set to the old path still works; move it to v1/.
A catalog entry can name the server it needs. requires_hop3 = "0.8" in an entry's catalog.toml makes an older server refuse hop3 catalog install by name, before it creates anything, where it used to stage the recipe and fail in the middle of the build. The publisher now works out what a recipe needs from the hop3.toml (php-fpm, scratch volumes, [proxy] deny, [access], a GitHub Nix source, and others) and from a hand-written hop3.nix (a setup step in before_run), records the feature names in the signed index, and refuses to publish a recipe with a requires_hop3 lower than those features need: Paheko, WordPress, Dolibarr and Kanboard needed php-fpm without declaring it. A server refuses an entry that names a feature it does not have, whatever its version number; requires_hop3 is checked only for catalogs published before feature names.

Email¶

The catch email backend has a sink: Mailpit. --with email installs it, pinned by version and SHA-256, and runs it on loopback under the process manager. The installer waits for the SMTP greeting, and selecting catch checks the sink; selecting catch used to point Postfix at a port nothing listened on, report success, and leave all messages deferred. hop3 server email caught lists the captured mail, and server email status now names the active backend, where it reported no transport for catch and direct. hop3-test selects catch for a run that includes an app declaring email. The 190-flask-email fixture checks the path end to end: a message sent through the injected relay, found in the sink.

Command line¶

uv tool install hop3 installs the client. The hop3 name on PyPI belonged to an unrelated 2024 server (0.1 to 0.2, now yanked) with no hop3 command. Its hop3 module overwrote hop3-server's in a shared environment. It is now a package with no code that installs hop3-cli and the hop3 command.
The SSH tunnel works after hop3 login under a shared ssh connection. With ControlMaster auto and ControlPersist in ~/.ssh/config, the login's connection stayed open, and the next command's tunnel handed the forward to it and exited. The command then failed with "ssh exited before the tunnel was ready", while the forward stayed open outside the CLI's control. The tunnel now always opens a separate connection.
A context is a server. A checkout is bound to one (ADR 042). One word used to name both a server connection (in config.toml) and a deploy environment (in hop3.toml), so hop3 context behaved differently in each directory and three functions disagreed on which context was current. Now:
A context lives only in config.toml: a name, a server and an optional bootstrap route. hop3 context add, remove and rename edit nothing else, and add refuses --app, --domain and --env.
hop3 context use <name> binds the checkout inside a project (.hop3-local.toml, gitignored) and sets the global current context outside one, or with --global. It asks first, naming the scope, the file and the address; --yes skips the question. hop3 context unset removes either.
A command is sent, in order, where -c, HOP3_CONTEXT (or HOP3_API_URL), the checkout's binding or, outside a project, the global current context points. A project with no binding is refused, and so is a name that matches no context, whichever step supplied it: a removed context named by HOP3_CONTEXT used to send the command to the default server.
Commands that contact a server print the destination and the reason on stderr: → demo (https://demo.example.com), bound in this checkout. --why shows the whole trace, the context's included.
[contexts.<name>] in hop3.toml sets what differs per deployment (app, domains, env), keyed by context name. A context matching none of a project's blocks is refused; a lone block is no longer selected by default.
A login records its context, and makes it the global current one only when none is set; it used to switch it on each login. Running hop3 login --context <name> with no target signs in again over the SSH route the context records.
A protected server accepts changes only when named for them. With PROTECTED = true in hop3-server.toml, a command above the viewer role is refused unless -c or HOP3_CONTEXT chose the server. The operator then types the app's name to confirm (--yes skips the prompt, for CI). The server enforces it from what the CLI reports, so a stale client configuration cannot get around it; a CLI older than 0.8 is refused such changes, and so is the TUI, which does not prompt for the confirmation yet.
hop3 settings set server and set token are refused, naming hop3 context add and hop3 login: no command read the keys they wrote. hop3 init --context <name> records the server under that name, as the help now states.
A command missing an argument fails, with the usage. It used to print the usage and exit 0, so hop3 addon postgres export > dump.sql without a name wrote "Usage: ..." into the dump and reported success. It now prints the usage from the command's help on stderr and exits 2.
hop3 passes each command the flags its help documents. The CLI refused --app for addon list, addon promote, backup list, app create and cert renew, so addon promote could not be run at all. The CLI split ps scale too, so the server ran ps instead. It also consumed --dry-run for all commands but deploy, with a warning, so system cleanup --dry-run, admin reencrypt-credentials --dry-run and app migrate --dry-run ran for real; it now hands the flag on, and a command without a preview refuses it. system info -v lists plugins again and system status --quiet prints one line. A test now checks the flags each command reads, the examples in its help and the command lines in the guides and tutorials against what the CLI sends.
hop3 addon show, status, destroy, attach and detach read an addon's type from the name, like addon expose and addon promote. Without --type they assumed postgres, so a Redis addon was looked up as a database that did not exist. A name two types share is refused until --type disambiguates it.
hop3 addon status works. It failed on each call, reading a type option it did not declare.
hop3 cert renew takes --app <app> and --force-renewal, in place of a positional app name and --force, which the CLI consumes: --force skipped a confirmation without reaching the server, so a forced renewal renewed only the certificates already due.
hop3 nix eject --app <app> > hop3.nix prints the generated expression. It wrote the file into the server's copy of the source, which the next deploy replaces. Save it in your project and remove the [nix] section from hop3.toml: a build refuses an app that has both, which the command used to claim it would ignore.
hop3 nix eject ejects. All invocations printed the command's help and exited.
A slow or failing server is no longer reported as a missing credential. Before a destructive command, the CLI checks the credential with the server, and any failure of that check, a timeout included, became "No credential for ...", exit 4, and the advice to log in again: a destroy sent while the server was still busy with an upgrade reported exactly that. Only a refused credential is now an authentication error; a timeout or a server error is reported as such. A read timeout, which no handler recognised, now reports that the server timed out. These messages are printed on stderr.
hop3 logout signs out after a login. A login records the server as the default context. The commands the CLI handles locally looked only for the default server older versions wrote: since 0.7.4, hop3 logout answered "Not logged in" and kept the token, and hop3 login --token or --browser without --url found no server.

Server installation and updates¶

Updating a server reconciles the platform's settings. hop3-deploy-server on an existing server updated the code and ran the installer only when features were requested, so a setting the platform added missed the servers installed before it; the nginx setting above first missed the very server it was written for. A plain update now runs the installer's --reconcile mode, which applies those settings and restarts no service that runs apps. The full installer no longer restarts uwsgi-hop3 unless the unit changed: each redeploy with --with bounced all apps.
A failed setup step aborts the server install, with the command's output. The systemd, SSL, nginx, PostgreSQL, MySQL, config-file and ACME steps caught their failures, printed a warning cut to 100 characters, and the install continued to report success: a failed config write gave a server without database credentials. Verification now fails on a stopped core service, a missing certificate or a missing hop3-server.toml, where it warned. On a host without systemd, the installer generates the environment and key files, installs no unit, and reports that, where systemctl failed and was reported as a "Systemd setup issue" on all container installs. See notes/installer-fail-loud-audit.md.
The server installer's toolchain and service steps abort at the failure. A failed install of npm's pinned pnpm, nodeenv or Go, a pnpm shadowed on PATH, Postfix for --with email, acme.sh, the S3 service or its token's group, the www-data and docker group memberships, an unsupported distribution, or the removal of the legacy sudoers fragment printed a warning and the install continued. The SSH key step could not fail at all: it printed "SSH keys configured" whatever happened. Rust, Leiningen and Elixir stay best-effort by default and abort when requested: --with clojure and --with elixir join --with rust.
The database steps of the server installer fail with the output. A failed PostgreSQL createuser or createdb was printed as a detail and followed by "role and database created"; a PostgreSQL cluster that failed to start fell back to an older one; MySQL statements that failed returned without a message and a later check reported a generic error; the Docker-network MySQL user and the bind-address restart could fail while the install continued. Each now raises where it fails. A failed MySQL statement names the operation, never the statement text, which contains the password. Redis is made a primary in the config file, where the runtime commands that did it were discarded by the restart after them. It is enabled under the Debian or Fedora name. The Redis check now authenticates: with a password set, the unauthenticated PING it made was refused.
hop3-install server fails when pip fails. A failed pip install of hop3-server, hop3-rootd or hop3-cli was ignored, and the installer printed "installed successfully" over it. It now fails there and shows pip's output.
hop3-install server fails when it cannot install Node 22. When NodeSource was unreachable it installed the distribution's Node 18, which is past end of life and too old for the Node apps in the catalog; the install succeeded and those apps failed later, at build. It now aborts there with the reason, and likewise when an older Node is still first on PATH after the install.
The server installer runs on Ubuntu 26.04. It requested libpcre3-dev, which 26.04 no longer ships, and ended at the first apt-get install; it now uses libpcre2-dev there, as on Debian 13. It also no longer requests postgresql-contrib, a transitional name for modules now in the server package, and a virtual package with two providers on 26.04. With --with mysql it ended at MySQL's service: the catalogue baseline's MariaDB headers made apt remove mysql-server, which on 26.04 refuses to share a host with them. The next run then mistook the leftover mysqld for an installed server. On 26.04 the client headers now come from MySQL. A requested feature's packages are always installed. The installer's hop3 MySQL account is created with the server's default authentication, as each app's account already was: MySQL 8.4 no longer loads the mysql_native_password plugin it named. A PHP app finds Ubuntu 26.04's php-fpm8.5: the toolchain looked for a list of versions that ended at 8.4, and now uses the highest versioned php-fpm on PATH.
The server installer sets a umask of 022. It inherited the caller's, and docker exec can pass 0000: files the installer created without an explicit mode were world-writable. Redis did not start: its init script created the pidfile world-writable, start-stop-daemon refused it, and the script exited 0.
An installer step no longer removes packages without saying so. apt resolves a conflict by removing the other side and still exits 0: on each install, --with mysql removed the MariaDB client headers and the catalogue baseline put them back, removing MySQL's. The package lists now install with --no-remove, so a conflict aborts the install, printing apt's list of what it would remove. --with mysql no longer adds a second set of client headers.
The server's npm is pinned (11.17.0), like its pnpm. Node 22 ships npm 10, which crashed installing a project scaffolded by the current NestJS CLI (Cannot read properties of null (reading 'edgesOut')). The installer reconciles npm on each install and fails if another npm shadows it on PATH.
The server installer installs the .NET SDKs 8 and 10, the long-term releases, wherever the distribution publishes them. It installed 8 and 9, but no feed publishes 9 for Ubuntu 24.04, so that install failed on all such servers and was printed as a warning. An SDK that is published and fails to install now aborts the install, as does a platform that publishes no SDK.
--with email waits for apt's lock. Its Postfix install was the one apt-get call without the bounded lock timeout the others include, so it failed at once when another apt still held the lock. A unit test now fails on any apt-get command without it.
The .NET SDK step explains why an SDK is missing. A failed apt-get update is fatal. So is failing to add Microsoft's repository on Debian, which has no .NET packages of its own, and on Ubuntu is named as the cause of any SDK then missing (the line read "not published"). On Fedora an installed SDK counts as available, which dnf list --available may not show.
The installer explains why a service would not start. Redis and MySQL failures aborted with only "did not restart"; the failing command's output and the unit's journal now come with it. Redis is given time to listen after a restart, which an init script does not wait for.
On a host without systemd, the installer's Redis runs the configuration the installer generates. The init script finds Redis by its pidfile, but the Redis the package starts on the Docker target has none. The installer's restart therefore reported success while the new Redis failed to bind next to the old one. The old one then answered the check that followed. The installer now asks Redis to shut down, waits for it to exit, and starts it again.
A PostgreSQL password that cannot be verified comes with the reason. The error now includes the ALTER USER and TCP-check output, which tell a refused connection from a wrong password from an ALTER that never ran.
hop3-deploy-server fails where it used to fall back. With --acme-email, a failed Let's Encrypt issuance, a placeholder address or a missing acme.sh fails the deploy, where it installed a self-signed certificate and reported success; without --acme-email, self-signed is the default, stated once. A failed admin user creation fails the deploy, where it continued to print login instructions for a user that did not exist. The CLI installer fails when the installed hop3 does not run or provides no command, and shows a failed package install's output without --verbose.
hop3-deploy-server installs Docker only on request (--with docker), on any target, as ADR 052 decided: Docker is a second-tier runtime. On the Docker target, where dockerd cannot run, --with all excludes Docker. An explicit --with docker is refused there with the reason, where it used to print "Failed to start Docker daemon" and continue.
hop3-deploy-server --docker fails when its image build fails. It used to start a bare ubuntu:24.04 instead, with a few packages added, and announce it in one line: a different machine from the one the tests describe.
hop3-deploy-server --docker waits for hop3-server to come up before it reports the services started. A local CLI it could not configure now fails the deploy. The login ran while the server was still starting and failed with "Could not connect", yet the deploy still ended with "Deployment complete!". --no-cli-setup skips the step.
On a Docker install, supervisor calls the uWSGI emperor uwsgi-hop3, the name of its systemd unit elsewhere. It was uwsgi. Use supervisorctl restart uwsgi-hop3. The next hop3-deploy-server rewrites the configuration.
hop3 system status works on a server with services under supervisor. It shelled out to systemctl, which a Docker install does not have, and failed outright. It now queries the process manager through hop3-rootd, and checks the proxy the server actually runs, so a Caddy or Traefik server no longer reports nginx as down. hop3 system status --json also returns the structured report it was documented to: the client took --json as an output switch and kept it from the command, so the "JSON" was the text report in a list.
hop3-rootd reports a service systemd has no unit for as unmanaged. It used to report it stopped. systemctl is-active prints inactive for a missing unit, so an uninstalled proxy read as installed and down.
The server's log messages are no longer lost to the database migrations. Alembic's logging setup, run in-process at startup, disabled all loggers created before it: among them, the catalog loader's report of the apps it excludes.

Documentation¶

Tutorial: your own Flask app, built hermetically by Nix (docs/tutorials/nix/nix-gen-flask.md). No Nix code and no Nix on the reader's machine: a [nix] template, a hash-pinned lockfile, and a deploy.
Tutorials teach what the platform accepts. Django records exact versions (pip freeze); Sinatra keeps its Gemfile.lock; Nuxt drops NITRO_PORT = "$PORT", which [env] passes literally; the NestJS, Nuxt and Astro scaffolds are pinned to a major version.
The tutorials attach addons the way the CLI expects, hop3 addon attach <addon> --app <app>: fifteen lines had the app first, as a positional, and nine told hop3 app logs to --tail, an option it does not have. hop3 backup list uses --app for the app in all guides.

For contributors: the test harness¶

The test suites run in a checkout with no catalog next to it. The tests that read the hop3-catalog recipes are marked catalog and run with make test-catalog (pytest --catalog=<checkout>); elsewhere they are deselected, and pytest's summary counts them. The two tests that used a catalog recipe as example input read a frozen copy instead.
A command run in a test container keeps its output against a remote Docker daemon. Over ssh://, docker-py reads an exec's output from the raw SSH channel, and an output that arrived with the response headers stayed in paramiko's buffer. A curl that had answered then read as "no HTTP response", at random, in the gate and proxy e2e tests. The harness's Docker client now reads that buffer first.
hop3-test installs the runtime the selected apps are built with. A Nix-built app brings --with nix and a Docker-built one --with docker, as their addons already brought theirs, now that no runtime is installed by default. On the Docker target, which cannot run Docker, Docker-built apps are left out before the deploy, each with the reason.
hop3-test fails a server deploy that continued past a warning. Each ⚠ line the installer or the deployer prints fails the run, unless an allowlist entry explains why it is safe to continue. Such lines sat unread in all run logs for weeks. Among them were two bugs and an install that failed on all Ubuntu 24.04 servers.
hop3-test fails an app that writes into src/ while running, and redeploys all apps. Each deploy replaces src/, so whatever an app puts there is lost at the next one, an upgrade included. The test lists the directories the app wrote to after starting, grouped for a [[volumes]] entry. It then deploys the unchanged source again and runs all validations a second time. --no-redeploy skips the second deploy.
An app with a sign-in only a browser can drive is no longer reported as passed without one. Ten catalog recipes build their sign-in form in JavaScript; their check verifies everything else and reports so, and hop3-test reported them passed, so no run verified their sign-in. A run without a browser now reports them as not verified, by name and with the status unverified, without failing them. hop3-test run --screenshots drives a real browser to each passed catalog recipe, signed in with its admin credential, saves the images into the recipe's screenshots/, and fails a browser-only app the browser cannot sign in to. It needs a server target and Playwright (uv run --with playwright). It replaces the catalog repository's check-catalog.py and shoot-catalog.py. The admin credential a check receives now also names the field the app's form expects (HOP3_ADMIN_LOGIN).
A Docker deploy takes 4 minutes instead of 17. The test image contains the packages the installer installs on Ubuntu 24.04, so a new container no longer downloads the base packages, the catalogue baseline, Go and the .NET SDKs. The installer still runs all apt steps and finds them installed. The image's lists are derived from the installer's code. A unit test fails when they drift, so the image cannot contain a package the installer would not install.
A recipe with a probe account that cannot be created fails its test. The deploy continues past a failed [probe].create, after which the app's check signs in as the operator. Six published recipes passed a whole run that way, their probe commands broken. hop3-test now fails such a test and quotes the deploy's reason. The check's summary no longer reads "no [probe]" when a probe exists.
A failed test with an app that still serves is headlined by the test's error. A failure the app's runtime does not explain (a check, a contains, the probe account) was listed under "Failed tests:" as "✓ ok" with the verdict "see sections below". The headline now states that the app serves and the test failed, and quotes the test's error.
A failed tutorial names the step that failed. All tutorial failures were classified build-failure and pointed at an empty build section, an output mismatch included: validoc's Error: Expected exit code matched the build-error pattern. A failed tutorial is now a step-failure with a headline quoting validoc's failed step and its error, pointing at the section with the step's output.
A Nix hash mismatch's headline gives the hash the build got. Nix ends its first line at the derivation, so a failed hop3-test app's verdict quoted that line and stopped before the value to put in the recipe. It now names the derivation, the recipe's hash and the one the build got. The text report also no longer prints "No failures recorded." beneath a list of failed tests when the target had none.
A command that hangs on a test target fails, naming itself. The Docker exec and the SSH channel do not bound a command, so one that hung left a hop3-test run silent, with no error, for as long as it hung. Commands the harness runs on a target are now bounded there (300 s unless the caller sets more; a tutorial gets 900 s). One that runs past the limit is killed and raises an error naming the command, the limit and the last output.
A test run names the tests it left out, and why, at any verbosity. Tests dropped for a missing service, a missing admin domain, or a Docker build on the Docker target were reported through warnings that --quiet silenced, absent from the report; they are now listed as they are left out and again after the summary. A failed app destroy at cleanup is shown the same way, and fails an upgrade chain, since the --host server keeps whatever the teardown left.
A test run no longer changes your CLI's default context. Docker deploys by the test harnesses (hop3-test, the pytest e2e targets) signed the developer's CLI in to the throwaway container and made it the default context, so once the run removed the container, plain hop3 commands went to a dead address. The harnesses now deploy with --no-cli-setup; a manual hop3-deploy-server --docker still configures the hop3-docker context.
The executable tutorials pass on Docker again. The test domain included the Docker target's SSH user (hop3@<ip>.sslip.io), so the hostname step of all tutorials was refused; on a target named by hostname it was <name>.sslip.io, which sslip.io does not resolve; and all 33 checked their app over http://, which the platform redirects. Tutorials now check https://; the harness accepts the test server's self-signed certificate. A tutorial also gets the server its hop3.toml requests (Nix, addons), as an app does.
One e2e session per machine. The Docker target has a fixed container name and ports, so a second run tore down the first's container mid-install. A run now locks ~/.hop3/docker-target.lock before touching Docker and refuses to start while another holds it.
The Docker tests run on the daemon the Docker context names, a remote one included: DOCKER_CONTEXT=fedora make test-e2e runs them on another machine, through the ports the container publishes there.
The upgrade-chain check cannot pass vacuously. The seeded account's rights on the first hop were taken as found, so a probe that failed there compared equal on all later hops; the seed now must have a plain account's rights.
Production code no longer checks whether it runs under pytest. Seven places skipped a hop3-rootd call, the Mailpit check or the port-25 probe when PYTEST_CURRENT_TEST was set; the test suite now replaces those dependencies with fakes it declares.
070-php-public expected the pre-2026-09-25 answer for /.env (the front controller). nginx now returns 404 for hidden files, which the fixture now checks.
hop3-tools catalog reports checks coverage against the catalog's maturity folders. It listed the directories directly under apps/, which have been the statuses since the catalog moved to them, so it claimed all experience reports described an app the catalog did not include, and could not report a published application without one. It now counts the applications behind the published entries, a variant counting as its application.
A test run states what chose its tests. Without apps, paths or --status, hop3-test run uses the smoke profile, nine Procfile fixtures, and a server rebuilt for the full catalog run passed it twice with "All 9 tests passed". The run now prints the selection before it starts and again under the summary, for example "profile smoke (tier fast; priority P0): 9 of 256 tests", with the options that widen it.