Skip to content

ADR 009: Nix Runtime Integration

Status: Deferred Type: Feature Created: 2024-07-17 Related-ADRs: 006, 008, 023, 035

Scope

Running Nix-built applications under Hop3's own runtime (uWSGI emperor + nginx) is not part of this ADR's scope: it is covered by ADR 035's RuntimeConfig contract. A Nix derivation produces $out/hop3/runtime.json; the deployer reads it and launches the workers with no further inference, across both hand-crafted and template-generated derivations.

This ADR concerns the further step of letting Nix manage the runtime itself — the host OS and backing services — rather than only the applications it builds:

  • Full NixOS module generation. Emit configuration.nix fragments so an operator can deploy Hop3 as a NixOS module rather than as a Python-installed daemon.
  • Nix-managed backing services. Provision PostgreSQL / MySQL / Redis / MinIO via nixpkgs services rather than via the OS-package-plus-config approach. This opens a path to per-app NixOS containers.
  • NixOS-native systemd integration. Use NixOS's systemd module system to express worker lifetimes rather than writing uWSGI vassal .ini files.

This is deferred. It delivers additional reproducibility guarantees (the OS itself becomes Nix-managed) at the cost of narrowing the installation target from "any Debian/Red-Hat host" to "NixOS host". That trade-off is not worth making before a quantitative evaluation (§5.5 of TR-01) has demonstrated the reproducibility benefits of the in-scope RuntimeConfig integration.

Two design questions remain open and are to be resolved against the results of that evaluation: the provisioning semantics for Nix-managed backing services, and the isolation boundary for NixOS containers.

Interaction with ADR 023 (Runtime Stack Replacement)

Nix support must work with both runtime stacks: - Current stack: uWSGI + nginx + supervisor - Proposed stack (ADR 023): Granian + Caddy + ProcessManager

The BuildArtifact.runtime field provides this abstraction — the run phase reads the artifact and starts the app without knowing how it was built.

Context and Goals

To enable the security, reliability, and operational efficiency of the Hop3 platform, it is critical to provide robust runtime isolation for the applications it hosts. While traditional containerization tools like Docker offer runtime isolation, they can introduce complexity and security vulnerabilities. Nix, with its purely functional package management system, is traditionally known for its build-time guarantees (reproducibility, determinism). However, Nix also has potential as a runtime tool, particularly in providing isolation between applications and managing services.

The goal of this ADR is to evaluate and leverage Nix's capabilities to ensure isolation between running applications and to manage applications along with their backing services (such as databases, email servers, certificates, etc.) in a controlled and consistent environment. This aligns with Hop3's objective of offering a secure, reliable, and user-friendly platform.

Architectural Context

Connection to BuildArtifact/RuntimeConfig (ADR 035):

Hop3's build/run separation model (ADR 035) aligns perfectly with Nix's philosophy:

  1. Build phase produces a BuildArtifact with RuntimeConfig containing:
  2. env_vars: Environment variables (PATH, PYTHONPATH, etc.)
  3. path_prepend: Paths to add to PATH

  4. workers: Process commands to run

  5. Run phase reads the artifact and applies the configuration - no detection or inference needed.

Nix naturally produces this model: - All paths are absolute (/nix/store/...) - All dependencies are explicit and immutable - Runtime environment is fully determined at build time

NixBuilder produces:
BuildArtifact {
    runtime: {
        env_vars: {
            "PATH": "/nix/store/abc-python/bin:/nix/store/def-app/bin",
            "PYTHONPATH": "/nix/store/ghi-deps/lib/python3.11/site-packages"
        },
        workers: {
            "web": "/nix/store/def-app/bin/gunicorn app:app"
        }
    }
}

Connection to Deployment Strategies (ADR 032):

Nix's immutable store paths enable clean versioning and rollback: - Each build produces a unique /nix/store/<hash>-app path - Rolling back = switching symlinks to previous store path - Blue-green deployments are natural with Nix's atomic symlink switches

Decision

Hop3 will use Nix not only for build-time management but also for runtime behavior, leveraging its declarative nature to provide isolation between running applications and managing services as resources. This will include managing dependencies, service configuration, and backing services (e.g., databases, email systems, storage) through Nix's package and environment management tools.

Key Components

Runtime Isolation with Nix

  1. Isolation Between Running Applications:

  2. Controlled Environments: Nix can create isolated environments for each application by ensuring each application has its own, unique set of dependencies and configurations. This can prevent conflicts between applications and ensure they run securely.

  3. Minimal Cross-Application Interference: Each application will be run in its own isolated environment, making it immune to changes in other applications' environments. This prevents issues such as dependency conflicts or one application affecting the performance of another.

  4. Managing Applications as Services:

  5. Service Management: Nix can be used to manage applications as long-running services, leveraging tools like systemd to handle service orchestration. Each application deployed in Hop3 can be managed declaratively using Nix, which will handle the starting, stopping, and restarting of services.

  6. Backing Services Management: Nix can also be extended to manage backing services such as databases, email services, and other resources like SSL certificates. By declaring these services in the Nix configuration, Hop3 can ensure that applications have access to the necessary resources and services, all while being managed in the same reproducible, isolated manner as the applications themselves.

Implementation Strategy

  1. Nix-Based Runtime Isolation:

  2. Nix Shells for Isolation: Use Nix shells or other isolation mechanisms to ensure that each application runs in its own environment with isolated dependencies and runtime configurations. This approach ensures runtime consistency and prevents dependency-related runtime failures.

  3. Declarative Service Management: Applications and their backing services will be declared in Nix, ensuring that the platform manages not only the deployment but also the runtime lifecycle of both the applications and the resources they depend on.

  4. Managing Backing Services:

  5. Service Integration: Use Nix to manage the lifecycle of databases (e.g., PostgreSQL, MySQL), email servers, and other essential services needed by the applications. Nix expressions can be written to declaratively define how these services are installed, started, and linked to the applications.

  6. Certificate and Resource Management: SSL certificates, storage systems, and other resources required for application operations can also be managed through Nix. By integrating certificate management (e.g., with Let's Encrypt) into the platform's configuration, Hop3 ensures secure, managed communication and availability of essential resources.

Managing Services with Nix or NixOS

Nix can help manage the lifecycle of backing services in several ways by leveraging its declarative, reproducible nature. How this will be done in practice is still under discussion and will be refined based on feedback and testing.

Here is the current thinking on how Nix can be used to manage services:

Declarative Service Configuration (on NixOS)

  • NixOS Modules: In NixOS (the operating system built around Nix), services such as databases (e.g., PostgreSQL, MySQL) and email servers (e.g., Postfix, Dovecot) can be defined declaratively using NixOS modules. These modules specify how services should be configured, started, and managed. The configuration is entirely reproducible, meaning that rebuilding or restarting the service will always produce the same result.

  • Declarative Resource Management: Nix allows you to declare not only the application and service dependencies but also the configuration of the services themselves (such as database ports, connection limits, etc.). For example, you can define how PostgreSQL should be set up (e.g., data directory, authentication methods) in the Nix expression, ensuring that the service behaves consistently across deployments.

Example for PostgreSQL:

services.postgresql = {
  enable = true;
  dataDir = "/var/lib/postgresql/data";
  authentication = {
    local = [{
      database = "all";
      method = "md5";
    }];
  };
};

This example shows how NixOS can manage the lifecycle of PostgreSQL, ensuring that the database server is started with the correct configuration and that it remains consistent even after reboots or migrations.

*Automatic Service Management (with systemd)

  • Systemd Integration: Nix (on NixOS) integrates seamlessly with systemd, the default service manager for Linux, to handle the automatic starting, stopping, and restarting of services. Each service is tied to a systemd unit file that is generated automatically by Nix. This ensures that services like PostgreSQL or an email server are automatically started when needed and are properly managed (restarted in case of failure, stopped on shutdown, etc.).

  • Service Dependencies: Nix can declaratively manage service dependencies. For example, if an application depends on a PostgreSQL database or an email server, Nix can ensure that those services are started before the application starts. This lifecycle management ensures that services are always available when needed.

Example:

systemd.services.myApp = {
  after = [ "postgresql.service" "postfix.service" ];
  wantedBy = [ "multi-user.target" ];
  serviceConfig = {
    ExecStart = "/path/to/myApp";
  };
};

This ensures that the application (myApp) only starts after PostgreSQL and Postfix services are up and running.

Managing Data Migrations and Backups

  • Database Migrations: Nix can manage migrations by defining version-specific configurations for services like PostgreSQL. Since Nix tracks dependencies and versions precisely, it can ensure that database migrations are applied at the correct time during the upgrade or deployment process. This ensures seamless upgrades of both applications and the databases they depend on.
  • Backups and Restores: With Nix, the configuration for backup services can be defined declaratively, ensuring consistent and automated backup schedules for databases or other services. Nix can integrate with existing backup tools to provide version-controlled and reliable backup solutions for databases, ensuring that they can be restored in case of data loss.

Reproducible Runtime Environments

  • Immutable Infrastructure: Because Nix configurations are immutable, every deployment of a service like PostgreSQL or an email server will be identical across environments. This ensures that developers can run the same version of PostgreSQL with the same configurations in both development and production, preventing "works on my machine" problems.

  • Version Pinning and Rollbacks: Nix allows pinning to specific versions of services and easily rolling back to previous versions. This ensures that upgrades to essential services (e.g., from PostgreSQL 13 to 14) can be done safely, with the ability to roll back if something goes wrong.

Example:

environment.systemPackages = with pkgs; [
  postgresql_13  # PostgreSQL 13 is explicitly specified.
];

This guarantees that the specific version of PostgreSQL will always be used unless explicitly changed.

SSL Certificates and Other Resources

  • Certificate Management: Nix can manage SSL certificates for services such as web servers or email servers. Using declarative tools like Let's Encrypt in combination with Nix, you can automatically issue, renew, and install SSL certificates for your services. This ensures that all certificates are managed centrally, making certificate rotation and renewal part of the system’s lifecycle management.

Example:

security.acme.certs = {
  mySite = {
    email = "admin@example.com";
    webroot = "/var/www";
    postRun = "systemctl restart nginx";
  };
};

This ensures that SSL certificates are automatically managed and applied to the services without manual intervention.

Orchestration of Backing Services

  • Orchestrating Multiple Services: Nix can orchestrate multiple backing services (databases, caches, email servers, etc.) needed by an application. By specifying the relationship between these services in the Nix configuration, the platform can ensure that all services are deployed and managed correctly as part of the application stack. For example, if an application requires PostgreSQL, Redis, and an email service, Nix can ensure that all these services are deployed together, started in the correct order, and managed declaratively.

Consequences

Benefits

  • Runtime Isolation: Nix provides fine-grained isolation for running applications, ensuring that each application runs in its own environment, independent of other applications. This reduces the risk of dependency conflicts and enhances runtime reliability.
  • Service and Resource Management: Nix’s declarative configuration model allows not only the applications but also their backing services (e.g., databases, email, certificates) to be managed as part of the same system. This creates a unified approach to managing all aspects of the application environment.
  • Security: By managing runtime dependencies and services in a controlled environment, Nix minimizes the attack surface and ensures that only the necessary components are included in the runtime.

Drawbacks

  • Complexity of Runtime Isolation: While Nix excels at managing build environments, its role in runtime behavior is still evolving. Using Nix for runtime isolation may introduce complexity or unstability that needs to be carefully managed.

Risks

  • Integration Complexity: Nix’s role as a runtime isolation tool is still less mature than its build-time capabilities. Ensuring that Nix provides adequate runtime isolation across a wide range of applications and services might be challenging. To mitigate this, Hop3 will leverage systemd and other NixOS tools known to manage services effectively.
  • Service Interoperability: Managing backing services like databases and certificates using Nix could face challenges when integrating with legacy or complex services. Mitigation includes extensive testing and community feedback to ensure compatibility.
  • Runtime Performance: Using Nix to manage the runtime environment might introduce performance overhead, especially in complex deployments with many services. Continuous performance optimization and monitoring are essential to minimize this impact.

Related ADRs: ADR 006: Nix Integration with Hop3, ADR 008: Template-Based Nix Expression Generation, ADR 023: Runtime Stack Replacement, ADR 035: Build Artifacts as Runtime Contract