ADR 022: Build and Deployment Plugin System¶
Status: Final Type: Feature Created: 2024-10-01 Related-ADRs: 020, 021, 030, 032, 033, 034, 035
Context¶
The original Hop3 architecture combined build and deployment logic into a monolithic class with hardcoded conditionals for different application types. Supporting new build systems (Docker, Nix) or deployment targets (Kubernetes, external orchestrators) required invasive core changes. With the pluggable architecture (ADR-020), build and deployment became the first two stages of the pipeline, with a key principle: the application's codebase determines which strategies are used.
Decision¶
We implement build and deployment as a two-stage plugin system with per-application auto-detection:
- Per-Application Selection: Each application auto-detects appropriate build and deployment strategies based on its codebase
- Two Protocol Interfaces:
BuilderandDeployerdefine interfaces withaccept()methods - Plugin Discovery: Strategies discovered via
get_builders()andget_deployers()hookspecs - Data Flow Pipeline:
DeploymentContext→BuildArtifact→DeploymentInfo - Orchestration:
do_deploy(app, deltas)coordinates the pipeline
Build Plugin Interface¶
Builder Protocol¶
class Builder(Protocol):
"""Interface for turning source code into a runnable artifact."""
name: str # Strategy identifier (e.g., "python", "docker")
context: DeploymentContext # Application context
def accept(self) -> bool:
"""Return True if this strategy can build the app.
Examine the application's source code to determine compatibility:
- DockerBuilder: Checks for Dockerfile/Containerfile
- PythonBuilder: Checks for requirements.txt, pyproject.toml, setup.py
- NodeBuilder: Checks for package.json
- StaticBuilder: Checks for HTML files, no build requirements
Returns:
bool: True if strategy can handle this application
"""
def build(self) -> BuildArtifact:
"""Execute build and return artifact descriptor.
Performs the build process:
1. Set up build environment (venv, node_modules, etc.)
2. Install dependencies
3. Compile/transpile code if needed
4. Return artifact descriptor
Returns:
BuildArtifact: Descriptor of the built artifact
Raises:
RuntimeError: If build fails
FileNotFoundError: If required files missing
"""
BuildArtifact Data Structure¶
@dataclass
class BuildArtifact:
"""Describes a build artifact produced by a Builder."""
kind: str # Artifact type: "buildpack", "docker-image", "nix-closure", "static"
location: str # Path or identifier: "/path/to/venv", "myapp:latest"
metadata: dict[str, Any] # Additional strategy-specific data
# Examples:
BuildArtifact(
kind="buildpack",
location="/home/hop3/apps/myapp/.venv",
metadata={"python_version": "3.11", "packages": 42}
)
BuildArtifact(
kind="docker-image",
location="myapp:f8a9c3d",
metadata={"size_mb": 250, "layers": 12}
)
BuildArtifact(
kind="static",
location="/home/hop3/apps/landing-page",
metadata={"files": 15, "total_size": 1024000}
)
DeploymentContext Input¶
@dataclass
class DeploymentContext:
"""Context information passed to build and deployment strategies."""
app_name: str # Application identifier
source_path: Path # Path to application source code
app_config: dict # Parsed Procfile and hop3.toml
app: App | None # Full App ORM object (optional)
# Example:
DeploymentContext(
app_name="myapp",
source_path=Path("/home/hop3/repos/myapp"),
app_config={
"workers": {
"web": "gunicorn app:app",
"worker": "celery -A tasks worker"
}
},
app=<App object>
)
Deployment Plugin Interface¶
Deployer Protocol¶
class Deployer(Protocol):
"""Interface for running a build artifact."""
name: str # Strategy identifier
context: DeploymentContext # Application context
artifact: BuildArtifact # Build artifact to deploy
def accept(self) -> bool:
"""Return True if this strategy can deploy the artifact.
Check artifact compatibility:
- UWSGIDeployer: Accepts "buildpack" artifacts (native builds)
- DockerDeployer: Accepts "docker-image" artifacts
- StaticDeployer: Accepts "static" artifacts
Returns:
bool: True if strategy can deploy this artifact
"""
def deploy(self, deltas: dict[str, int] | None = None) -> DeploymentInfo:
"""Deploy the artifact and return connection information.
Performs deployment:
1. Create/update worker processes
2. Configure environment variables
3. Start/restart services
4. Return connection details for proxy
Args:
deltas: Worker scaling adjustments {"web": 1, "worker": -1}
Positive = add workers, Negative = remove workers
Returns:
DeploymentInfo: Connection details for reverse proxy
Raises:
RuntimeError: If deployment fails
"""
def scale(self, deltas: dict[str, int] | None = None) -> None:
"""Scale worker processes up or down.
Args:
deltas: Worker adjustments {"web": 2} adds 2 web workers
"""
def stop(self) -> None:
"""Stop all workers for this application."""
DeploymentInfo Data Structure¶
@dataclass
class DeploymentInfo:
"""Connection information returned by deployment strategies."""
protocol: str # "http", "uwsgi", "fastcgi"
address: str # Socket path or IP: "/tmp/app.sock", "127.0.0.1"
port: int | None # Port number or None for Unix sockets
# Examples:
DeploymentInfo(
protocol="uwsgi",
address="/tmp/myapp-web.sock",
port=None
)
DeploymentInfo(
protocol="http",
address="127.0.0.1",
port=8000
)
Plugin Registration¶
Build Plugin Example¶
from hop3.core.hooks import hookimpl
class NativeBuildPlugin:
"""Plugin providing native build strategies for various languages."""
name = "native-build"
@hookimpl
def get_builders(self) -> list:
"""Return list of build strategy classes."""
return [
PythonBuilder,
NodeBuilder,
RubyBuilder,
GoBuilder,
StaticBuilder,
]
plugin = NativeBuildPlugin()
Deployment Plugin Example¶
class UWSGIPlugin:
"""Plugin providing uWSGI deployment strategy."""
name = "uwsgi-deploy"
@hookimpl
def get_deployers(self) -> list:
"""Return list of deployment strategy classes."""
return [UWSGIDeployer]
plugin = UWSGIPlugin()
Configuration¶
Auto-Detection¶
Auto-detection is the default. Applications are detected based on files present:
# Python app with requirements.txt
$ ls myapp/
requirements.txt app.py Procfile
# → Auto-selects: PythonBuilder → UWSGIDeployer
# Node.js app with package.json
$ ls webapp/
package.json server.js Procfile
# → Auto-selects: NodeBuilder → UWSGIDeployer
# Static HTML site
$ ls landing/
index.html style.css images/
# → Auto-selects: StaticBuilder → StaticDeployer
Explicit Configuration¶
Auto-detection can be overridden in hop3.toml, for the edge cases where the codebase does not unambiguously select a strategy:
[build]
strategy = "docker" # Force Docker build
[deploy]
strategy = "docker" # Force Docker deployment
Complete Example: Python Application¶
PythonBuilder Implementation¶
@dataclass
class PythonBuilder:
"""Build strategy for Python applications."""
name: str = "python"
context: DeploymentContext
def accept(self) -> bool:
"""Check if this is a Python application."""
source = self.context.source_path
return (
(source / "requirements.txt").exists()
or (source / "pyproject.toml").exists()
or (source / "setup.py").exists()
)
def build(self) -> BuildArtifact:
"""Build Python application with virtualenv."""
venv_path = self.context.source_path / ".venv"
# 1. Create virtualenv
run(["python3", "-m", "venv", str(venv_path)])
# 2. Install dependencies
pip = venv_path / "bin" / "pip"
run([str(pip), "install", "-r", "requirements.txt"])
# 3. Return artifact descriptor
return BuildArtifact(
kind="buildpack",
location=str(venv_path),
metadata={
"python_version": self._get_python_version(),
"packages": self._count_packages(venv_path),
}
)
UWSGIDeployer Implementation¶
@dataclass
class UWSGIDeployer:
"""Deployment strategy using uWSGI application server."""
name: str = "uwsgi"
context: DeploymentContext
artifact: BuildArtifact
def accept(self) -> bool:
"""Check if artifact is compatible."""
return self.artifact.kind == "buildpack"
def deploy(self, deltas: dict[str, int] | None = None) -> DeploymentInfo:
"""Deploy using uWSGI."""
workers = self.context.app_config["workers"]
sockets = {}
# 1. Create uWSGI config for each worker
for worker_name, command in workers.items():
socket_path = f"/tmp/{self.context.app_name}-{worker_name}.sock"
sockets[worker_name] = socket_path
uwsgi_config = self._generate_uwsgi_config(
app_name=self.context.app_name,
command=command,
socket=socket_path,
venv=self.artifact.location,
)
config_file = Path(f"/tmp/{worker_name}.ini")
config_file.write_text(uwsgi_config)
# 2. Start/restart uWSGI processes
for worker_name, socket in sockets.items():
run(["uwsgi", "--ini", f"/tmp/{worker_name}.ini"])
# 3. Return connection info for first worker (web)
return DeploymentInfo(
protocol="uwsgi",
address=sockets["web"],
port=None
)
def scale(self, deltas: dict[str, int] | None = None) -> None:
"""Scale workers up/down."""
for worker_name, delta in (deltas or {}).items():
if delta > 0:
# Add workers
for _ in range(delta):
self._spawn_worker(worker_name)
elif delta < 0:
# Remove workers
for _ in range(abs(delta)):
self._terminate_worker(worker_name)
def stop(self) -> None:
"""Stop all uWSGI workers."""
run(["pkill", "-f", f"uwsgi.*{self.context.app_name}"])
Orchestration: do_deploy()¶
The orchestrator coordinates the pipeline:
def do_deploy(app: App, deltas: dict[str, int] | None = None) -> None:
"""Deploy application using pluggable strategies.
Args:
app: Application to deploy
deltas: Optional worker scaling adjustments
"""
# 1. Create deployment context
context = DeploymentContext(
app_name=app.name,
source_path=app.src_path,
app_config=AppConfig.from_dir(app.app_path).to_dict(),
app=app,
)
# 2. Select and run builder
builder = get_builder(context) # Auto-detects via accept()
log(f"Building with: {builder.name}")
artifact = builder.build()
log(f"Built: {artifact.kind} at {artifact.location}")
# 3. Select and run deployment strategy
deployer = get_deployment_strategy(context, artifact)
log(f"Deploying with: {deployer.name}")
deployment_info = deployer.deploy(deltas)
log(f"Deployed: {deployment_info.protocol}://{deployment_info.address}")
# (Proxy configuration happens separately - see ADR-021)
Strategy Selection Functions¶
def get_builder(context: DeploymentContext) -> Builder:
"""Find and instantiate appropriate builder.
Tries each registered builder's accept() method until one returns True.
Args:
context: Application context
Returns:
Builder instance ready to build
Raises:
RuntimeError: If no strategy accepts the application
"""
pm = get_plugin_manager()
strategy_classes = flatten(pm.hook.get_builders())
for strategy_class in strategy_classes:
strategy = strategy_class(context)
if strategy.accept():
return strategy
raise RuntimeError("No build strategy found for this application")
def get_deployment_strategy(
context: DeploymentContext,
artifact: BuildArtifact
) -> Deployer:
"""Find and instantiate appropriate deployment strategy.
Tries each registered strategy's accept() method until one accepts the artifact.
Args:
context: Application context
artifact: Build artifact from build stage
Returns:
Deployer instance ready to deploy
Raises:
RuntimeError: If no strategy accepts the artifact
"""
pm = get_plugin_manager()
strategy_classes = flatten(pm.hook.get_deployers())
for strategy_class in strategy_classes:
strategy = strategy_class(context, artifact)
if strategy.accept():
return strategy
raise RuntimeError(f"No deployment strategy for artifact kind '{artifact.kind}'")
Rationale¶
Why Per-Application (Not Server-Wide)?¶
Unlike proxy (server-wide, ADR-021), build and deployment must be per-application:
- Different Requirements: Python apps need pip/venv, Node apps need npm, static sites need no build
- No Shared Resource: Each app has independent build and processes (unlike shared reverse proxy)
- Flexibility: Different apps on same server use different strategies
- Isolation: Build/deploy failures isolated to individual applications
Alternative Rejected: Server-wide build/deploy strategy would force all applications to use the same approach.
Why Auto-Detection (Not Explicit Config)?¶
Convention over configuration:
- Developer Experience: Most applications "just work" without configuration
- Simplicity: No need to learn strategy names for common cases
- Flexibility: Explicit configuration remains available for edge cases
Example: Application with requirements.txt automatically uses PythonBuilder → UWSGIDeployer
Why Two Separate Stages?¶
Separating build from deployment enables composition:
- Mix and Match: Docker build + Kubernetes deploy, Nix build + uWSGI deploy
- Reuse Artifacts: Build once, deploy to multiple environments
- Clear Separation: Build concerns (dependencies) vs runtime concerns (processes)
Consequences¶
Benefits¶
- Application Flexibility: Each app uses the approach that suits its needs
- Automatic Detection: Most apps work without configuration
- Extensibility: New strategies added as plugins
- Clear Data Flow: Explicit data structures between stages
- Composability: Build and deployment strategies can be mixed
Drawbacks¶
- Complexity: More moving parts than monolithic deployer
- Configuration Overhead: When auto-detection fails, users need to understand plugin system
- Strategy Conflicts: Multiple strategies might accept same application (resolved by ordering)
Trade-offs Considered¶
| Aspect | Decision | Alternative | Why Rejected |
|---|---|---|---|
| Scope | Per-application | Server-wide | Apps have different build requirements |
| Selection | Auto-detection | Explicit config | Better UX, config available for edge cases |
| Stages | Separate build/deploy | Combined | Enables composition and artifact reuse |
| Interface | Protocol | ABC | Structural typing more Pythonic |
Strategies¶
The Builder protocol has three production implementations and the Deployer protocol three, spanning the native, container, and Nix toolchains:
Build:
- LocalBuilder (NativeBuildPlugin) - native builds for Python, Node.js, Ruby, Go, Clojure, and static sites
- DockerBuilder - Docker-based builds
- NixBuilder - Nix closure builds
Deployment:
- UWSGIDeployer - uWSGI application server with Unix sockets
- StaticDeployer - static file serving via reverse proxy
- DockerDeployer - Docker Compose container deployment
The BuildArtifact → DeploymentInfo data flow is the stable contract between the two stages, independent of which strategies fill them. ADR 035 formalises this data flow as the runtime contract.
Prior Art¶
- Heroku Buildpacks: Auto-detection based on files (e.g.,
requirements.txt) - Cloud Foundry Buildpacks: Similar auto-detection with pluggable strategies
- Kubernetes Operators: Separate concerns into distinct controllers
- GitLab CI/CD: Pipeline stages with clear data flow
References¶
- ADR-020: Pluggable Architecture
- ADR-021: Proxy Plugin System
- Heroku Buildpacks
- Python Protocol (PEP 544)
Related ADRs: ADR 020: Pluggable Architecture for Core Deployment Workflow, ADR 021: Proxy Plugin System for Reverse Proxy Configuration, ADR 030: Two-Level Build Architecture, ADR 032: Deployment Strategies and Artifact Lifecycle, ADR 033: Docker Integration Strategy, ADR 034: Streaming Deployment Logs, ADR 035: Build Artifacts as Runtime Contract