ADR-0011: Platform Engineering Blueprint For The Homelab Workload Plane

Status

Accepted, with four open platform choices tracked at the end of the Consequences section.

Context

YourBOX is one application among several that target the same homelab. Weather-app already runs on the cluster, GOAD lives in a hostile VLAN, future applications (a Symfony admin, a few internal tools) follow the same path. Without a workspace level platform blueprint, each application re negotiates the GitOps engine, the observability stack, the secret backend and the runtime topology. That leads to drift, duplicated work and inconsistent operational surfaces.

The homelab platform repository (homelab-platform) holds the infrastructure blueprint and is the natural place to standardize. Several decisions are already settled in practice and need to be recorded explicitly so that every application onboarding refers to one source of truth. A few decisions remain open and are tracked here as proposed work, not implemented choices.

Decision

The homelab workload plane standardizes on the building blocks below. Every application chart consumes them through the same conventions and every application documentation references this ADR for the rationale.

GitOps Engine: Argo CD

Argo CD is the GitOps engine for every workload in the homelab. The previous Flux based wiring used by weather-app during the 2024 to 2026 transition is migrated to Argo CD Application and ApplicationSet resources. The selection rests on:

• A first class UI for operators and reviewers.
• Native multi cluster support through ClusterSecret resources, which the homelab needs as soon as a second cluster joins for blue green migrations or DR drills.
• The ApplicationSet controller fits the per environment generator pattern (staging, production, smoke) used by every application.
• Strong RBAC integration with Authentik for human access to the UI.

The app of apps pattern groups applications under homelab-platform/kubernetes/apps/<app>/application.yaml and a parent Application reconciles the directory.

Observability: Prometheus, Loki and Grafana

Every workload exposes a Prometheus /metrics endpoint, ships JSON logs to stdout collected by Promtail into Loki, and renders dashboards in Grafana. The Prometheus Operator, the Loki Helm chart and the Grafana Operator live in the monitoring namespace and are owned by the platform repository.

Workloads register themselves through ServiceMonitor resources and through Grafana dashboard ConfigMaps labeled grafana_dashboard: "1". The DORA template in gitlab-ci-templates feeds the workspace DORA dashboards.

Forensic Search: ELK Pipeline

Audit and security relevant events flow through an ELK pipeline: Filebeat or the Monolog Elasticsearch handler ships events, Logstash enriches them, and Elasticsearch (or OpenSearch under the same pipeline) indexes them. Kibana is the search UX.

The ELK pipeline is dedicated to long retention forensic search across applications. The Loki retention is shorter and tuned for operational troubleshooting. The two systems do not duplicate work: Loki keeps the last 30 days of all logs, ELK keeps the last 365 days of the audit and security channels only.

Secrets: OpenBao + External Secrets Operator

ADR-0008 sets the YourBOX side. The platform side commits to OpenBao plus the External Secrets Operator for every workload, with SealedSecrets reserved for bootstrap credentials. The pattern is identical to what weather-app already follows.

Identity: Authentik

ADR-0004 sets the YourBOX side. Authentik is the single identity provider for every workload through OIDC. Workload specific roles map to Authentik groups.

Supply Chain: Plumber, cosign and Sigstore

ADR-0009 mandates digest pinning. The Sigstore policy controller verifies cosign signatures at admission. Plumber gates every pipeline. The cosign public key lives at homelab-platform/kubernetes/platform/security/cosign-pub.yaml.

Open Platform Choices

Four platform choices are tracked here as open work. Each one becomes its own ADR once decided. Until then, application charts use abstract Helm values that an adapter ADR will instantiate.

Platform Substrate

The initial implementation assumed Proxmox as the bare-metal substrate. That assumption is reopened by ADR-0013, which compares Proxmox with a NixOS plus Incus platform. Until ADR-0013 is accepted, platform documentation must treat Proxmox-specific material as current-state evidence, not immutable target architecture.

Evaluation criteria:

• Host reproducibility from versioned configuration.
• VM and container support for OPNsense, trusted services, Kubernetes nodes and GOAD.
• Storage, backup and restore model on the available SSD/HDD hardware.
• Security hardening surface for users, SSH, firewalling, updates and auditing.
• Automation fit with Terraform, Ansible, Packer, Nix and Incus tooling.

Kubernetes Distribution

Talos Linux remains one candidate distribution, but the decision is not yet approved. The evaluation covers Talos, k3s and RKE2 against the operational criteria below. The application charts target standard Kubernetes APIs only so they remain distribution agnostic.

Evaluation criteria:

• Patch and upgrade cadence (immutable OS preferred).
• Footprint on the selected substrate (RAM, disk, control plane overhead).
• Integration with the homelab Argo CD and OpenBao plane.
• Operator surface for cluster lifecycle (Talos talosctl vs k3s vs RKE2).

The selected distribution lands as ADR-0014 once the evaluation concludes.

High Performance Cache

A Redis cache is proposed for every workload that benefits from a shared cache plane (Symfony cache pool, session storage, rate limiter storage, deduplication of Meilisearch upserts). The candidate is a shared multi tenant Redis cluster managed by the Bitnami Redis Helm chart with per workload AUTH and per workload database number.

Application charts already include a cache: block in their values, wired to a CACHE_DSN secret. Without the cache layer the application falls back to the in process Symfony filesystem cache; with the cache layer the same code path uses Redis.

The Redis adoption lands as ADR-0015 once the cluster sizing and the AUTH model are decided. The current open questions: standalone vs Redis Sentinel vs Redis Cluster, and whether the platform exposes Valkey instead of Redis to dodge license changes.

Messaging Bus For Write Path Optimization

ADR-0006 puts Symfony Messenger at the heart of the application domains. Today the in process transport is sufficient. For workloads that need to decouple writes (video transcoding, audit log fan out, search index updates that survive a database write peak), a dedicated broker is the next step.

Two candidates are tracked:

• Apache Kafka if event sourcing and replayable streams become a shared platform feature (audit log fan out, cross application event bus, DORA event ingestion). Kafka has the highest operational cost. Candidate operator: Strimzi.
• RabbitMQ if the workload only needs work queues with acknowledgements, dead letter routing and operational simplicity. Candidate operator: RabbitMQ Cluster Operator.

The decision drives whether the Symfony Messenger transport switches from doctrine:// (today) to amqp:// (RabbitMQ) or kafka:// (Kafka). The platform exposes one or the other, not both.

The messaging adoption lands as ADR-0016 once a trial proves out the operational cost on the homelab footprint.

Consequences

• Every workload chart consumes the platform through the same Helm value conventions: externalSecrets, serviceMonitor, networkPolicy, cache, messaging, httpRoute. Onboarding a new application copies the YourBOX chart structure.
• The platform owns the lifecycle of Argo CD, Prometheus, Loki, Grafana, ELK, OpenBao, ESO, Authentik, Kyverno and the Sigstore policy controller. Application teams (in this homelab: one engineer) never touch those components directly.
• The Flux to Argo CD migration is a one shot operation for weather-app. The weather-app helmrelease.yaml is rewritten as an Argo CD Application and the Flux controllers are uninstalled. The migration runbook lives in the homelab platform repository.
• The open choices (platform substrate, Kubernetes distribution, cache, messaging) leave follow up ADRs explicitly visible. Application charts ship the consumption hooks now so that the eventual adoption is a values change, not a code change.
• Cross application contracts (Authentik OIDC, OpenBao KV paths, Argo CD Application schema, Prometheus ServiceMonitor labels) are documented once and referenced from every service overview.
• The ELK pipeline duplicates a small fraction of the Loki content intentionally. The cost is one Logstash pipeline and one Elasticsearch cluster, the benefit is a forensic search horizon that Loki does not cover.
• Reviewers refuse any application Helm chart that references a non Argo CD GitOps controller, a non Authentik identity provider or a custom secret backend.