YourBOX Observability

Purpose

ADR-0011 sets the workspace observability baseline: Prometheus, Loki and Grafana for metrics, logs and dashboards, with an ELK pipeline for the audit log forensic search. This page documents how YourBOX plugs into the baseline: the metrics it exposes, the log schema, the dashboards, the alert rules and the SLOs.

Metrics

The application exposes /metrics in the Prometheus exposition format. The metrics are produced by app/src/Platform/Infrastructure/Metrics/ which wraps the promphp/prometheus_client_php library. Four categories of metrics are registered:

HTTP: request count, latency histogram, status code distribution, route cardinality. Labels: method, route, status, firewall.
Domain: video uploads count, transcoding latency histogram, search query count, comment count, vote count. Labels: context, outcome.
Infrastructure: PostgreSQL connection pool usage, Meilisearch query latency, Flysystem operation count and latency by adapter, Messenger queue depth, Redis cache hit ratio.
Build: image version, build SHA, build timestamp. One label per dimension, exported once at process boot.

The Symfony bundle wraps the registry through a Redis storage backend so that metrics persist across PHP-FPM workers in a multi process model.

The Prometheus operator picks up the pod through a ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: yourbox
  namespace: yourbox
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: yourbox
  endpoints:
    - port: http
      path: /metrics
      interval: 15s

Logs

The application writes JSON logs to stdout through Monolog. Promtail collects the logs from the Kubernetes node and forwards them to Loki with the labels {namespace, app, instance, environment}. The log schema is:

{
  "ts": "2026-06-20T12:34:56.789Z",
  "level": "INFO",
  "channel": "request",
  "message": "GET /catalog/show resolved",
  "request_id": "01J0X...",
  "user_id": "01H7QZ1X...",
  "username": "thonyan32",
  "route": "catalog_show",
  "method": "GET",
  "status": 200,
  "duration_ms": 42,
  "ip": "203.0.113.10"
}

Sensitive fields are scrubbed by a Monolog processor before serialization: passwords, session cookies, OIDC tokens, signed URL query strings and the Authorization header never reach the log line.

A second Monolog handler ships the audit channel to Logstash through a Filebeat sidecar on the pod node. Logstash enriches the event with the geo IP from the source address and indexes it in Elasticsearch (or OpenSearch under the same ELK pipeline) at yourbox-audit-YYYY.MM. The hot retention on the index is 90 days, then a cold tier of 365 days.

Dashboards

Three Grafana dashboards ship with the application chart as ConfigMaps labeled grafana_dashboard: "1":

YourBOX Overview: request rate, error rate, latency percentiles (p50, p95, p99), active session count, OIDC login success rate, transcoding queue depth, search latency. Default time range: last 6 hours.
YourBOX Catalog: upload rate, transcoding success rate and duration, storage operation count by adapter, top categories, video count growth.
YourBOX Engagement: comment rate, vote rate, top liked videos in the last 24 hours, moderation queue depth, report rate.

The DORA dashboards in the workspace ingest the deployment events emitted by the CI pipeline (DORA template) and surface deployment frequency, lead time, change failure rate and mean time to restore.

Alerts And SLOs

The SLO targets:

Availability: 99.5 percent monthly on GET / and GET /health measured by the Kubernetes black box probe.
Latency: p95 of HTTP requests below 300 ms over a 5 minute window.
Transcoding: 95 percent of videos reach the published state within 5 minutes of upload.
Search: p95 search latency below 150 ms.

Each SLO becomes a Prometheus alert and a Grafana SLO panel. Alerts fire to Alertmanager which routes to the homelab notification channel. Severity mapping:

critical for availability dips below 99 percent over 15 minutes.
warning for latency or transcoding slowness above the threshold for 10 minutes.
info for moderation queue depth above the threshold.

The Alertmanager routing tree, the silences policy and the notification channel configuration live in the homelab platform repository.

Black Box Probes

A Blackbox exporter probes the public hostname from the VPS edge and from a cluster external probe. The probes exercise:

GET /health returns 200 in under 500 ms.
GET / returns 200 with the homepage content match.
The OIDC discovery endpoint of the YourBOX client in Authentik resolves.

A probe failure surfaces on the Overview dashboard and feeds Alertmanager.

Tracing

OpenTelemetry tracing is a follow up step. When tracing is enabled, the Symfony exporter adds the OpenTelemetry SDK and the OTLP exporter targets a Tempo backend (or Jaeger) in the homelab platform. The trace context flows through Messenger so that the upload request links to the transcoding span.

ELK And Forensic Search

The audit log lands in Elasticsearch (or OpenSearch under the same ELK pipeline) through Logstash. The forensic search UX is Kibana (or the OSI fork OpenSearch Dashboards depending on the workspace ADR outcome) for cross service correlation.

A YourBOX audit query example in KQL:

index=yourbox-audit-* action=video.deleted
| where actor_username != target_uploader_username
| sort -@timestamp

Logstash also normalizes the Authentik event log (login success or failure, group changes) so that a single Kibana query joins YourBOX moderation actions with the corresponding Authentik authentication events.

Operational Runbooks

Each alert links to a runbook on this documentation site. The runbook lists the exact Grafana panel, the corresponding Loki query and the exact ELK filter that surfaces the failing population.