paulserban.eu

Portfolio Edition

Paul Serban

AI & full-stack engineer · agentic platforms · distributed systems

← Back to portfolio

Featured

Observability Hub

Gated, self-hosted telemetry for a four-surface static platform

Role: Architect & lead implementer

  • #architecture
  • #aws
  • #clickhouse
  • #cloudflare
  • #cloudfront

Problem

Four independently deployed static surfaces had no honest telemetry story - the AWS console is not observability, cookie analytics is the wrong privacy default, and a static site has nothing for Prometheus to scrape.

Approach

Designed a gated platform - prove CDN questions locally against data AWS already emits, ship cookie-less visitor analytics as a JS beacon without waiting on a VPS, then add owned Umami, uptime, and unified dashboards only after those cheap paths are real. Compose over Kubernetes; Loki for Grafana logs; clickhouse-local for ad-hoc SQL.

Outcome

Phase 0 is live on the laptop. Phase 0b is decided and documented (Cloudflare Web Analytics, prod-only snippets) - visitor analytics no longer waits on Lightsail. Remaining phases stay gated.

phases
8
phase_0
done
phase_0b
documented
surfaces
4
hosting_spend_phase_0
$0
View code

Problem

prj--personal-portfolio--v3 is four static surfaces behind CloudFront - portfolio, blog, quiz, news-feed - not a process you can scrape. There is no app server, no /metrics, no request log on a box you SSH into. "Is it healthy?" lived in the AWS console. "Who is reading the blog?" defaulted to a third-party JS tracker and a cookie banner. "Did the quiz PWA just 5xx at the edge?" was a guess until someone emailed.

That is not an observability gap you close with Datadog because you have used Datadog at work. A personal JAMstack platform has a different failure mode: almost no runtime, almost no budget, and a privacy posture you actually have to mean. Prometheus against S3 is theatre. Google Analytics on a site that argues for craft is a contradiction. Kubernetes on a single VPS is resume-driven architecture.

I needed a platform contract for telemetry: which questions each signal can answer, which it cannot, what ships before any hosting invoice, and how a fifth project joins later without a second stack.

Solution

prj--observability-hub is a separate repo on purpose. The portfolio Terraform owns how sites are served (distributions, private buckets, logging_config). This repo owns how those sites are observed - identity, local tooling, and later the always-on VPS. That split is the same seam as content-vs-app in the content pipeline: ownership follows change cadence, not a single "infra" folder.

The stack is phased with exit gates. Phase 0 runs entirely on the laptop against production CloudFront data. Phase 0b is a JS beacon in the four apps - still $0 hosting, still no VPS. Phases 1–6 are designed, spiked, and ADR'd; they are not shipped. Shipping a VPS before the CDN questions are answerable - or before a cookie-less visitor signal exists - would be spending to feel busy.

Target shape, once the gates are met:

ConcernToolWhen it exists
Reverse proxy / TLSTraefikPhase 1 (VPS)
Page analytics (bridge)Cloudflare Web AnalyticsPhase 0b (prod snippet, no VPS)
Page analytics (owned)Umami + PostgresPhase 2
Core Web VitalsCloudflare beacon first; web-vitals → Umami only if neededPhase 0b; Phase 3 optional
UptimeUptime KumaPhase 4
Infra metricsPrometheus + node_exporterPhase 5
CDN metricsGrafana CloudWatch DS (YACE optional)Phase 0 local → Phase 5 VPS
Access-log SQLclickhouse-localPhase 0 (CLI, stays CLI)
Access-log UIGrafana Loki + AlloyPhase 0 local (Phase 6 practiced early)
Error trackingGlitchTipPhase 6, only if islands justify it

Scope and features

In scope

Out of scope (named, not forgotten)

Honest privacy note: "privacy-friendly" here means no cookie consent theatre, not "no third party." Phase 0b sends beacons to Cloudflare. CloudFront standard logs still contain client IPs. Loki stores them in the log line (never as labels). ClickHouse can uniqExact(c_ip). That is CDN forensics plus a vendor dashboard, not "we have no identifiers." Pretending otherwise would be the kind of product copy I would reject in a design review. Umami (Phase 2) is the path to hold the event rows ourselves - it does not erase the Phase 0b choice.

Architecture

Two diagrams matter: the target platform, and what is actually running today.

Target (Phases 0-6)

Visitors
   │
   ▼
Route 53 → CloudFront x 4 → private S3 x 4
              │                    │
              │                    └── site objects (SSG / quiz SPA)
              ├── CloudWatch metrics (AWS/CloudFront, Region=Global)
              └── standard access logs → s3://cf-access-logs.paulserban.eu
                                            │
                     ┌──────────────────────┼──────────────────────┐
                     ▼                      ▼                      ▼
              Grafana CloudWatch     Alloy → Loki            clickhouse-local
              (health KPIs)          (Grafana log UI)        (ad-hoc SQL CLI)

Browser JS (Phase 0b, prod only)
  beacon.min.js → Cloudflare Web Analytics
  (page views, referrers, LCP/INP/CLS — not in Grafana, no API)

VPS (not started) — Traefik TLS
  analytics.  → Umami + Postgres     (owned page views; later optional CWV)
  status.     → Uptime Kuma          (HTTP checks from outside the laptop)
  grafana.    → Grafana              (CloudWatch + Prometheus + Loki + Umami)
  errors.     → GlitchTip            (optional, Phase 6)
  Prometheus + node_exporter + Traefik metrics

Phase 0 as-built (shipped)

Laptop (127.0.0.1 only)
  grafana :3000
    ├── CloudWatch DS  (IAM keys, defaultRegion us-east-1)
    └── Loki DS        (compose network only)
  loki                 (filesystem, ~14d retention, no auth)
  alloy                (parse CloudFront TSV .gz → Loki)
  cf-log-sync          (aws s3 sync every 300s)

AWS
  CloudFront x 4  →  CloudWatch
                  →  s3://cf-access-logs.paulserban.eu  (60-day lifecycle)
  IAM user observability-hub-readonly
    CloudWatch GetMetric* + CloudFront List/Get + S3 GetObject on the log bucket

Not in Compose
  docker run clickhouse/clickhouse-server clickhouse-local
    s3() table function → saved SQL in infrastructure/local/clickhouse/queries/

Ownership is a first-class architecture decision, not a README footnote:

RepoOwns
prj--personal-portfolio--v3Distributions, OAC, the shared log bucket, logging_config on site/blog/quiz/news
prj--observability-hubRead-only IAM, local Compose, Grafana provisioning, Alloy parse pipeline, ClickHouse query pack, later VPS Compose

A clone of the hub repo cannot enable logging. That is correct: logging is a property of the edge, not of the dashboard.

Approach

As architect and lead implementer I treated observability the way I treat hosting and CI: seams first, spend second, orchestration last.

Platform questions before tool shopping

The original notes were a tool list (Umami vs Plausible vs Cloudflare Web Analytics, UptimeRobot vs Kuma, Sentry vs GlitchTip). That is how you accidentally buy four SaaS free tiers that do not compose. I inverted it: what questions must this platform answer, then which signal is allowed to answer them.

QuestionHonest signalWrong signal
Is CloudFront serving cleanly this week?CloudWatch request/error/byte metricsPage-view JS (bots, cache, and 5xx at the edge are invisible or lying)
Which paths 404, miss cache, or dominate bytes?Access logs (Loki + ClickHouse)CloudWatch (no URI dimension on standard CF metrics)
Are unique IPs / edge POPs moving?clickhouse-local (uniqExact)Loki labels (cardinality bomb)
Who is reading the blog as a person?Cloudflare Web Analytics (Phase 0b); Umami later (Phase 2)Access-log IPs (NAT, crawlers, shared egress)
What LCP did a real phone see?Cloudflare CWV (Phase 0b); owned web-vitals only if neededLighthouse on my laptop
Is the apex down from the public internet?Uptime Kuma on a VPS (Phase 4)Grafana on 127.0.0.1
Is the VPS itself dying?Prometheus + node_exporter (Phase 5)CloudFront metrics

If a tool cannot be placed on that table, it does not enter the compose file.

Validate before you rent

Phase 0 is the design spike the console-first hosting case study already argued for: prove the contract on real traffic before freezing spend. CloudFront logging is pennies of S3. CloudWatch basic metrics are free. Docker on the laptop is free. A VPS is not free, and Lightsail's $5 bundle will not hold this stack anyway.

Right-size the runtime

Spikes killed three fashionable defaults:

Traefik over Caddy is the other proxy call: label-based Docker routing, no shared Caddyfile to edit per phase, and it matches the local Traefik mesh already used for the four apps. Consistency of operations beats a slightly simpler ACME story.

Two query paths on purpose

Grafana does not speak ClickHouse. Interactive access-log dashboards are Loki. Exact unique IPs, top IPs, and edge POPs are ClickHouse CLI. That looks like indecision. It is cardinality and operational weight:

Stack (Phase 0, real)

LayerChoices
IdentityTerraform IAM user, least-privilege, local state, keys in gitignored .env
Metrics UIGrafana 11, provisioned CloudWatch DS (us-east-1 / Global)
Log UILoki 3 + Alloy 1.7, CloudFront TSV parse, ~14 day filesystem retention
Log shipaws s3 sync loop every 5 minutes - not Kinesis, not S3 notifications
SQLclickhouse-local 24.12, s3() + saved queries, PrettyCompact CLI
Bind127.0.0.1:3000 only - this is not a shared team Grafana

Phased delivery

Each phase has an objective, decisions, an observation contract, extractable metrics, an outcome, and the limits I would put in a design review. Do not start the next VPS phase until the current gate is met. Phase 0 has met its gate. Phase 0b is the next $0 slice - it does not wait on Phase 1.

Phase 0 — Local AWS log aggregation (done)

Objective. Prove CDN/traffic visibility from data AWS already has, on one machine, before renting anything.

What shipped. CloudFront standard logging on all four prod distributions into cf-access-logs.paulserban.eu (60-day lifecycle). Read-only IAM via Terraform. Local Compose: Grafana + Loki + Alloy + cf-log-sync. Two provisioned dashboards. Seven saved ClickHouse queries behind ./scripts/query.sh. Makefile targets matching the portfolio repo's compose_up habit.

Decisions.

What you can actually observe.

Metrics you can extract today.

SourceMetrics
CloudWatch AWS/CloudFrontRequests, 4xxErrorRate, 5xxErrorRate, BytesDownloaded (all four distributions; Region=Global)
Loki / access logsvolume, status, result_type, cache hit %, top error paths, top cache-miss paths, top content/asset paths, bytes by path, referrer hosts, direct vs referred, 404 timeseries, sample client IPs, raw log search
clickhouse-localtop 10 stems + error rate (7d), status histogram, cache hit % (Hit / Miss+RefreshHit), top 20 referrers, uniqExact(c_ip), top 20 IPs, top 20 x-edge-location POPs

Live check after logging was enabled: top paths populated; roughly 84% 200s already visible. That is a real number from production objects, not a dashboard screenshot of fixture data.

Outcome. Exit gate met: local Grafana answers "is CloudFront serving cleanly this week"; saved SQL answers "top paths and error rate, last 7 days"; both reproduce from a clean laptop with keys in .env. Hosting spend: $0.

Limits I will not dress up.

Phase 0b — Cloudflare Web Analytics (documented, implement in the apps)

Objective. Page views, referrers, and coarse Core Web Vitals from real browsers, on all four production hostnames, without renting a VPS.

This is the original napkin item I first treated as "still a third party, wait for Umami." That was the wrong gate. Visitor analytics is a script tag. Coupling it to Lightsail would have left a product question unanswered for a purity goal Phase 0 already broke (AWS has the IPs). ADR-003 records the reversal. The how-to lives in the hub repo: 02 - adding-cloudflare-web-analytics.md.

Decisions.

What you will be able to observe. Visits and page views for browsers that execute the snippet. Referrers, browser/OS, country. Page load (FP/FCP). Core Web Vitals with a debug view of offending elements. Quiz client-route changes, if spa: true is set.

Metrics.

SourceMetrics
Cloudflare Web Analyticsvisits, page views, referrers, duration, browser/OS/country, FP/FCP, LCP/INP/CLS (P75 + element debug)

Not in Grafana. There is no Web Analytics API worth wiring. This dashboard stays in Cloudflare. That is a real product cost of the $0 bridge.

Outcome (gate). One real production page view visible per hostname; zero from local.* / test.* / stage.*. Snippets are in the four apps and CI; the gate is unmet until production Environment variables CF_BEACON_TOKEN_* are set and a prod release ships.

Limits I will not dress up.

Phase 1 — VPS and platform foundation (not started)

Objective. Host the pieces that must be public and always-on: uptime from outside my network, later Grafana and Umami. Visitor analytics no longer waits on this box - that is Phase 0b.

Decisions (designed, not applied).

What you will be able to observe. Almost nothing new about the sites. This phase observes whether the platform can exist: TLS on a placeholder, Compose reproducible from a clone, SSH key-only, firewall tight. The exit gate is https://status.paulserban.eu (or equivalent) with a valid cert - a platform heartbeat, not a product dashboard.

Metrics. Instance up, Traefik ACME success, port policy. Visitor metrics already come from Phase 0b; this gate is still about the box, not about page views.

Limits. A public Grafana/status host is a new attack surface. Basic auth at Traefik is the v0 answer; Authelia is a later hardening phase I already wrote down and should not pretend is in Phase 1. Single AZ, no HA: if Lightsail in eu-central-1 dies, analytics and uptime die together. For a personal site that is acceptable; for a team SLO it would not be.

Phase 2 — Owned page analytics / Umami (not started)

Objective. Hold page-view rows ourselves, still without a cookie banner, across all four domains.

Decisions.

What you will be able to observe. Sessions and page views for browsers that execute the Umami snippet. Referrers as the client reported them. Per-site breakdowns. Still not bots that skip JS. Still not CloudFront cache hits. Still not 5xx that never reached HTML.

Metrics. Page views, unique visitors (Umami's definition, not uniqExact(c_ip)), top pages, referrers, bounce-ish rates, per-hostname series.

Limits. Client-side analytics under-counts privacy-hardened browsers. It will not replace Phase 0. Anyone who turns off CloudFront logs after Umami is live has misunderstood the architecture. Dual-running with Cloudflare without a kill date is how you collect two dashboards and trust neither.

Phase 3 — Owned RUM / Core Web Vitals (optional, default skip)

Objective. Know P75 LCP / CLS / INP from real visits in a system we own - only if Cloudflare's CWV view (Phase 0b) is not enough.

Default: skip or shrink. Dual-instrumenting CWV so Grafana can draw a line we already have in Cloudflare is vanity.

If revisited (Umami live, Cloudflare dropped, or we need owned histograms / non-Chromium / Grafana overlay):

What you will be able to observe (if this phase runs). Field CWV for visits that completed the beacon. Missing data from bounced/abandoned loads is structural. No server timing.

Metrics. P75 LCP, CLS, INP per hostname, 7-day trend, sample count. Exit gate is a 7-day trend on at least one domain outside the Cloudflare UI.

Limits. Umami custom events are a crude histogram. Cloudflare already has element-level debug. If I run this phase just to say "we have RUM in Grafana," I am lying about the requirement.

Phase 4 — Uptime / synthetic monitoring (not started)

Objective. Know within minutes if a hostname is down, checked from outside the laptop.

Decisions.

What you will be able to observe. Binary up/down, latency from one probe, SSL expiry if configured, alert delivery. Not "the page is slow for users in Brazil." Not "the quiz client route 404s" unless you keyword-check a specific path.

Metrics. Uptime %, probe latency, time-to-alert on a deliberate outage (the actual exit gate). If I cannot produce an alert by breaking a monitor on purpose, the phase is not done - dashboards without pages are decoration.

Limits. Putting the probe on the same VPS as Grafana means a dead VPS looks like "all sites down" and "monitoring down" at once. For personal ops that is acceptable. A lead signing off a paid SLO would put probes off-box. I am not signing off a paid SLO.

Phase 5 — Infra metrics and unified Grafana (not started)

Objective. Migrate Phase 0 dashboards onto the always-on VPS, add VPS health, one URL for "is the box healthy, is the CDN clean, are CWV in range."

Decisions.

What you will be able to observe. VPS CPU/mem/disk, Traefik request/latency, CloudFront health (same as Phase 0, now always on), CWV trend if Phase 3 landed. Still not a trace. Still not unique humans.

Metrics. node_*, Traefik entrypoint/router series, existing CloudWatch KPIs, optional SQL against Umami. Exit gate is one Grafana URL, not five bookmarks.

Limits. Always-on Grafana is now a public (or Traefik-auth'd) surface. Provisioned dashboards must travel with the repo; click-ops panels on the VPS will drift. Retention on Prometheus needs an explicit answer or the disk fills. I have not picked a remote_write/Grafana Cloud hybrid - and I should not "just add Grafana Cloud" to feel enterprise.

Phase 6 — Log aggregation and error tracking (optional, highest effort)

Objective. Centralize logs and capture client JS errors when something actually breaks.

Decisions.

What you will be able to observe. LogQL over CloudFront + Traefik from anywhere with Grafana access. Deliberate JS errors in GlitchTip within a minute (exit gate, if deployed). Still no distributed traces.

Metrics. Same access-log metrics as Phase 0, plus Traefik status/latency from logs, plus GlitchTip event rate / affected users if present.

Limits. Loki on a small VPS with CloudFront log volume is the first thing that will OOM if retention and label cardinality are sloppy. The Phase 0 label discipline (filename dropped, ip never a label) is load-bearing - it is not style. Skipping this phase is a valid architectural outcome, not a failure to finish the roadmap.

Design decisions that mattered

What shipped

Phase 0 is done. Phase 0b is decided and written down - snippets are not in the apps yet:

Phases 1–6 exist as a gated plan. Phase 3 is optional if Cloudflare CWV is enough. I would rather show a finished Phase 0, a written 0b, and a contract than a half-installed VPS.

Outcome

The platform has an observability contract that matches how the sites are actually built: static at the edge, logs in S3, metrics in CloudWatch, visitor beacons as a $0 Cloudflare bridge, questions split by what each signal can tell the truth about. Phase 0 de-risked the CDN half. Phase 0b refuses to wait on a VPS for "is anyone reading this."

What a lead should see is scope control: reject k8s and Prometheus-as-identity, split ownership from the serving stack, reverse a purity call when it delayed a real question (ADR-003), and write down what you will not observe. The remaining phases are sequenced so each one adds a question the previous phase cannot answer - not another logo on a slide.

Part of prj--personal-portfolio--v3

This hub is the telemetry counterpart to the monorepo's serving and delivery seams: