Problem
prj--personal-portfolio--v3 is four static surfaces behind CloudFront - portfolio, blog, quiz, news-feed - not a process you can scrape. There is no app server, no /metrics, no request log on a box you SSH into. "Is it healthy?" lived in the AWS console. "Who is reading the blog?" defaulted to a third-party JS tracker and a cookie banner. "Did the quiz PWA just 5xx at the edge?" was a guess until someone emailed.
That is not an observability gap you close with Datadog because you have used Datadog at work. A personal JAMstack platform has a different failure mode: almost no runtime, almost no budget, and a privacy posture you actually have to mean. Prometheus against S3 is theatre. Google Analytics on a site that argues for craft is a contradiction. Kubernetes on a single VPS is resume-driven architecture.
I needed a platform contract for telemetry: which questions each signal can answer, which it cannot, what ships before any hosting invoice, and how a fifth project joins later without a second stack.
Solution
prj--observability-hub is a separate repo on purpose. The portfolio Terraform owns how sites are served (distributions, private buckets, logging_config). This repo owns how those sites are observed - identity, local tooling, and later the always-on VPS. That split is the same seam as content-vs-app in the content pipeline: ownership follows change cadence, not a single "infra" folder.
The stack is phased with exit gates. Phase 0 runs entirely on the laptop against production CloudFront data. Phase 0b is a JS beacon in the four apps - still $0 hosting, still no VPS. Phases 1–6 are designed, spiked, and ADR'd; they are not shipped. Shipping a VPS before the CDN questions are answerable - or before a cookie-less visitor signal exists - would be spending to feel busy.
Target shape, once the gates are met:
| Concern | Tool | When it exists |
|---|---|---|
| Reverse proxy / TLS | Traefik | Phase 1 (VPS) |
| Page analytics (bridge) | Cloudflare Web Analytics | Phase 0b (prod snippet, no VPS) |
| Page analytics (owned) | Umami + Postgres | Phase 2 |
| Core Web Vitals | Cloudflare beacon first; web-vitals → Umami only if needed | Phase 0b; Phase 3 optional |
| Uptime | Uptime Kuma | Phase 4 |
| Infra metrics | Prometheus + node_exporter | Phase 5 |
| CDN metrics | Grafana CloudWatch DS (YACE optional) | Phase 0 local → Phase 5 VPS |
| Access-log SQL | clickhouse-local | Phase 0 (CLI, stays CLI) |
| Access-log UI | Grafana Loki + Alloy | Phase 0 local (Phase 6 practiced early) |
| Error tracking | GlitchTip | Phase 6, only if islands justify it |
Scope and features
In scope
- Four production hostnames:
paulserban.eu,blog.paulserban.eu,quiz.paulserban.eu,news-feed.paulserban.eu - CDN health without living in the AWS console (requests, 4xx/5xx, bytes, cache behaviour)
- Path-level traffic, referrers, status breakdown, and forensic access-log search
- Privacy-friendly page analytics across all surfaces without a cookie banner - Cloudflare Web Analytics now (Phase 0b), Umami later if we want to own the rows
- Real-user Core Web Vitals, not only Lighthouse on a laptop
- Uptime checks that are not "curl from the same machine that is down"
- One Grafana as the eventual single pane; domain-agnostic so a fifth project is a site entry + a monitor, not a new platform
- Cost and complexity bounded to a single-node Compose stack
Out of scope (named, not forgotten)
- Distributed tracing / Grafana Tempo - static SSG has no server spans; the quiz PWA is not a mesh
- Real-time CloudFront logs via Kinesis - priced for a problem I do not have
- Multi-node HA, k3s, EKS - rejected in ADR-001
- Athena + Glue as the Phase 0 query path - superseded by ADR-002
- Selling this as a product or a team SRE platform - it is personal-scale, solo-operated, and should read that way
Honest privacy note: "privacy-friendly" here means no cookie consent theatre, not "no third party." Phase 0b sends beacons to Cloudflare. CloudFront standard logs still contain client IPs. Loki stores them in the log line (never as labels). ClickHouse can uniqExact(c_ip). That is CDN forensics plus a vendor dashboard, not "we have no identifiers." Pretending otherwise would be the kind of product copy I would reject in a design review. Umami (Phase 2) is the path to hold the event rows ourselves - it does not erase the Phase 0b choice.
Architecture
Two diagrams matter: the target platform, and what is actually running today.
Target (Phases 0-6)
Visitors
│
▼
Route 53 → CloudFront x 4 → private S3 x 4
│ │
│ └── site objects (SSG / quiz SPA)
├── CloudWatch metrics (AWS/CloudFront, Region=Global)
└── standard access logs → s3://cf-access-logs.paulserban.eu
│
┌──────────────────────┼──────────────────────┐
▼ ▼ ▼
Grafana CloudWatch Alloy → Loki clickhouse-local
(health KPIs) (Grafana log UI) (ad-hoc SQL CLI)
Browser JS (Phase 0b, prod only)
beacon.min.js → Cloudflare Web Analytics
(page views, referrers, LCP/INP/CLS — not in Grafana, no API)
VPS (not started) — Traefik TLS
analytics. → Umami + Postgres (owned page views; later optional CWV)
status. → Uptime Kuma (HTTP checks from outside the laptop)
grafana. → Grafana (CloudWatch + Prometheus + Loki + Umami)
errors. → GlitchTip (optional, Phase 6)
Prometheus + node_exporter + Traefik metrics
Phase 0 as-built (shipped)
Laptop (127.0.0.1 only)
grafana :3000
├── CloudWatch DS (IAM keys, defaultRegion us-east-1)
└── Loki DS (compose network only)
loki (filesystem, ~14d retention, no auth)
alloy (parse CloudFront TSV .gz → Loki)
cf-log-sync (aws s3 sync every 300s)
AWS
CloudFront x 4 → CloudWatch
→ s3://cf-access-logs.paulserban.eu (60-day lifecycle)
IAM user observability-hub-readonly
CloudWatch GetMetric* + CloudFront List/Get + S3 GetObject on the log bucket
Not in Compose
docker run clickhouse/clickhouse-server clickhouse-local
s3() table function → saved SQL in infrastructure/local/clickhouse/queries/
Ownership is a first-class architecture decision, not a README footnote:
| Repo | Owns |
|---|---|
prj--personal-portfolio--v3 | Distributions, OAC, the shared log bucket, logging_config on site/blog/quiz/news |
prj--observability-hub | Read-only IAM, local Compose, Grafana provisioning, Alloy parse pipeline, ClickHouse query pack, later VPS Compose |
A clone of the hub repo cannot enable logging. That is correct: logging is a property of the edge, not of the dashboard.
Approach
As architect and lead implementer I treated observability the way I treat hosting and CI: seams first, spend second, orchestration last.
Platform questions before tool shopping
The original notes were a tool list (Umami vs Plausible vs Cloudflare Web Analytics, UptimeRobot vs Kuma, Sentry vs GlitchTip). That is how you accidentally buy four SaaS free tiers that do not compose. I inverted it: what questions must this platform answer, then which signal is allowed to answer them.
| Question | Honest signal | Wrong signal |
|---|---|---|
| Is CloudFront serving cleanly this week? | CloudWatch request/error/byte metrics | Page-view JS (bots, cache, and 5xx at the edge are invisible or lying) |
| Which paths 404, miss cache, or dominate bytes? | Access logs (Loki + ClickHouse) | CloudWatch (no URI dimension on standard CF metrics) |
| Are unique IPs / edge POPs moving? | clickhouse-local (uniqExact) | Loki labels (cardinality bomb) |
| Who is reading the blog as a person? | Cloudflare Web Analytics (Phase 0b); Umami later (Phase 2) | Access-log IPs (NAT, crawlers, shared egress) |
| What LCP did a real phone see? | Cloudflare CWV (Phase 0b); owned web-vitals only if needed | Lighthouse on my laptop |
| Is the apex down from the public internet? | Uptime Kuma on a VPS (Phase 4) | Grafana on 127.0.0.1 |
| Is the VPS itself dying? | Prometheus + node_exporter (Phase 5) | CloudFront metrics |
If a tool cannot be placed on that table, it does not enter the compose file.
Validate before you rent
Phase 0 is the design spike the console-first hosting case study already argued for: prove the contract on real traffic before freezing spend. CloudFront logging is pennies of S3. CloudWatch basic metrics are free. Docker on the laptop is free. A VPS is not free, and Lightsail's $5 bundle will not hold this stack anyway.
Right-size the runtime
Spikes killed three fashionable defaults:
- Prometheus as the first tool - pull-based metrics need a scrape target. A static origin has none. Grafana's native CloudWatch data source answers CDN health without an exporter. Prometheus earns its keep in Phase 5 for the VPS and Traefik, not for the sites.
- Kubernetes - one node, one replica of everything. k3s would tax ~700MB-1GB before a container starts and roughly double VPS cost. ADR-001 accepts Compose and names the revisit trigger (actual multi-node need, or a deliberate homelab/content goal - not "it looks good on a CV").
- Athena - pay-per-query and a Glue catalog for a volume I can scan from the laptop. ADR-002 picks
clickhouse-localvia Docker; no binary install, no catalog, transferable SQL.
Traefik over Caddy is the other proxy call: label-based Docker routing, no shared Caddyfile to edit per phase, and it matches the local Traefik mesh already used for the four apps. Consistency of operations beats a slightly simpler ACME story.
Two query paths on purpose
Grafana does not speak ClickHouse. Interactive access-log dashboards are Loki. Exact unique IPs, top IPs, and edge POPs are ClickHouse CLI. That looks like indecision. It is cardinality and operational weight:
- Loki labels are only
job,site,status,result_type. Path, IP, UA, referrer stay in the log line (logfmt). Promotingipto a label would explode the index the first week a crawler showed up. - A persistent ClickHouse server on a 2-4GB VPS duplicates Loki and eats RAM (~1GB baseline) for no new Grafana panel I need in Phase 0.
- User-Agent is left out of Alloy's logfmt (spaces and
=break naive parsing). UA analysis is ClickHouse or it does not happen. That is a documented hole, not a surprise.
Stack (Phase 0, real)
| Layer | Choices |
|---|---|
| Identity | Terraform IAM user, least-privilege, local state, keys in gitignored .env |
| Metrics UI | Grafana 11, provisioned CloudWatch DS (us-east-1 / Global) |
| Log UI | Loki 3 + Alloy 1.7, CloudFront TSV parse, ~14 day filesystem retention |
| Log ship | aws s3 sync loop every 5 minutes - not Kinesis, not S3 notifications |
| SQL | clickhouse-local 24.12, s3() + saved queries, PrettyCompact CLI |
| Bind | 127.0.0.1:3000 only - this is not a shared team Grafana |
Phased delivery
Each phase has an objective, decisions, an observation contract, extractable metrics, an outcome, and the limits I would put in a design review. Do not start the next VPS phase until the current gate is met. Phase 0 has met its gate. Phase 0b is the next $0 slice - it does not wait on Phase 1.
Phase 0 — Local AWS log aggregation (done)
Objective. Prove CDN/traffic visibility from data AWS already has, on one machine, before renting anything.
What shipped. CloudFront standard logging on all four prod distributions into cf-access-logs.paulserban.eu (60-day lifecycle). Read-only IAM via Terraform. Local Compose: Grafana + Loki + Alloy + cf-log-sync. Two provisioned dashboards. Seven saved ClickHouse queries behind ./scripts/query.sh. Makefile targets matching the portfolio repo's compose_up habit.
Decisions.
- Native CloudWatch Grafana DS, not YACE. Phase 0 has no Prometheus. An exporter would be a service to run so that I could scrape a translation of an API Grafana already speaks.
- Loki practiced early. The written plan parked logs in Phase 6. Grafana still needed a log UI once Athena was rejected and ClickHouse was CLI-only. Pulling Phase 6's ship path forward locally is non-linear on the Gantt chart and correct on the architecture.
- Poll S3, do not stream.
cf-log-syncis a shellwhile true; aws s3 sync; sleep 300. It is crude. It is also the right cost model: standard logs already lag minutes to hours; a 5-minute sync does not make them worse in a way a visitor would notice. - IAM user + access keys on the laptop. Fine for a bind-to-localhost stack. The wrong long-term pattern on a VPS (Lightsail instance profiles are weak; this will get revisited in Phase 1, not papered over).
- Local Terraform state. One operator, one laptop. Same named trade-off as the CI/CD case study.
What you can actually observe.
- Whether each distribution is taking traffic, erroring, or shipping bytes - coarse, per-distribution, not per-path
- Cache hit ratio and result types (
Hit/Miss/Error/RefreshHit/ …) from logs, which CloudWatch will not give you at URI granularity - Top content vs asset paths, bytes by path, internal vs external referrer hosts, direct vs referred
- 404s over time, top client IPs in Grafana (from logfmt, high-cardinality but not labelled)
- Exact unique IPs, top IPs with path diversity, edge POP codes - CLI only
Metrics you can extract today.
| Source | Metrics |
|---|---|
CloudWatch AWS/CloudFront | Requests, 4xxErrorRate, 5xxErrorRate, BytesDownloaded (all four distributions; Region=Global) |
| Loki / access logs | volume, status, result_type, cache hit %, top error paths, top cache-miss paths, top content/asset paths, bytes by path, referrer hosts, direct vs referred, 404 timeseries, sample client IPs, raw log search |
| clickhouse-local | top 10 stems + error rate (7d), status histogram, cache hit % (Hit / Miss+RefreshHit), top 20 referrers, uniqExact(c_ip), top 20 IPs, top 20 x-edge-location POPs |
Live check after logging was enabled: top paths populated; roughly 84% 200s already visible. That is a real number from production objects, not a dashboard screenshot of fixture data.
Outcome. Exit gate met: local Grafana answers "is CloudFront serving cleanly this week"; saved SQL answers "top paths and error rate, last 7 days"; both reproduce from a clean laptop with keys in .env. Hosting spend: $0.
Limits I will not dress up.
- This is not real-time. Standard access logs lag. The sync adds up to five more minutes. If I needed seconds, I would pay for real-time logs - I do not.
- Unique IPs are not unique humans. Carrier NAT, IPv6 privacy addresses, office egress, and bots all inflate or collapse that count. Treat it as a CDN forensic, not a marketing KPI.
- CloudWatch cannot tell you
/portfolio/prj--…/is hot. If the health dashboard looks green and a slug is 404ing, you need the traffic dashboard. That split is the product. - Grafana is
admin/adminin.env.example, no SSO, no alerting, bound to localhost. You still have to look. Phase 0 does not page anyone. - Alloy drops CloudFront
#Version/#Fieldsheaders and does not index UA. Incomplete parse is a choice; it is also a footgun if someone greps Loki for browsers. - I practiced Phase 6 before Phase 1. The plan document is slightly behind the repo. The ADRs and README are the source of truth - that is how this should work, and it is also drift I have to maintain.
Phase 0b — Cloudflare Web Analytics (documented, implement in the apps)
Objective. Page views, referrers, and coarse Core Web Vitals from real browsers, on all four production hostnames, without renting a VPS.
This is the original napkin item I first treated as "still a third party, wait for Umami." That was the wrong gate. Visitor analytics is a script tag. Coupling it to Lightsail would have left a product question unanswered for a purity goal Phase 0 already broke (AWS has the IPs). ADR-003 records the reversal. The how-to lives in the hub repo: 02 - adding-cloudflare-web-analytics.md.
Decisions.
- Manual snippet, not orange-cloud auto-inject. The sites are CloudFront. Cloudflare cannot rewrite HTML at the edge. Pretending otherwise is a silent no-op.
- One token per hostname. Same isolation as one distribution per surface. Sharing a token would mix portfolio and quiz in one dashboard.
- Production only. Tokens as GitHub Environment variables on
production, passed into_build-site.yaml. Local Traefik, DEV Pages, TEST, STAGE stay unset so fixture and rehearsal traffic never land in the series. - Quiz gets
spa: true. TanStack Router uses the History API. Hash routing is unsupported by the beacon - do not switch. - Umami is not cancelled. Phase 2 is still how we own rows. Cloudflare is the bridge. After a week of overlap, keep or drop - that is a revisit trigger, not a forever vendor.
- Phase 3 shrinks. The beacon already reports LCP / INP / CLS (Chromium-first, with element debug). A second
web-vitalspipeline is dual-instrumentation until Cloudflare's view is insufficient.
What you will be able to observe. Visits and page views for browsers that execute the snippet. Referrers, browser/OS, country. Page load (FP/FCP). Core Web Vitals with a debug view of offending elements. Quiz client-route changes, if spa: true is set.
Metrics.
| Source | Metrics |
|---|---|
| Cloudflare Web Analytics | visits, page views, referrers, duration, browser/OS/country, FP/FCP, LCP/INP/CLS (P75 + element debug) |
Not in Grafana. There is no Web Analytics API worth wiring. This dashboard stays in Cloudflare. That is a real product cost of the $0 bridge.
Outcome (gate). One real production page view visible per hostname; zero from local.* / test.* / stage.*. Snippets are in the four apps and CI; the gate is unmet until production Environment variables CF_BEACON_TOKEN_* are set and a prod release ships.
Limits I will not dress up.
- Data leaves the AWS account. Cookie-less is not "no third party."
- Under-counts privacy-hardened browsers and anything that never ran JS. Will disagree with access logs (assets, bots, 5xx). If the two series match, the pipeline is wrong.
- CWV is Chromium-first. Safari/Firefox are incomplete.
- Tokens are in public HTML. Treat them as identifiers, not secrets, but still do not commit them.
- If a CSP is added at CloudFront later, allowlist
static.cloudflareinsights.comandcloudflareinsights.comor the beacon dies silently. - This does not replace Phase 0. Turn off CloudFront logging after the beacon and you have lost CDN truth.
Phase 1 — VPS and platform foundation (not started)
Objective. Host the pieces that must be public and always-on: uptime from outside my network, later Grafana and Umami. Visitor analytics no longer waits on this box - that is Phase 0b.
Decisions (designed, not applied).
- Lightsail over Hetzner/Contabo/Oracle free as the default, because the rest of the platform is already AWS: one bill, no cross-account IAM, Terraform
aws_lightsail_*in the same provider. Contabo wins raw €/GB. Oracle Always Free is capacity-constrained and ARM-only. This is account cohesion, not cheapest-possible. - Do not buy the $5 Lightsail bundle. The plan table's "$5-10" line is a trap. Umami+Postgres+Grafana+Loki needs ~2-3GB; budget 2GB ($12) or 4GB ($24). Under-sizing this is how you learn about swap thrashing instead of telemetry.
- Terraform owns the box, Compose owns the processes. Instance, static IP (Lightsail public IP changes on stop/start), ports 22/80/443, Route 53 records. Containers stay in git. Managing Grafana as a Terraform resource would be the wrong abstraction.
- DNS ownership must be picked once.
analytics./status./grafana./errors.A records either live in hub Terraform (passhosted_zone_id) or stay in the portfolio prod stack. Two roots fighting overpaulserban.euis how you get a Saturday outage. - Lightsail IAM is weaker than EC2 instance profiles. Phase 0's access keys on a VPS is the lazy path. Call that out at apply time; do not discover it after the box is public.
What you will be able to observe. Almost nothing new about the sites. This phase observes whether the platform can exist: TLS on a placeholder, Compose reproducible from a clone, SSH key-only, firewall tight. The exit gate is https://status.paulserban.eu (or equivalent) with a valid cert - a platform heartbeat, not a product dashboard.
Metrics. Instance up, Traefik ACME success, port policy. Visitor metrics already come from Phase 0b; this gate is still about the box, not about page views.
Limits. A public Grafana/status host is a new attack surface. Basic auth at Traefik is the v0 answer; Authelia is a later hardening phase I already wrote down and should not pretend is in Phase 1. Single AZ, no HA: if Lightsail in eu-central-1 dies, analytics and uptime die together. For a personal site that is acceptable; for a team SLO it would not be.
Phase 2 — Owned page analytics / Umami (not started)
Objective. Hold page-view rows ourselves, still without a cookie banner, across all four domains.
Decisions.
- Umami + Postgres, one instance, one site entry per hostname. Cloudflare Web Analytics is already the third-party bridge (Phase 0b). Umami is data ownership, not "first analytics." Plausible hosted is $9/mo for a problem Compose already solves.
- Same layout include points as the Cloudflare beacon - the three
BaseTemplate.astrofiles and quizmain.tsx. There is no shared layout package; consistency is a checklist, not a magic shared component. - Postgres now, ClickHouse later. Umami can swap the events table to ClickHouse at higher volume. Personal traffic does not need that.
- JS still runs in the browser. Cookie-less is not zero-cost. Two snippets (Cloudflare + Umami) is worse. After ~7 days of overlap, keep or drop Cloudflare (ADR-003 revisit). Do not run both forever out of cowardice.
- Expect disagreement. Umami vs Cloudflare vs access logs are three definitions of "a visit." Matching 1:1 means the pipeline is wrong.
What you will be able to observe. Sessions and page views for browsers that execute the Umami snippet. Referrers as the client reported them. Per-site breakdowns. Still not bots that skip JS. Still not CloudFront cache hits. Still not 5xx that never reached HTML.
Metrics. Page views, unique visitors (Umami's definition, not uniqExact(c_ip)), top pages, referrers, bounce-ish rates, per-hostname series.
Limits. Client-side analytics under-counts privacy-hardened browsers. It will not replace Phase 0. Anyone who turns off CloudFront logs after Umami is live has misunderstood the architecture. Dual-running with Cloudflare without a kill date is how you collect two dashboards and trust neither.
Phase 3 — Owned RUM / Core Web Vitals (optional, default skip)
Objective. Know P75 LCP / CLS / INP from real visits in a system we own - only if Cloudflare's CWV view (Phase 0b) is not enough.
Default: skip or shrink. Dual-instrumenting CWV so Grafana can draw a line we already have in Cloudflare is vanity.
If revisited (Umami live, Cloudflare dropped, or we need owned histograms / non-Chromium / Grafana overlay):
web-vitals+navigator.sendBeacon, ingested as Umami custom events - simplest path, no new service.- Datadog RUM stays skipped. Cloudflare is the cheap RUM we actually turned on. Datadog would be the k8s mistake again.
- Lab scores stay in CI if at all. Field vs lab are different questions.
What you will be able to observe (if this phase runs). Field CWV for visits that completed the beacon. Missing data from bounced/abandoned loads is structural. No server timing.
Metrics. P75 LCP, CLS, INP per hostname, 7-day trend, sample count. Exit gate is a 7-day trend on at least one domain outside the Cloudflare UI.
Limits. Umami custom events are a crude histogram. Cloudflare already has element-level debug. If I run this phase just to say "we have RUM in Grafana," I am lying about the requirement.
Phase 4 — Uptime / synthetic monitoring (not started)
Objective. Know within minutes if a hostname is down, checked from outside the laptop.
Decisions.
- Uptime Kuma on the VPS, HTTP (+ optional keyword) monitors for all four public URLs, alert to Telegram/Discord/email.
- Skip Route 53 health checks. They cost money to duplicate a container I am already running.
- One vantage point. This is "up from the VPS region" (planned
eu-central-1), not Catchpoint. CloudFront can be fine in Frankfurt and broken at a single POP. Kuma will not see that. Access-log POP queries and CloudWatch 5xx will. Use both; do not pretend Kuma is global synthetics.
What you will be able to observe. Binary up/down, latency from one probe, SSL expiry if configured, alert delivery. Not "the page is slow for users in Brazil." Not "the quiz client route 404s" unless you keyword-check a specific path.
Metrics. Uptime %, probe latency, time-to-alert on a deliberate outage (the actual exit gate). If I cannot produce an alert by breaking a monitor on purpose, the phase is not done - dashboards without pages are decoration.
Limits. Putting the probe on the same VPS as Grafana means a dead VPS looks like "all sites down" and "monitoring down" at once. For personal ops that is acceptable. A lead signing off a paid SLO would put probes off-box. I am not signing off a paid SLO.
Phase 5 — Infra metrics and unified Grafana (not started)
Objective. Migrate Phase 0 dashboards onto the always-on VPS, add VPS health, one URL for "is the box healthy, is the CDN clean, are CWV in range."
Decisions.
- Prometheus + node_exporter for the VPS. This is the first time Prometheus has a legitimate scrape target. Traefik's metrics endpoint joins it. That is the opposite of Phase 0's "do not invent exporters for static files."
- Keep Grafana's CloudWatch DS unless PromQL unification is actually painful. YACE is optional. Phase 0 already proved native CloudWatch works. Adding YACE so that every panel is PromQL is aesthetic unless I am alerting from Prom rules across CDN + node.
- Umami Postgres as a Grafana data source for CWV / traffic overlays - possible, not mandatory. Do not build a second analytics UI in Grafana if Umami's own UI is enough.
What you will be able to observe. VPS CPU/mem/disk, Traefik request/latency, CloudFront health (same as Phase 0, now always on), CWV trend if Phase 3 landed. Still not a trace. Still not unique humans.
Metrics. node_*, Traefik entrypoint/router series, existing CloudWatch KPIs, optional SQL against Umami. Exit gate is one Grafana URL, not five bookmarks.
Limits. Always-on Grafana is now a public (or Traefik-auth'd) surface. Provisioned dashboards must travel with the repo; click-ops panels on the VPS will drift. Retention on Prometheus needs an explicit answer or the disk fills. I have not picked a remote_write/Grafana Cloud hybrid - and I should not "just add Grafana Cloud" to feel enterprise.
Phase 6 — Log aggregation and error tracking (optional, highest effort)
Objective. Centralize logs and capture client JS errors when something actually breaks.
Decisions.
- Most likely phase to skip. Phases 0-5 already cover traffic, RUM, uptime, infra. Build this on the VPS only if log-level debugging is a real pain.
- The parse pipeline is already written. Alloy config, Loki labels, dashboards - Phase 0 did the hard part locally. VPS work is shipping + Traefik JSON access logs, not invention.
- GlitchTip only if Astro islands / the quiz PWA throw enough client errors to care. A static blog with almost no JS does not need a Sentry-compatible service. The quiz PWA is the plausible consumer. One DSN per app, not a single dumped bucket.
What you will be able to observe. LogQL over CloudFront + Traefik from anywhere with Grafana access. Deliberate JS errors in GlitchTip within a minute (exit gate, if deployed). Still no distributed traces.
Metrics. Same access-log metrics as Phase 0, plus Traefik status/latency from logs, plus GlitchTip event rate / affected users if present.
Limits. Loki on a small VPS with CloudFront log volume is the first thing that will OOM if retention and label cardinality are sloppy. The Phase 0 label discipline (filename dropped, ip never a label) is load-bearing - it is not style. Skipping this phase is a valid architectural outcome, not a failure to finish the roadmap.
Design decisions that mattered
- Gates over a big-bang compose file. A 12-service
docker-compose.ymlon day one would have hidden that CloudWatch already answered half the CDN questions. - Compose over k3s. Single-node, single-replica, cost-sensitive. ADR-001 is the lead-engineer artifact: alternatives table, resource tax, revisit trigger.
- clickhouse-local over Athena. Same $0 constraint, less AWS surface, real SQL, CLI-shaped. ADR-002. Grafana stays on Loki.
- Two repos, two owners. Logging config is an edge concern; dashboards are an observability concern. Crossing that line is how you get circular Terraform and "just apply both."
- Cardinality as a first-class constraint. Loki labels bounded; high-cardinality forensics in ClickHouse. This is the same instinct as not putting user-id on Prometheus labels at work - applied at personal scale on purpose.
- Traefik for label routing, matching local app-dev, so Phase 1 does not invent a second proxy story.
- Cloudflare Web Analytics now, Umami later. Visitor analytics is a script tag. Waiting for a VPS was the wrong gate. ADR-003. Tokens prod-only; quiz
spa: true; no Grafana API. - Trade-offs named up front. Long-lived IAM keys on the laptop; local TF state; log lag; no HA; no traces; Phase 6 optional; Lightsail IAM ugliness deferred to Phase 1; Cloudflare as a third party for $0 visitor/RUM; Umami JS cost vs ownership; one-probe uptime. A case study that hides those is a brochure.
What shipped
Phase 0 is done. Phase 0b is decided and written down - snippets are not in the apps yet:
- CloudFront access logging on four prod distributions into a shared bucket (in the portfolio Terraform, 60-day lifecycle)
- Terraform IAM user
observability-hub-readonly(CloudWatch metrics, CloudFront list, S3 log-bucket read) - Local Compose: Grafana 11, Loki 3, Alloy,
cf-log-sync - Dashboards: CloudFront health (CloudWatch KPIs + Loki cache/error/miss ops) and CloudFront traffic (paths, referrers, forensics)
- clickhouse-local query pack:
top-paths,status-breakdown,cache-hit-ratio,referrers,unique-ips,top-ips,edge-pops - ADRs for orchestration, query engine, and Cloudflare Web Analytics; spikes that killed k8s-first, Prometheus-first, and Athena-first
- How-to for Phase 0b: four tokens, layout insertion points, prod-only CI, quiz SPA flag
make compose_up/compose_down/ logs - operable, not a pile of undocumented containers
Phases 1–6 exist as a gated plan. Phase 3 is optional if Cloudflare CWV is enough. I would rather show a finished Phase 0, a written 0b, and a contract than a half-installed VPS.
Outcome
The platform has an observability contract that matches how the sites are actually built: static at the edge, logs in S3, metrics in CloudWatch, visitor beacons as a $0 Cloudflare bridge, questions split by what each signal can tell the truth about. Phase 0 de-risked the CDN half. Phase 0b refuses to wait on a VPS for "is anyone reading this."
What a lead should see is scope control: reject k8s and Prometheus-as-identity, split ownership from the serving stack, reverse a purity call when it delayed a real question (ADR-003), and write down what you will not observe. The remaining phases are sequenced so each one adds a question the previous phase cannot answer - not another logo on a slide.
Part of prj--personal-portfolio--v3
This hub is the telemetry counterpart to the monorepo's serving and delivery seams:
- AWS multi-subdomain hosting - the CloudFront x 4 topology whose logs and metrics this stack reads
- Multi-env CI/CD & Terraform delivery - prod is where logging is enabled; other envs are not the observability target
- Content pipeline SSG - four surfaces from one artifact; this stack watches those four hostnames
- Local HTTPS Traefik + Docker Compose - the proxy pattern Phase 1 reuses on the VPS
- Spaced-repetition quiz PWA - the SPA most likely to justify GlitchTip later