Observability rebuild

Placeholder content — replace with the real write-up.

The problem

Incidents took too long to diagnose. Every service logged in its own format, nothing carried a request ID across boundaries, and the only reliable debugging tool was someone who happened to remember how a subsystem behaved under load.

What I built

A single instrumentation library that every service adopts, emitting structured events with a shared trace context. The events land in a columnar store, which makes the “show me every slow request that touched this dependency” question answerable in seconds rather than an afternoon.

The rollout was deliberately incremental: the library shipped as a no-op wrapper first, so teams could adopt it without behaviour change, then instrumentation was switched on service by service.

What I’d do differently

I under-invested in sampling early on and paid for it in storage costs for two quarters. Deciding what not to record turned out to be the more important design question.