How long did this engagement take, from audit to a live-tested result?

The full engagement ran about 3 months, covering the caching and autoscaling rebuild, dashboard and SLO work, and the Locust-based load testing. Rehearsal broadcasts scheduled ahead of the flagship event served as the final validation step, so the team went into the real night with results from a full-scale dry run rather than theory.

Does this approach still work if a team isn't using Kubernetes or the exact CDN stack shown here (CloudFront and Fastly)?

Yes — the core techniques (custom-metric autoscaling instead of CPU-only, standardized cache-key composition with stale-while-revalidate, and correlated CDN/origin/player telemetry) are patterns, not vendor lock-in. Any autoscaler that accepts external metrics and any CDN that supports cache-key rules and edge logging can implement the same approach, just with different configuration syntax.

What was the biggest risk during the migration to custom-metric autoscaling?

The main risk was replacing a working, if imprecise, CPU-based trigger before the new queue-depth and P99-latency signals were proven reliable. We mitigated this by running both systems in parallel during rehearsal events, comparing scaling decisions side by side before cutting over fully, so a bad metric definition would surface in a low-stakes rehearsal rather than during a flagship broadcast.

Would this be overkill for a smaller streaming team without flagship-scale traffic spikes?

The full program (unified dashboards, error budgets, dedicated rehearsal events) is sized for teams that face real correlated spikes, but the individual pieces scale down well. Even a small team can adopt standardized cache keys and one or two custom autoscaling metrics without needing the entire observability and rehearsal apparatus.

Back to case studies

Media & EntertainmentCloud InfrastructureDevOps Consulting

Live Events & Streaming Scale

Surviving traffic spikes for global broadcasts with resilient edge and observability — turning flagship live events from a source of dread into a rehearsed, measured, and repeatable operation.

Stable streams and predictable autoscaling under peak load

Client

Confidential — media & entertainment

Industry

Media & Entertainment

Timeline

3 months

Technologies

7+ tools

The Challenge

!Every flagship broadcast produced a terrifying traffic curve: viewer counts could triple within a two-minute pre-show window as social promotion and push notifications drove synchronized arrivals, and the origin APIs serving manifests and session tokens had no way to absorb that kind of correlated spike without shedding requests.

!Cache behavior differed wildly from one CDN region to another because cache-key rules had grown organically over several years of feature additions, so a viewer in one region might get a fresh manifest in under a second while a viewer in another region triggered an origin fetch storm during the exact same minute.

!The team had almost no real-time signal for what viewers were actually experiencing — rebuffering, start-time delay, and bitrate drops were invisible until support tickets and social media complaints arrived, by which point the live moment causing them was already over.

!Autoscaling was tied almost entirely to CPU utilization on origin and API pods, which lagged badly behind the actual demand signal; by the time CPU climbed enough to trigger new capacity, the spike had often already caused latency and error-rate damage that could not be undone.

!Post-event post-mortems routinely stalled because nobody could say with confidence whether a given slowdown originated at the CDN edge, the origin cluster, or the player itself, so engineering time was wasted debating blame instead of fixing root cause.

!Encore and rebroadcast traffic patterns — viewers catching up on a missed live segment right after it ended — were not modeled separately from live traffic, so capacity planning consistently underestimated the secondary wave that followed the primary spike.

!There was no standing playbook for graceful degradation, so when an upstream dependency (ad decisioning, DRM licensing, or authentication) slowed down, the whole streaming experience degraded uniformly instead of shedding non-critical features first.

!Load testing, where it existed, used steady ramping traffic patterns that bore no resemblance to the bursty, geographically clustered fan-out of a real global broadcast, so tests routinely passed while real events still produced incidents.

Our Solution

We audited and rebuilt the multi-layer caching strategy from the CDN edge down to the origin shield layer, standardizing cache-key composition and introducing stale-while-revalidate so that a burst of concurrent requests for the same manifest triggered a single origin fetch instead of a stampede.

We replaced CPU-only autoscaling with policies driven by custom metrics — queue depth, request rate, and P99 latency — feeding Kubernetes HPA and cluster autoscaler decisions so new capacity came online ahead of the demand curve rather than behind it.

We built unified dashboards in Grafana that correlated CDN edge metrics, origin API latency, and client-side player telemetry on a single timeline, so an on-call engineer could see within seconds whether a spike was a CDN cache-miss problem, an origin saturation problem, or a player-side issue.

We defined explicit error budgets and SLOs for start-time and rebuffering ratio, and wired burn-rate alerts so the team got early warning of degradation trends minutes before viewers would notice, not after.

We designed and ran Locust-based load tests that specifically modeled regional fan-out patterns and encore/rebroadcast waves, replaying realistic arrival curves from prior broadcasts rather than generic linear ramps.

We used rehearsal events — smaller live broadcasts scheduled ahead of flagship nights — as full-scale fire drills, validating that autoscaling, caching, and dashboards behaved correctly under real (if smaller) production load before the moment that mattered.

We wrote runbooks covering brownout scenarios, CDN or origin failover procedures, and graceful degradation paths that shed non-essential features (personalization, certain ad formats) first while protecting core video delivery.

We codified all networking, caching, and autoscaling configuration in Terraform so that the same tuned, tested topology could be reproduced identically across environments and quickly rolled back if a change introduced regression.

Measurable Impact

Peak capacity

3x traffic surge absorbed with zero shedding

Rehearsal events at realistic scale validated headroom for a tripling of viewers within a two-minute window, well ahead of flagship broadcasts, removing the guesswork from capacity planning.

Incident clarity

Root-cause identification cut from hours to under 10 minutes

Mean time to identify root cause dropped sharply once CDN, origin, and player telemetry lived on one correlated timeline.

Viewer experience

Rebuffering ratio held under 1% during peak spikes

Rebuffering ratio and start-time metrics stayed within target SLOs even during the highest-traffic minutes of live spikes.

Cache efficiency

Origin fetch storms cut by roughly 80%

Standardized cache keys and stale-while-revalidate eliminated most origin fetch storms triggered by concurrent viewer arrivals.

Cost

20-25% lower origin infrastructure spend

Rightsized origin fleets and smarter TTLs reduced over-provisioning without sacrificing headroom for spikes.

Team confidence

Zero live-event P1 incidents across 3 flagship broadcasts post-rollout

On-call engineers moved from reactive firefighting to running rehearsed, documented playbooks during live events.

“Our biggest night of the year used to be a coin flip. This time we knew exactly what healthy looked like, we had rehearsed the failure modes in advance, and we had real levers to pull when the internet misbehaved. That shift in confidence changed how the whole team approaches live events now.”
V
VP Engineering
VP Engineering, streaming platform (NDA)

Technology stack

AWS CloudFrontFastlyKubernetesPrometheusGrafanaLocustTerraform