Skip to content

Engineering Case Study

LivePulse

Real-Time Sports Intelligence Platform

A continuously running, event-driven sports data platform built on real football/soccer data — not a simulator. LivePulse ingests live match data from a real external provider (API-Football), detects meaningful changes, publishes internal domain events, and pushes real-time updates to the browser over WebSockets, backed by PostgreSQL for durable history, Redis for live state, and Kafka as the internal event backbone.

Overview

What LivePulse is

Live sports data changes constantly and unpredictably, and a browser wants to reflect that without a manual refresh. LivePulse ingests real match data on a schedule, figures out what actually changed since the last poll, and pushes only that change to connected clients — through a pipeline designed to fail in isolated, understandable ways rather than all at once.

Live and running against real data, not a local-only demo: ingestion, change detection, Kafka, WebSockets, observability, and automated testing are all built and verified against real API-Football and football-data.org data. An AI-features phase is deliberately deferred and kept separate from the core pipeline.

Architecture

How data moves through the system

  1. 1API-Football (external) → polled on tiered intervals
  2. 2Ingestion service → normalizes provider data, writes to PostgreSQL
  3. 3Change Detector → compares against last-known state, emits domain events only on real changes
  4. 4Kafka (6 topics) → scores / stats / alerts consumer groups, independently scalable
  5. 5Redis → live state cache-aside + pub/sub bridge to WebSocket gateway
  6. 6WebSocket gateway → subscribe/snapshot/update protocol to the browser

Why it's built this way

PostgreSQL stays the source of truth so durable history never depends on Kafka or Redis being healthy. Redis is used purely for speed — every cached value carries a freshness timestamp and falls back to Postgres on a miss. Kafka decouples scores, stats, and alerts into independent consumer groups so a slow one can never block another.

Explore the interactive Architecture Lab →

Verified, Not Assumed

Engineering Highlights

Real Engineering Incident

The WebSocket subscribe race

The WebSocket subscribe race: OPEN doesn't mean subscribed

2026-09-09

Test failed

A Playwright E2E test published a Redis update once the client's WebSocket reached readyState OPEN, then asserted the new score appeared. It passed reliably on a local machine but failed consistently in CI — the page kept showing the original fixture score.

Initial assumption

The failure looked like a timing/flakiness issue, so the first attempt bumped the assertion timeout to 10 seconds. It didn't help — nothing was slow, something was simply lost.

Investigation

The CI trace's actual page snapshot showed the pre-update score still on screen, ruling out 'just needs a longer timeout' before that fix was even tried.

Root cause

A client socket reaching readyState OPEN only proves the WebSocket handshake finished — it says nothing about whether the server has issued its Redis SUBSCRIBE for that match's channel yet, which happens asynchronously after the server receives the client's subscribe message. The test published to Redis in exactly that window. Redis pub/sub has no delivery guarantee for a message published before a subscriber exists (a documented ADR-006 tradeoff) — CI's colder first database query widened the race window enough to lose it consistently, where it was apparently always won locally.

Fix

Changed the test to wait for the actual match:snapshot message — which the server sends only after it has subscribed to Redis — instead of waiting on the client socket's readyState. No application code changed; the gateway's subscribe-then-snapshot ordering was already correct, only the test's synchronization assumption was wrong.

Verification

6/6 runs passed locally across 3 repeated executions against a freshly seeded, isolated backend mirroring CI's own setup, each completing in under a second — where the old race-prone version needed the full retry-and-timeout path to occasionally pass at all.

Engineering lesson

A connection being OPEN is not the same claim as 'the server-side subscription this connection depends on is ready.' Any test or client that synchronizes on transport-level readiness instead of an application-level confirmation is racing the exact gap between the two — wait for the signal that means what you actually need it to mean.

8 Architecture Decision Records

Architecture Decisions

ADR-001

Sports API Selection

API-Football chosen on its free tier behind a provider abstraction — the only evaluated provider offering genuine live in-play data at zero cost.

ADR-002

REST Polling Strategy

Batched, tiered polling designed around a real 100-requests/day budget rather than assuming a faster refresh rate was affordable.

ADR-003

Kafka Architecture

Six matchId-keyed topics and three consumer groups behind an EventBus interface — an intentional architectural choice, honestly documented as more than the current ~100-event/day workload needs.

ADR-004

Redis Strategy

A documented key schema and cache-aside strategy where Redis speeds things up but Postgres remains the correctness fallback.

ADR-005

PostgreSQL Schema

A normalized, provider-independent schema with deterministic UUIDs so swapping data providers never requires a schema change.

ADR-006

WebSocket Architecture

A subscribe/snapshot/update protocol over Redis pub/sub, designed to let gateway instances scale horizontally without knowing about each other.

ADR-007

Provider Abstraction

A SportsDataProvider interface that keeps a provider's response shape from becoming the domain model, validated when a second provider was added for standings.

ADR-008

Free Deployment Strategy

Portfolio Mode topology (single process, Neon Postgres, Upstash Redis, Redis Streams) chosen after discovering several "cardless" free tiers actually required a card in practice.

Explore all 8 ADRs in the Architecture Lab →

Scaling

Measured today, projected tomorrow

LivePulse deliberately separates what has actually been run from what the architecture is designed to support later. The Scaling Demo exists specifically to turn one of those projections into a measurement.

Stage 1 — Current

Measured

Portfolio Mode, as actually deployed at €0/month: ingestion, API, and WebSocket gateway run as one Node process on a single Oracle Cloud Always Free VM, with Neon Postgres, Upstash Redis, and Redis Streams as the event transport.

Stage 2 — Higher Traffic

Architectural projection

Not built, not measured. The first moves as real traffic grows: split ingestion / API / WebSocket-gateway into separate deployables, run multiple backend instances behind a load balancer, and switch the event bus to managed Kafka via an existing config flag — no code change required.

Stage 3 — Large Scale

Architectural projection

A documented, not-built production topology: a load balancer in front of N API instances, a right-sized Kafka cluster, independently scaled consumer groups, a Redis cluster, PostgreSQL read replicas, and dedicated WebSocket gateway instances behind sticky-session routing. No performance numbers are claimed at this stage.

Scaling Demo: real fan-out measurement

Measured

LivePulse's own engineering review named a real gap: the WebSocket gateway's horizontal-scaling story was a design, never measured — the project has only ever run exactly one instance in production.

The Scale Demo runs two real, unmodified LivePulse backend processes against one shared Redis, opens real WebSocket connections split across both instances, subscribes them to the same real match, publishes one update to the exact Redis channel a Kafka/Redis-Streams consumer would use, and measures whether clients on both instances receive it.

ConnectionsDeliveredBoth instancesp50p99
80 (40 + 40)80/806.0 ms6.9 ms
300 (150 + 150)300/30011.5 ms13.7 ms

Validates: ADR-006's fan-out design — consumers PUBLISH to a per-match Redis channel, each gateway instance independently subscribes only while it has a local client interested — works correctly across real, separate processes, with 100% delivery both times.

Does not validate:

  • Single machine: both gateway instances and the load generator ran on one host, not separate machines or a real network.
  • Nowhere near the documented 500-connections-per-instance ceiling — this tested 40–150 per instance, not a saturation test.
  • LivePulse's own public deployment still runs exactly one backend process — this result validates the design out-of-band; it is not a claim about what's running in production.
Explore Scale Demo →

Numbers That Are Real

Measured Metrics

Messages produced by one real live-poll tick

Measured

48 match.updated · 34 score-changed · 28 status-changed · 2 event-created

Unit tests

Measured

52 tests (backend/test/unit)

Standings rows reconciled across two providers

Measured

96 rows across 5 leagues

WebSocket connection ceiling (Portfolio Mode)

Measured

500 concurrent connections — a configured limit, not a benchmark result

API-Football daily request budget

Measured

100 requests/day, 10/minute (free tier)

Architecture Decision Records

Measured

8 ADRs, all accepted

Technology

Stack

Next.jsReactTypeScriptTanStack QueryFastifyPostgreSQLRedisKafkaioredisWebSockets (ws)OpenTelemetryPrometheusPlaywrightDocker ComposeVercelOracle Cloud