LiveOps

Test failed

Built a complete second EventBus implementation on Kafka and ran the same load test used to validate the Postgres transport. Ingest performed comparably — but the drain check (how long the async pipeline takes to catch up) plateaued at a small residual lag and never reached zero, for the entire 60-second timeout.

Initial assumption

The obvious read: Kafka's consumers were stuck, or messages were being silently dropped somewhere in the new produce/consume path.

Investigation

Kafka's own admin API (rpk group describe) told a different story: TOTAL-LAG 0 for both consumer groups — the broker believed everything had been consumed. That ruled out a stuck consumer immediately and reframed the question: not 'why isn't Kafka finishing' but 'why does this project's own measurement disagree with Kafka's own record of what happened.' Re-running after a real fix to the new Kafka code changed the plateau's size but never closed it — proof that fix wasn't the actual mechanism, and cause for a second investigation.

Root cause

Two separate causes, found in sequence. First: the load-test script itself — unchanged and trusted since an earlier phase — generated overlapping event IDs across its concurrency waves, so a large fraction of 'unique' requests were actually duplicates. Correctly rejected by idempotency, but a rejected insert still consumes a database sequence value with no row created, inflating the drain target past the true number of real events every prior load test in the project had ever measured against. Second, after fixing that and re-running on a genuinely clean database: Kafka consumer sessions could time out under sustained backlog processing, since each message did two sequential database round trips with no explicit heartbeat in between — a real client-tuning gap, not a data-loss bug (every message's actual side effect always completed, cross-checked against the business tables directly).

Fix

Gave every load-test wave a distinct ID prefix, eliminating the cross-wave collisions entirely. Widened the Kafka consumer's session timeout and set an explicit heartbeat interval, addressing the rebalance risk directly instead of papering over its symptom.

Verification

Reset the database to a genuinely clean state and re-ran both transports back to back on identical, current code: Postgres reproducibly drained 6,200 real events in ~45 seconds — the first accurate number this project ever produced for that figure, meaningfully higher than what earlier, duplicate-cushioned runs had reported. The session-timeout fix cut the Kafka bookkeeping discrepancy by roughly 95%, reported as exactly that rather than rounded up to fully resolved.

Test failed

A custom load test pushed a 3,000-event burst through ingest, then measured how long the async pipeline took to drain. Ingest itself scaled cleanly to ~3,700 req/s. But one consumer group, projection-worker, was still over 5,600 events behind a full 60 seconds later — draining at roughly 9 events per second, nowhere close to keeping pace with a pipeline that had just accepted thousands of events per second.

Initial assumption

The obvious suspect was the transport — maybe Postgres itself couldn't keep up as a message queue under this load, which would have been read as early evidence that a Kafka migration was needed sooner than expected.

Investigation

Checked each of the three consumer groups' lag independently instead of treating 'the pipeline' as one thing: audit-log and workflow-trigger were both already at zero lag. Only projection-worker was behind — ruling out the transport itself and pointing at that one consumer's own code.

Root cause

refreshWorkflowProjection was issuing one INSERT ... ON CONFLICT statement per changed workflow row, in a plain sequential loop, on every single tick. Invisible at demo scale (tens of rows). At load-test scale, with 3,000+ workflow executions touched per tick, this was 3,000+ sequential round trips competing for the same connection pool every other consumer group and the API shared.

Fix

Replaced the per-row loop with one bulk INSERT ... SELECT ... JOIN (VALUES ...) ON CONFLICT statement — the VALUES list is bounded by the number of distinct workflow definitions (currently one), not the number of executions, so the whole refresh became one round trip regardless of backlog size.

Verification

Before: still 5,600+ events behind after 60 seconds. After: a fresh, more than doubled load test fully drained in what was measured at the time as 23.5–28 seconds — a number Phase 10's later investigation found had itself been measured against an inflated target (see the Kafka migration incident above); the fix was real and correct, the exact seconds it bought were revised once a more accurate measurement existed.