Testing a real-time app for real,
and taming flaky tests.

QuizClash is a real-time quiz game: up to eight players in a room, ten seconds per question, one shared reveal. After Part 1 the server runs the round, and after Part 2 the room survives its host leaving. Each of those is a claim about many clients, a server, a database and its security rules behaving together over time.

Pure logic is easy to test and we tested it heavily. The things that actually break real-time apps are elsewhere:

• a rule that refuses a write the app makes, so a feature silently never works;
• two clients racing for the same step;
• a screen that changes its URL before it has rendered its button;
• a browser that throttles timers in a tab nobody is looking at.

None of those show up unless you run real browsers against the real backend. So that's what we did, and then we spent a surprising amount of time making those runs trustworthy.

Most checks are cheap and pure. A few are expensive and real. Both kinds are required in CI.

The bottom layers carry most of the weight. Every rule of play is a pure function with a unit test, and the Go port is checked against fixtures generated from the JavaScript, so the two can't drift. Rules tests run both rule files on the Firebase emulators under a demo project id, so nothing ever reaches the real project. They cover the writes the app really makes and the abuse cases, and we confirm the new cases fail against the old rules before trusting them.

The top layer is three browser suites: smoke (sign in, solo game, admin tools, history), vs (a 1v1 match, rematch, forfeit) and group (a room of players, someone dropping, the host leaving). The rest of this post is mostly about that top layer, because that's where all the lessons were.

The app has no runtime npm dependencies, and we wanted the tests to keep it that way: no Playwright, no Puppeteer, no browser downloads in CI. Chrome already speaks a full automation protocol, the Chrome DevTools Protocol (CDP), over a WebSocket. Node 22+ has a built-in WebSocket. That's enough.

The driver is about 120 lines. It launches Chrome headless with a remote-debugging port, opens a page per player, and gives each page a handful of methods: goto, eval, waitFor, click, type, shot. It also collects every console message and uncaught exception per page, so a test can assert "no errors" at the end.

The tests run the production build with the real response headers, so the strict Content Security Policy is enforced while they run. A CSP violation is a test failure, not a warning in someone's console.

The first browser suites ran against an offline demo mode: a backend implementation that keeps everything in localStorage, so the UI can be developed without Firebase. It was convenient and fast. It was also lying to us in two ways:

• it can't enforce security rules, so anything the rules would refuse looked fine;
• every write rewrites the whole room in localStorage, and tabs see each other's writes late, so concurrent actions (two rematch votes) were lost and tests flaked for reasons production doesn't have.

PR #14 moved all three suites onto the Firebase emulators (Auth, Firestore, Realtime Database, with the generated rules) and the real Go game server. The runner builds and starts the server and the site, loads the question bank through the admin's own import screen, signs players in as emulated Google users, and runs the suites. The admin in the smoke suite became a real admin instead of a stubbed isAdmin = () => true.

The first run on the real stack found three bugs, one of them live in production:

The demo mode still exists for UI work. It's just not allowed to be the thing that tells us the app works.

The obvious way to simulate a group is one Chrome with a tab per player. It failed in two different ways, and each taught us something about browsers.

Connections. The Firestore emulator speaks HTTP/1.1, and Chrome allows six connections per host. A handful of tabs, each holding long-lived Firestore and database channels, used them all up, and the rest of the players simply stalled. Production Firestore uses HTTP/2, which multiplexes, so this is purely a test-environment limit, but it's a real one.

Background timers. After the server took over the round (Part 1), rounds move forward when a client's timer says a step is due. Chrome doesn't reliably run timers in a background tab, even with the throttling flags turned off. In a one-Chrome setup, only the focused tab was a reliable player.

So the group suite gives every player their own browser process, each with its own visible page. That's how real players look anyway: one person, one screen. Together with the browser the runner uses to seed the question bank, a group run starts nine Chromes.

That has a cost. A cold CI runner can take well over ten seconds to start Chrome, and with nine launches the odds of one slow start add up. PR #8 gave Chrome 30 seconds to answer, and the launcher kills the process and reports Chrome's own error output if it never does. A Chrome left running keeps Node, and the CI job, alive until its timeout.

CI on main once failed in the 1v1 suite with guest: nothing to click for .setup-cta .btn. The guest clicked Ready as soon as the URL hash became #/lobby/. The hash changes before the lobby renders its button. On a fast machine the button was always there in time. On a busy runner it wasn't.

The fix (PR #35) was in the driver, not the test: click waits for its element, like waitFor does, instead of failing at once. It's still a state wait (the element exists), never a fixed sleep.

async click(selector, text, timeout = 10_000) {
  const find = findExpression(selector, text); // first match whose text includes `text`
  const start = Date.now();
  do {
    const clicked = await page.eval(`(() => { const el = ${find}; el?.click(); return !!el; })()`);
    if (clicked) return;
    await sleep(100);                       // poll interval, not a guess at timing
  } while (Date.now() - start < timeout);
  throw new Error(`nothing to click for ${selector} "${text}"`);
}

The same principle fixed our worst flaky test. The group check "a player who drops mid-match doesn't hold up the reveal" asserted a timing: the reveal should happen well before the ten-second timer. Sometimes it didn't:

The real cause was product behaviour (Part 2: presence went offline late, and hostless rounds paid the grace period twice). Once that was fixed, the check was rewritten to assert state instead of duration: the dropped host shows offline, and the next host is the one driving the round. A state check passes or fails for a reason you can read. A timing check fails when the machine is busy.

Before we understood the dropped-player failure, it was blocking merges and the first automated deploy. We had three choices: keep re-running until green, delete the check, or quarantine it. We quarantined it, with rules (PR #3, issue #2):

• An issue, not a comment. Issue #2 holds the failure output, the runs that passed and failed, what we tried that didn't work, and the two suspects.
• Say what coverage is lost. The PR stated that the behaviour was no longer browser-tested, and that it was still unit-tested in the core rules.
• A known-gaps section in the architecture doc pointing at the issue, so it isn't forgotten.
• An exit criterion. To restore the check: capture the failing state, fix the cause, then 10 passing runs in a row locally and 3 on CI.

The issue was closed by the product fix in PR #32, and the check came back as the state-based version. Quarantine with an exit criterion is a promise. Quarantine without one is deletion with extra steps.

Not every flake was the app. Issue #34 collected the ones that were ours:

Two lessons from that list. Every layer of a harness needs a timeout, including the lowest one, and a pending call should be rejected when its connection closes. And an "ignore these console errors" list must be as narrow as possible, written down with the reason, or it slowly turns into "ignore errors".

One more habit paid for itself: keep the whole log. One early failure said "console errors:" and our output filter dropped the lines that followed. We never learned what they were. Since then, failures print everything.

The sibling apps (Pebble for expenses, Cadence for routines, Grove for shared spaces) are simpler, but the same approach found real bugs in each:

The common pattern: tests that run in conditions the developer's laptop doesn't have, such as another timezone, a slow network, a slow CPU, or a fresh write racing a read. Bugs live in the conditions you don't usually run in.