Investigate why dropping socket.io polling fallback degrades p95 under load (#7756 follow-up) #7767

Activity

1. JohnMcLear commented on May 15, 2026

   

   JohnMcLear

   on May 15, 2026

   Last edited by JohnMcLear

   MemberAuthor

   More actions

   Diagnosis

   Confirmed hypothesis 4 by reading the engine.io 6.6.5 transport code shipped with etherpad core. The polling fallback is a natural coalescer; WebSocket isn't.

   Engine.io socket buffering (`engine.io/build/socket.js:flush`)

   sendPacket(type, data, ...) {
     this.writeBuffer.push(packet);
     this.flush();
   }
   
   flush() {
     if (closed || !transport.writable || !writeBuffer.length) return;
     const wbuf = this.writeBuffer;
     this.writeBuffer = [];
     this.transport.send(wbuf);   // entire buffer hands off in one call
   }

   So packets queue in `writeBuffer` and `flush()` is a no-op while `transport.writable === false`.

   Polling transport (`transports/polling.js`)

   `writable` is false between HTTP poll requests. While a poll response is being written and until the client opens its next long-poll, every `emit()` piles up in `writeBuffer`. Then on the next poll:

   send(packets) {
     this.parser.encodePayload(packets, doWrite);
     // -> single HTTP response containing ALL queued packets
   }

   N packets → 1 HTTP response.

   WebSocket transport (`transports/websocket.js`)

   `writable` is briefly false during each frame write, then true again essentially immediately. Each `emit()` triggers its own `flush()` while the buffer is still small. And even when the buffer accumulates:

   send(packets) {
     for (let i = 0; i < packets.length; i++) {
       const packet = packets[i];
       this.parser.encodePacket(packet, ...);  // ONE WS frame per packet
     }
   }

   So N packets → N WS frames, regardless of batching at the socket layer.

   Why this matters

   At 200 authors in our dive, the server emits 6 600 `NEW_CHANGES` per second per client. Polling sends those as roughly 25 HTTP responses/sec (default poll cadence ~25 ms) carrying ~250 packets each — i.e. 25 system calls instead of 6 600. WebSocket sends ~6 600 individual frames per second per client. Hence baseline apply_mean ≈ 5 ms vs ws-only ≈ 13 ms: WebSocket is starving the apply path of CPU on syscall overhead.

   This is not an etherpad bug. It's a deliberate engine.io WS-transport design choice. Engine.io's wire protocol can pack multiple packets per WS message (the "payload" encoding the polling transport uses), but the WS transport always sends one packet per frame.

   Where to take this

   Two paths in priority order:

   1. Application-level coalesce (cheap): `NEW_CHANGES_BATCH` message type carrying an array of revisions in a single emit per socket. Cuts the engine.io packet count proportionally, regardless of transport. This is what the dive doc already recommends; the WS finding makes it more urgent.
   2. Transport-level pack (deeper): PR engine.io upstream to use payload-encoding on the WS transport — pack multiple queued packets into one WS message. Would help any high-emit-rate socket.io application. Worth filing in socketio/engine.io once we've prototyped it locally.

   Keeping `socketTransportProtocols: ["websocket", "polling"]` (current default) remains correct until at least path 1 lands.
2. JohnMcLear

   self-assigned this

   on May 16, 2026
3. JohnMcLear

   added a commit that references this issue on May 16, 2026

   docs(scaling-dive): rewrite with cliff-finding + Qodo fixes

   ...

   80b3b74
4. JohnMcLear commented on May 16, 2026

   

   JohnMcLear

   on May 16, 2026

   MemberAuthor

   More actions

   Prototype scored, did not help

   Closed #7772. The simple monkey-patch (pack only when send(packets.length > 1)) almost never fires under load because engine.io's socket.flush() drains the writeBuffer immediately whenever transport.writable === true, and WS writes flip writable back within microseconds. Net result: the writeBuffer rarely accumulates more than one packet, so the patch passes through to the legacy single-frame path 99% of the time.

   The actual change needed

   To get real coalescing on the WS transport, we'd need to deliberately delay the flush — buffer multiple sendPacket calls within a small time window before draining via transport.send. Two viable strategies:

   1. Microtask deferral. Replace the synchronous flush() call inside sendPacket() with queueMicrotask(() => flush()). Multiple sends within the same task accumulate. Zero added wall-clock latency (microtasks drain before the next task). Smallest behaviour change.

   2. Configurable small delay. Add a setTimeout(flush, debounceMs) with debounceMs like 1-2 ms. Slightly more accumulation across event-loop iterations; adds latency proportional to the delay. Bigger behaviour change but reaches more flush opportunities.

   Either path is invasive enough that it should probably be a focused engine.io upstream PR rather than a monkey-patch. Worth opening once we have a clear hypothesis on how much delay is needed to materially shift the WS-frame-per-packet ratio.

   Secondary methodology finding

   The scoring run also showed websocket-only as the best lever — contradicting every previous dive. The cause is that each GHA matrix entry runs on a separate physical runner, so within-run cross-lever comparisons are cross-hardware. Runner noise can flip lever conclusions. Future dive runs need N ≥ 3 trials per lever to filter that noise out before drawing strong conclusions. The scaling dive doc PR needs a follow-up note about this; existing conclusions in the doc may need re-validation.
5. JohnMcLear commented on May 16, 2026

   

   JohnMcLear

   on May 16, 2026

   MemberAuthor

   More actions

   Resolved by #7774

   The investigation traced WS-only's performance degradation to engine.io sending one WS frame per packet (vs polling's encodePayload batched HTTP responses). Two prototype directions came out of this:

   • First attempt — #7772 — patched transport.send(packets[]) to use encodePayload for multi-packet sends. Almost never fired because engine.io's Socket.flush() drains the writeBuffer immediately when transport.writable === true, and WS writable returns to true within microseconds. Closed.

   • Real fix — #7774 — patches Socket.prototype.sendPacket to schedule a coalesced flush via queueMicrotask. Multiple sendPacket calls in the same task accumulate before drain. N=3 scoring shows modest mid-load tail-latency reduction (step 350 max: 122 → 110 ms; step 300 max: 71 → 58 ms). Wire bytes unchanged.

   Net effect on the original question — "why does WS-only degrade under load?" — is now answered: socket.io's polling fallback was acting as a natural coalescer at the HTTP boundary; the WS transport doesn't coalesce, so high-emit-rate workloads pay a per-frame syscall cost the polling path naturally amortises. Production should keep socketTransportProtocols: ["websocket", "polling"] (the default), and the flush-deferral approach in #7774 is the right way to claw back some of polling's coalescing property on the WS path without changing wire behaviour.

   Closing. The full diagnosis is captured in the scaling-dive doc, section "Lever 8b — engine.io socket flush deferral".
6. JohnMcLear

   closed this as completedon May 16, 2026