Multi-Hoster-Upload/tasks
Administrator 7f636258d4 perf(uploads): make the batch-start file-stat non-blocking — kill the 336ms main stall (v3.3.103)
The v3.3.102 log (real 224-job batch) confirmed History virtualization works and
steady-state uploads are pristine (fps=32, event-loop mean 11.8ms). The remaining
residual was a ~6s spin-up burst at batch start: the main event loop blocked for
336ms with cpu=0%core (i.e. blocked on I/O, not computing) and the renderer janked
196-391ms, then everything settled clean.

A multi-agent investigation plus an adversarial review corrected the obvious-looking
hypothesis. The renderer's uncapped progress-batch drain is NOT the cause:
handleProgress only mutates plain JS state and schedules already-coalesced renders
(one per frame), and main coalesces progress to ~50 latest-per-job entries per
100ms. Chunking that drain would fix nothing — and the reviewer showed it would
REGRESS correctness: requestAnimationFrame throttles to ~0 when the window is
minimized (the common state for a background uploader), so a rAF-chunked drain would
grow an unbounded backlog and defer persistQueueStateSoon for every buffered item,
losing terminal 'done' events on close (the queue-persistence ghost-fix class). So
that path is deliberately not taken.

The real cause (cpu=0%core = blocked on I/O) is a synchronous fs.statSync storm in
UploadManager.startBatch: the dedup loop ran up to DEDUP_CHUNK=200 synchronous
fs.statSync calls in a single tick before yielding (200 x ~1.68ms on the user's VM
= the exact 336ms), on a disk already saturated by the 1MB read-ahead.

Fix — make the batch-start stats non-blocking:
- The dedup loop now dedupes synchronously (cheap Map work) and then stats the
  unique files in parallel via await Promise.all(fs.promises.stat ...) per chunk, so
  the stat I/O runs on the libuv threadpool and the main thread never blocks. The
  results-Map shape ({name,size,results:[]}) and dedup semantics (size 0 on failure)
  are unchanged.
- The per-job statSync fallback is converted to await fs.promises.stat for
  consistency (it sits in an async function before the first real await; the cached
  size from dedup already lets nearly every job skip it).

Tests: the upload-manager mocks override fs.statSync; they now also override
fs.promises.stat with the same fake sizes (upload-manager.test.js x2,
suspect-reject-alternates.test.js). 407 tests pass; clean Electron boot.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-21 20:53:48 +02:00
..
lessons.md perf(config): split history out of the hot config file — the real 38.5MB main-thread thrash (v3.3.99) 2026-06-21 18:37:12 +02:00
todo.md perf(uploads): make the batch-start file-stat non-blocking — kill the 336ms main stall (v3.3.103) 2026-06-21 20:53:48 +02:00