76 lines
6.1 KiB
Markdown
76 lines
6.1 KiB
Markdown
# High-concurrency lag audit (v3.3.90) — "lag ist immernoch da, ich vermute ab X gleichzeitig muss er alle Zeilen gebündelt updaten"
|
||
|
||
Method: 44-agent high-concurrency audit of the full upload→IPC→render path + Blink benchmark of the
|
||
renderer queue table (Playwright/Chromium = same Blink engine), targeting the user's NEW hypothesis:
|
||
"with ~100 concurrent uploads the renderer has to update ALL rows bundled rather than cleanly per-row."
|
||
|
||
## The user's hypothesis is MEASURED-REFUTED — the renderer is NOT the bottleneck.
|
||
Blink benchmark over scenarios Q=150..1000, M=10 active, 60 ticks each:
|
||
- renderQueueTable virtualizes at ≥200 rows; <200 = change-detecting in-place update.
|
||
- _updateRowInPlace is change-detecting (no forced reflow, no layout reads).
|
||
- median render <1 ms at Q=1000; only ~4/60 renders are full rebuilds even with progress-crossing sorts.
|
||
- progress is coalesced main-side (_progressByJob Map keyed by jobId + 100ms flush → one batch sized by
|
||
active-job count, ~10/sec); renderer iterates the batch with cheap per-row handleProgress.
|
||
DOM amplification is ruled out by measurement. "Laggy at ~40% CPU / 8 cores" = ONE core at 100% =
|
||
main-thread saturation / synchronous blocking, not DOM.
|
||
|
||
## SHIPPED (v3.3.90) — the two real main-thread blockers, both behavior-preserving
|
||
1. lib/clouddrop-upload.js `_uploadChunked`: was reading each 16 MB chunk with `fs.readSync` SYNCHRONOUSLY
|
||
on the main event loop — unique among the 5 uploaders (the other 4 stream async). Each read blocks the
|
||
WHOLE loop (~5–9 ms SSD, 30–100 ms slow disk) → freezes all progress/IPC/render/other-uploads, scaling
|
||
with the number of concurrent clouddrop uploads. Fits "laggy when uploading, worse with more concurrent."
|
||
User uses clouddrop. Fix: `fs.openSync`/`readSync`/`closeSync` → `fs.promises.open` + `await fh.read` +
|
||
`await fh.close()`. Byte-equivalence verified by SHA-256 over all chunk-boundary cases (full chunk,
|
||
partial last chunk, 2/3/4-chunk, single byte) before shipping — a chunk-read bug = corrupt upload.
|
||
2. lib/upload-manager.js rotation-retry (944) + suspect-alternate (1075) progressCb: both called
|
||
`_emitProgress` (a synchronous `emit('progress')` + fresh object spread) on EVERY stream chunk
|
||
(hundreds/sec per job) — they were missing the 250 ms `lastEmitTime` gate that the primary path (631)
|
||
has. With many concurrent uploads in rotation/suspect mode that's real main-thread emit amplification.
|
||
Mirrored the gate exactly: activeEntry mutation stays UNGATED (stats/speed-monitor stay fresh), only the
|
||
emit is throttled to 4/sec. Behavior-preserving.
|
||
397/397 tests pass, eslint clean (1 pre-existing unrelated warning at line 554).
|
||
|
||
## DROPPED (advisor: measured fine, don't chase perception)
|
||
- Lowering the virtual-row threshold below 200: the Blink benchmark shows <200 in-place updates are already
|
||
sub-ms; no change warranted.
|
||
|
||
## DISCRIMINATOR ANSWERED (user, 2026-06-21)
|
||
(a) Lag NOT clouddrop-specific — other hosters. (b) Parallel counts RAISED deliberately (10+).
|
||
(c) 50+ uploading SIMULTANEOUSLY active. → This is the TRUE high-concurrency main-thread-funnel branch,
|
||
NOT clouddrop. v3.3.90 stands but does not target this user's case.
|
||
|
||
## v3.3.91 — instrument first, don't refactor the upload core off elimination-reasoning
|
||
Advisor reframe: two LIVE hypotheses need OPPOSITE fixes — (A) main thread CPU-blocked (TLS/crypto/sync) →
|
||
event loop stalls → a cap/workers help; (B) main thread fine but IO-STARVED (libuv threadpool/sockets) →
|
||
loop stays responsive, uploads just queue → workers are WASTED, config fixes it. A worker/child-process
|
||
upload refactor touches throttle/rotation/abort/progress/credentials and is hard to reverse — DO NOT ship it
|
||
off sandbox elimination. One measurement splits the hypotheses and must run in the REAL app.
|
||
|
||
SHIPPED (both reversible, zero upload-core refactor):
|
||
1. main.js: `perf_hooks.monitorEventLoopDelay({resolution:10})` enabled at startup; logged via logInfo every
|
||
~5 s WHILE uploading (state==='uploading' && activeJobs>0) as
|
||
`eventloop-delay active=N mean=..ms p99=..ms max=..ms stddev=..ms threadpool=..`. Pure numbers, no secret
|
||
→ does NOT touch the redaction surface. This is the GROUND TRUTH: high mean/p99 → CPU-blocking → workers
|
||
justified; low delay while uploads stall → IO-bound → workers wasted, threadpool/sockets is the fix.
|
||
2. main.js (first statement, before require('electron')): `UV_THREADPOOL_SIZE = env || '64'`. Default is 4;
|
||
every async uploader feeds undici from fs.createReadStream (+ clouddrop fh.read) and DNS getaddrinfo goes
|
||
through the same pool → 50 concurrent vs 4 threads = reads/DNS serialize 4-at-a-time = a hard cliff at a
|
||
small connection count = the "ab X connections" symptom. Threads are created lazily on demand → 64-max
|
||
costs nothing if unused (zero-risk, reversible). The advisor's prescribed one-env-var hypothesis test.
|
||
|
||
CAVEAT (honest): synthetic sandbox benches could NOT confirm the threadpool is the bottleneck — pbkdf2 is
|
||
CPU-core-bound (masks pool size); DNS .invalid returns instantly; real-RTT DNS showed NO pool benefit because
|
||
WINDOWS serializes getaddrinfo via the OS DNS Client service (so on Windows the DNS half of the cliff is
|
||
masked by the resolver, though the fs-read half still benefits). This is exactly why the ELD number must come
|
||
from the user's real load, not the sandbox. Per-uploader undici Agent audit: clouddrop has a shared
|
||
module-level Agent (connections:50); doodstream/voe/vidmoly use the global dispatcher (pooled per origin, NO
|
||
per-call agent explosion) — so no agent fix needed.
|
||
|
||
## NEXT (gated on the real-app ELD number + user's explicit nod)
|
||
User runs their 50-concurrent load once; the `eventloop-delay` log lines decide:
|
||
- mean/p99 HIGH (tens–hundreds ms) → CPU-blocked → propose worker_threads/child-process upload pool OR a
|
||
smart concurrency cap (WITH the user's nod — it's hard to reverse and touches credentials/abort/rotation).
|
||
- delay LOW while it still lags → IO-bound → threadpool bump already addresses it; if not, look at socket
|
||
caps / undici Agent connection limits / per-origin pooling, NOT workers.
|
||
Do NOT build the worker refactor before this number exists.
|