Multi-Hoster-Upload/tasks/todo.md

High-concurrency lag audit (v3.3.90) — "lag ist immernoch da, ich vermute ab X gleichzeitig muss er alle Zeilen gebündelt updaten"

Method: 44-agent high-concurrency audit of the full upload→IPC→render path + Blink benchmark of the renderer queue table (Playwright/Chromium = same Blink engine), targeting the user's NEW hypothesis: "with ~100 concurrent uploads the renderer has to update ALL rows bundled rather than cleanly per-row."

The user's hypothesis is MEASURED-REFUTED — the renderer is NOT the bottleneck.

Blink benchmark over scenarios Q=150..1000, M=10 active, 60 ticks each:

• renderQueueTable virtualizes at ≥200 rows; <200 = change-detecting in-place update.
• _updateRowInPlace is change-detecting (no forced reflow, no layout reads).
• median render <1 ms at Q=1000; only ~4/60 renders are full rebuilds even with progress-crossing sorts.
• progress is coalesced main-side (_progressByJob Map keyed by jobId + 100ms flush → one batch sized by active-job count, ~10/sec); renderer iterates the batch with cheap per-row handleProgress. DOM amplification is ruled out by measurement. "Laggy at ~40% CPU / 8 cores" = ONE core at 100% = main-thread saturation / synchronous blocking, not DOM.

SHIPPED (v3.3.90) — the two real main-thread blockers, both behavior-preserving

1. lib/clouddrop-upload.js _uploadChunked: was reading each 16 MB chunk with fs.readSync SYNCHRONOUSLY on the main event loop — unique among the 5 uploaders (the other 4 stream async). Each read blocks the WHOLE loop (~5–9 ms SSD, 30–100 ms slow disk) → freezes all progress/IPC/render/other-uploads, scaling with the number of concurrent clouddrop uploads. Fits "laggy when uploading, worse with more concurrent." User uses clouddrop. Fix: fs.openSync/readSync/closeSync → fs.promises.open + await fh.read + await fh.close(). Byte-equivalence verified by SHA-256 over all chunk-boundary cases (full chunk, partial last chunk, 2/3/4-chunk, single byte) before shipping — a chunk-read bug = corrupt upload.
2. lib/upload-manager.js rotation-retry (944) + suspect-alternate (1075) progressCb: both called _emitProgress (a synchronous emit('progress') + fresh object spread) on EVERY stream chunk (hundreds/sec per job) — they were missing the 250 ms lastEmitTime gate that the primary path (631) has. With many concurrent uploads in rotation/suspect mode that's real main-thread emit amplification. Mirrored the gate exactly: activeEntry mutation stays UNGATED (stats/speed-monitor stay fresh), only the emit is throttled to 4/sec. Behavior-preserving. 397/397 tests pass, eslint clean (1 pre-existing unrelated warning at line 554).

DROPPED (advisor: measured fine, don't chase perception)

• Lowering the virtual-row threshold below 200: the Blink benchmark shows <200 in-place updates are already sub-ms; no change warranted.

DISCRIMINATOR ANSWERED (user, 2026-06-21)

(a) Lag NOT clouddrop-specific — other hosters. (b) Parallel counts RAISED deliberately (10+). (c) 50+ uploading SIMULTANEOUSLY active. → This is the TRUE high-concurrency main-thread-funnel branch, NOT clouddrop. v3.3.90 stands but does not target this user's case.

v3.3.91 — instrument first, don't refactor the upload core off elimination-reasoning

Advisor reframe: two LIVE hypotheses need OPPOSITE fixes — (A) main thread CPU-blocked (TLS/crypto/sync) → event loop stalls → a cap/workers help; (B) main thread fine but IO-STARVED (libuv threadpool/sockets) → loop stays responsive, uploads just queue → workers are WASTED, config fixes it. A worker/child-process upload refactor touches throttle/rotation/abort/progress/credentials and is hard to reverse — DO NOT ship it off sandbox elimination. One measurement splits the hypotheses and must run in the REAL app.

SHIPPED (both reversible, zero upload-core refactor):

1. main.js: perf_hooks.monitorEventLoopDelay({resolution:10}) enabled at startup; logged via logInfo every ~5 s WHILE uploading (state==='uploading' && activeJobs>0) as eventloop-delay active=N mean=..ms p99=..ms max=..ms stddev=..ms threadpool=... Pure numbers, no secret → does NOT touch the redaction surface. This is the GROUND TRUTH: high mean/p99 → CPU-blocking → workers justified; low delay while uploads stall → IO-bound → workers wasted, threadpool/sockets is the fix.
2. main.js (first statement, before require('electron')): UV_THREADPOOL_SIZE = env || '64'. Default is 4; every async uploader feeds undici from fs.createReadStream (+ clouddrop fh.read) and DNS getaddrinfo goes through the same pool → 50 concurrent vs 4 threads = reads/DNS serialize 4-at-a-time = a hard cliff at a small connection count = the "ab X connections" symptom. Threads are created lazily on demand → 64-max costs nothing if unused (zero-risk, reversible). The advisor's prescribed one-env-var hypothesis test.

CAVEAT (honest): synthetic sandbox benches could NOT confirm the threadpool is the bottleneck — pbkdf2 is CPU-core-bound (masks pool size); DNS .invalid returns instantly; real-RTT DNS showed NO pool benefit because WINDOWS serializes getaddrinfo via the OS DNS Client service (so on Windows the DNS half of the cliff is masked by the resolver, though the fs-read half still benefits). This is exactly why the ELD number must come from the user's real load, not the sandbox. Per-uploader undici Agent audit: clouddrop has a shared module-level Agent (connections:50); doodstream/voe/vidmoly use the global dispatcher (pooled per origin, NO per-call agent explosion) — so no agent fix needed.

v3.3.92 — make the single measurement decisive + breadth audit of un-checked main.js hot paths

Enriched the ELD log line with process.getActiveResourcesInfo() as a compact type-histogram: eventloop-delay active=N mean/p99/max/stddev ms threadpool=64 resources=K {TCPSocketWrap:50,FSReqCallback:4,...}. Now ONE run splits all three readings in a single line: high mean/p99 → CPU-blocked (workers/cap); low delay + many TCP/FS/GetAddrInfo resources → IO-bound queueing (threadpool/sockets, NOT workers); low delay + few resources → not saturated (lag elsewhere / perception). Pure numbers, no redaction surface.

Breadth audit this round (4th /goal re-fire, code I wrote, NOT re-measuring cleared render/persist):

• main.js logging (debug/rot/upload): all buffered + ASYNC fs.appendFile (write-guard flag, 500ms timer, setImmediate re-flush). Sync appendFileSync ONLY in crash/signal/exit handlers (correct there). CLEAN.
• main.js progress coalescing (_progressByJob Map + 100ms batch → one upload-progress-batch via safeSend): non-terminal = Map.set (keeps latest/job); gated upstream to 4/sec/job. CLEAN at N=50.
• _appendJobLog: capped in-memory ring buffer (Map, FIFO-evict). CLEAN.
• All 5 uploaders: doodstream/voe/vidmoly/clouddrop-simple stream via async createReadStream + for-await + async throttle.consume; clouddrop-chunked now async fh.read. NONE block the main loop per chunk. CLEAN (clouddrop's old readSync was the unique outlier, fixed v3.3.90).

NEXT (gated on the real-app ELD number + user's explicit nod)

User runs their 50-concurrent load once; the enriched eventloop-delay log line decides:

• mean/p99 HIGH (tens–hundreds ms) → CPU-blocked → propose worker_threads/child-process upload pool OR a smart concurrency cap (WITH the user's nod — it's hard to reverse and touches credentials/abort/rotation).
• delay LOW while it still lags → IO-bound → threadpool bump already addresses it; if not, look at socket caps / undici Agent connection limits / per-origin pooling, NOT workers. Do NOT build the worker refactor before this number exists.