Compare commits

...

3 Commits

Author SHA1 Message Date
Administrator
f63ca53f2f release: v3.3.91 2026-06-21 04:11:21 +02:00
Administrator
e135655c95 docs(tasks): high-concurrency discriminator answered — instrument-first (v3.3.91), gate worker refactor on real-app ELD number
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-21 04:10:47 +02:00
Administrator
2fd26add1a perf(main): bump UV_THREADPOOL_SIZE to 64 + instrument event-loop delay
The user runs 50+ concurrent uploads (parallel counts raised deliberately). Every async uploader feeds undici from fs.createReadStream and resolves DNS via getaddrinfo — both go through the libuv threadpool, whose default size is 4. At 50 concurrent uploads, file reads and lookups serialize 4-at-a-time: a hard cliff at a small connection count that matches the 'lags from X connections onward' symptom. Raise the cap to 64 as the first statement (before require('electron'), so libuv reads it when it lazily inits the pool; an explicit env override still wins). Threads are created on demand, so a higher max costs nothing when unused — reversible, zero upload-core change.

Also enable perf_hooks.monitorEventLoopDelay and log mean/p99/max/stddev every ~5s while uploading. This is the ground-truth instrument that splits the two competing explanations for the lag: a high event-loop delay means the main thread is CPU-blocked (TLS/crypto) and only workers or a concurrency cap will help; a low delay while uploads stall means the work is IO-bound and the threadpool/socket config is the lever, not workers. The numbers are pure metrics — they never touch the credential-redaction path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-21 04:10:43 +02:00
3 changed files with 62 additions and 9 deletions

22
main.js
View File

@ -1,3 +1,5 @@
process.env.UV_THREADPOOL_SIZE = process.env.UV_THREADPOOL_SIZE || '64';
const { monitorEventLoopDelay } = require('perf_hooks');
const { app, BrowserWindow, ipcMain, dialog, clipboard, nativeTheme, Tray, Menu } = require('electron'); const { app, BrowserWindow, ipcMain, dialog, clipboard, nativeTheme, Tray, Menu } = require('electron');
nativeTheme.themeSource = 'dark'; nativeTheme.themeSource = 'dark';
const path = require('path'); const path = require('path');
@ -25,6 +27,10 @@ const stats = require('./lib/stats');
const { createCollectors } = require('./lib/diagnostics-collectors'); const { createCollectors } = require('./lib/diagnostics-collectors');
const { createAgent } = require('./lib/diagnostics-agent'); const { createAgent } = require('./lib/diagnostics-agent');
const _eventLoopDelay = monitorEventLoopDelay({ resolution: 10 });
_eventLoopDelay.enable();
let _eldLastLog = 0;
let mainWindow; let mainWindow;
let _lastImportPath = null; let _lastImportPath = null;
let dropTargetWindow = null; let dropTargetWindow = null;
@ -181,6 +187,21 @@ function logMarker(label, fields) {
debugLog(`────── ${label}${extra} ──────`); debugLog(`────── ${label}${extra} ──────`);
} }
function _maybeLogEventLoopDelay(activeJobs) {
const now = Date.now();
if (now - _eldLastLog < 5000) return;
_eldLastLog = now;
try {
const ns = 1e6;
const mean = (_eventLoopDelay.mean / ns).toFixed(1);
const max = (_eventLoopDelay.max / ns).toFixed(1);
const p99 = (_eventLoopDelay.percentile(99) / ns).toFixed(1);
const stddev = (_eventLoopDelay.stddev / ns).toFixed(1);
logInfo('perf', `eventloop-delay active=${activeJobs} mean=${mean}ms p99=${p99}ms max=${max}ms stddev=${stddev}ms threadpool=${process.env.UV_THREADPOOL_SIZE}`);
_eventLoopDelay.reset();
} catch {}
}
// Dedicated account-rotation log so users can trace fallback decisions // Dedicated account-rotation log so users can trace fallback decisions
// without wading through general debug output. Writes to account-rotation.log // without wading through general debug output. Writes to account-rotation.log
// in the same directory as fileuploader.log (honors user's configured path). // in the same directory as fileuploader.log (honors user's configured path).
@ -1705,6 +1726,7 @@ ipcMain.handle('start-upload', (_event, payload) => {
if (data.state === 'uploading' && data.activeJobs > 0) { if (data.state === 'uploading' && data.activeJobs > 0) {
const speedMb = ((Number(data.globalSpeedKbs) || 0) / 1024).toFixed(1); const speedMb = ((Number(data.globalSpeedKbs) || 0) / 1024).toFixed(1);
updateTrayTooltip(`Upload: ${data.activeJobs} aktiv - ${speedMb} MB/s`); updateTrayTooltip(`Upload: ${data.activeJobs} aktiv - ${speedMb} MB/s`);
_maybeLogEventLoopDelay(data.activeJobs);
} else { } else {
updateTrayTooltip('Multi-Hoster-Upload'); updateTrayTooltip('Multi-Hoster-Upload');
} }

View File

@ -1,6 +1,6 @@
{ {
"name": "multi-hoster-uploader", "name": "multi-hoster-uploader",
"version": "3.3.90", "version": "3.3.91",
"description": "Upload files to doodstream, voe, vidmoly, byse simultaneously", "description": "Upload files to doodstream, voe, vidmoly, byse simultaneously",
"main": "main.js", "main": "main.js",
"scripts": { "scripts": {

View File

@ -34,11 +34,42 @@ main-thread saturation / synchronous blocking, not DOM.
- Lowering the virtual-row threshold below 200: the Blink benchmark shows <200 in-place updates are already - Lowering the virtual-row threshold below 200: the Blink benchmark shows <200 in-place updates are already
sub-ms; no change warranted. sub-ms; no change warranted.
## OPEN — discriminator question to the user (do NOT declare victory blind) ## DISCRIMINATOR ANSWERED (user, 2026-06-21)
With this user's real config (parallelCount 2 × 5 hosters ≈ 10 max concurrent), "100 concurrent" is only (a) Lag NOT clouddrop-specific — other hosters. (b) Parallel counts RAISED deliberately (10+).
reachable if the parallel counts were raised — otherwise "100" is the QUEUE size and only ~10 upload at once. (c) 50+ uploading SIMULTANEOUSLY active. → This is the TRUE high-concurrency main-thread-funnel branch,
Ask: (a) is the lag specifically during clouddrop uploads? (b) did you raise the per-hoster/global parallel NOT clouddrop. v3.3.90 stands but does not target this user's case.
counts above 2? If it's true high concurrency (dozens of simultaneous undici streams funneling decode +
progress callbacks through the one main JS thread), that needs a concurrency cap or a worker/child process — ## v3.3.91 — instrument first, don't refactor the upload core off elimination-reasoning
NOT a micro-fix. The two shipped fixes are genuine improvements regardless; the answer decides whether a Advisor reframe: two LIVE hypotheses need OPPOSITE fixes — (A) main thread CPU-blocked (TLS/crypto/sync) →
bigger architectural change is the next step. event loop stalls → a cap/workers help; (B) main thread fine but IO-STARVED (libuv threadpool/sockets) →
loop stays responsive, uploads just queue → workers are WASTED, config fixes it. A worker/child-process
upload refactor touches throttle/rotation/abort/progress/credentials and is hard to reverse — DO NOT ship it
off sandbox elimination. One measurement splits the hypotheses and must run in the REAL app.
SHIPPED (both reversible, zero upload-core refactor):
1. main.js: `perf_hooks.monitorEventLoopDelay({resolution:10})` enabled at startup; logged via logInfo every
~5 s WHILE uploading (state==='uploading' && activeJobs>0) as
`eventloop-delay active=N mean=..ms p99=..ms max=..ms stddev=..ms threadpool=..`. Pure numbers, no secret
→ does NOT touch the redaction surface. This is the GROUND TRUTH: high mean/p99 → CPU-blocking → workers
justified; low delay while uploads stall → IO-bound → workers wasted, threadpool/sockets is the fix.
2. main.js (first statement, before require('electron')): `UV_THREADPOOL_SIZE = env || '64'`. Default is 4;
every async uploader feeds undici from fs.createReadStream (+ clouddrop fh.read) and DNS getaddrinfo goes
through the same pool → 50 concurrent vs 4 threads = reads/DNS serialize 4-at-a-time = a hard cliff at a
small connection count = the "ab X connections" symptom. Threads are created lazily on demand → 64-max
costs nothing if unused (zero-risk, reversible). The advisor's prescribed one-env-var hypothesis test.
CAVEAT (honest): synthetic sandbox benches could NOT confirm the threadpool is the bottleneck — pbkdf2 is
CPU-core-bound (masks pool size); DNS .invalid returns instantly; real-RTT DNS showed NO pool benefit because
WINDOWS serializes getaddrinfo via the OS DNS Client service (so on Windows the DNS half of the cliff is
masked by the resolver, though the fs-read half still benefits). This is exactly why the ELD number must come
from the user's real load, not the sandbox. Per-uploader undici Agent audit: clouddrop has a shared
module-level Agent (connections:50); doodstream/voe/vidmoly use the global dispatcher (pooled per origin, NO
per-call agent explosion) — so no agent fix needed.
## NEXT (gated on the real-app ELD number + user's explicit nod)
User runs their 50-concurrent load once; the `eventloop-delay` log lines decide:
- mean/p99 HIGH (tenshundreds ms) → CPU-blocked → propose worker_threads/child-process upload pool OR a
smart concurrency cap (WITH the user's nod — it's hard to reverse and touches credentials/abort/rotation).
- delay LOW while it still lags → IO-bound → threadpool bump already addresses it; if not, look at socket
caps / undici Agent connection limits / per-origin pooling, NOT workers.
Do NOT build the worker refactor before this number exists.