The v3.3.103 log (real 2464-job, 4-hoster batch ramping to 95 concurrent) confirmed
the statSync fix held (batch-start main spike 336→231ms with 10× more jobs) and the
whole 90s ramp to 95 active was pristine (event-loop mean ~11ms, fps=32,
longtasks=0). The one residual: rapidly clicking tabs DURING the 95-active upload
produced 210-221ms renderer long-tasks (proc=0ms → layout/paint, not JS).
Cause: the Recent-uploads panel rendered every sessionFilesData row (up to 2000)
into the DOM non-virtualized — the exact analog of the History table before it was
virtualized in v3.3.102. Switching to that view laid out ~2000 rows (~210ms on the
RDP VM).
Fix — virtualize renderRecentUploadsPanel, mirroring the queue/History pattern:
- The tbody gets only the visible rows plus top/bottom spacer <tr> sized from
VIRTUAL_ROW_HEIGHT. A rAF-coalesced scroll handler and a ResizeObserver on
.recent-files-table-wrap re-render the visible window (the ResizeObserver also
serves as the show-trigger when the hidden panel gains size). _recentWorking holds
the sorted set. The insertAdjacentHTML append-only fast path is dropped — a
~40-row window re-render is cheap, so every render just re-renders the window; on
prepend (date desc) the scroll position is preserved (scrollTop=0 at top, else
+= added*ROW_HEIGHT).
- Selection stays correct: _buildRecentRowHtml already stamps the selected class
from selectedRecentIds.has(row.order) per row, so an off-screen-selected row
renders selected when scrolled into view; selectedRecentIds remains the source of
truth and shift-select already reads the sort cache, not the DOM.
- styles.css gives .recent-file-row a fixed 28px height so the virtualization math
is exact (the table already had table-layout:fixed, so no column-jump fix needed).
- _renderRecentVirtualRows returns early when there are no rows, so it never wipes
the empty-state message.
Verified with Playwright at 2000 rows (bounded container): view-show layout drops
from ~118ms to ~2.4ms, the DOM stays at 29-39 rows, scrolling maps to the correct
rows, the scrollbar height is exact, an off-screen-selected row renders with the
selected class, and the row height is exactly 28. Every large table (Queue,
History, Recent) is now virtualized.
407 tests pass; clean Electron boot.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The v3.3.102 log (real 224-job batch) confirmed History virtualization works and
steady-state uploads are pristine (fps=32, event-loop mean 11.8ms). The remaining
residual was a ~6s spin-up burst at batch start: the main event loop blocked for
336ms with cpu=0%core (i.e. blocked on I/O, not computing) and the renderer janked
196-391ms, then everything settled clean.
A multi-agent investigation plus an adversarial review corrected the obvious-looking
hypothesis. The renderer's uncapped progress-batch drain is NOT the cause:
handleProgress only mutates plain JS state and schedules already-coalesced renders
(one per frame), and main coalesces progress to ~50 latest-per-job entries per
100ms. Chunking that drain would fix nothing — and the reviewer showed it would
REGRESS correctness: requestAnimationFrame throttles to ~0 when the window is
minimized (the common state for a background uploader), so a rAF-chunked drain would
grow an unbounded backlog and defer persistQueueStateSoon for every buffered item,
losing terminal 'done' events on close (the queue-persistence ghost-fix class). So
that path is deliberately not taken.
The real cause (cpu=0%core = blocked on I/O) is a synchronous fs.statSync storm in
UploadManager.startBatch: the dedup loop ran up to DEDUP_CHUNK=200 synchronous
fs.statSync calls in a single tick before yielding (200 x ~1.68ms on the user's VM
= the exact 336ms), on a disk already saturated by the 1MB read-ahead.
Fix — make the batch-start stats non-blocking:
- The dedup loop now dedupes synchronously (cheap Map work) and then stats the
unique files in parallel via await Promise.all(fs.promises.stat ...) per chunk, so
the stat I/O runs on the libuv threadpool and the main thread never blocks. The
results-Map shape ({name,size,results:[]}) and dedup semantics (size 0 on failure)
are unchanged.
- The per-job statSync fallback is converted to await fs.promises.stat for
consistency (it sits in an async function before the first real await; the cached
size from dedup already lets nearly every job skip it).
Tests: the upload-manager mocks override fs.statSync; they now also override
fs.promises.stat with the same fake sizes (upload-manager.test.js x2,
suspect-reject-alternates.test.js). 407 tests pass; clean Electron boot.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
v3.3.101's gate removed the get-history parse on tab switch, but the log still
showed tab clicks at ~216ms with a ~197ms renderer long-task and NO get-history
(proc=0ms → pure browser layout, not JS). Cause: `.view{display:none}` ->
`.active{display:flex}` and the History table built up to 2000 non-virtualized
<tr>, so making the view visible laid out 2000 rows (~197ms on the RDP VM). The
queue was already virtualized; the History table was the only large non-virtual
one.
Measured the options with Playwright on the real DOM (this machine; the user's VM
is ~1.7x slower):
- current 2000 rows (auto layout): 118ms
- content-visibility + table-layout:fixed: 117ms (no help — rows still laid out)
- cap to 300 rows: 16ms, but rejected: History rows are per-file-per-hoster, so a
single 1280-file batch is ~3840 rows and a small cap would hide a recent batch's
links
- virtualize (40 visible of 2000): 2.3ms
Fix — virtualize renderHistoryTable, mirroring the queue's _renderVirtualRows:
- The header is always rendered; tbody#historyBody gets only the visible rows plus
top/bottom spacer <tr> sized from VIRTUAL_ROW_HEIGHT. A rAF-coalesced scroll
handler and a ResizeObserver on #historyContainer re-render the visible window;
the ResizeObserver also serves as the show-trigger (a hidden 0x0 container that
gains size on tab activation re-renders at the correct height). Sorting resets
scrollTop and re-renders; the copy-link / sort-header click delegation is
unchanged. _historyWorking holds the sorted working set.
- styles.css: .history-table gets table-layout:fixed plus scoped column widths so
columns do not jump as rows scroll in and out. This is scoped to .history-table
and does not touch the .col-* classes the queue shares.
Verified end-to-end with Playwright at 6000 rows: show cost 1.1ms (was 118ms), DOM
stays 32-42 rows, scroll maps to the right rows (top/middle/bottom), scrollbar
height exact, column widths stable, rows update on scroll. All rows remain
scrollable (no UX loss); the show cost drops ~100x.
407 tests pass; clean Electron boot.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
v3.3.100's interaction instrument pinpointed the residual UI lag exactly: every
slow click was a tab switch into the History view (button.tab / nav.tab-bar,
200-840ms), each coupled to `ipc get-history wall=150-200ms sync` +
`main-longtask blocked=242-289ms lastIpc=get-history` + a 200-248ms renderer long
task. Uploads themselves are pristine (event-loop mean 11.7ms; the only spikes line
up with the get-history tab switches).
Root cause: the History tab handler called loadHistory() UNCONDITIONALLY on every
activation (renderer/app.js) even though it tracks a `_historyDirty` flag it never
checked. Each call synchronously readFileSync + JSON.parse's the ~185MB /
30000-entry electron-history.json (no cache), ships all 30000 batches across IPC,
and the renderer flattens ~120000 row objects before .slice(-2000) for the DOM (the
DOM is already capped at 2000).
Fix (the adversary-verified safe subset):
- Gate the History-tab load: only reload when `_historyDirty || !_historyEverLoaded`.
Added `_historyEverLoaded`, and both flags are now set inside loadHistory() after
the fetch succeeds (so a failed load retries). Dirty-coverage is complete — every
history append routes through batch-done -> appendHistory and upload-batch-done ->
handleBatchDone which sets _historyDirty=true. Result: repeat History tab switches
with no new uploads do zero IPC/parse/flatten and are instant. (The first open
after a new batch still parses once ~450ms; removing that needs a parse-cache or
JSONL storage, deferred.)
- Diagnostics history regression: getHistory() read loadConfig().history, but since
the v3.3.99 history split load() returns history:[] in packaged mode, so remote
diagnostics reported totalBatches:0 despite 30000 real batches. It now reads
loadHistory() (injected via the collector deps), with a backward-compatible
fallback to loadConfig().history when not provided.
407 tests pass (2 new diagnostics regression tests); clean Electron boot.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the last measurement gap. The main process was already fully instrumented
(every ipcMain handler timed at >=50ms, a 100ms main-thread long-task monitor with
channel attribution, config load/serialize timing). Account switches were already
covered too: switchAccount is a trivial synchronous Map-set and the rotation work
is async, so a switch cannot block the main thread, and any block that did occur
would surface in the long-task monitor.
The real gap was the renderer side: the renderer-perf line was gated on active
uploads (idle clicks were never logged) and only reported an aggregate longtask
count — no per-interaction latency and no element attribution. So a switch/sort/tab
that janked in the renderer (not main) was invisible.
renderer/app.js (additive, self-silencing, wrapped in try/catch):
- An Event Timing observer (PerformanceObserver type:'event', durationThreshold:50,
buffered) logs `renderer-interaction <type> dur=Xms proc=Yms target=<el>` for every
user interaction whose latency exceeds 50ms — always on, idle or under load — and
names the element (id / first class / data-action / aria-label / title). This is
the direct click->reaction latency the user feels.
- The longtask observer now also logs `renderer-longtask dur=Xms` immediately for any
single renderer long task >=100ms, regardless of upload state.
405 tests pass; clean Electron boot. Every action — main or renderer, idle or under
load — now names itself in the log if it is slow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The v3.3.98 instrument exposed the actual cause of the 1-2s UI button lag, and it
was none of the read-path suspects. electron-config.json had grown to 38.5MB and
was being load()/structuredClone()/JSON.stringify()'d ~137 times in a 73s window
(140-592ms each) on the Electron main thread — roughly 47% main-thread occupancy.
That is the lag: a button click lands while the thread is mid-clone of a 38.5MB
object. The 1MB read-ahead could not touch it.
The bulk is history, not the queue. Each batch-done appended the upload-manager
summary verbatim, including the full per-file result list (per-hoster URLs), into
config.history; default historyRetention='all' never prunes, so it grew unbounded
(75 batches ≈ 38.5MB). The "queue=undefined" in the perf log was a logging bug
(reading .length on the pendingQueue object), not an empty queue. Writes drove the
storm: queue-persistence (save-global-settings) did two loads + one 38.5MB
serialize per call, and _atomicWrite nulls the cache so the next load is a full
38.5MB reparse; even cache hits structuredCloned the whole 38.5MB. The per-job
upload path makes zero config calls, so pending=1280 was never the driver.
Fix — move history into its own file so it is never parsed/cloned/serialized on a
config op (a 5-agent investigation + adversarial review chose this over three
read-side half-fixes; it is the only change that removes the clone AND the
serialize AND the post-write reparse at once):
- History lives in electron-history.json. _migrateHistory() runs once at init
(packaged only) and is fail-safe: it writes history.json (tmp → fsync → rename),
re-reads and verifies the entry count, and keeps a permanent
electron-config.json.pre-history-split.bak BEFORE the config is ever allowed to
drop its history. If verification fails it leaves history in the config (retry
next launch). _loadImpl returns history:[] once migrated, so the cached object is
tiny (cheap clones); the config file shrinks to ~KB on the first save (cheap
reparse) and _serializeForDisk writes ~KB (cheap serialize).
loadHistory/appendHistory/pruneHistory/clearHistory go through history.json on
their own write-queue with a no-clobber guard; the legacy config path stays as a
fallback when migration did not run.
- Validated against the real 185MB / 30000-entry bench fixture: migrate 1.5s once,
every entry preserved + .bak kept; load() 631ms cold once → 0.1ms after the first
save strips the file (185MB → 2.1KB); loadHistory() still returns all 30000.
Dropped per review (one-variable + risk): loadShallow (moot after the split),
cache-repopulate (its gate can never fire), and a per-batch resolution cache (stale
account pools → the rotation/byse failover-regression class — the one thing that
could silently corrupt uploads).
Instrumentation (the user asked to measure everything; all additive, threshold-
gated, MHU_PERF=0 disables):
- ipcMain.handle/.on are centrally wrapped to log `ipc <channel> wall=Xms sync=Yms`
over 50ms — the button-press→response latency — hardened (Promise.resolve(p)
.finally + try/catch'd logging) so a logging failure can never break IPC.
- A 100ms main-process drift monitor logs `main-longtask blocked=Xms lastIpc=… gc=…`
for any single main-thread turn over 100ms (catches GC, fs scans, serialize that
the IPC timing structurally cannot see).
- config-store perf lines gain via=<caller> and wqDepth=, and the queue= logging
bug is fixed (now reads pendingQueue.queueJobs.length).
405 tests pass (9 new migration tests: preserve-count, round-trip, save() never
loses history, crash-window fallback, idempotency). Clean Electron boot, no repo
pollution (migration is packaged-only).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
v3.3.97 (UV_THREADPOOL_SIZE 64→8) was a decisive win — mean event-loop-delay at
70 active uploads dropped 200ms→~11ms (18×), rss 577→287MB, renderer healthy in
14/15 windows. But the user reports it is still not perfectly smooth. A focused
multi-agent investigation plus an adversarial review localized the residual to
TWO distinct, separately-measured spike sources:
1. Read-bursts. In the tail windows the file-read histogram inverts: FSReqCallback
climbs to 66-70 against threadpool=8 (~8.75× queue depth) while SimpleWriteWrap
(socket writes) collapses to 4-24 and mean delay rises to 30-42ms. GC is ruled
out (gcMax ≤27ms in every window). The clean inversion at a stable active=70 /
pending=1287 shows the reads are causal, not a symptom of a block elsewhere.
2. A suspected synchronous config-persist stall. save() → load() reparses the whole
electron-config.json — which now carries the 1287-job pending queue nested in
globalSettings plus full history — on every persist (because _atomicWrite nulls
the read cache), then _serializeForDisk JSON.stringify(…, null, 2) of all of it.
One tail sample (max 1021ms, heap spiking to 142MB) fits a large synchronous
structuredClone+stringify, but it is a single confounded point, so this build
only INSTRUMENTS the path rather than asserting the cause.
This release ships one behavioral change (kept to a single variable so the next
log attributes cleanly) plus measurement:
- highWaterMark 256KB→1MB in all five streaming read loops (lib/hosters.js,
doodstream/voe/vidmoly CHUNK_SIZE consts, and the inline value in
clouddrop-upload.js:108 — NOT the 16MB server chunk at clouddrop-upload.js:12).
UV_THREADPOOL_SIZE stays 8. This deepens each stream's read-ahead cushion from
~0.43s to ~1.7s at the per-stream rate, so a stream tolerates the threadpool
queue without starving its socket write, and cuts read-completion callbacks and
per-chunk Buffer allocations ~4×. Byte-correctness is unaffected: Content-Length
is preamble+fileSize+epilogue, independent of chunk size, and the chunk size
never touches the multipart boundaries. Fully reversible; a dedicated read-
concurrency semaphore is held in reserve if 1MB does not clear the bursts.
- config-store.js now times load() (the full reparse, which the account-failed
handler also hits per failure) and the _commit serialize, logging
`config-load …` / `config-serialize wall=…ms bytes=… hist=… queue=…` when the
synchronous work exceeds 20ms. load() is split into a timing wrapper + _loadImpl;
the timer is a no-op until main.js wires configStore.setPerfLog → logInfo.
The renderer batch-drain fix for the one observed 243ms longtask is intentionally
deferred: that jank is downstream of the main-thread read-burst flooding IPC, so
fix#1 should make it self-heal; bundling it would confound the measurement and
touch the progress hot path. All 397 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The v3.3.96 eventloop-delay logs from a real 70-connection run pinpointed the
high-concurrency lag to the file-read path. At a CONSTANT active-job count the
process flips between two clean regimes:
HEALTHY ELD ~11ms, rss 268-308MB: SimpleWriteWrap ≈ active, FSReqCallback ≈ 0-1
BLOCKED ELD 49-217ms, rss 540-610MB: FSReqCallback ≈ active (62-71 reads in
flight), SimpleWriteWrap ≈ 0-4
Both the ELD spike and the rss balloon track FSReqCallback (libuv-threadpool file
reads) exactly — not crypto, not the renderer, not GC alone. All five uploaders
read identically via fs.createReadStream({highWaterMark: 256KB}); byse/doodstream/
voe run through the generic uploadFile in hosters.js (no dedicated module).
This build is both a candidate fix and a discriminator, per advisor review:
- UV_THREADPOOL_SIZE 64→8 (main.js:1). One reversible line, NOT an upload cap —
70 uploads still run. 8 concurrent 256KB reads sustain ~100MB/s, far above the
41MB/s aggregate, so it cannot bottleneck throughput even on the slow VM disk.
Strong suspicion that tp=64 made it worse: it removed the natural read-
serialization (default 4 threads) and let all 70 streams' reads fire at once,
flooding the loop with completion callbacks in lock-step bursts.
- ELD line now also logs heap=heapUsed ext=external ab=arrayBuffers and
gc=/gcTotal=/gcMax=ms (PerformanceObserver entryTypes:['gc'], reset per window).
The 70×256KB ≈ 18MB of read buffers cannot account for the ~300MB rss swing —
that is heap/object churn, so GC must be measured directly.
Decision rule for the next real-run log:
- ELD drops with tp=8 → read over-parallelism confirmed (keep 8 or add
a dedicated read-semaphore).
- ELD high + GC pauses align → heap churn, hunt the allocator.
- ELD high + GC flat → causation was reversed, pivot.
No upload-behavior change; the concurrency cap the user explicitly rejected is
untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The v3.3.94 measurement computed the main-process event-loop delay, CPU%, per-hoster connection distribution and transient error counts, but logged only '[INFO ] perf': logInfo(ctx, msg) treats a string first arg as the whole message and discards the second, so the payload was thrown away (confirmed in the user's upload-debug.log). Pass the line as a single argument so the real numbers reach the log.
Also: assets/ was missing from the electron-builder files list, so app_icon.ico was never packaged into the asar — new Tray() threw an unhandled rejection on every startup and left no tray icon. Package the icons and wrap createTray in try/catch with a nativeImage fallback so a missing/invalid icon can never crash tray creation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Clicking 'install update' while many uploads were running left the update stuck at 'Update wird heruntergeladen...' forever, requiring a manual restart. Two causes, both fixed:
1) The 95 MB installer download competed with the active uploads for the (saturated) main process, CPU, network and connection pool, so it barely progressed. The install handler now calls uploadManager.cancel() first — the app quits and relaunches for the update anyway, and the queue is already persisted (non-terminal jobs restore as 'Bereit'), so freeing the resources is correct and makes the download fast.
2) The download read loop had no timeout, so a stalled stream hung indefinitely with no error. Added a 45 s stall-timeout: if no data arrives, the download aborts with a clear message ('Download haengt — seit 45 s keine Daten ... bitte erneut versuchen') instead of freezing, so the retry button works.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
User asked to measure everything so we can pinpoint the lag/ECONNRESET at 70 concurrent uploads. All additive instrumentation, no upload-behavior change.
main process: the eventloop-delay log line now also reports cpu=X%core (process.cpuUsage delta over wall time; >100% means multiple cores), rss, per-hoster live connection distribution (active-by-hoster), cumulative transient (ECONNRESET-class) error count, and pending count — pulled from a new UploadManager.getDiagnostics(). This shows whether the main thread is CPU-bound and which hoster is oversubscribed.
renderer: a PerformanceObserver('longtask') plus a requestAnimationFrame frame-time monitor log 'renderer-perf' every 5s while uploading — fps, jankFrames (>33ms), worstFrame, longtask count and max. This is the direct renderer ground truth (real frame rate under load), which the component-timing harness could not capture. Low fps / high jank => renderer is blocked; ~60fps while it still feels laggy => the cpu/eld numbers in the same log decide CPU-bound vs IO-bound in the main process.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
maybeAddSessionFile computed formatDateTime(new Date()) at the top, before the status==='done' guard that early-returns for every other status. formatDateTime runs two Intl locale formats (~83us measured), so it executed on every progress event — onUploadProgressBatch loops the M-item batch into handleProgress -> maybeAddSessionFile, i.e. 10xM times per second — and threw the result away for all non-done events (the vast majority while uploading).
That made each progress batch a synchronous main-thread block scaling with the active-upload count: ~2.4ms at 25 concurrent, ~5ms at 61, every 100ms, on top of render and sort — enough to blow the 16ms frame budget and stutter scrolling. It is per-event, not per-render, so it janks regardless of scroll position, matching the user's report that the lag appears above ~50 connections and when the uploading rows are scrolled out of view.
Move the formatDateTime call inside the dedup block so it runs once per genuinely-new completed upload. A faithful Blink benchmark at the user's regime (500 rows virtualized, dynamic progress sort, scrolling) shows the per-batch cost drop from 1.7/3.2/4.1ms at 25/50/61 concurrent to a flat 0ms, and frame P95 at 61 concurrent from 7.3ms to 4.2ms.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make the single high-concurrency measurement decisive. The ELD line now appends process.getActiveResourcesInfo() as a compact type-histogram, so one run splits all three readings in a single line: high mean/p99 means the main thread is CPU-blocked (workers/cap justified); low delay with many TCPSocketWrap/FSReqCallback/GetAddrInfoReqWrap resources means IO-bound queueing (threadpool/socket tuning, not workers); low delay with few resources means it is not saturated at all. Pure metrics — no credential-redaction path touched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The user runs 50+ concurrent uploads (parallel counts raised deliberately). Every async uploader feeds undici from fs.createReadStream and resolves DNS via getaddrinfo — both go through the libuv threadpool, whose default size is 4. At 50 concurrent uploads, file reads and lookups serialize 4-at-a-time: a hard cliff at a small connection count that matches the 'lags from X connections onward' symptom. Raise the cap to 64 as the first statement (before require('electron'), so libuv reads it when it lazily inits the pool; an explicit env override still wins). Threads are created on demand, so a higher max costs nothing when unused — reversible, zero upload-core change.
Also enable perf_hooks.monitorEventLoopDelay and log mean/p99/max/stddev every ~5s while uploading. This is the ground-truth instrument that splits the two competing explanations for the lag: a high event-loop delay means the main thread is CPU-blocked (TLS/crypto) and only workers or a concurrency cap will help; a low delay while uploads stall means the work is IO-bound and the threadpool/socket config is the lever, not workers. The numbers are pure metrics — they never touch the credential-redaction path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The rotation-retry and suspect-alternate progressCb callbacks called _emitProgress (a synchronous EventEmitter emit plus a fresh object spread) on every stream chunk — hundreds per second per job — because they lacked the 250 ms lastEmitTime gate the primary upload path already has. With many concurrent uploads in rotation or suspect mode that is real main-thread emit amplification.
Mirror the primary path's gate exactly: the activeEntry speed/bytes mutation stays ungated so the stats timer and speed monitor keep seeing fresh values; only the _emitProgress call is throttled to ~4/sec. Behavior-preserving — identical progress data, fewer emits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_uploadChunked read each 16 MB chunk with fs.readSync on the main JS thread — the only one of the five uploaders that blocks synchronously (the other four stream async). Each readSync stalls the whole event loop ~5-9 ms on SSD, 30-100 ms on a slow disk, freezing all progress emits, IPC, renders and every other concurrent upload for that window. The stall scales with the number of simultaneous clouddrop uploads, matching the 'feels laggy while uploading, worse with more at once' symptom at modest CPU (one core pinned, ~40% of 8).
Swap fs.openSync/readSync/closeSync for fs.promises.open + await fh.read + await fh.close. Buffer reuse and the partial-last-chunk subarray view are unchanged. Verified byte-identical to the old loop via SHA-256 over every chunk-boundary case (full chunk, partial last chunk, 2/3/4-chunk files, single byte) before shipping — a chunk-read bug would corrupt the upload.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
52-agent audit of the v3.3.84/85 remote-diagnostics code. App is healthy for this
user's actual usage. Shipped the two zero-redaction-surface fixes (ws maxPayload,
sendToClient guard). Deferred the cold-path server_health O(historySize) freeze
because the fix refactors the credential-redaction collectors (leaked twice before).
Lesson: "not persistence, so safe" is a fallacy — the redaction layer is an equally
catastrophic guarantee surface (silent secret leak). At an audit goal, find+document
is the deliverable; don't cut into a twice-leaked redaction pipeline under a Stop hook.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two isolated hardenings of the opt-in remote/diagnostics WS server, surfaced by
the session-wide diagnostics audit:
1. WebSocketServer was created with no maxPayload, so ws defaults to 100 MiB per
message. The connection handler runs JSON.parse(raw) on the FIRST message
(the auth frame) before authentication, so any peer past the IP allowlist
could send a huge payload and force a synchronous multi-MB JSON.parse on the
main-process event loop — an unbounded freeze/DoS sink. Diag, auth and WebRTC
signaling messages are all small; cap maxPayload at 256 KiB to close it.
2. sendToClient did ws.send(JSON.stringify(data)) with no readyState/try guard
(unlike broadcast, which checks ws.readyState === 1). A send on a closing
socket, or a stringify throw, escaped the diag-response callback as an
uncaughtException — a potential crash. Mirror broadcast: send only when
readyState === 1, wrapped in try/catch.
Both are isolated to the transport layer with zero redaction surface. The audit's
larger finding — server_health doing O(historySize) synchronous work per request
(6-7 full-config clones + unbounded history walks) — is a real freeze, but ONLY on
the cold opt-in diagnostics path with a large history (this user: 23 rows), and the
safe fix cuts into the credential-redaction collectors (which have leaked twice);
deferred and documented in tasks/todo.md rather than operated under risk.
397/397 tests pass, eslint clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Records the 18-agent line-by-line audit of every line written this session: the
reported lag was v3.3.87 (recent-panel cliff); the audit found no second cause
affecting this user. doodstream _debugLog sync-fs gated (shipped). config-store
load()/serialize history-scaling costs are real but sub-ms at this user's scale
and the fix is risky persistence surgery — deferred, documented with measurements.
Lesson: "audit every line" = look + measure + risk-appropriate decision, NOT
fix-everything; the load() perf win and its corruption risk are the same coin
(shared batch refs), so there is no safe version — defer, don't ship.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
doodstream-upload.js's _debugLog ran with NO verbose gate: every call did a
synchronous statSync (via maybeRotateLogFile) + appendFileSync directly on the
main-process event loop, ~8-15x per upload (server probe, response, redirect,
result page, filecode parse, hidden fields, submit/follow, plus retry branches).
DoodstreamUploader runs in the main process, so each pair of sync fs syscalls
blocked the event loop while uploading — delaying IPC, upload-progress-batch
forwarding, tray-tooltip and webhook handling for every other concurrent upload.
A constant per-upload main-thread tax (not history-scaling), surfaced by the
session-wide lag audit and confirmed as the one finding that bites in the real
upload scenario.
Fix: gate _debugLog behind the existing globalSettings.logVerbose setting
(default false), exactly mirroring main.js logDebug/_logVerbose. A module-level
_debugVerbose flag + setDebugVerbose() setter, an early-return at the top of
_debugLog, and one wire at main.js's single setLogVerbose chokepoint (covers
boot + save-config + the verbose toggle). When verbose is off the doodstream
trace simply isn't written — same contract as the main debug.log — and the
per-upload sync fs disappears. When a doodstream issue needs tracing, enabling
verbose restores the full trace.
The config-store audit findings (load() history-clone cost, per-write history
serialize) are real but scale only with history size — tens of microseconds at
this user's 8-batch config, and the safe fix is risky persistence surgery on
credential-bearing code for a latent micro-cost; deferred and documented in
tasks/todo.md rather than shipped.
397/397 tests pass, eslint clean, gate wiring verified (shared module instance,
default-off, toggles).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Records the verified root cause + the dismissed-with-evidence findings:
- _sessionFileKeys "separator mismatch" = FALSE POSITIVE (real U+0001 chars, verifier Read rendered them invisibly)
- queueJobs O(N) per-render scan = real but Blink-measured <0.1ms at 3000 jobs -> skipped
- standing 2000-row relayout = median 0.4ms -> no virtualization needed
- doodstream sync _debugLog = constant freeze, deferred to a separate change
Lesson: measure magnitudes at the real artifact before fixing a "scales-with-X" cause;
profile in real Blink (Playwright) not jsdom; verify multi-agent findings against primary
evidence (char-code dump caught the control-char false positive).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The recent-uploads panel had a cheap append-only fast path, but it was gated on
`rows.length > _recentLastRenderedLen`. maybeAddSessionFile caps sessionFilesData
by push-then-slice (2000 -> 2001 -> sliced back to 2000), so once a session
produces more than SESSION_FILES_CAP rows the length is pinned at the cap and the
gate is false forever. Every subsequent completion then fell through to the full
`tbody.innerHTML = rows.map(...).join('')` rebuild of all ~2000 rows.
The cap is per (link x file x hoster), so with 4-5 selected hosters the 2000 cap
is hit at only ~400-500 distinct files — very reachable in a long folder-monitor
session. Profiled in Chromium (same Blink engine as Electron, table-layout:fixed):
the full 2000-row rebuild costs ~80ms and ran on EVERY completion past the cap — a
repeating ~80ms main-thread freeze. That is the "fine on a fresh start, gets laggy
after many uploads while CPU (~40%) and RAM (~6GB, stable) stay normal" symptom: a
render-thread stall, not CPU saturation or a memory leak.
Fix: track newly-pushed rows in _recentPendingAppends (incremented in
maybeAddSessionFile, consumed every render) and gate the fast path on
`pendingAppends > 0` instead of length growth, so it survives the cap. Prepend the
new rows, then evict the same overflow count from the DOM bottom (oldest rows,
which is where the date-desc view places the front-of-array entries the cap slices
off). DOM work is O(added) again. The fast path is gated behind an explicit
`appendOnly` flag passed only by scheduleRecentRender's rAF, so selection / delete
/ clear / sort / batch-done renders stay full rebuilds and cannot wrong-evict or
double-prepend.
Verified in Blink over a simulated 5000-completion session (3/frame, far past the
cap): per-frame render 80ms -> median 7.4ms (>10x), and the DOM stays exactly equal
to the data (cap held at 2000, newest-on-top, oldest evicted, zero duplicates).
397/397 tests pass, eslint clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A 13-agent hunt pinned the "wird mit der Zeit laggy" symptom (CPU/RAM normal, UI
sluggish after many uploads) to the main process re-doing config I/O that scales
with the ever-growing history, stalling the synchronous main event loop so the
renderer's IPC round-trips feel laggy. The renderer render path was already
optimized (virtualized queue, capped panels) — confirmed clean.
This commit lands the two contained fixes (T1 + T3); the queue-persist rewrite (T2)
follows separately.
config-store (T1):
- load() now has an in-memory cache keyed on the file mtime+size. The processed
config (merged + credential-decrypted) is re-read/re-parsed/re-DPAPI-decrypted
ONLY when the file actually changes; our writes invalidate it, external edits
change mtime/size so the cache misses. Eliminates a full disk read + JSON.parse of
the whole growing history + per-credential decrypt on the vast majority of the ~38
load() call sites. load() always returns a structuredClone so callers can mutate
freely without corrupting the cache.
- _serializeForDisk clones ONLY the hosters subtree (the only thing encryptCredentials
touches) instead of JSON.parse(JSON.stringify(whole config)) — no more deep-cloning
an 8 MB history on every write.
- _atomicWrite refreshes the .bak with a raw fs.copyFileSync instead of
read + JSON.parse + write (it was re-parsing the full config a 2nd time per write).
main.js (T3):
- The log-flush paths resolved the log file via configStore.load() ~8x/second during
uploads (re-reading + cloning the whole config just to read logMode/logFilePath).
Cache those two strings in module scope, invalidate on the settings-save handlers.
Verified: 26 config-store tests (incl. new cache-correctness: independent clones,
external-change invalidation, save invalidation) + full 394-test suite green, lint 0
errors. Benchmark (8000-batch / 4.6 MB history): log flush no longer calls load() at
all; the remaining per-write history serialize is what T2 removes from the hot path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the one link the unit/wiring tests covered only by composition: binds 0.0.0.0,
allowlists a real LAN IPv4, and asserts auth-ok over a real socket (the Tailscale path).
Skips when no non-internal IPv4 interface exists.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Matches the Real-Debrid-Downloader's rd-diagnostics-mcp model so the read-only
diagnostics agent is reachable over Tailscale (or any private tunnel) the same way
the downloader is, instead of requiring an SSH local-forward.
- lib/ip-allowlist.js (NEW): fail-closed IP allowlist — normalizeIp strips
::ffff:, loopback is always allowed, an empty allowlist accepts loopback ONLY
(fail-closed), exact IP + CIDR (incl. the Tailscale CGNAT range 100.64.0.0/10) +
wildcard rules. The real socket peer IP is the authority (never a forwarded header).
- remote-server.js: rejects non-allowlisted peers at connection (close 4005). Opt-in
via config.allowlist (the existing remote-control server, which passes none, is
unaffected). Loopback always passes, so local + SSH-forward use keeps working.
- Two bind modes (config diagnostics.bindMode): "local" -> 127.0.0.1 (default),
"network" -> 0.0.0.0 but ONLY when a non-empty allowlist is set (else it stays
loopback, fail-closed). The allowlist + token gate access; the tunnel
(Tailscale/WireGuard) is the confidentiality layer (transport is still plaintext ws://).
- The connection code now carries the host: mhu1_<base64url{v,h,p,t,n,fp?,s?}>. The
gateway decode is tolerant of the legacy {port,token,label} keys; connect_server
takes the host from the code (host arg is an optional override). Proven end-to-end:
the integration harness now connects with NO host arg and resolves it from the code.
- Renderer: Sichtbarkeit selector (local/network), public-host input with
suggested-host chips (os.networkInterfaces — the Tailscale IP shows up there),
allowlist textarea (network mode), and network-requires-allowlist validation.
- main.js: bindMode->host, getSuggestedRemoteHosts, host-in-code, allowlist plumbed
into startDiagnosticAgent + the diagnostics IPC (get/save/status).
- docs: rewritten for Tailscale (set the allowlist to your tailnet, put the Tailscale
IP/MagicDNS in the code address — no SSH forward needed).
This supersedes the v3.3.85 hard loopback-lock with the downloader's allowlist model.
Tests: lib/ip-allowlist (8) + remote-server allowlist wiring/loopback (2) + gateway
decode (host short-key + legacy tolerance). 393 app tests + 9 gateway tests + e2e +
host-in-code integration + adversarial all green; lint 0 errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Repo-side hardening and tooling from the intensive test round (none of this ships
in the app installer).
- gateway: registry.json (holds bearer tokens) now gets a best-effort owner-only
NTFS ACL on Windows via `icacls /inheritance:r /grant:r <user>:F` (the chmod
0600 is a no-op on NTFS); verified the file ends up <user>:(F) only.
- gateway: connect_server now reads the app version from the real get_system_info
shape (data.app.version / data.agent.version), so "connected to vX.Y.Z" works.
- gateway: read_log tool description documents grep as a case-insensitive substring
filter with "|" alternation (not a regex), matching the agent-side change.
- gateway: standalone verification harnesses moved to gateway/verify/ (so
`node --test` only sweeps real unit tests) and exposed via `npm run verify`:
e2e-verify, integration-mcp (live gateway-MCP <-> agent, all 14 tools), and
adversarial-probe (redaction fuzz + ReDoS + lockout). `npm test` runs the units.
- eslint: gateway/** now lints as ESM (sourceType module) via a dedicated block;
global ignores fixed so `eslint .` is clean across the whole project (0 errors).
- docs/remote-diagnostics-setup.md: made the transport story honest — the agent
speaks plaintext ws:// over enforced loopback; the SSH/WireGuard tunnel is the
ONLY confidentiality layer (wss/TLS + cert-pin is a documented future mode, not
active). Removed the stale "bind to a LAN/VPN IP" guidance (loopback is enforced).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Intensive end-to-end testing (a live gateway-MCP <-> agent integration harness +
an adversarial redaction/abuse probe + an independent security audit) surfaced
three real issues in the shipped read-only diagnostic agent. All run in lib/**,
which is packaged in the app.
1. grep ReDoS froze the Electron main process. read_log compiled the
client-supplied grep into `new RegExp(grep, 'i')` and ran it synchronously over
the log tail IN the main process. A catastrophic pattern (e.g. "(a+)+$" against
a long line) hangs the whole app — empirically confirmed (8s timeout, killed).
JS regex is synchronous and uncancellable, so grep is now a case-insensitive
literal substring filter with "|" alternation ("error|timeout|502"). Provably
linear-time; covers the real diagnostic need.
2. Prototype-chain whitelist bypass. The op table was a plain object literal, so
handle("constructor" | "toString" | "valueOf", ...) resolved an inherited
Object.prototype function, passed the `typeof fn === 'function'` guard and
returned {ok:true}. Harmless functions today, but a whitelist-integrity hole.
Now guarded with a string check + Object.prototype.hasOwnProperty.
3. Redaction defense-in-depth gaps. redactLogText now also scrubs: basic-auth URL
passwords (scheme://user:pass@host), Authorization: Basic, JWTs (eyJ...x.y.z),
and bare/JSON session= values. Mostly theoretical in today's readable logs
(secret-bearing bodies go to the excluded doodstream-debug.log; other hosters
throw static strings) but matters as the verbose-logging surface grows.
Verified: 383 app tests (incl. new regression tests for all three), the live
gateway-MCP integration harness (all 14 tools, zero leaks, error paths), the
adversarial probe (14/14+ secret shapes scrubbed, ReDoS 1ms, lockout, malformed
args), e2e gate, lint 0 errors. Only residual: a standalone high-entropy blob with
zero key/Bearer/URL context — inherent to any denylist, acknowledged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
registry.json holds each server's bearer token (the connection secret). It was
written with the process default umask, leaving it group/world-readable on POSIX
multi-user hosts. Write with mode 0o600 and chmod the existing file (writeFile
only applies mode on creation). No-op on Windows (NTFS uses ACLs, and the file
already sits under the user profile and is gitignored), effective on Linux/macOS
where the gateway may run.
Gateway-only change — not part of the app installer or auto-updater, so no
version bump.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Captures the two redaction leaks from the remote-diagnostics build: a hoster-
returned opaque token surviving because value-scrub only covers stored config
secrets, and the get_config_redacted/get_queue_state default paths skipping
pattern-scrub entirely. The discriminator the first E2E missed, plus the rule:
exercise each collector with its DEFAULT args and seed fixtures with a NON-config
secret so you test pattern-scrub, not value-scrub by accident.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
