The v3.3.102 log (real 224-job batch) confirmed History virtualization works and
steady-state uploads are pristine (fps=32, event-loop mean 11.8ms). The remaining
residual was a ~6s spin-up burst at batch start: the main event loop blocked for
336ms with cpu=0%core (i.e. blocked on I/O, not computing) and the renderer janked
196-391ms, then everything settled clean.
A multi-agent investigation plus an adversarial review corrected the obvious-looking
hypothesis. The renderer's uncapped progress-batch drain is NOT the cause:
handleProgress only mutates plain JS state and schedules already-coalesced renders
(one per frame), and main coalesces progress to ~50 latest-per-job entries per
100ms. Chunking that drain would fix nothing — and the reviewer showed it would
REGRESS correctness: requestAnimationFrame throttles to ~0 when the window is
minimized (the common state for a background uploader), so a rAF-chunked drain would
grow an unbounded backlog and defer persistQueueStateSoon for every buffered item,
losing terminal 'done' events on close (the queue-persistence ghost-fix class). So
that path is deliberately not taken.
The real cause (cpu=0%core = blocked on I/O) is a synchronous fs.statSync storm in
UploadManager.startBatch: the dedup loop ran up to DEDUP_CHUNK=200 synchronous
fs.statSync calls in a single tick before yielding (200 x ~1.68ms on the user's VM
= the exact 336ms), on a disk already saturated by the 1MB read-ahead.
Fix — make the batch-start stats non-blocking:
- The dedup loop now dedupes synchronously (cheap Map work) and then stats the
unique files in parallel via await Promise.all(fs.promises.stat ...) per chunk, so
the stat I/O runs on the libuv threadpool and the main thread never blocks. The
results-Map shape ({name,size,results:[]}) and dedup semantics (size 0 on failure)
are unchanged.
- The per-job statSync fallback is converted to await fs.promises.stat for
consistency (it sits in an async function before the first real await; the cached
size from dedup already lets nearly every job skip it).
Tests: the upload-manager mocks override fs.statSync; they now also override
fs.promises.stat with the same fake sizes (upload-manager.test.js x2,
suspect-reject-alternates.test.js). 407 tests pass; clean Electron boot.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
v3.3.101's gate removed the get-history parse on tab switch, but the log still
showed tab clicks at ~216ms with a ~197ms renderer long-task and NO get-history
(proc=0ms → pure browser layout, not JS). Cause: `.view{display:none}` ->
`.active{display:flex}` and the History table built up to 2000 non-virtualized
<tr>, so making the view visible laid out 2000 rows (~197ms on the RDP VM). The
queue was already virtualized; the History table was the only large non-virtual
one.
Measured the options with Playwright on the real DOM (this machine; the user's VM
is ~1.7x slower):
- current 2000 rows (auto layout): 118ms
- content-visibility + table-layout:fixed: 117ms (no help — rows still laid out)
- cap to 300 rows: 16ms, but rejected: History rows are per-file-per-hoster, so a
single 1280-file batch is ~3840 rows and a small cap would hide a recent batch's
links
- virtualize (40 visible of 2000): 2.3ms
Fix — virtualize renderHistoryTable, mirroring the queue's _renderVirtualRows:
- The header is always rendered; tbody#historyBody gets only the visible rows plus
top/bottom spacer <tr> sized from VIRTUAL_ROW_HEIGHT. A rAF-coalesced scroll
handler and a ResizeObserver on #historyContainer re-render the visible window;
the ResizeObserver also serves as the show-trigger (a hidden 0x0 container that
gains size on tab activation re-renders at the correct height). Sorting resets
scrollTop and re-renders; the copy-link / sort-header click delegation is
unchanged. _historyWorking holds the sorted working set.
- styles.css: .history-table gets table-layout:fixed plus scoped column widths so
columns do not jump as rows scroll in and out. This is scoped to .history-table
and does not touch the .col-* classes the queue shares.
Verified end-to-end with Playwright at 6000 rows: show cost 1.1ms (was 118ms), DOM
stays 32-42 rows, scroll maps to the right rows (top/middle/bottom), scrollbar
height exact, column widths stable, rows update on scroll. All rows remain
scrollable (no UX loss); the show cost drops ~100x.
407 tests pass; clean Electron boot.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
v3.3.100's interaction instrument pinpointed the residual UI lag exactly: every
slow click was a tab switch into the History view (button.tab / nav.tab-bar,
200-840ms), each coupled to `ipc get-history wall=150-200ms sync` +
`main-longtask blocked=242-289ms lastIpc=get-history` + a 200-248ms renderer long
task. Uploads themselves are pristine (event-loop mean 11.7ms; the only spikes line
up with the get-history tab switches).
Root cause: the History tab handler called loadHistory() UNCONDITIONALLY on every
activation (renderer/app.js) even though it tracks a `_historyDirty` flag it never
checked. Each call synchronously readFileSync + JSON.parse's the ~185MB /
30000-entry electron-history.json (no cache), ships all 30000 batches across IPC,
and the renderer flattens ~120000 row objects before .slice(-2000) for the DOM (the
DOM is already capped at 2000).
Fix (the adversary-verified safe subset):
- Gate the History-tab load: only reload when `_historyDirty || !_historyEverLoaded`.
Added `_historyEverLoaded`, and both flags are now set inside loadHistory() after
the fetch succeeds (so a failed load retries). Dirty-coverage is complete — every
history append routes through batch-done -> appendHistory and upload-batch-done ->
handleBatchDone which sets _historyDirty=true. Result: repeat History tab switches
with no new uploads do zero IPC/parse/flatten and are instant. (The first open
after a new batch still parses once ~450ms; removing that needs a parse-cache or
JSONL storage, deferred.)
- Diagnostics history regression: getHistory() read loadConfig().history, but since
the v3.3.99 history split load() returns history:[] in packaged mode, so remote
diagnostics reported totalBatches:0 despite 30000 real batches. It now reads
loadHistory() (injected via the collector deps), with a backward-compatible
fallback to loadConfig().history when not provided.
407 tests pass (2 new diagnostics regression tests); clean Electron boot.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the last measurement gap. The main process was already fully instrumented
(every ipcMain handler timed at >=50ms, a 100ms main-thread long-task monitor with
channel attribution, config load/serialize timing). Account switches were already
covered too: switchAccount is a trivial synchronous Map-set and the rotation work
is async, so a switch cannot block the main thread, and any block that did occur
would surface in the long-task monitor.
The real gap was the renderer side: the renderer-perf line was gated on active
uploads (idle clicks were never logged) and only reported an aggregate longtask
count — no per-interaction latency and no element attribution. So a switch/sort/tab
that janked in the renderer (not main) was invisible.
renderer/app.js (additive, self-silencing, wrapped in try/catch):
- An Event Timing observer (PerformanceObserver type:'event', durationThreshold:50,
buffered) logs `renderer-interaction <type> dur=Xms proc=Yms target=<el>` for every
user interaction whose latency exceeds 50ms — always on, idle or under load — and
names the element (id / first class / data-action / aria-label / title). This is
the direct click->reaction latency the user feels.
- The longtask observer now also logs `renderer-longtask dur=Xms` immediately for any
single renderer long task >=100ms, regardless of upload state.
405 tests pass; clean Electron boot. Every action — main or renderer, idle or under
load — now names itself in the log if it is slow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The v3.3.98 instrument exposed the actual cause of the 1-2s UI button lag, and it
was none of the read-path suspects. electron-config.json had grown to 38.5MB and
was being load()/structuredClone()/JSON.stringify()'d ~137 times in a 73s window
(140-592ms each) on the Electron main thread — roughly 47% main-thread occupancy.
That is the lag: a button click lands while the thread is mid-clone of a 38.5MB
object. The 1MB read-ahead could not touch it.
The bulk is history, not the queue. Each batch-done appended the upload-manager
summary verbatim, including the full per-file result list (per-hoster URLs), into
config.history; default historyRetention='all' never prunes, so it grew unbounded
(75 batches ≈ 38.5MB). The "queue=undefined" in the perf log was a logging bug
(reading .length on the pendingQueue object), not an empty queue. Writes drove the
storm: queue-persistence (save-global-settings) did two loads + one 38.5MB
serialize per call, and _atomicWrite nulls the cache so the next load is a full
38.5MB reparse; even cache hits structuredCloned the whole 38.5MB. The per-job
upload path makes zero config calls, so pending=1280 was never the driver.
Fix — move history into its own file so it is never parsed/cloned/serialized on a
config op (a 5-agent investigation + adversarial review chose this over three
read-side half-fixes; it is the only change that removes the clone AND the
serialize AND the post-write reparse at once):
- History lives in electron-history.json. _migrateHistory() runs once at init
(packaged only) and is fail-safe: it writes history.json (tmp → fsync → rename),
re-reads and verifies the entry count, and keeps a permanent
electron-config.json.pre-history-split.bak BEFORE the config is ever allowed to
drop its history. If verification fails it leaves history in the config (retry
next launch). _loadImpl returns history:[] once migrated, so the cached object is
tiny (cheap clones); the config file shrinks to ~KB on the first save (cheap
reparse) and _serializeForDisk writes ~KB (cheap serialize).
loadHistory/appendHistory/pruneHistory/clearHistory go through history.json on
their own write-queue with a no-clobber guard; the legacy config path stays as a
fallback when migration did not run.
- Validated against the real 185MB / 30000-entry bench fixture: migrate 1.5s once,
every entry preserved + .bak kept; load() 631ms cold once → 0.1ms after the first
save strips the file (185MB → 2.1KB); loadHistory() still returns all 30000.
Dropped per review (one-variable + risk): loadShallow (moot after the split),
cache-repopulate (its gate can never fire), and a per-batch resolution cache (stale
account pools → the rotation/byse failover-regression class — the one thing that
could silently corrupt uploads).
Instrumentation (the user asked to measure everything; all additive, threshold-
gated, MHU_PERF=0 disables):
- ipcMain.handle/.on are centrally wrapped to log `ipc <channel> wall=Xms sync=Yms`
over 50ms — the button-press→response latency — hardened (Promise.resolve(p)
.finally + try/catch'd logging) so a logging failure can never break IPC.
- A 100ms main-process drift monitor logs `main-longtask blocked=Xms lastIpc=… gc=…`
for any single main-thread turn over 100ms (catches GC, fs scans, serialize that
the IPC timing structurally cannot see).
- config-store perf lines gain via=<caller> and wqDepth=, and the queue= logging
bug is fixed (now reads pendingQueue.queueJobs.length).
405 tests pass (9 new migration tests: preserve-count, round-trip, save() never
loses history, crash-window fallback, idempotency). Clean Electron boot, no repo
pollution (migration is packaged-only).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
v3.3.97 (UV_THREADPOOL_SIZE 64→8) was a decisive win — mean event-loop-delay at
70 active uploads dropped 200ms→~11ms (18×), rss 577→287MB, renderer healthy in
14/15 windows. But the user reports it is still not perfectly smooth. A focused
multi-agent investigation plus an adversarial review localized the residual to
TWO distinct, separately-measured spike sources:
1. Read-bursts. In the tail windows the file-read histogram inverts: FSReqCallback
climbs to 66-70 against threadpool=8 (~8.75× queue depth) while SimpleWriteWrap
(socket writes) collapses to 4-24 and mean delay rises to 30-42ms. GC is ruled
out (gcMax ≤27ms in every window). The clean inversion at a stable active=70 /
pending=1287 shows the reads are causal, not a symptom of a block elsewhere.
2. A suspected synchronous config-persist stall. save() → load() reparses the whole
electron-config.json — which now carries the 1287-job pending queue nested in
globalSettings plus full history — on every persist (because _atomicWrite nulls
the read cache), then _serializeForDisk JSON.stringify(…, null, 2) of all of it.
One tail sample (max 1021ms, heap spiking to 142MB) fits a large synchronous
structuredClone+stringify, but it is a single confounded point, so this build
only INSTRUMENTS the path rather than asserting the cause.
This release ships one behavioral change (kept to a single variable so the next
log attributes cleanly) plus measurement:
- highWaterMark 256KB→1MB in all five streaming read loops (lib/hosters.js,
doodstream/voe/vidmoly CHUNK_SIZE consts, and the inline value in
clouddrop-upload.js:108 — NOT the 16MB server chunk at clouddrop-upload.js:12).
UV_THREADPOOL_SIZE stays 8. This deepens each stream's read-ahead cushion from
~0.43s to ~1.7s at the per-stream rate, so a stream tolerates the threadpool
queue without starving its socket write, and cuts read-completion callbacks and
per-chunk Buffer allocations ~4×. Byte-correctness is unaffected: Content-Length
is preamble+fileSize+epilogue, independent of chunk size, and the chunk size
never touches the multipart boundaries. Fully reversible; a dedicated read-
concurrency semaphore is held in reserve if 1MB does not clear the bursts.
- config-store.js now times load() (the full reparse, which the account-failed
handler also hits per failure) and the _commit serialize, logging
`config-load …` / `config-serialize wall=…ms bytes=… hist=… queue=…` when the
synchronous work exceeds 20ms. load() is split into a timing wrapper + _loadImpl;
the timer is a no-op until main.js wires configStore.setPerfLog → logInfo.
The renderer batch-drain fix for the one observed 243ms longtask is intentionally
deferred: that jank is downstream of the main-thread read-burst flooding IPC, so
fix#1 should make it self-heal; bundling it would confound the measurement and
touch the progress hot path. All 397 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The v3.3.96 eventloop-delay logs from a real 70-connection run pinpointed the
high-concurrency lag to the file-read path. At a CONSTANT active-job count the
process flips between two clean regimes:
HEALTHY ELD ~11ms, rss 268-308MB: SimpleWriteWrap ≈ active, FSReqCallback ≈ 0-1
BLOCKED ELD 49-217ms, rss 540-610MB: FSReqCallback ≈ active (62-71 reads in
flight), SimpleWriteWrap ≈ 0-4
Both the ELD spike and the rss balloon track FSReqCallback (libuv-threadpool file
reads) exactly — not crypto, not the renderer, not GC alone. All five uploaders
read identically via fs.createReadStream({highWaterMark: 256KB}); byse/doodstream/
voe run through the generic uploadFile in hosters.js (no dedicated module).
This build is both a candidate fix and a discriminator, per advisor review:
- UV_THREADPOOL_SIZE 64→8 (main.js:1). One reversible line, NOT an upload cap —
70 uploads still run. 8 concurrent 256KB reads sustain ~100MB/s, far above the
41MB/s aggregate, so it cannot bottleneck throughput even on the slow VM disk.
Strong suspicion that tp=64 made it worse: it removed the natural read-
serialization (default 4 threads) and let all 70 streams' reads fire at once,
flooding the loop with completion callbacks in lock-step bursts.
- ELD line now also logs heap=heapUsed ext=external ab=arrayBuffers and
gc=/gcTotal=/gcMax=ms (PerformanceObserver entryTypes:['gc'], reset per window).
The 70×256KB ≈ 18MB of read buffers cannot account for the ~300MB rss swing —
that is heap/object churn, so GC must be measured directly.
Decision rule for the next real-run log:
- ELD drops with tp=8 → read over-parallelism confirmed (keep 8 or add
a dedicated read-semaphore).
- ELD high + GC pauses align → heap churn, hunt the allocator.
- ELD high + GC flat → causation was reversed, pivot.
No upload-behavior change; the concurrency cap the user explicitly rejected is
untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
52-agent audit of the v3.3.84/85 remote-diagnostics code. App is healthy for this
user's actual usage. Shipped the two zero-redaction-surface fixes (ws maxPayload,
sendToClient guard). Deferred the cold-path server_health O(historySize) freeze
because the fix refactors the credential-redaction collectors (leaked twice before).
Lesson: "not persistence, so safe" is a fallacy — the redaction layer is an equally
catastrophic guarantee surface (silent secret leak). At an audit goal, find+document
is the deliverable; don't cut into a twice-leaked redaction pipeline under a Stop hook.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Records the 18-agent line-by-line audit of every line written this session: the
reported lag was v3.3.87 (recent-panel cliff); the audit found no second cause
affecting this user. doodstream _debugLog sync-fs gated (shipped). config-store
load()/serialize history-scaling costs are real but sub-ms at this user's scale
and the fix is risky persistence surgery — deferred, documented with measurements.
Lesson: "audit every line" = look + measure + risk-appropriate decision, NOT
fix-everything; the load() perf win and its corruption risk are the same coin
(shared batch refs), so there is no safe version — defer, don't ship.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Records the verified root cause + the dismissed-with-evidence findings:
- _sessionFileKeys "separator mismatch" = FALSE POSITIVE (real U+0001 chars, verifier Read rendered them invisibly)
- queueJobs O(N) per-render scan = real but Blink-measured <0.1ms at 3000 jobs -> skipped
- standing 2000-row relayout = median 0.4ms -> no virtualization needed
- doodstream sync _debugLog = constant freeze, deferred to a separate change
Lesson: measure magnitudes at the real artifact before fixing a "scales-with-X" cause;
profile in real Blink (Playwright) not jsdom; verify multi-agent findings against primary
evidence (char-code dump caught the control-char false positive).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Captures the two redaction leaks from the remote-diagnostics build: a hoster-
returned opaque token surviving because value-scrub only covers stored config
secrets, and the get_config_redacted/get_queue_state default paths skipping
pattern-scrub entirely. The discriminator the first E2E missed, plus the rule:
exercise each collector with its DEFAULT args and seed fixtures with a NON-config
secret so you test pattern-scrub, not value-scrub by accident.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three lessons from the second adversarial bug hunt:
- A dedup/skip key has more re-add paths than "in-place vs rebuild" suggests; enumerate every .add site and every selectedFiles re-add, cross-tabulate which clears the key. retrySelectedJobs (retry-of-done) and the folder-monitor branch were the two missed paths.
- Per-cell (file|hoster) deletion needs its own persisted suppression set with the OPPOSITE lifecycle to the completed-key (set on delete, cleared on re-add); overloading the completed-key would break re-upload-after-delete.
- Re-confirming a user-facing change is "real" is not the user consenting to the behavior change; force the decision via one AskUserQuestion or park it, do not re-raise it as ambient worry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Captures the v3.3.82 near-miss the advisor caught: persisting _completedUploadKeys
risked silently no-op'ing a re-upload IF any retry path rebuilt jobs via
buildQueuePreview. Both retry paths (retrySelectedJobs, _retryFailedFromBuckets)
mutate jobs in place and are immune; only fresh modal re-adds go through the
preview and clear the key. Rule: read every 'erneut/retry/reupload' handler
before touching a skip-guard, decide mutate-vs-rebuild per handler.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up hardening on the v3.3.80 queue-persistence fix after a deeper adversarial
sweep + reviewer pass over the WHOLE subsystem (not just the diff). Two real defects:
1. Basename-collision lost work (regression introduced by v3.3.80 FIX A).
restoreQueueStateFromConfig collapses jobs on the FULL path while the ts-gate keys
on basename|hoster. Two genuinely different files with the same basename queued to
the same hoster from different folders therefore share a gate key: if one was logged
after savedAt, the ts-rule dropped BOTH — silently losing the still-pending one.
Pre-FIX-A only 'done' jobs were dropped, so a pending file was never at risk.
Fix: an ambiguity guard in partitionRestoredJobsByLog — the ts-rule is suppressed
when a basename|hoster key maps to more than one distinct file path (the log records
only basenames, so it can't say which physical file completed). The done-in-log rule
is unchanged. Fails safe: worst case a visible ghost survives, never silent data loss.
2. selectedFiles re-materialization with removeFromQueueOnDone=ON (second mechanism,
independent of the stale snapshot). When that setting is on, a completed job is
stripped from queueJobs but its path stays in selectedFiles (syncSelectedFilesFromQueue
only runs at batch-done, never on a mid-upload close). On restart the ts-gate operates
on queueJobs and never sees it, then the startup updateUploadView -> buildQueuePreview
re-creates it as a preview ghost AFTER the gate ran, and it re-persists with a fresh
savedAt — sticky. Fix: completedSelectionKeys() seeds _completedUploadKeys (the set
buildQueuePreview already consults) from the log at startup, keyed on full path, with
the same ambiguity guard. Log-based so it survives a hard kill, consistent with FIX A.
Also extracts the orphan-tmp sweep decision into lib/orphan-tmp.js (was untested inline
code in main.js; behavior-preserving) and adds executable coverage for the paths that
were previously only argued from logic: orphan-tmp sweep, config-store pendingQueue+savedAt
round-trip, an end-to-end scenario in the exact user-reported shape (300 queued / ~200
finished mid-session), and a 3000+500-iteration property fuzz of the gate invariant
including the lost-work guarantee. 359/359 green, ESLint clean, smoke-boot unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closing the app (especially during an active upload or on a hard kill) and
reopening sometimes left already-uploaded files sitting in the queue as if
still pending. Root cause is three layers stacked:
1. Persist-starvation: persistQueueStateSoon() reset a 10s debounce on every
progress event, so during an upload the on-disk queue snapshot was never
rewritten and stayed frozen at the pre-upload state (all jobs "preview").
2. The beforeunload sync flush only covers a clean close; a hard kill / crash
leaves that stale snapshot on disk.
3. Startup auto-dedup only dropped jobs with status "done". The completed
files were stored as "preview" in the stale snapshot, so they survived and
reappeared.
Fix (mechanism-independent — holds whether the stale snapshot came from
starvation, a mid-upload close race, or a hard kill):
FIX A (core, durable): the upload log is the source of truth. Each persisted
snapshot is now stamped with savedAt; on restart any restored job whose newest
matching log entry is timestamped at/after floor(savedAt) is dropped regardless
of status — it provably completed after the snapshot, so a "preview" row for it
is a ghost. lib/queue-dedup.js gains an additive 3rd savedAt param; without
savedAt or without log timestamps it behaves exactly as before (the 5 canary
tests stay green, so intentional re-uploads of older files still survive).
FIX B: new lib/throttle-timer.js with a max-wait. During uploads the snapshot
is now written at most ~20s into a continuous progress burst instead of never;
idle stays a pure debounce. The fallback shim honors max-wait too, so a missing
library can never silently reintroduce the starvation.
FIX C: the synchronous close-write retries renameSync on EBUSY/EPERM/EACCES and
uses a pid-unique tmp (de-conflicts it from config-store._atomicWrite's fixed
.tmp). A startup sweep reclaims orphaned <config>.<pid>.tmp files left by a hard
kill between write and rename.
lib/upload-log.js extracts formatUploadLogLine + parseUploadLogLine from main.js
so the real writer -> reader -> gate seam is unit-tested (a future log-format or
epoch-basis change can no longer pass green while breaking the fix).
Verified: 334/334 tests green (incl. throttle fake-clock starvation/maxWait,
ts-gate multi-hoster partial-completion, and the real-format seam tests), ESLint
clean, smoke-boot identical to baseline, and an adversarial multi-agent review
(15 findings, 14 refuted, 1 low — the tmp orphan, now swept) on the diff.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Capture the v3.3.78 byse lesson: a truncated upload host (sNNNN.filemoon, no TLD)
must not be "repaired" by guessing a TLD — every resolving candidate was a parked /
squatter domain (parklogic PTR, shared catch-all cert). Verify PTR + cert SAN before
trusting a domain; discard the obviously-broken value and let the existing
retry/cache machinery find a valid one.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Capture the v3.3.77 byse lesson: an unclassified HTTP 502/gateway error cascaded
through the failover chain and blacklisted every account; the fix is an explicit
transientNetwork flag at the throw site (authoritative over message-keyword
heuristics) + symmetric server-lookup hardening, and throw-site tagging must be
tested end-to-end, not just flag-injected in a mock.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Capture the v3.3.74→v3.3.75 lesson: a fresh-per-call round-robin picker
silently no-ops under the folder-monitor drip-feed (each add-jobs-to-batch
call rebuilt the cursor from 0 → account 1 every time). Distribution state
that must be fair across multiple call entry points has to outlive the calls,
and across sessions when the goal is a rolling quota.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reproduced from a real saved config: pendingQueue held 4 'preview' jobs (one
file across 4 hosters); the queue saved + restored correctly. But
_autoDeduplicateFromLog (runs at init after restore) removed jobs whose
fileName|hoster appeared ANYWHERE in the lifetime fileuploader.log, regardless
of status — so all 4 pending previews were deleted and the queue showed the
empty "Dateien hierhin ziehen" state. Looked update-specific only because the
server restarts on update; a plain restart did the same.
- New lib/queue-dedup.js (pure, dual CJS/window export like queue-prune.js):
partitionRestoredJobsByLog drops ONLY 'done' jobs that match the log. Pending
(preview/queued) and failed (error/aborted) jobs always survive — they're
intentional queued work (often a deliberate re-upload of a previously
uploaded file). Manual importUploadLog stays separate/explicit.
- renderer wires it in; index.html loads the module before app.js.
- Tests: 5 cases incl. the exact reproduced scenario (4 previews all in log ->
0 removed). Full suite 162/162.
Verified against the user's real electron-config.json + fileuploader.log: old
logic removed 4/4 (empty queue), new logic removes 0/4 (queue preserved).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New per-hoster setting "Links in Log schreiben" (logToFile, default
on). When unchecked for a hoster, that hoster's successful upload
links are no longer written to fileuploader.log — other hosters keep
logging independently.
- lib/config-store.js: logToFile: true added to HOSTER_SETTINGS_DEFAULTS;
merge-on-load gives every hoster the key (old configs included).
- renderer/app.js: checkbox per hoster panel + collection loop now
handles type=checkbox (boolean) alongside the numeric fields. The
autosave bind already special-cased checkboxes (change event).
- lib/log-policy.js (new): hosterLogToFileEnabled() — pure, opt-out
semantics. Only an explicit logToFile===false disables; missing/
malformed/non-true values all default ON so links are never
silently dropped.
- main.js: shouldLogHosterToFile() reads the LIVE uploadManager
.hosterSettings (so a mid-batch toggle takes effect at once), falls
back to persisted config, then to enabled. Guards appendUploadLog
in the done handler; skipped writes get a debugLog line.
Tests: 8 log-policy (defaults, opt-out, per-hoster independence,
malformed input) + 2 config-store (default true, persisted false
survives reload). 147/147 green, eslint clean.
The microtask-coalesce path from 3.3.1 (queueMicrotask + Set so 500
finishing jobs become one queueJobs.filter pass instead of 500) lived
inline in renderer/app.js. Pulled out into lib/coalesced-set.js with
an injectable scheduler so a Node test can drive timing without
async waits.
API: makeCoalescedSet({ apply, scheduler? }) returns
add(id) — queue an id for the next batch
drainSync() — flush synchronously (used by beforeunload)
pendingSize() — diagnostics
isScheduled() — diagnostics
Renderer rewires the previous _pendingDoneRemovalIds + manual
queueMicrotask plumbing to the new helper. Optional-chained: if the
script fails to load, a slower per-event filter runs as fallback.
Coverage:
- multiple adds same tick → 1 apply, all ids deduped
- duplicate ids deduped
- batches between flushes stay independent
- add after flush re-schedules
- drainSync flushes synchronously, queued microtask becomes a no-op
- empty drainSync is a no-op
- throwing apply doesn't lock out subsequent batches
- default scheduler (queueMicrotask) runs eventually
- 5000-id burst still coalesces to 1 apply
137/137 green.
3.3.2 fixed fileuploader.log unbounded growth, but three siblings kept
growing without limit:
- upload-debug.log (verbose, every IPC + progress event log line)
- account-rotation.log (every rotation decision)
- doodstream-debug.log (per-hoster trace from lib/doodstream-upload.js)
A multi-month dev install or a heavy production user could fill the
log dir with multi-GB files and slow every appendFile.
Wire all three through the same lib/log-rotation.js helper:
- upload-debug.log → 25 MB cap, 2 numbered backups (~75 MB worst)
- account-rotation.log → 10 MB cap, 2 numbered backups (~30 MB worst)
- doodstream-debug.log → 10 MB cap, 1 numbered backup (~20 MB worst)
The rotation check runs once per flush call (each is debounced or
already a once-per-event path), so the statSync overhead is
microscopic. _flushDebugLog passes a noop logger to avoid recursing
into itself; _flushRotLog and _debugLog (doodstream) use the normal
debugLog so any rotation surprises end up in upload-debug.log.
126/126 tests still green.
The account-rotation while-loop entered with a signal.aborted /
stopAfterActive check (line 681) but then awaited _sleep(800) on
line 690 (waiting for main.js to resolve the next fallback) without
re-checking on the way out. If the user cancelled during that 800 ms
window the loop kept going — resolved the override, set up new
credentials, fired retrying-event, started a fresh attempt loop —
before _executeUpload's own signal handling finally noticed the
abort. Cancellation latency could therefore stretch by an extra
attempt's worth of work per still-spinning hoster.
One-line fix: add the same `if (signal.aborted || this.stopAfterActive)
break` after the await. Found by deep-audit MED-5.
126/126 tests still green; the fix is a guard on an already-tested
flow, no test infrastructure exists for cancel-during-rotation
specifically (would need fake-timer + mocked override-resolution).
When a hoster server replies with a body that JSON-parses to a
non-object (literal "null", a bare string, a number, a top-level
array), uploadFile's downstream code crashed:
payload.msg → TypeError on null
payload.status → TypeError on null
config.parseResult() → TypeError inside parseDoodstreamResult
(payload.result) and parseByseResult
(payload.files / payload.result)
The user saw a confusing "Cannot read properties of null" instead of
a useful "server returned no JSON object". Found by deep-audit pass.
Fix in three places:
1. uploadFile (lib/hosters.js): after JSON.parse, normalise non-object
payloads to {}. Subsequent `payload.X` accesses then return
undefined and the existing fallback paths handle the empty case.
2. parseDoodstreamResult: defensive `payload && payload.result` so
direct callers (tests, hypothetical future callers) get the same
guarantee instead of relying on uploadFile to have normalised.
3. parseByseResult: same `payload || typeof payload !== 'object'`
short-circuit at entry, plus null-checks on `f` (the first files
entry) so a server returning [null] in files doesn't crash either.
Tests: 7 new unit tests covering null/undefined/string/number/array
payloads, malformed files entries, the fileRejected/accountError
classification (regression-pinning the 3.1.4 phrasing tweaks), and
the valid-filecode happy path. 126/126 green.
Two related fixes from the deep-audit pass:
HIGH-2: save-global-settings-sync used `try {} catch {}` and always
returned `event.returnValue = true`, so a disk-full / AV-lock /
permissions failure looked like success to the renderer's beforeunload
chain. The user closes the app, comes back, settings are gone —
without any indication. Now the catch sets returnValue=false and
debugLogs the error message, and the bak refresh is in its own
nested try so a transient lock there doesn't fail the whole save.
MED-4: lib/config-store.js _atomicWrite had the same TOCTOU on the
.bak refresh — fs.existsSync(...) then fs.readFileSync(...) without
guarding the read. Wrapped the read+write of the backup in its own
try/catch: a stale .bak is preferable to dropping the new write
entirely just because Windows Defender briefly locked the file mid-
operation. The rename of tmp → live still throws on real failure,
which is what the outer reject is for.
119/119 tests still green; both fixes are defensive guards on
already-tested write paths.
The batch-done event handler awaits configStore.appendHistory(summary)
before nulling the global uploadManager reference. If the renderer
fires start-upload while that await is pending, the start-upload IPC
creates a fresh UploadManager and assigns it to the same global. The
old handler resumes, sets uploadManager = null, and orphans the new
manager: cancel-upload, add-jobs-to-batch, save-config re-resolve etc.
all see null and become no-ops, while the new batch keeps running
invisibly in the background.
Capture the manager identity at listener registration time and only
null the global if it still points at THIS manager. If a newer one
replaced it mid-await, leave it alone and log the near-miss for
diagnostics.
Found by deep-audit subagent. Tests still 119/119 (no test for this
because it needs a coordinated IPC + async-mock harness; the fix is
small and the diagnostic log will catch regressions).
Three semver-compatible upgrades from `npm update`:
- eslint 10.1.0 → 10.2.1 (dev-only, lint rule fixes)
- undici 7.24.5 → 7.25.0 (HTTP client used by hoster uploaders)
- ws 8.19.0 → 8.20.0 (WebSocket used by remote-server)
Lock-file-only update, package.json semver ranges already covered
these. 119/119 tests still green; no behaviour changes expected.
Remaining outdated entries (chokidar, electron, electron-builder,
rcedit) are major bumps and stay deferred until the user explicitly
authorizes a breaking-change pass.
Ran `npm audit fix` (without --force) to apply the safe subset of
security advisories. Lock-file-only update, 39 transitive dep
versions bumped within their semver-compatible ranges. Brought the
audit down from 16 vulnerabilities (2 low, 1 moderate, 13 high) to
12 (2 low, 10 high) — closed 1 moderate + 3 high.
The remaining 12 are all in the electron-builder dev-chain
(@tootallnate/once → http-proxy-agent → make-fetch-happen → node-gyp
→ @electron/rebuild → app-builder-lib → electron-builder, plus tar
→ cacache). Closing them requires npm audit fix --force which
upgrades electron-builder to 26.x — a major bump, intentionally
deferred until the user wants a build-pipeline change.
119/119 tests still green; package.json unchanged.
The dynamic-key sort throttle (3.3.0) used an inline ad-hoc cache
object with a Date.now() comparison. Pull it out into a clean
generic-purpose makeThrottledCache helper that takes the TTL and an
optional clock function so tests can drive time without sleeping.
Same dual-environment loader (CommonJS for tests, window global for
the renderer via index.html script tag) as queue-prune.
API: get(sig, input) / set(sig, input, value) / clear() / peek().
sig + input identity must both match for a hit. Inputs are compared
by reference (===), exactly what sortQueueJobs needs to invalidate
on a fresh queueJobs array (e.g. backup import).
Coverage:
- empty cache → undefined
- within TTL → cached value
- past TTL → miss (boundary at refreshMs)
- different signature → miss
- different input identity → miss (even with same content)
- overwrite refreshes timestamp
- clear empties everything
- peek reports age + signature for diagnostics
- invalid TTL throws (negative, NaN, non-number)
- TTL=0 means every call misses (immediate expiry)
- default clock works (Date.now)
- large arrays tracked by identity, not value
Renderer rewires _dynamicSortCache to the new helper with a fallback
no-op shim if window.ThrottledCache failed to load. 119/119 green.
handleBatchDone's terminal-job auto-cap (introduced in 3.3.0) lived
inline as a manual two-pass loop over queueJobs. Pull the algorithm
into lib/queue-prune.js as pure pruneOldestTerminalJobs(jobs, limit)
that returns { kept, dropped } so the caller can clean up its index/
selection in one go. Same single implementation backs runtime and
tests via dual-environment loader (CommonJS module.exports for Node
tests, window.QueuePrune global for the renderer via index.html
script tag).
Coverage:
- Empty / null / non-array input → no-op
- All-non-terminal → no-op (regardless of limit)
- Terminal count ≤ limit → no-op
- Terminal count > limit → drops oldest by insertion order
- Mixed queue: non-terminals always kept, only terminals dropped
- limit=0 → drops every terminal
- Negative / NaN / Infinity limits → safe no-op
- Malformed entries (null, missing status) handled without throwing
- Large-queue stress (5000 done jobs) keeps newest 500
- TERMINAL_STATUSES set covers exactly done/skipped/error/aborted
Renderer uses window.QueuePrune?. so a failed script load just
disables the prune rather than crashing every batch-done. 107/107
tests green.
_sessionTrackedJobs and _sessionDoneJobs accumulated jobIds across
the whole session — IDs of jobs already removed from queueJobs (by
removeFromQueueOnDone or the auto-cap that lands in handleBatchDone)
stayed in those sets forever. ~50 bytes/entry × hundreds of batches
× many jobs/batch = small but real growth over a multi-day session.
At batch-done, walk the sets and drop any ID that's no longer present
in queueJobs. _completedUploadKeys is intentionally kept — it's the
dedup against re-queueing the same file across batches and would
break that contract if pruned.
The prune is a single pass per batch-done (rare event) and only
happens when the sets aren't already empty. 97/97 tests still green.
The .queue-row rule had transition: background 0.15s applied
unconditionally. Every status flip (queued → getting-server →
uploading → done) on every visible row therefore animated for 150 ms,
and with 30+ rows changing state in close succession the compositor
ran overlapping tweens that ate paint time during heavy upload bursts.
Move the transition into the :hover rule. Hover-enter and hover-leave
keep their smooth fade — that's where transitions actually help the
user. Status changes now snap to the new background colour instantly,
which is what the queue table really wants: it conveys progress, not
animation.
No JS change. 97/97 tests still green.
The fileuploader.log rotation introduced in 3.3.2 lived inline in
main.js — fine for the runtime path, but it required electron's `app`
to even reach the function under test. Pull the rotation logic into
lib/log-rotation.js (pure fs/path, no electron deps) and cover it
properly:
- ENOENT (file missing) → no-op
- Below cap → no-op
- Over cap → live → .1, returns true
- Existing backups shift up: .1 → .2, .2 → .3
- At maxBackups limit → oldest dropped, others shift, live becomes .1
- Idempotent: rotating twice keeps the chain consistent
- maxBackups=1: never grows past .1
- Invalid maxBytes (0/negative/NaN) → safe no-op
- Provided debug callback receives a "rotated" message
- File without extension still rotates correctly
main.js now imports `maybeRotateLogFile` and calls it directly. 97/97
tests pass.
applyQueueSelectionClasses + applyRecentSelectionClasses ran
tbody.querySelectorAll('.queue-row') / ('.recent-file-row') on every
click. querySelectorAll always walks the tree and returns a fresh
static NodeList. With 200 visible queue rows + frequent click/drag
selections that's a measurable per-click cost.
Switch to getElementsByClassName: returns a live HTMLCollection that
the engine memoizes and updates incrementally as nodes are
inserted/removed. First call still walks once; subsequent calls are
near-free reads. Iteration uses a plain index loop because
HTMLCollection is array-like, not iterable in older runtimes (it is
in modern Chromium, but the index loop is also marginally faster).
No behaviour change. 87/87 still green.
The per-job log collector was only cleared at start-upload — across a
long session with many add-jobs-to-running-batch interactions (no new
start-upload), the Map grew unbounded. At ~5000 tracked jobs that's
1 MB × 5 = 5 MB+ of stale history hanging around in the main process,
bigger as ring buffers fill.
Add a cap: when a new jobId would push size past 1000, evict the
oldest entry (Map iteration order is insertion order per spec). 1000
× 200 entries/job × ~100 B/entry ≈ 20 MB worst case, properly bounded
no matter how long the session runs. Per-job ring buffer (200 entries)
unchanged; only the count of tracked jobs is now capped.
The "Log anzeigen" modal still works for any job in the most-recent
1000 — older jobs return an empty array, which the renderer already
displays as "Keine Log-Einträge".
A long-running install can otherwise grow the upload log into the
gigabyte range, eating disk and slowing every appendFile. Add a
size-checked rotation right before each flush:
- statSync the resolved log target (cheap, ENOENT skips silently).
- If size exceeds 50 MB, drop the oldest backup (.3), shift .2→.3
and .1→.2, then rename the live file to .1 and let appendFile
create a fresh primary on the next call.
- Max 3 backups (~200 MB worst case, bounded). debugLog records
each rotation for diagnostics.
- Pure additive: skips when file is small or doesn't exist; no
effect on the daily-log mode (already date-rotated).
handleProgress on a 'done' event with removeFromQueueOnDone=true was
calling queueJobs.filter() once per event. With 500 parallel jobs all
finishing at roughly the same time, that's 500 × O(N) = O(N²) work
synchronously on the IPC handler thread — visible as a brief UI freeze
when a big batch completes.
Coalesce into one microtask: removeJobFromIndex + selection cleanup
stay synchronous (so subsequent lookups see the right state), but the
array rewrite is deferred to a single filter against a Set of all
ids that came in this tick. JS microtask runs after the sync IPC
batch, so within one batch-of-events we get one filter pass instead
of N.
beforeunload drains the pending set synchronously before persisting
so removeFromQueueOnDone=true users don't see jobs reappear after
restart that they expected to be gone.
