User asked to measure everything so we can pinpoint the lag/ECONNRESET at 70 concurrent uploads. All additive instrumentation, no upload-behavior change.
main process: the eventloop-delay log line now also reports cpu=X%core (process.cpuUsage delta over wall time; >100% means multiple cores), rss, per-hoster live connection distribution (active-by-hoster), cumulative transient (ECONNRESET-class) error count, and pending count — pulled from a new UploadManager.getDiagnostics(). This shows whether the main thread is CPU-bound and which hoster is oversubscribed.
renderer: a PerformanceObserver('longtask') plus a requestAnimationFrame frame-time monitor log 'renderer-perf' every 5s while uploading — fps, jankFrames (>33ms), worstFrame, longtask count and max. This is the direct renderer ground truth (real frame rate under load), which the component-timing harness could not capture. Low fps / high jank => renderer is blocked; ~60fps while it still feels laggy => the cpu/eld numbers in the same log decide CPU-bound vs IO-bound in the main process.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
maybeAddSessionFile computed formatDateTime(new Date()) at the top, before the status==='done' guard that early-returns for every other status. formatDateTime runs two Intl locale formats (~83us measured), so it executed on every progress event — onUploadProgressBatch loops the M-item batch into handleProgress -> maybeAddSessionFile, i.e. 10xM times per second — and threw the result away for all non-done events (the vast majority while uploading).
That made each progress batch a synchronous main-thread block scaling with the active-upload count: ~2.4ms at 25 concurrent, ~5ms at 61, every 100ms, on top of render and sort — enough to blow the 16ms frame budget and stutter scrolling. It is per-event, not per-render, so it janks regardless of scroll position, matching the user's report that the lag appears above ~50 connections and when the uploading rows are scrolled out of view.
Move the formatDateTime call inside the dedup block so it runs once per genuinely-new completed upload. A faithful Blink benchmark at the user's regime (500 rows virtualized, dynamic progress sort, scrolling) shows the per-batch cost drop from 1.7/3.2/4.1ms at 25/50/61 concurrent to a flat 0ms, and frame P95 at 61 concurrent from 7.3ms to 4.2ms.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make the single high-concurrency measurement decisive. The ELD line now appends process.getActiveResourcesInfo() as a compact type-histogram, so one run splits all three readings in a single line: high mean/p99 means the main thread is CPU-blocked (workers/cap justified); low delay with many TCPSocketWrap/FSReqCallback/GetAddrInfoReqWrap resources means IO-bound queueing (threadpool/socket tuning, not workers); low delay with few resources means it is not saturated at all. Pure metrics — no credential-redaction path touched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The user runs 50+ concurrent uploads (parallel counts raised deliberately). Every async uploader feeds undici from fs.createReadStream and resolves DNS via getaddrinfo — both go through the libuv threadpool, whose default size is 4. At 50 concurrent uploads, file reads and lookups serialize 4-at-a-time: a hard cliff at a small connection count that matches the 'lags from X connections onward' symptom. Raise the cap to 64 as the first statement (before require('electron'), so libuv reads it when it lazily inits the pool; an explicit env override still wins). Threads are created on demand, so a higher max costs nothing when unused — reversible, zero upload-core change.
Also enable perf_hooks.monitorEventLoopDelay and log mean/p99/max/stddev every ~5s while uploading. This is the ground-truth instrument that splits the two competing explanations for the lag: a high event-loop delay means the main thread is CPU-blocked (TLS/crypto) and only workers or a concurrency cap will help; a low delay while uploads stall means the work is IO-bound and the threadpool/socket config is the lever, not workers. The numbers are pure metrics — they never touch the credential-redaction path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The rotation-retry and suspect-alternate progressCb callbacks called _emitProgress (a synchronous EventEmitter emit plus a fresh object spread) on every stream chunk — hundreds per second per job — because they lacked the 250 ms lastEmitTime gate the primary upload path already has. With many concurrent uploads in rotation or suspect mode that is real main-thread emit amplification.
Mirror the primary path's gate exactly: the activeEntry speed/bytes mutation stays ungated so the stats timer and speed monitor keep seeing fresh values; only the _emitProgress call is throttled to ~4/sec. Behavior-preserving — identical progress data, fewer emits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_uploadChunked read each 16 MB chunk with fs.readSync on the main JS thread — the only one of the five uploaders that blocks synchronously (the other four stream async). Each readSync stalls the whole event loop ~5-9 ms on SSD, 30-100 ms on a slow disk, freezing all progress emits, IPC, renders and every other concurrent upload for that window. The stall scales with the number of simultaneous clouddrop uploads, matching the 'feels laggy while uploading, worse with more at once' symptom at modest CPU (one core pinned, ~40% of 8).
Swap fs.openSync/readSync/closeSync for fs.promises.open + await fh.read + await fh.close. Buffer reuse and the partial-last-chunk subarray view are unchanged. Verified byte-identical to the old loop via SHA-256 over every chunk-boundary case (full chunk, partial last chunk, 2/3/4-chunk files, single byte) before shipping — a chunk-read bug would corrupt the upload.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
52-agent audit of the v3.3.84/85 remote-diagnostics code. App is healthy for this
user's actual usage. Shipped the two zero-redaction-surface fixes (ws maxPayload,
sendToClient guard). Deferred the cold-path server_health O(historySize) freeze
because the fix refactors the credential-redaction collectors (leaked twice before).
Lesson: "not persistence, so safe" is a fallacy — the redaction layer is an equally
catastrophic guarantee surface (silent secret leak). At an audit goal, find+document
is the deliverable; don't cut into a twice-leaked redaction pipeline under a Stop hook.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two isolated hardenings of the opt-in remote/diagnostics WS server, surfaced by
the session-wide diagnostics audit:
1. WebSocketServer was created with no maxPayload, so ws defaults to 100 MiB per
message. The connection handler runs JSON.parse(raw) on the FIRST message
(the auth frame) before authentication, so any peer past the IP allowlist
could send a huge payload and force a synchronous multi-MB JSON.parse on the
main-process event loop — an unbounded freeze/DoS sink. Diag, auth and WebRTC
signaling messages are all small; cap maxPayload at 256 KiB to close it.
2. sendToClient did ws.send(JSON.stringify(data)) with no readyState/try guard
(unlike broadcast, which checks ws.readyState === 1). A send on a closing
socket, or a stringify throw, escaped the diag-response callback as an
uncaughtException — a potential crash. Mirror broadcast: send only when
readyState === 1, wrapped in try/catch.
Both are isolated to the transport layer with zero redaction surface. The audit's
larger finding — server_health doing O(historySize) synchronous work per request
(6-7 full-config clones + unbounded history walks) — is a real freeze, but ONLY on
the cold opt-in diagnostics path with a large history (this user: 23 rows), and the
safe fix cuts into the credential-redaction collectors (which have leaked twice);
deferred and documented in tasks/todo.md rather than operated under risk.
397/397 tests pass, eslint clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Records the 18-agent line-by-line audit of every line written this session: the
reported lag was v3.3.87 (recent-panel cliff); the audit found no second cause
affecting this user. doodstream _debugLog sync-fs gated (shipped). config-store
load()/serialize history-scaling costs are real but sub-ms at this user's scale
and the fix is risky persistence surgery — deferred, documented with measurements.
Lesson: "audit every line" = look + measure + risk-appropriate decision, NOT
fix-everything; the load() perf win and its corruption risk are the same coin
(shared batch refs), so there is no safe version — defer, don't ship.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
doodstream-upload.js's _debugLog ran with NO verbose gate: every call did a
synchronous statSync (via maybeRotateLogFile) + appendFileSync directly on the
main-process event loop, ~8-15x per upload (server probe, response, redirect,
result page, filecode parse, hidden fields, submit/follow, plus retry branches).
DoodstreamUploader runs in the main process, so each pair of sync fs syscalls
blocked the event loop while uploading — delaying IPC, upload-progress-batch
forwarding, tray-tooltip and webhook handling for every other concurrent upload.
A constant per-upload main-thread tax (not history-scaling), surfaced by the
session-wide lag audit and confirmed as the one finding that bites in the real
upload scenario.
Fix: gate _debugLog behind the existing globalSettings.logVerbose setting
(default false), exactly mirroring main.js logDebug/_logVerbose. A module-level
_debugVerbose flag + setDebugVerbose() setter, an early-return at the top of
_debugLog, and one wire at main.js's single setLogVerbose chokepoint (covers
boot + save-config + the verbose toggle). When verbose is off the doodstream
trace simply isn't written — same contract as the main debug.log — and the
per-upload sync fs disappears. When a doodstream issue needs tracing, enabling
verbose restores the full trace.
The config-store audit findings (load() history-clone cost, per-write history
serialize) are real but scale only with history size — tens of microseconds at
this user's 8-batch config, and the safe fix is risky persistence surgery on
credential-bearing code for a latent micro-cost; deferred and documented in
tasks/todo.md rather than shipped.
397/397 tests pass, eslint clean, gate wiring verified (shared module instance,
default-off, toggles).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Records the verified root cause + the dismissed-with-evidence findings:
- _sessionFileKeys "separator mismatch" = FALSE POSITIVE (real U+0001 chars, verifier Read rendered them invisibly)
- queueJobs O(N) per-render scan = real but Blink-measured <0.1ms at 3000 jobs -> skipped
- standing 2000-row relayout = median 0.4ms -> no virtualization needed
- doodstream sync _debugLog = constant freeze, deferred to a separate change
Lesson: measure magnitudes at the real artifact before fixing a "scales-with-X" cause;
profile in real Blink (Playwright) not jsdom; verify multi-agent findings against primary
evidence (char-code dump caught the control-char false positive).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The recent-uploads panel had a cheap append-only fast path, but it was gated on
`rows.length > _recentLastRenderedLen`. maybeAddSessionFile caps sessionFilesData
by push-then-slice (2000 -> 2001 -> sliced back to 2000), so once a session
produces more than SESSION_FILES_CAP rows the length is pinned at the cap and the
gate is false forever. Every subsequent completion then fell through to the full
`tbody.innerHTML = rows.map(...).join('')` rebuild of all ~2000 rows.
The cap is per (link x file x hoster), so with 4-5 selected hosters the 2000 cap
is hit at only ~400-500 distinct files — very reachable in a long folder-monitor
session. Profiled in Chromium (same Blink engine as Electron, table-layout:fixed):
the full 2000-row rebuild costs ~80ms and ran on EVERY completion past the cap — a
repeating ~80ms main-thread freeze. That is the "fine on a fresh start, gets laggy
after many uploads while CPU (~40%) and RAM (~6GB, stable) stay normal" symptom: a
render-thread stall, not CPU saturation or a memory leak.
Fix: track newly-pushed rows in _recentPendingAppends (incremented in
maybeAddSessionFile, consumed every render) and gate the fast path on
`pendingAppends > 0` instead of length growth, so it survives the cap. Prepend the
new rows, then evict the same overflow count from the DOM bottom (oldest rows,
which is where the date-desc view places the front-of-array entries the cap slices
off). DOM work is O(added) again. The fast path is gated behind an explicit
`appendOnly` flag passed only by scheduleRecentRender's rAF, so selection / delete
/ clear / sort / batch-done renders stay full rebuilds and cannot wrong-evict or
double-prepend.
Verified in Blink over a simulated 5000-completion session (3/frame, far past the
cap): per-frame render 80ms -> median 7.4ms (>10x), and the DOM stays exactly equal
to the data (cap held at 2000, newest-on-top, oldest evicted, zero duplicates).
397/397 tests pass, eslint clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A 13-agent hunt pinned the "wird mit der Zeit laggy" symptom (CPU/RAM normal, UI
sluggish after many uploads) to the main process re-doing config I/O that scales
with the ever-growing history, stalling the synchronous main event loop so the
renderer's IPC round-trips feel laggy. The renderer render path was already
optimized (virtualized queue, capped panels) — confirmed clean.
This commit lands the two contained fixes (T1 + T3); the queue-persist rewrite (T2)
follows separately.
config-store (T1):
- load() now has an in-memory cache keyed on the file mtime+size. The processed
config (merged + credential-decrypted) is re-read/re-parsed/re-DPAPI-decrypted
ONLY when the file actually changes; our writes invalidate it, external edits
change mtime/size so the cache misses. Eliminates a full disk read + JSON.parse of
the whole growing history + per-credential decrypt on the vast majority of the ~38
load() call sites. load() always returns a structuredClone so callers can mutate
freely without corrupting the cache.
- _serializeForDisk clones ONLY the hosters subtree (the only thing encryptCredentials
touches) instead of JSON.parse(JSON.stringify(whole config)) — no more deep-cloning
an 8 MB history on every write.
- _atomicWrite refreshes the .bak with a raw fs.copyFileSync instead of
read + JSON.parse + write (it was re-parsing the full config a 2nd time per write).
main.js (T3):
- The log-flush paths resolved the log file via configStore.load() ~8x/second during
uploads (re-reading + cloning the whole config just to read logMode/logFilePath).
Cache those two strings in module scope, invalidate on the settings-save handlers.
Verified: 26 config-store tests (incl. new cache-correctness: independent clones,
external-change invalidation, save invalidation) + full 394-test suite green, lint 0
errors. Benchmark (8000-batch / 4.6 MB history): log flush no longer calls load() at
all; the remaining per-write history serialize is what T2 removes from the hot path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the one link the unit/wiring tests covered only by composition: binds 0.0.0.0,
allowlists a real LAN IPv4, and asserts auth-ok over a real socket (the Tailscale path).
Skips when no non-internal IPv4 interface exists.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Matches the Real-Debrid-Downloader's rd-diagnostics-mcp model so the read-only
diagnostics agent is reachable over Tailscale (or any private tunnel) the same way
the downloader is, instead of requiring an SSH local-forward.
- lib/ip-allowlist.js (NEW): fail-closed IP allowlist — normalizeIp strips
::ffff:, loopback is always allowed, an empty allowlist accepts loopback ONLY
(fail-closed), exact IP + CIDR (incl. the Tailscale CGNAT range 100.64.0.0/10) +
wildcard rules. The real socket peer IP is the authority (never a forwarded header).
- remote-server.js: rejects non-allowlisted peers at connection (close 4005). Opt-in
via config.allowlist (the existing remote-control server, which passes none, is
unaffected). Loopback always passes, so local + SSH-forward use keeps working.
- Two bind modes (config diagnostics.bindMode): "local" -> 127.0.0.1 (default),
"network" -> 0.0.0.0 but ONLY when a non-empty allowlist is set (else it stays
loopback, fail-closed). The allowlist + token gate access; the tunnel
(Tailscale/WireGuard) is the confidentiality layer (transport is still plaintext ws://).
- The connection code now carries the host: mhu1_<base64url{v,h,p,t,n,fp?,s?}>. The
gateway decode is tolerant of the legacy {port,token,label} keys; connect_server
takes the host from the code (host arg is an optional override). Proven end-to-end:
the integration harness now connects with NO host arg and resolves it from the code.
- Renderer: Sichtbarkeit selector (local/network), public-host input with
suggested-host chips (os.networkInterfaces — the Tailscale IP shows up there),
allowlist textarea (network mode), and network-requires-allowlist validation.
- main.js: bindMode->host, getSuggestedRemoteHosts, host-in-code, allowlist plumbed
into startDiagnosticAgent + the diagnostics IPC (get/save/status).
- docs: rewritten for Tailscale (set the allowlist to your tailnet, put the Tailscale
IP/MagicDNS in the code address — no SSH forward needed).
This supersedes the v3.3.85 hard loopback-lock with the downloader's allowlist model.
Tests: lib/ip-allowlist (8) + remote-server allowlist wiring/loopback (2) + gateway
decode (host short-key + legacy tolerance). 393 app tests + 9 gateway tests + e2e +
host-in-code integration + adversarial all green; lint 0 errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Repo-side hardening and tooling from the intensive test round (none of this ships
in the app installer).
- gateway: registry.json (holds bearer tokens) now gets a best-effort owner-only
NTFS ACL on Windows via `icacls /inheritance:r /grant:r <user>:F` (the chmod
0600 is a no-op on NTFS); verified the file ends up <user>:(F) only.
- gateway: connect_server now reads the app version from the real get_system_info
shape (data.app.version / data.agent.version), so "connected to vX.Y.Z" works.
- gateway: read_log tool description documents grep as a case-insensitive substring
filter with "|" alternation (not a regex), matching the agent-side change.
- gateway: standalone verification harnesses moved to gateway/verify/ (so
`node --test` only sweeps real unit tests) and exposed via `npm run verify`:
e2e-verify, integration-mcp (live gateway-MCP <-> agent, all 14 tools), and
adversarial-probe (redaction fuzz + ReDoS + lockout). `npm test` runs the units.
- eslint: gateway/** now lints as ESM (sourceType module) via a dedicated block;
global ignores fixed so `eslint .` is clean across the whole project (0 errors).
- docs/remote-diagnostics-setup.md: made the transport story honest — the agent
speaks plaintext ws:// over enforced loopback; the SSH/WireGuard tunnel is the
ONLY confidentiality layer (wss/TLS + cert-pin is a documented future mode, not
active). Removed the stale "bind to a LAN/VPN IP" guidance (loopback is enforced).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Intensive end-to-end testing (a live gateway-MCP <-> agent integration harness +
an adversarial redaction/abuse probe + an independent security audit) surfaced
three real issues in the shipped read-only diagnostic agent. All run in lib/**,
which is packaged in the app.
1. grep ReDoS froze the Electron main process. read_log compiled the
client-supplied grep into `new RegExp(grep, 'i')` and ran it synchronously over
the log tail IN the main process. A catastrophic pattern (e.g. "(a+)+$" against
a long line) hangs the whole app — empirically confirmed (8s timeout, killed).
JS regex is synchronous and uncancellable, so grep is now a case-insensitive
literal substring filter with "|" alternation ("error|timeout|502"). Provably
linear-time; covers the real diagnostic need.
2. Prototype-chain whitelist bypass. The op table was a plain object literal, so
handle("constructor" | "toString" | "valueOf", ...) resolved an inherited
Object.prototype function, passed the `typeof fn === 'function'` guard and
returned {ok:true}. Harmless functions today, but a whitelist-integrity hole.
Now guarded with a string check + Object.prototype.hasOwnProperty.
3. Redaction defense-in-depth gaps. redactLogText now also scrubs: basic-auth URL
passwords (scheme://user:pass@host), Authorization: Basic, JWTs (eyJ...x.y.z),
and bare/JSON session= values. Mostly theoretical in today's readable logs
(secret-bearing bodies go to the excluded doodstream-debug.log; other hosters
throw static strings) but matters as the verbose-logging surface grows.
Verified: 383 app tests (incl. new regression tests for all three), the live
gateway-MCP integration harness (all 14 tools, zero leaks, error paths), the
adversarial probe (14/14+ secret shapes scrubbed, ReDoS 1ms, lockout, malformed
args), e2e gate, lint 0 errors. Only residual: a standalone high-entropy blob with
zero key/Bearer/URL context — inherent to any denylist, acknowledged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
registry.json holds each server's bearer token (the connection secret). It was
written with the process default umask, leaving it group/world-readable on POSIX
multi-user hosts. Write with mode 0o600 and chmod the existing file (writeFile
only applies mode on creation). No-op on Windows (NTFS uses ACLs, and the file
already sits under the user profile and is gitignored), effective on Linux/macOS
where the gateway may run.
Gateway-only change — not part of the app installer or auto-updater, so no
version bump.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Captures the two redaction leaks from the remote-diagnostics build: a hoster-
returned opaque token surviving because value-scrub only covers stored config
secrets, and the get_config_redacted/get_queue_state default paths skipping
pattern-scrub entirely. The discriminator the first E2E missed, plus the rule:
exercise each collector with its DEFAULT args and seed fixtures with a NON-config
secret so you test pattern-scrub, not value-scrub by accident.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
get_queue_state with includeJobs:true (the DEFAULT path) scrubbed the job list
with value-scrub only, so an opaque token a hoster returns inside a job error
(token=... that is NOT one of the user's stored credentials) survived in the
response. Same leak class already fixed for get_config_redacted and
server_health, still open on the queue collector's default path.
The first end-to-end gate missed it on two coincidences: server_health calls
getQueueState with includeJobs:false (no job error ever serialized), and the
fixture's queue error used a value that WAS a config secret (so value-scrub
caught it anyway). The direct get_queue_state{includeJobs:true} path with a
non-config token was never exercised.
- getQueueState job list and getRotationState now go through _deepRedact
(per-leaf pattern + value scrub), matching the other collectors.
- e2e-verify.mjs now plants a non-config token in a queue job error and asserts
both get_queue_state{includeJobs:true} and the default-args call leak nothing.
- Added a main-suite regression test for the default includeJobs path.
386 app tests + 9 gateway tests + e2e gate pass; lint 0 errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the connect-by-code side of remote diagnostics and closes two real
secret-leak vectors that an end-to-end gateway<->agent test surfaced.
Gateway (gateway/, local stdio MCP, Claude connects once):
- 14 read-only tools (server_health hub, read_log, list_logs, list_errors,
get_queue_state, get_history, get_config_redacted, get_system_info,
get_rotation_state, get_app_events + connect/disconnect/list/current).
- The HOST is always supplied by the operator, never taken from the code.
- TLS fingerprint pinning is enforced in the socket 'open' handler BEFORE the
token is sent (wss opt-in); plain ws is loopback-only.
- registry.json (holds bearer tokens) is gitignored; only an empty example ships.
Security hardening (gates every off-box payload):
- redactLogText now scrubs opaque bearer/token-family secrets that are NOT
stored config credentials (e.g. a session token a hoster returns inside an
error string): bare token/auth_token/refresh_token/session_token + standalone
"Bearer <opaque>". Benign "token bucket" prose is left intact.
- get_config_redacted deep-redacts every string leaf (JSON-safe, per-leaf, so
the cookie/sess line patterns can't gobble across a compact-JSON field) and
drops the history subtree (served by get_history with its own per-error
redaction). This plugs leaks via globalSettings.pendingQueue[].error etc.
Bind-address safety:
- _safeDiagBindAddress() forces the diagnostic agent to 127.0.0.1/::1; the
0.0.0.0 UI option is removed. Direct LAN/Internet bind stays disabled until
encrypted transport (wss) exists — remote access goes through an SSH/VPN
tunnel to loopback. (Never plaintext ws:// on all interfaces.)
Tests: end-to-end gateway<->agent gate (connect -> server_health/read_log/
get_config_redacted, asserts zero secret leakage, rejects doodstream log, path
traversal and write ops); + redaction regression tests in the main suite.
385 app tests + 9 gateway tests pass; lint 0 errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the server/app half of a remote-diagnostics system so Claude (via a local
MCP gateway, added separately) can read a server's full state to diagnose
problems: logs, errors, queue/app state, redacted config, system health. All
read-only; security review folded in as hard requirements.
Architecture (design-verified):
- A SECOND, independent RemoteServer instance (diagnosticMode) on its own port
(default 9110, bind 127.0.0.1) with NO capture/input callbacks, so screen
capture and sendInputEvent are structurally unreachable from the diagnostic
path. The existing screen-share server (port 9100) is byte-for-byte unchanged.
- lib/remote-server.js: a `host` bind option, a post-auth `diag-request` branch
that delegates to onDiagnosticRequest and replies with a reqId-correlated
`diag-response`, a `diagnosticMode` guard (capture never spawns), a
timing-safe token compare (length-guarded), and getLastAccess().
- lib/diagnostics-agent.js: handle(op,args) enforces a HARDCODED read-only op
whitelist as the sole authority — no write/exec op, no run_health_check.
- lib/diagnostics-collectors.js: pure, dependency-injected collectors
(server_health one-shot hub, read_log, list_errors, get_queue_state,
get_history, get_config_redacted, get_system_info, list_logs, get_app_events,
get_rotation_state, get_health) reusing support-bundle + stats.
Security (all mandatory, implemented):
- Redaction gates every off-box payload. support-bundle now adds webhookUrl +
diagToken to CRED_KEYS, and exports redactLogText (value-scrub of live secret
strings + pattern-scrub of Discord webhooks / Bearer / api_key= / cookies /
sess ids) + valueScrub + collectSecretValues. read_log takes a logical NAME
(no path traversal); doodstream-debug.log is excluded from the readable set
(it logs live api-key-bearing HTML). grep is length-capped (ReDoS guard).
- diagnostics config subtree (enabled/port/token/label/codeIssuedAt/bindAddress)
defaults OFF, bind 127.0.0.1; deep-merge makes it migration-free. The
server-owned token is preserved against renderer clobber in both
save-global-settings handlers; the diagnostics:* IPC is the only mutator.
- app.requestSingleInstanceLock() (also the approved fix#6) so a relaunch can't
EADDRINUSE-kill the agent and two instances can't clobber the config.
main.js: buildDiagnosticCode (mhu1_ base64url, no host embedded), start/stop +
auto-start-on-launch + stop-on-quit, the four diagnostics:* IPC handlers.
renderer: a "Diagnose-Zugriff" settings subtab (toggle, port, bind-address,
copyable connection code + regenerate, status, read-only security note).
382 tests pass (incl. 11 new: collectors+redaction, agent whitelist, and a live
RemoteServer protocol test proving auth->diag-response correlation, that a
diagnostic client never triggers the capture window, and brute-force lockout).
Lint clean (0 errors). Smoke boots identically to baseline. The standalone MCP
gateway package + operator setup docs land in a follow-up commit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three lessons from the second adversarial bug hunt:
- A dedup/skip key has more re-add paths than "in-place vs rebuild" suggests; enumerate every .add site and every selectedFiles re-add, cross-tabulate which clears the key. retrySelectedJobs (retry-of-done) and the folder-monitor branch were the two missed paths.
- Per-cell (file|hoster) deletion needs its own persisted suppression set with the OPPOSITE lifecycle to the completed-key (set on delete, cleared on re-add); overloading the completed-key would break re-upload-after-delete.
- Re-confirming a user-facing change is "real" is not the user consenting to the behavior change; force the decision via one AskUserQuestion or park it, do not re-raise it as ambient worry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A 36-agent adversarial sweep over the queue-persistence context surfaced three
NEW, reachable defects (all renderer-only) that the v3.3.80-82 work missed:
#1 retrySelectedJobs (the reuploadBtn / "erneut hochladen" path) is a THIRD
re-add path that never cleared the completed-dedup guard. _completedUploadKeys
gains file|hoster when a job reaches done; retry of a done row reset the job to
pending and re-added its path to selectedFiles but kept the key. On a restart
before the re-upload re-completes, restoreQueueStateFromConfig re-seeds the key
and buildQueuePreview then refuses to recreate the row, so the deliberate
re-upload silently vanished (lost work). Retry now clears the exact file|hoster
keys it re-activates (per key, NOT per path, so a sibling hosters completed
ghost is not resurrected).
#7 The folder-monitor pre-selected-hosters branch re-adds a path and calls
buildQueuePreview WITHOUT going through applyHosterSelection, so the v3.3.82
key-clear never ran. With removeFromQueueOnDone ON, a re-encoded / re-dropped
file whose row was auto-removed was suppressed forever (silent no-upload that
even survived a restart). The branch now clears the dedup keys for the freshly
re-added paths, matching the manual modal flow.
#4 Deleting one hosters row of a multi-hoster file was silently undone. A
manually deleted non-completed file|hoster pair is recreated by buildQueuePreview
the next time it runs (adding another file, or a folder-monitor drop), because
the recreate guard only consulted _completedUploadKeys, never the per-cell
deletion. Result: the file got uploaded to a hoster the user explicitly
removed, burning quota and publishing an unwanted link. Introduces
_suppressedPreviewKeys: a manual delete of a non-done job whose file stays
selected (pinned by a sibling job) suppresses that exact file|hoster from
re-creation; the suppression is persisted in pendingQueue.suppressedKeys and
re-seeded on restore (mirroring completedKeys), and is cleared whenever the
user deliberately re-adds the file (modal or folder monitor) so a real re-add
still works.
The clear-on-re-add logic for both sets is unified into clearDedupKeysForPaths()
and reused by applyHosterSelection and the folder-monitor branch.
Held for a separate decision/round (surfaced to the user): single-instance lock,
the sync-close write-sequence guard (its ghost symptom is masked by the existing
microtask-ordered removal + restore-time ts-gate; its real residual is rotation/
history integrity), and the upload-log timezone/same-basename edges (need a log
format change). 362 tests pass, lint clean, smoke boots identically to baseline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Captures the v3.3.82 near-miss the advisor caught: persisting _completedUploadKeys
risked silently no-op'ing a re-upload IF any retry path rebuilt jobs via
buildQueuePreview. Both retry paths (retrySelectedJobs, _retryFailedFromBuckets)
mutate jobs in place and are immune; only fresh modal re-adds go through the
preview and clear the key. Rule: read every 'erneut/retry/reupload' handler
before touching a skip-guard, decide mutate-vs-rebuild per handler.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The completed-upload dedup guard (_completedUploadKeys, "file|hoster") is the
single source of truth that keeps a finished job from being re-materialised as a
"Bereit" preview by buildQueuePreview(). Three independent paths leaked ghosts
back into the queue:
A) removeJobFromIndex() unconditionally DELETED the dedup key. The
removeFromQueueOnDone auto-remove path (handleProgress), the batch-done sweep,
and the terminal-job prune all call removeJobFromIndex on a *finished* job, so
the guard handleProgress had just added was immediately wiped and
buildQueuePreview re-created the job as a preview both mid-session and after
restart. removeJobFromIndex now takes keepCompletedKey; the three auto-removal
call sites pass true (keep the guard), while the manual-delete sites keep the
old behaviour (drop the guard so the user can re-add the file).
D/E) The dedup guard lived only in memory, so every restart began with an empty
set and buildQueuePreview rebuilt ghosts from the persisted selectedFiles.
buildPersistedQueueState() now serialises the keys whose file is still in the
snapshot (completedKeys) and restoreQueueStateFromConfig() re-seeds them. To
keep deliberate re-uploads working, applyHosterSelection() clears the
persisted key for every freshly (re-)added file: re-adding through the hoster
modal is the explicit "upload this again" signal. Both retry paths
(retrySelectedJobs, _retryFailedFromBuckets) mutate the existing job in place
instead of relying on buildQueuePreview, so they are unaffected.
H) buildQueuePreview() excluded error jobs from its existingKeys set, so a
file+hoster that already had an 'error' row got a second 'preview' row stacked
beside it. Error jobs now count as existing.
B/C) parseUploadLogLine() took the filename from a fixed field index and trimmed
it. A pipe inside the link shifted the field (entry lost from the log-based
dedup) and trimming broke matching against the untrimmed OS basename used as
the queue-job key. The filename is now the last non-empty field and is no
longer trimmed, so log lines with pipes in the URL and leading-space filenames
both match end-to-end.
362 tests pass, lint clean on all touched files.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up hardening on the v3.3.80 queue-persistence fix after a deeper adversarial
sweep + reviewer pass over the WHOLE subsystem (not just the diff). Two real defects:
1. Basename-collision lost work (regression introduced by v3.3.80 FIX A).
restoreQueueStateFromConfig collapses jobs on the FULL path while the ts-gate keys
on basename|hoster. Two genuinely different files with the same basename queued to
the same hoster from different folders therefore share a gate key: if one was logged
after savedAt, the ts-rule dropped BOTH — silently losing the still-pending one.
Pre-FIX-A only 'done' jobs were dropped, so a pending file was never at risk.
Fix: an ambiguity guard in partitionRestoredJobsByLog — the ts-rule is suppressed
when a basename|hoster key maps to more than one distinct file path (the log records
only basenames, so it can't say which physical file completed). The done-in-log rule
is unchanged. Fails safe: worst case a visible ghost survives, never silent data loss.
2. selectedFiles re-materialization with removeFromQueueOnDone=ON (second mechanism,
independent of the stale snapshot). When that setting is on, a completed job is
stripped from queueJobs but its path stays in selectedFiles (syncSelectedFilesFromQueue
only runs at batch-done, never on a mid-upload close). On restart the ts-gate operates
on queueJobs and never sees it, then the startup updateUploadView -> buildQueuePreview
re-creates it as a preview ghost AFTER the gate ran, and it re-persists with a fresh
savedAt — sticky. Fix: completedSelectionKeys() seeds _completedUploadKeys (the set
buildQueuePreview already consults) from the log at startup, keyed on full path, with
the same ambiguity guard. Log-based so it survives a hard kill, consistent with FIX A.
Also extracts the orphan-tmp sweep decision into lib/orphan-tmp.js (was untested inline
code in main.js; behavior-preserving) and adds executable coverage for the paths that
were previously only argued from logic: orphan-tmp sweep, config-store pendingQueue+savedAt
round-trip, an end-to-end scenario in the exact user-reported shape (300 queued / ~200
finished mid-session), and a 3000+500-iteration property fuzz of the gate invariant
including the lost-work guarantee. 359/359 green, ESLint clean, smoke-boot unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
