v3.3.97 (UV_THREADPOOL_SIZE 64→8) was a decisive win — mean event-loop-delay at
70 active uploads dropped 200ms→~11ms (18×), rss 577→287MB, renderer healthy in
14/15 windows. But the user reports it is still not perfectly smooth. A focused
multi-agent investigation plus an adversarial review localized the residual to
TWO distinct, separately-measured spike sources:
1. Read-bursts. In the tail windows the file-read histogram inverts: FSReqCallback
climbs to 66-70 against threadpool=8 (~8.75× queue depth) while SimpleWriteWrap
(socket writes) collapses to 4-24 and mean delay rises to 30-42ms. GC is ruled
out (gcMax ≤27ms in every window). The clean inversion at a stable active=70 /
pending=1287 shows the reads are causal, not a symptom of a block elsewhere.
2. A suspected synchronous config-persist stall. save() → load() reparses the whole
electron-config.json — which now carries the 1287-job pending queue nested in
globalSettings plus full history — on every persist (because _atomicWrite nulls
the read cache), then _serializeForDisk JSON.stringify(…, null, 2) of all of it.
One tail sample (max 1021ms, heap spiking to 142MB) fits a large synchronous
structuredClone+stringify, but it is a single confounded point, so this build
only INSTRUMENTS the path rather than asserting the cause.
This release ships one behavioral change (kept to a single variable so the next
log attributes cleanly) plus measurement:
- highWaterMark 256KB→1MB in all five streaming read loops (lib/hosters.js,
doodstream/voe/vidmoly CHUNK_SIZE consts, and the inline value in
clouddrop-upload.js:108 — NOT the 16MB server chunk at clouddrop-upload.js:12).
UV_THREADPOOL_SIZE stays 8. This deepens each stream's read-ahead cushion from
~0.43s to ~1.7s at the per-stream rate, so a stream tolerates the threadpool
queue without starving its socket write, and cuts read-completion callbacks and
per-chunk Buffer allocations ~4×. Byte-correctness is unaffected: Content-Length
is preamble+fileSize+epilogue, independent of chunk size, and the chunk size
never touches the multipart boundaries. Fully reversible; a dedicated read-
concurrency semaphore is held in reserve if 1MB does not clear the bursts.
- config-store.js now times load() (the full reparse, which the account-failed
handler also hits per failure) and the _commit serialize, logging
`config-load …` / `config-serialize wall=…ms bytes=… hist=… queue=…` when the
synchronous work exceeds 20ms. load() is split into a timing wrapper + _loadImpl;
the timer is a no-op until main.js wires configStore.setPerfLog → logInfo.
The renderer batch-drain fix for the one observed 243ms longtask is intentionally
deferred: that jank is downstream of the main-thread read-burst flooding IPC, so
fix#1 should make it self-heal; bundling it would confound the measurement and
touch the progress hot path. All 397 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
doodstream-upload.js's _debugLog ran with NO verbose gate: every call did a
synchronous statSync (via maybeRotateLogFile) + appendFileSync directly on the
main-process event loop, ~8-15x per upload (server probe, response, redirect,
result page, filecode parse, hidden fields, submit/follow, plus retry branches).
DoodstreamUploader runs in the main process, so each pair of sync fs syscalls
blocked the event loop while uploading — delaying IPC, upload-progress-batch
forwarding, tray-tooltip and webhook handling for every other concurrent upload.
A constant per-upload main-thread tax (not history-scaling), surfaced by the
session-wide lag audit and confirmed as the one finding that bites in the real
upload scenario.
Fix: gate _debugLog behind the existing globalSettings.logVerbose setting
(default false), exactly mirroring main.js logDebug/_logVerbose. A module-level
_debugVerbose flag + setDebugVerbose() setter, an early-return at the top of
_debugLog, and one wire at main.js's single setLogVerbose chokepoint (covers
boot + save-config + the verbose toggle). When verbose is off the doodstream
trace simply isn't written — same contract as the main debug.log — and the
per-upload sync fs disappears. When a doodstream issue needs tracing, enabling
verbose restores the full trace.
The config-store audit findings (load() history-clone cost, per-write history
serialize) are real but scale only with history size — tens of microseconds at
this user's 8-batch config, and the safe fix is risky persistence surgery on
credential-bearing code for a latent micro-cost; deferred and documented in
tasks/todo.md rather than shipped.
397/397 tests pass, eslint clean, gate wiring verified (shared module instance,
default-off, toggles).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Direct improvement to the login/web path (no API key needed): we were POSTing a
stale field set — sess_id + utype=reg + file — but doodstream's CURRENT upload
form dropped `utype` and added file_title / fakefilepc / submit_btn. Submitting
an incomplete/stale field set can make the CDN node accept the bytes but skip
the registration step (→ the empty result form with no fn). Now we parse the
live upload form (already fetched in _getUploadServer) and replicate ALL its
non-file fields faithfully — exactly what the browser submits — while keeping
sess_id (the fresh node token) and utype as a harmless compatibility extra.
- _parseUploadFormFields(html): pull every named input/button from the upload
form, excluding the file input (streamed separately). Adapts to whatever
fields doodstream uses now rather than hardcoding.
- upload() builds the multipart from those fields; minimal known-good fallback
if the form wasn't parsed.
- Tests: real-form extraction (incl. file-input exclusion) + no-form safety. 183/183.
Low regression risk (superset of the previously-working fields). Whether it
resolves the large-file empty-form is for the server run; the API path
(3.3.31/32) remains the reliable route when a key is available/derivable.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The user uploads with username/password (login), so 3.3.30's "use API when a
key is configured" did nothing for them — and the web-form upload keeps failing
with empty forms on large files. Fix the LOGIN path itself: after logging in,
pull the account's API key out of the logged-in session and upload via the
official doodapi API (which returns result[0].filecode directly, no empty form).
The user keeps using login and configures nothing.
How the key is derived without knowing doodstream's (cookie-gated, unseen)
settings DOM: brute-force candidate extraction + API validation.
- DoodstreamUploader.deriveApiKey(): fetch the logged-in settings page
(?op=my_account / /settings), pull every plausible long token from form-field
values + element contents (ranked: tokens near an "api" mention first), and
validate each against doodapi.co/api/account/info — only the account's real
key returns status 200. A wrong guess is therefore harmless (fails validation
→ web fallback). Logs the raw settings HTML when nothing validates, so the
scrape can be refined from a real capture if doodstream's markup differs.
- upload-manager: doodstream login-path now resolves the key ONCE per batch
(cached by accountId; '' = tried-none) and routes to the API when found, else
the existing web-form upload. Keyless accounts: one extra probe-login per
batch, then unchanged.
- Tests: candidate extraction (value/textarea/api_key shapes, api-context
ranking), validate-then-pick, null→web-fallback, preset short-circuit. 178/178.
If derivation works the login path now uploads via the API. It does NOT change
doodstream's backend; the server run confirms. Falls back safely if no key.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The "kein Filecode — Server gab leeren Link zurueck" error was treated as a
generic upload failure → after retries exhausted, the manager called mark-failed
and added the account to _failedAccounts → next batch re-primed with
primedFailed=1 → pre-job-swap-blocked because no fallback override exists for a
single-account hoster. One server-side flake permanently poisoned the session.
It's not an account problem — same account + same file works on a later try.
This is a doodstream-backend processing flake (empty CDN form, no fn / no st),
the same class as a transient network error: don't blacklist, just fail this
file cleanly.
- doodstream-upload.js: tag the empty-form throw with err.hosterTransient=true
(explicit flag, primary signal — matches the err.accountError / err.fileRejected
pattern already used elsewhere).
- upload-manager.js: new _isHosterTransientError classifier (flag first, message
regex as defensive fallback). In the retry loop: break on first hit (server
flake won't clear in 3 s, re-uploading the file 4× is pure bandwidth waste).
Post-loop: dedicated branch that emits the final error WITHOUT blacklisting
the account — same shape as the existing transient-network branch.
- Tests: classifier unit tests (flag path, regex path, negatives) + regression
test that proves the account is NOT added to _failedAccounts and mark-failed
does NOT fire. Drops the hoster-transient test from ~19 s to ~1.5 ms,
confirming the in-loop fast-break works.
We now fail fast on this error class instead of retrying — the next-batch
manual retry is the recovery path, and the account stays usable for it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
After 3.3.26 fixed the filecode parsing, the remaining intermittent failure is
a generic "fetch failed" — a transient network error on one of the requests
around the multi-minute upload. Can't tell from one log line whether it's the
server-discovery GET or the post-upload result-submit, so harden both:
- _fetch (the native-fetch chokepoint for discovery, redirects, result-submit):
retry up to 3x with short backoff on a thrown network error, each attempt
bounded by a 20s timeout (Node fetch has none by default). Caller aborts are
not retried. The big file upload (undici) is retried at the upload-manager
level, not here.
- result-submit is now best-effort: if it still fails after retries but we
already hold the filecode from the CDN response, return that instead of
discarding a completed upload.
- label the undici upload-POST error with phase + MB sent + node, preserving the
original message so transient classification still matches.
- eslint: add AbortSignal to globals.
- Tests: _fetch transient-retry path (10 doodstream tests total).
"fetch failed" is already classified transient by upload-manager, so this is
additive resilience; next logs will show if anything still slips through.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The 3.3.25 diagnostics captured the live upload page: doodstream moved the
upload server from a `srv_url` JS variable into the multipart form's action,
e.g. action="https://xxx.cloudatacdn.com/upload/01?SESSID", with a per-page
session token in the query that matches the page's hidden sess_id input. The
old parser found neither and fell through to the stale hardcoded node, which
returns an empty filecode.
- Parse the upload server from the form action (matched via the /upload/ path),
un-escaping & in the query string.
- Refresh this.sessId from the SAME page (only on action match) so the
multipart sess_id field matches the node URL's token; login-time and node
tokens otherwise diverge. Keep the existing sessId if the input is absent.
- Keep the legacy ?op=upload_server JSON and srv_url paths as fallbacks; the
fail-fast throw from 3.3.25 stays as the last resort.
- Tests: form-action parse, sess_id refresh, & un-escape (9 total).
Whether this fully resolves the uploads is for the next server logs to confirm;
both the node and sess_id fixes are individually correct.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Real root cause from the 3.3.24 diagnostics: the failing upload used CDN
"tr1128ve.cloudatacdn.com/upload/01" — character-for-character the hardcoded
last-resort fallback in _getUploadServer(). The CDN form came back with only
op=upload_result and NO fn/NO st, i.e. the bytes went into a stale node that
returns an empty form. So _getUploadServer can no longer extract the current
upload server (Doodstream likely changed the upload_server response/format) and
silently fell back to a dead node — wasting ~90s/95MB per attempt.
- Remove the silent hardcoded-node fallback; throw a clear error when discovery
fails so the upload fails instantly instead of 90s later with a cryptic msg.
- Embed the raw upload_server response (status, content-type, body) and
upload-page URL hints in the error AND debug log, to pin the format change.
- Tests: getUploadServer JSON path, srv_url HTML fallback, and the no-silent-
fallback throw (asserts the hardcoded node never leaks into the error).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The "upload_result Seite hat keinen filecode" error fired with no actionable
detail when Doodstream's CDN returned an empty filecode (fn). Root cause is
server-side: the page structure is unchanged, the link is just missing —
Doodstream's backend refused the file (copyright/hash match, duplicate, size,
quota). XFileSharing reports the reason in the `st` field, which we ignored.
- Surface `st`: non-OK status now throws "Doodstream lehnt Datei ab (Status: …)".
- Enrich the generic error with st, fn-state, and the CDN node for diagnosis.
- Fix debug-log path: wrote to __dirname/.. which is read-only (app.asar) in
packaged builds, so production captured zero traces. Now uses Electron's
writable userData dir, with repo-root fallback for tests/plain node.
- Add tests/doodstream-upload.test.js (4 tests) pinning the parse/error paths.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
3.3.2 fixed fileuploader.log unbounded growth, but three siblings kept
growing without limit:
- upload-debug.log (verbose, every IPC + progress event log line)
- account-rotation.log (every rotation decision)
- doodstream-debug.log (per-hoster trace from lib/doodstream-upload.js)
A multi-month dev install or a heavy production user could fill the
log dir with multi-GB files and slow every appendFile.
Wire all three through the same lib/log-rotation.js helper:
- upload-debug.log → 25 MB cap, 2 numbered backups (~75 MB worst)
- account-rotation.log → 10 MB cap, 2 numbered backups (~30 MB worst)
- doodstream-debug.log → 10 MB cap, 1 numbered backup (~20 MB worst)
The rotation check runs once per flush call (each is debounced or
already a once-per-event path), so the statSync overhead is
microscopic. _flushDebugLog passes a noop logger to avoid recursing
into itself; _flushRotLog and _debugLog (doodstream) use the normal
debugLog so any rotation surprises end up in upload-debug.log.
126/126 tests still green.
- Upload button no longer gets permanently stuck if startUpload()
throws after health check (try-catch with uploading=false reset)
- Wait for running health check instead of silently blocking upload
- Add abort signal check in VOE/Vidmoly upload generators
- Escape filenames with quotes/backslashes in multipart form headers
(all 4 uploaders: doodstream, voe, vidmoly, byse)
- Validate backup import structure before overwriting config
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When Doodstream requires 2FA, the account modal now dynamically
shows an OTP input field so the user can enter the code from
their email and complete the login without restarting.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Guard startBatch against null uploadManager in nextTick (race on fast cancel)
- Fix updateSettings not creating globalThrottle when none existed at start
- Fix updateSettings not updating globalSemaphore limit live
- Fix retry pause: 2500ms → 3000ms as intended
- Remove dead isError code in history (was always false after continue)
- Add signal.aborted check in API upload generator (hosters.js)
- Add extra signal check in throttle consume loop for faster abort
- Fix doodstream debug log path (process.cwd → __dirname)
- Fix updater fetchJson signal listener leak
- Make progress column sortable in queue table
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix DoodStream upload: parse <textarea> fields (not just <input hidden>)
- Fix DoodStream upload: handle redirect responses from upload server
- Fix DoodStream upload: submit upload_result to doodstream.com (not CDN)
- Fix DoodStream speed display: switch to async generator streaming
- Add "Start Selected" toolbar button to upload only selected queue items
- Move "Always on Top" from context menu to Settings
- Remove "Shutdown after Finish" from context menu
- Hide error entries from upload history (only show successful uploads)
- Disable background throttling to prevent UI lag on focus switch
- Add debug logging for DoodStream upload troubleshooting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Stats tab in recent panel (queue counts, sizes, speed, ETA, run time)
- Aborted jobs persist across restart (saved as queued)
- Doodstream: throttle support, better error messages with HTTP status
- Recent panel tab switching (Files / Stats)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix doodstream login: handle redirect on success (server returns HTML dashboard instead of JSON)
- Fix sess_id extraction: match hidden input field format
- Files are now only added to queue after clicking "Uebernehmen" in hoster modal
- Cancel/Escape/click-outside discards pending files
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add doodstream.com web login (email+password) as alternative to API key
- Fix doodstream login: use X-Requested-With header for JSON response
- Add "Aus der Queue entfernen bei Abschluss" setting
- Fix byse.sx download URLs to use /d/ prefix
- Make config writes async to prevent race conditions
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
