c7fa422d9b
5 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Administrator
|
7f636258d4 |
perf(uploads): make the batch-start file-stat non-blocking — kill the 336ms main stall (v3.3.103)
The v3.3.102 log (real 224-job batch) confirmed History virtualization works and
steady-state uploads are pristine (fps=32, event-loop mean 11.8ms). The remaining
residual was a ~6s spin-up burst at batch start: the main event loop blocked for
336ms with cpu=0%core (i.e. blocked on I/O, not computing) and the renderer janked
196-391ms, then everything settled clean.
A multi-agent investigation plus an adversarial review corrected the obvious-looking
hypothesis. The renderer's uncapped progress-batch drain is NOT the cause:
handleProgress only mutates plain JS state and schedules already-coalesced renders
(one per frame), and main coalesces progress to ~50 latest-per-job entries per
100ms. Chunking that drain would fix nothing — and the reviewer showed it would
REGRESS correctness: requestAnimationFrame throttles to ~0 when the window is
minimized (the common state for a background uploader), so a rAF-chunked drain would
grow an unbounded backlog and defer persistQueueStateSoon for every buffered item,
losing terminal 'done' events on close (the queue-persistence ghost-fix class). So
that path is deliberately not taken.
The real cause (cpu=0%core = blocked on I/O) is a synchronous fs.statSync storm in
UploadManager.startBatch: the dedup loop ran up to DEDUP_CHUNK=200 synchronous
fs.statSync calls in a single tick before yielding (200 x ~1.68ms on the user's VM
= the exact 336ms), on a disk already saturated by the 1MB read-ahead.
Fix — make the batch-start stats non-blocking:
- The dedup loop now dedupes synchronously (cheap Map work) and then stats the
unique files in parallel via await Promise.all(fs.promises.stat ...) per chunk, so
the stat I/O runs on the libuv threadpool and the main thread never blocks. The
results-Map shape ({name,size,results:[]}) and dedup semantics (size 0 on failure)
are unchanged.
- The per-job statSync fallback is converted to await fs.promises.stat for
consistency (it sits in an async function before the first real await; the cached
size from dedup already lets nearly every job skip it).
Tests: the upload-manager mocks override fs.statSync; they now also override
fs.promises.stat with the same fake sizes (upload-manager.test.js x2,
suspect-reject-alternates.test.js). 407 tests pass; clean Electron boot.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Administrator
|
8b2a1d7c1f |
fix(byse): treat 5xx/gateway upload failures as transient infra, not account faults
A single byse.sx gateway hiccup (HTTP 502 "<!doctype html>..." or a mid-upload
ECONNRESET) was cascading through the whole failover chain and blacklisting every
account for the batch. Root cause: the upload-POST throw sites threw PLAIN errors
with no classification, so a 502 was treated as a GENERIC error -> mark-failed ->
emit('account-failed') -> failover to the next account (which hits the same
gateway) -> repeat until the chain is exhausted, then poison sibling files in the
batch via the blacklist. The screenshots showed exactly this: "Primär ... Fallback
#3" all failing with the same 502. byse did NOT change their API (the upload
contract still matches their docs and was live-probed: GET /upload/server -> POST
{key}+file -> files[0].filecode); these are transient infrastructure failures.
A 5xx gateway error is not an account fault: every account hits the same gateway,
so failing over is pointless and blacklisting is harmful. Fail open instead — retry
the SAME account and, if byse stays down, fail the file cleanly without touching the
account or the rotation cursor.
- upload-manager: _isTransientNetworkError() now honors an explicit err.transientNetwork
flag (checked before the empty-message guard) and, as a defensive fallback, matches
/HTTP 5\d\d/, Bad Gateway, Service Unavailable, Gateway Time-out. The flag is made
authoritative in _isFileRejectedError and _shouldSkipRetryOnAccountError (both return
early when it is set) so a 5xx whose HTML body happens to contain a rejection/auth
keyword can never be mis-binned as file/account. The post-rotation retry loop also
breaks on a transient error (parity with the primary loop) to avoid burning the
retry budget re-uploading on a fallback.
- hosters: the upload POST throw sites tag err.transientNetwork when statusCode >= 500
(non-JSON body and non-2xx-with-JSON) and when the 2xx status-envelope carries
status:500; 401/403/429 stay PLAIN so they remain account errors. The server-lookup
path (apiGet/getUploadServer) is hardened symmetrically: a 5xx there tags
transientNetwork and getUploadServer preserves the flag onto its wrapped error, so
the heuristic shouldRetryServerLookup() can no longer be defeated by a 5xx body that
contains an auth keyword.
- hosters: fixed a separate real bug surfaced during investigation — _fetchByseFileList
built the wrong URL https://api.byse.sx/api/file/list (the /api/ prefix is correct
only for doodstream's doodapi.co host; byse already carries the api. subdomain).
Live-probed: /api/file/list 302-redirects to the docs page, /file/list returns 200
JSON. The wrong path made the byse async-recovery poll silently dead (always []),
removing the safety net that reclaims a large file that registered despite a 502.
Now https://api.byse.sx/file/list.
ECONNRESET was already transient and is unchanged. doodstream's 2xx empty-form stays
hosterTransient (one attempt, no re-upload). vidmoly/clouddrop use their own uploaders
and are unaffected; doodstream/voe apiKey uploads share uploadFile and correctly
benefit from the same 5xx-is-infra logic.
Deferred follow-up: poll-first-on-5xx dedup (route a 5xx through the now-working
recovery poll before retrying, to reclaim a registered-but-502 file instead of
re-uploading). It does not add new dupe risk vs the prior cascade and a pure 502
means the backend was never reached, so retry-same-account is dupe-safe for it.
Investigated and reviewed by two multi-agent workflows (4-lens investigation with a
synthesized fix spec; 4-lens adversarial review with per-finding verification): no
API drift, 0 confirmed defects. Tests: classifier units (flag-above-message-guard,
5xx-transient, 4xx-stay-account), an end-to-end 502/503/500-envelope tag through
uploadFile, and a 502-retries-same-account-no-cascade integration case. Suite 311/311.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Administrator
|
7a04060e9b |
feat(rotation): per-hoster toggle to disable the "Bekanntes Größen-Limit" pre-skip
The byse suspect size-memo (softened in v3.3.73 to arm only after two confirmed suspect rejections on the same account and only for strictly larger files) still occasionally pre-skips a primary account with "Bekanntes Größen-Limit auf diesem Account", which the user finds annoying when they would rather just let every file attempt the upload for real. Add a per-hoster opt-out: a new "Größen-Limit merken" checkbox (Accounts tab, default ON so existing behavior is preserved for everyone) that, when unchecked, disables the memo-based pre-skip entirely for that hoster. - config-store.js / upload-manager.js DEFAULT_SETTINGS: sizeMemoEnabled: true. - upload-manager.js _suspectMemoBlocks(): returns false up front when the hoster has sizeMemoEnabled === false. That single guard covers BOTH consult sites — the pre-job block (skips the guaranteed-fail upload) and the alternate-account skip — since both route through _suspectMemoBlocks. The memo is still recorded; it is simply never allowed to block. This moves the flow toward "fail open": with the toggle off, every file always gets a real attempt on its account. - The genuine suspect-reject alternates walk is gated on err.suspectReject, not the memo, so disabling the memo does not weaken real failover — a file the account actually rejects still rolls over to the next account. - renderer/app.js: checkbox data-hs="sizeMemoEnabled", checked when hs.sizeMemoEnabled !== false; persisted by the existing generic checkbox path in saveHosterSettingsFromDom(). The running manager picks it up on the next batch (fresh UploadManager) and immediately on save (updateSettings). Tests: suspect-reject-alternates gains a disabled-memo case proving the larger third file still gets a real primary attempt (key1 tried 3x, no suspect-memo-skip event); config-store gains a default-true + persist-false case. Full suite 305/305. Reviewed by a 4-lens adversarial workflow (completeness / composition / persistence-ui / conventions): 0 findings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Administrator
|
87f976faf0 |
fix(rotation): soften the byse suspect size-memo so one blip cannot poison a batch
The per-batch size memo previously armed on a SINGLE confirmed suspect rejection
and blocked every same-or-larger file on that account (fileSize >= memoSize).
A single spurious byse "Not video file format" at e.g. 1.3GB therefore pre-failed
the whole series of ~1.3GB files in that batch with "Bekanntes Größen-Limit auf
diesem Account", even when byse could actually take that size.
Now the memo:
- arms only after the 2nd confirmed rejection on the same account (count >= 2),
so a lone byse aussetzer no longer short-circuits anything; and
- blocks only STRICTLY LARGER files (fileSize > memoSize), so a same-size file
always still gets one real attempt.
_suspectSizeMemo now stores { size: smallest-rejected, count } instead of a bare
number. Still per-batch (cleared at batch start, never re-primed across batches).
Updated the two memo tests to the new semantics (added a size-aware statSync mock
so a strictly-larger file can be exercised). Full suite 284 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Administrator
|
497317dc32 |
fix(rotation): suspect byse rejections now reach every account instead of dying on the first
Root cause of "4 accounts configured, rotation only ever uses 1-3": byse returns HTTP 200 + per-file status "Not video file format" for valid MKVs above an account's size tier (proven from the live rotation log: account 4ha0 accepted 1158 files up to 2.62 GB and deterministically rejected all 6 files >= 2.81 GB with that status, each costing a full ~20 min upload). The parser tagged this fileRejected, which (a) skipped the 30s recovery poll that was originally BUILT for this misleading status (guard regression from "byse skips 30s recovery poll on explicit reject") and (b) made the upload manager skip rotation entirely - so the remaining accounts were never tried and the files failed forever. Account selection is primary-first failover, so a later account only ever runs when every earlier one is hard-failed or disabled, which matches the reported "account 4 only gets used when I disable the others". The fix, layered: - parseByseResult flags "Not video file format" as err.suspectReject (alongside fileRejected). Genuine rejections (Duplicate, too small, account-level disk-full) behave exactly as before. - uploadFile runs the byse recovery poll again for suspect rejections (the file often registers asynchronously despite the status), but skips it when the caller's file probe says the upload is genuinely not a video (new opts.probeIsVideoLike, threaded from the upload manager's probe). Poll helpers now swallow an abort during their sleep and return null so a cancelled poll surfaces the original error instead of a bare AbortError. - upload-manager: on a suspect rejection of a probe-verified video, the file gets ONE attempt on each remaining pool account (new accountPools from main.js, refreshed on save-config) - WITHOUT blacklisting the rejecting account, which keeps working for smaller files. A per-batch size memo (hoster:account -> smallest rejected size) plus a known-good account preference prevent re-uploading every following oversized file through the whole chain: the second file skips known-limited accounts outright and goes straight to the account that took the first one. Alternates that fail with a genuine account error (quota/ban/full) are marked failed in-batch so no later file burns a multi-GB upload on them; deliberately no account-failed emit there, as repointing the hoster-wide override would reroute normal-sized files away from a healthy primary. - Rotation hardening from the adversarial review: the rotated-to retry loop now fast-breaks on file-rejected/hoster-transient errors instead of burning maxAttempts full uploads, and a file-class error on a rotated account no longer blacklists that account (it fails only the file). Cancelling mid-alternates or mid-rotation now records the job as aborted instead of error (this also stops the batch webhook from firing on fully user-cancelled batches). New upload-failure rot-logs are guarded against logging user aborts as failures; rotation retries are now visible in the rotation log at all (previously failures on a rotated-to account were never logged, which is why account 2's death was invisible in the support log). - file-probe: the mpeg-ts signature required only a single leading 0x47 byte, so GIFs and "G"-prefixed text classified as video and would have qualified junk for pool-wide alternate uploads; it now requires TS sync-byte periodicity (0x47 at offsets 0/188/376) and the probe head read grew from 64 to 512 bytes to make that check possible. GIF gets its own non-video signature. Tests: 9 new/reworked across suspect-reject-alternates.test.js (alternate walk, no blacklist, memo short-circuit, account-error marking, abort labeling, probe gating) and byse-reject-recovery.test.js (suspect polls + recovers, empty poll throws suspectReject, Duplicate still fast-fails, probe-confirmed non-video skips the poll) plus a TS/GIF probe regression test. Full suite 276 green. |