Multi-Hoster-Upload

Administrator/Multi-Hoster-Upload

Fork 0

Commit Graph

Author	SHA1	Message	Date
Administrator	6b0c515a9b	fix(queue): close two lost-work / sticky-ghost edge cases found by intensive testing Follow-up hardening on the v3.3.80 queue-persistence fix after a deeper adversarial sweep + reviewer pass over the WHOLE subsystem (not just the diff). Two real defects: 1. Basename-collision lost work (regression introduced by v3.3.80 FIX A). restoreQueueStateFromConfig collapses jobs on the FULL path while the ts-gate keys on basename\|hoster. Two genuinely different files with the same basename queued to the same hoster from different folders therefore share a gate key: if one was logged after savedAt, the ts-rule dropped BOTH — silently losing the still-pending one. Pre-FIX-A only 'done' jobs were dropped, so a pending file was never at risk. Fix: an ambiguity guard in partitionRestoredJobsByLog — the ts-rule is suppressed when a basename\|hoster key maps to more than one distinct file path (the log records only basenames, so it can't say which physical file completed). The done-in-log rule is unchanged. Fails safe: worst case a visible ghost survives, never silent data loss. 2. selectedFiles re-materialization with removeFromQueueOnDone=ON (second mechanism, independent of the stale snapshot). When that setting is on, a completed job is stripped from queueJobs but its path stays in selectedFiles (syncSelectedFilesFromQueue only runs at batch-done, never on a mid-upload close). On restart the ts-gate operates on queueJobs and never sees it, then the startup updateUploadView -> buildQueuePreview re-creates it as a preview ghost AFTER the gate ran, and it re-persists with a fresh savedAt — sticky. Fix: completedSelectionKeys() seeds _completedUploadKeys (the set buildQueuePreview already consults) from the log at startup, keyed on full path, with the same ambiguity guard. Log-based so it survives a hard kill, consistent with FIX A. Also extracts the orphan-tmp sweep decision into lib/orphan-tmp.js (was untested inline code in main.js; behavior-preserving) and adds executable coverage for the paths that were previously only argued from logic: orphan-tmp sweep, config-store pendingQueue+savedAt round-trip, an end-to-end scenario in the exact user-reported shape (300 queued / ~200 finished mid-session), and a 3000+500-iteration property fuzz of the gate invariant including the lost-work guarantee. 359/359 green, ESLint clean, smoke-boot unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-19 06:51:48 +02:00
Administrator	eeec1d150c	fix(queue): completed files no longer reappear in the queue after restart Closing the app (especially during an active upload or on a hard kill) and reopening sometimes left already-uploaded files sitting in the queue as if still pending. Root cause is three layers stacked: 1. Persist-starvation: persistQueueStateSoon() reset a 10s debounce on every progress event, so during an upload the on-disk queue snapshot was never rewritten and stayed frozen at the pre-upload state (all jobs "preview"). 2. The beforeunload sync flush only covers a clean close; a hard kill / crash leaves that stale snapshot on disk. 3. Startup auto-dedup only dropped jobs with status "done". The completed files were stored as "preview" in the stale snapshot, so they survived and reappeared. Fix (mechanism-independent — holds whether the stale snapshot came from starvation, a mid-upload close race, or a hard kill): FIX A (core, durable): the upload log is the source of truth. Each persisted snapshot is now stamped with savedAt; on restart any restored job whose newest matching log entry is timestamped at/after floor(savedAt) is dropped regardless of status — it provably completed after the snapshot, so a "preview" row for it is a ghost. lib/queue-dedup.js gains an additive 3rd savedAt param; without savedAt or without log timestamps it behaves exactly as before (the 5 canary tests stay green, so intentional re-uploads of older files still survive). FIX B: new lib/throttle-timer.js with a max-wait. During uploads the snapshot is now written at most ~20s into a continuous progress burst instead of never; idle stays a pure debounce. The fallback shim honors max-wait too, so a missing library can never silently reintroduce the starvation. FIX C: the synchronous close-write retries renameSync on EBUSY/EPERM/EACCES and uses a pid-unique tmp (de-conflicts it from config-store._atomicWrite's fixed .tmp). A startup sweep reclaims orphaned <config>.<pid>.tmp files left by a hard kill between write and rename. lib/upload-log.js extracts formatUploadLogLine + parseUploadLogLine from main.js so the real writer -> reader -> gate seam is unit-tested (a future log-format or epoch-basis change can no longer pass green while breaking the fix). Verified: 334/334 tests green (incl. throttle fake-clock starvation/maxWait, ts-gate multi-hoster partial-completion, and the real-format seam tests), ESLint clean, smoke-boot identical to baseline, and an adversarial multi-agent review (15 findings, 14 refuted, 1 low — the tmp orphan, now swept) on the diff. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-19 04:35:27 +02:00
Administrator	af51bebaf7	fix(queue): stop auto-dedup from deleting pending jobs on restart/update Reproduced from a real saved config: pendingQueue held 4 'preview' jobs (one file across 4 hosters); the queue saved + restored correctly. But _autoDeduplicateFromLog (runs at init after restore) removed jobs whose fileName\|hoster appeared ANYWHERE in the lifetime fileuploader.log, regardless of status — so all 4 pending previews were deleted and the queue showed the empty "Dateien hierhin ziehen" state. Looked update-specific only because the server restarts on update; a plain restart did the same. - New lib/queue-dedup.js (pure, dual CJS/window export like queue-prune.js): partitionRestoredJobsByLog drops ONLY 'done' jobs that match the log. Pending (preview/queued) and failed (error/aborted) jobs always survive — they're intentional queued work (often a deliberate re-upload of a previously uploaded file). Manual importUploadLog stays separate/explicit. - renderer wires it in; index.html loads the module before app.js. - Tests: 5 cases incl. the exact reproduced scenario (4 previews all in log -> 0 removed). Full suite 162/162. Verified against the user's real electron-config.json + fileuploader.log: old logic removed 4/4 (empty queue), new logic removes 0/4 (queue preserved). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-25 01:08:59 +02:00

Author

SHA1

Message

Date

Administrator

6b0c515a9b

fix(queue): close two lost-work / sticky-ghost edge cases found by intensive testing

Follow-up hardening on the v3.3.80 queue-persistence fix after a deeper adversarial
sweep + reviewer pass over the WHOLE subsystem (not just the diff). Two real defects:

1. Basename-collision lost work (regression introduced by v3.3.80 FIX A).
restoreQueueStateFromConfig collapses jobs on the FULL path while the ts-gate keys
on basename|hoster. Two genuinely different files with the same basename queued to
the same hoster from different folders therefore share a gate key: if one was logged
after savedAt, the ts-rule dropped BOTH — silently losing the still-pending one.
Pre-FIX-A only 'done' jobs were dropped, so a pending file was never at risk.
Fix: an ambiguity guard in partitionRestoredJobsByLog — the ts-rule is suppressed
when a basename|hoster key maps to more than one distinct file path (the log records
only basenames, so it can't say which physical file completed). The done-in-log rule
is unchanged. Fails safe: worst case a visible ghost survives, never silent data loss.

2. selectedFiles re-materialization with removeFromQueueOnDone=ON (second mechanism,
independent of the stale snapshot). When that setting is on, a completed job is
stripped from queueJobs but its path stays in selectedFiles (syncSelectedFilesFromQueue
only runs at batch-done, never on a mid-upload close). On restart the ts-gate operates
on queueJobs and never sees it, then the startup updateUploadView -> buildQueuePreview
re-creates it as a preview ghost AFTER the gate ran, and it re-persists with a fresh
savedAt — sticky. Fix: completedSelectionKeys() seeds _completedUploadKeys (the set
buildQueuePreview already consults) from the log at startup, keyed on full path, with
the same ambiguity guard. Log-based so it survives a hard kill, consistent with FIX A.

Also extracts the orphan-tmp sweep decision into lib/orphan-tmp.js (was untested inline
code in main.js; behavior-preserving) and adds executable coverage for the paths that
were previously only argued from logic: orphan-tmp sweep, config-store pendingQueue+savedAt
round-trip, an end-to-end scenario in the exact user-reported shape (300 queued / ~200
finished mid-session), and a 3000+500-iteration property fuzz of the gate invariant
including the lost-work guarantee. 359/359 green, ESLint clean, smoke-boot unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-19 06:51:48 +02:00

Administrator

eeec1d150c

fix(queue): completed files no longer reappear in the queue after restart

Closing the app (especially during an active upload or on a hard kill) and
reopening sometimes left already-uploaded files sitting in the queue as if
still pending. Root cause is three layers stacked:

1. Persist-starvation: persistQueueStateSoon() reset a 10s debounce on every
   progress event, so during an upload the on-disk queue snapshot was never
   rewritten and stayed frozen at the pre-upload state (all jobs "preview").
2. The beforeunload sync flush only covers a clean close; a hard kill / crash
   leaves that stale snapshot on disk.
3. Startup auto-dedup only dropped jobs with status "done". The completed
   files were stored as "preview" in the stale snapshot, so they survived and
   reappeared.

Fix (mechanism-independent — holds whether the stale snapshot came from
starvation, a mid-upload close race, or a hard kill):

FIX A (core, durable): the upload log is the source of truth. Each persisted
snapshot is now stamped with savedAt; on restart any restored job whose newest
matching log entry is timestamped at/after floor(savedAt) is dropped regardless
of status — it provably completed after the snapshot, so a "preview" row for it
is a ghost. lib/queue-dedup.js gains an additive 3rd savedAt param; without
savedAt or without log timestamps it behaves exactly as before (the 5 canary
tests stay green, so intentional re-uploads of older files still survive).

FIX B: new lib/throttle-timer.js with a max-wait. During uploads the snapshot
is now written at most ~20s into a continuous progress burst instead of never;
idle stays a pure debounce. The fallback shim honors max-wait too, so a missing
library can never silently reintroduce the starvation.

FIX C: the synchronous close-write retries renameSync on EBUSY/EPERM/EACCES and
uses a pid-unique tmp (de-conflicts it from config-store._atomicWrite's fixed
.tmp). A startup sweep reclaims orphaned <config>.<pid>.tmp files left by a hard
kill between write and rename.

lib/upload-log.js extracts formatUploadLogLine + parseUploadLogLine from main.js
so the real writer -> reader -> gate seam is unit-tested (a future log-format or
epoch-basis change can no longer pass green while breaking the fix).

Verified: 334/334 tests green (incl. throttle fake-clock starvation/maxWait,
ts-gate multi-hoster partial-completion, and the real-format seam tests), ESLint
clean, smoke-boot identical to baseline, and an adversarial multi-agent review
(15 findings, 14 refuted, 1 low — the tmp orphan, now swept) on the diff.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-19 04:35:27 +02:00

Administrator

af51bebaf7

fix(queue): stop auto-dedup from deleting pending jobs on restart/update

Reproduced from a real saved config: pendingQueue held 4 'preview' jobs (one
file across 4 hosters); the queue saved + restored correctly. But
_autoDeduplicateFromLog (runs at init after restore) removed jobs whose
fileName|hoster appeared ANYWHERE in the lifetime fileuploader.log, regardless
of status — so all 4 pending previews were deleted and the queue showed the
empty "Dateien hierhin ziehen" state. Looked update-specific only because the
server restarts on update; a plain restart did the same.

- New lib/queue-dedup.js (pure, dual CJS/window export like queue-prune.js):
  partitionRestoredJobsByLog drops ONLY 'done' jobs that match the log. Pending
  (preview/queued) and failed (error/aborted) jobs always survive — they're
  intentional queued work (often a deliberate re-upload of a previously
  uploaded file). Manual importUploadLog stays separate/explicit.
- renderer wires it in; index.html loads the module before app.js.
- Tests: 5 cases incl. the exact reproduced scenario (4 previews all in log ->
  0 removed). Full suite 162/162.

Verified against the user's real electron-config.json + fileuploader.log: old
logic removed 4/4 (empty queue), new logic removes 0/4 (queue preserved).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

2026-05-25 01:08:59 +02:00

3 Commits