Administrator/Multi-Hoster-Upload
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 262 Wiki Activity
f7c8d308fc
Multi-Hoster-Upload/tasks/todo.md
Administrator eeec1d150c fix(queue): completed files no longer reappear in the queue after restart 
Closing the app (especially during an active upload or on a hard kill) and
reopening sometimes left already-uploaded files sitting in the queue as if
still pending. Root cause is three layers stacked:

1. Persist-starvation: persistQueueStateSoon() reset a 10s debounce on every
   progress event, so during an upload the on-disk queue snapshot was never
   rewritten and stayed frozen at the pre-upload state (all jobs "preview").
2. The beforeunload sync flush only covers a clean close; a hard kill / crash
   leaves that stale snapshot on disk.
3. Startup auto-dedup only dropped jobs with status "done". The completed
   files were stored as "preview" in the stale snapshot, so they survived and
   reappeared.

Fix (mechanism-independent — holds whether the stale snapshot came from
starvation, a mid-upload close race, or a hard kill):

FIX A (core, durable): the upload log is the source of truth. Each persisted
snapshot is now stamped with savedAt; on restart any restored job whose newest
matching log entry is timestamped at/after floor(savedAt) is dropped regardless
of status — it provably completed after the snapshot, so a "preview" row for it
is a ghost. lib/queue-dedup.js gains an additive 3rd savedAt param; without
savedAt or without log timestamps it behaves exactly as before (the 5 canary
tests stay green, so intentional re-uploads of older files still survive).

FIX B: new lib/throttle-timer.js with a max-wait. During uploads the snapshot
is now written at most ~20s into a continuous progress burst instead of never;
idle stays a pure debounce. The fallback shim honors max-wait too, so a missing
library can never silently reintroduce the starvation.

FIX C: the synchronous close-write retries renameSync on EBUSY/EPERM/EACCES and
uses a pid-unique tmp (de-conflicts it from config-store._atomicWrite's fixed
.tmp). A startup sweep reclaims orphaned <config>.<pid>.tmp files left by a hard
kill between write and rename.

lib/upload-log.js extracts formatUploadLogLine + parseUploadLogLine from main.js
so the real writer -> reader -> gate seam is unit-tested (a future log-format or
epoch-basis change can no longer pass green while breaking the fix).

Verified: 334/334 tests green (incl. throttle fake-clock starvation/maxWait,
ts-gate multi-hoster partial-completion, and the real-format seam tests), ESLint
clean, smoke-boot identical to baseline, and an adversarial multi-agent review
(15 findings, 14 refuted, 1 low — the tmp orphan, now swept) on the diff.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 04:35:27 +02:00

59 lines
3.8 KiB
Markdown
Raw Blame History

# Queue-Persistenz Bug: fertige Dateien tauchen nach Neustart wieder auf
## Symptom
User: 300 Dateien, 100 übrig, Programm schließen + öffnen → manchmal sind bereits
fertig hochgeladene Dateien wieder in der Liste.
## Root Cause (verifiziert im Code)
- **RC-1 (Persist-Starvation, code-confirmed):** `persistQueueStateSoon()` setzt bei
jedem Progress-Event den Timer per `clearTimeout` zurück; Delay während Upload war
10000ms. Progress-Events feuern öfter als alle 10s → Timer feuert NIE während eines
aktiven Uploads. Der Disk-Snapshot bleibt auf dem Stand VOR Upload-Start stehen
(alle Jobs `preview`).
- **RC-2 (unzuverlässiger Close-Flush):** beforeunload-Sync-Flush existiert
(app.js:4605) und fängt den sauberen Close ab. Bei hartem Kill / Crash / OS-Kill
läuft er nicht → der stale Snapshot bleibt liegen.
- **RC-3 (Dedup-Asymmetrie):** `_autoDeduplicateFromLog` droppt beim Start nur Jobs
mit Status `done`. Die Ghosts aus dem stale Snapshot stehen aber als `preview` da
→ werden NICHT gedroppt → fertige Dateien erscheinen erneut.
## Fix (mechanismus-unabhängig, vom Kern auf)
- [x] **FIX A — Timestamp-gated Dedup (Kern-Fix, durable):** Beim Start jeden restored
Job droppen, dessen file+hoster im Log mit `ts >= floor(savedAt)` steht — egal ob
`preview` oder `done`. Fängt Ghosts auch nach hartem Kill (hängt vom Log ab, nicht
vom Snapshot). `lib/queue-dedup.js` additiver 3. Param `savedAt`; `buildPersistedQueueState`
stempelt `savedAt`; `restoreQueueStateFromConfig` merkt `_restoredSnapshotSavedAt`;
Log-Zeile → `ts` geparst; `_autoDeduplicateFromLog` reicht savedAt durch.
- [x] **FIX B — Throttle mit max-wait:** `lib/throttle-timer.js` (neu). Upload: delay 500
+ maxWait 20000 → Snapshot alle ~20s statt nie. Idle: reine Debounce. Fallback-Shim
honoriert maxWait (kein stilles Starvation-Reintro).
- [x] **FIX C — Close-Write-Härtung:** `save-global-settings-sync` renameSync-Retry bei
EBUSY/EPERM/EACCES + pid-unique tmp + tmp-cleanup. Startup-Sweep `_sweepOrphanConfigTmps`
räumt verwaiste `<config>.<pid>.tmp` toter PIDs (gegen Orphan-Akkumulation).
- [x] **Seam-Extraktion (Advisor #2):** `lib/upload-log.js` (neu) — `formatUploadLogLine`
+ `parseUploadLogLine` aus main.js gezogen; Test fährt den ECHTEN Writer→Reader→Gate-
Vertrag (kein Mirror) → fängt künftige Format-/Epoch-Brüche.
## Tests
- [x] `tests/throttle-timer.test.js`: Starvation ohne maxWait → 0 Fires; mit maxWait →
periodische Fires; last-write-wins (distinct fn); flushSync/cancel.
- [x] `tests/queue-dedup.test.js`: ts>=savedAt→DROP; ts<savedAt→KEEP; same-second→DROP;
max-ts; Multi-Hoster Teilabschluss (reale Bug-Form); ohne savedAt/ohne ts→Legacy.
- [x] `tests/upload-log.test.js`: realer Writer→Reader-Roundtrip + Seam-Drop/Keep.
- [x] 334/334 grün, ESLint clean, Smoke-Boot identisch zu Baseline (kein Regress).
## Review
- **Adversariale Multi-Agent-Review (4 Dimensionen, 15 Findings):** 14 refuted (meist
"ist korrekt"-Bestätigungen, Kommentar-Drift, Test-Härtungs-Vorschläge). 1 confirmed
(LOW): pid-unique tmp konnte bei Hard-Kill zwischen write und rename verwaisen → mit
Startup-Sweep behoben. Stale-Kommentare (queue-dedup Header + _autoDeduplicateFromLog)
auf die Zwei-Regel-Logik korrigiert.
- **Was bewiesen ist:** Komponenten-Logik (Unit-Tests inkl. realer Format-Seam),
Code-getraceter Wiring-Pfad, adversariale Gegenprüfung. Der Fix ist
MECHANISMUS-UNABHÄNGIG: greift egal ob der stale Snapshot von Starvation, einem
Mid-Upload-Close-Race ODER einem Hard-Kill kommt.
- **Ehrliche Einschränkung:** KEIN Live-Repro mit echtem byse-Key (Key unter anderem
Windows-Profil verschlüsselt, nicht entschlüsselbar). Symptom tritt nur auf bei
Close WÄHREND aktivem Upload oder Hard-Kill — ein sauberer Idle-Close war schon
vorher korrekt.
Reference in New Issue View Git Blame Copy Permalink