perf(queue): coalesce removeFromQueueOnDone removals into one filter pass
handleProgress on a 'done' event with removeFromQueueOnDone=true was calling queueJobs.filter() once per event. With 500 parallel jobs all finishing at roughly the same time, that's 500 × O(N) = O(N²) work synchronously on the IPC handler thread — visible as a brief UI freeze when a big batch completes. Coalesce into one microtask: removeJobFromIndex + selection cleanup stay synchronous (so subsequent lookups see the right state), but the array rewrite is deferred to a single filter against a Set of all ids that came in this tick. JS microtask runs after the sync IPC batch, so within one batch-of-events we get one filter pass instead of N. beforeunload drains the pending set synchronously before persisting so removeFromQueueOnDone=true users don't see jobs reappear after restart that they expected to be gone.
This commit is contained in:
parent
4af89d7aa3
commit
38ecc6a4cb
@ -47,6 +47,10 @@ const _sessionTrackedJobs = new Set(); // Job IDs already counted for totalBytes
|
|||||||
const _sessionDoneJobs = new Set(); // Job IDs already counted for uploadedBytes
|
const _sessionDoneJobs = new Set(); // Job IDs already counted for uploadedBytes
|
||||||
const _completedUploadKeys = new Set(); // 'filepath|hoster' keys for done uploads (survives removeFromQueueOnDone)
|
const _completedUploadKeys = new Set(); // 'filepath|hoster' keys for done uploads (survives removeFromQueueOnDone)
|
||||||
const _deletedJobIds = new Set(); // IDs of jobs explicitly deleted by user (prevents re-creation from stale progress callbacks)
|
const _deletedJobIds = new Set(); // IDs of jobs explicitly deleted by user (prevents re-creation from stale progress callbacks)
|
||||||
|
// Coalesce removeFromQueueOnDone removals into one filter pass per microtask
|
||||||
|
// to avoid O(N²) behaviour when a burst of jobs finish at once.
|
||||||
|
let _pendingDoneRemovalIds = new Set();
|
||||||
|
let _doneRemovalScheduled = false;
|
||||||
const queueSortState = { key: 'filename', direction: 'asc' };
|
const queueSortState = { key: 'filename', direction: 'asc' };
|
||||||
|
|
||||||
// History state
|
// History state
|
||||||
@ -1925,11 +1929,26 @@ function handleProgress(data) {
|
|||||||
_completedUploadKeys.add(`${job.file}|${job.hoster}`);
|
_completedUploadKeys.add(`${job.file}|${job.hoster}`);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Remove finished jobs from queue immediately if setting is enabled
|
// Remove finished jobs from queue if setting is enabled. Coalesce the
|
||||||
|
// actual array filter into one microtask: a burst of 500 done events
|
||||||
|
// would otherwise fire 500 individual O(N) filters = O(N²) work, visible
|
||||||
|
// as a brief UI freeze when a big batch finishes. Index/selection are
|
||||||
|
// updated synchronously so subsequent lookups see the right state — only
|
||||||
|
// the array rewrite is deferred.
|
||||||
if (job.status === 'done' && config.globalSettings && config.globalSettings.removeFromQueueOnDone) {
|
if (job.status === 'done' && config.globalSettings && config.globalSettings.removeFromQueueOnDone) {
|
||||||
removeJobFromIndex(job);
|
removeJobFromIndex(job);
|
||||||
selectedJobIds.delete(job.id);
|
selectedJobIds.delete(job.id);
|
||||||
queueJobs = queueJobs.filter(j => j !== job);
|
_pendingDoneRemovalIds.add(job.id);
|
||||||
|
if (!_doneRemovalScheduled) {
|
||||||
|
_doneRemovalScheduled = true;
|
||||||
|
queueMicrotask(() => {
|
||||||
|
_doneRemovalScheduled = false;
|
||||||
|
if (_pendingDoneRemovalIds.size === 0) return;
|
||||||
|
const drop = _pendingDoneRemovalIds;
|
||||||
|
_pendingDoneRemovalIds = new Set();
|
||||||
|
queueJobs = queueJobs.filter(j => !drop.has(j.id));
|
||||||
|
});
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Status changes (done/error/etc) get one coalesced update per frame so a
|
// Status changes (done/error/etc) get one coalesced update per frame so a
|
||||||
@ -3683,6 +3702,14 @@ window.addEventListener('beforeunload', () => {
|
|||||||
}
|
}
|
||||||
clearTimeout(queuePersistTimer);
|
clearTimeout(queuePersistTimer);
|
||||||
queuePersistTimer = null;
|
queuePersistTimer = null;
|
||||||
|
// Drain pending done-removals synchronously before persisting so jobs the
|
||||||
|
// user expected to disappear (removeFromQueueOnDone=true) don't reappear
|
||||||
|
// on next launch. Microtask wouldn't run before the sync IPC below.
|
||||||
|
if (_pendingDoneRemovalIds.size > 0) {
|
||||||
|
const drop = _pendingDoneRemovalIds;
|
||||||
|
_pendingDoneRemovalIds = new Set();
|
||||||
|
queueJobs = queueJobs.filter(j => !drop.has(j.id));
|
||||||
|
}
|
||||||
const globalSettings = {
|
const globalSettings = {
|
||||||
...(config.globalSettings || {}),
|
...(config.globalSettings || {}),
|
||||||
pendingQueue: buildPersistedQueueState()
|
pendingQueue: buildPersistedQueueState()
|
||||||
|
|||||||
@ -1,32 +1,25 @@
|
|||||||
# 3.3.0 — Performance + Log-Recovery
|
# Verbesserungs-Loop — open items
|
||||||
|
|
||||||
## Probleme (User-Bericht)
|
## Released
|
||||||
1. Lag beim Tab-Wechsel + Scrollen nach langer Session.
|
- ✅ 3.3.0 — Performance-Fixes (queue-cap, sort-throttle, history-delegation, recent-cap) + Log-Recovery
|
||||||
2. File-Uploader-Log: bei einem von zwei Hostern gar keine Log-Einträge — möglicherweise nach Datei/Verzeichnis-Löschung kein Recovery.
|
|
||||||
|
|
||||||
## Root Causes (aus Audit)
|
## Open items (priorisiert)
|
||||||
|
|
||||||
|
### Stabilität
|
||||||
|
- [ ] **removeFromQueueOnDone O(N²)** in `handleProgress` (renderer/app.js) — bei 500 gleichzeitig fertig werdenden Jobs 500× ein O(N)-`queueJobs.filter()` aufgerufen. Coalesce über microtask + Set.
|
||||||
|
- [ ] **fileuploader.log wächst unbounded** — automatische Rotation bei >50MB (rename → .1, neue Datei).
|
||||||
|
- [ ] **`_jobLogCollector`** (main.js) — wird nur bei start-upload geleert, nicht bei batch-done. Bei vielen Batches ohne neuen start-upload wächst es. Cleanup bei batch-done für jobs die nicht mehr in queueJobs sind.
|
||||||
|
|
||||||
### Performance
|
### Performance
|
||||||
- **queueJobs wächst unbounded** (Default `removeFromQueueOnDone=false`). Sortierung + Stats + In-place-Render skalieren O(N) und O(N log N) je render. Nach ~5000 Jobs spürbar.
|
- [ ] **`applyQueueSelectionClasses`** (renderer/app.js:891) — `tbody.querySelectorAll` bei jedem Klick. Bei 5000-Jobs-Queue O(N) per click. Cache last rendered range.
|
||||||
- **`sortQueueJobs` cached nur für static keys** (filename, host). Bei status/speed/progress: voller Sort jedes Mal.
|
|
||||||
- **`renderHistoryTable`** bindet per-row click listener + voller `innerHTML`-Rebuild bei jedem Tab-Switch.
|
|
||||||
- **`renderRecentUploadsPanel`** baut komplettes innerHTML bei sort-change neu, `sessionFilesData` Array ungecapt.
|
|
||||||
|
|
||||||
### Log-Bug
|
### Code-Qualität
|
||||||
- `_flushUploadLog` clear buffer **vor** appendFile. Bei `ENOENT` (Datei oder Dir wurde gelöscht) → silent log + buffer verloren.
|
- [ ] **Test-Coverage für 3.3.0** — keine Tests für die queue-cap-prune-Logik in handleBatchDone, sortQueueJobs dynamic-throttle, log-error-recovery.
|
||||||
- `_resolveUploadLogTarget` cached den Path. Bei mid-session Verzeichnis-Löschung wird der cached target wiederverwendet → permanent ENOENT bis Neustart.
|
|
||||||
|
|
||||||
## Plan
|
### UX-Politur
|
||||||
- [ ] Log: bei error in `appendFile` → cache invalidieren + buffer prepend + retry
|
- [ ] **CSS `.queue-row` transition** auf `:hover` scopen (aktuell auf jedem row → unnötiger Repaint bei Status-Flip).
|
||||||
- [ ] Log: vor jedem flush mkdirSync (idempotent, recreated deleted dir)
|
- [ ] Module-level Sets `_sessionTrackedJobs`/`_sessionDoneJobs`/`_completedUploadKeys` werden nie geleert — minor memory growth.
|
||||||
- [ ] queueJobs: auto-prune nach `handleBatchDone` für älteste 'done' jobs jenseits Cap (z.B. 500)
|
|
||||||
- [ ] sortQueueJobs: für dynamic keys → coalesce auf 1× pro UI_UPDATE_INTERVAL (200ms)
|
|
||||||
- [ ] renderHistoryTable: delegated click-listener auf tbody (nicht per-row), short-circuit wenn `_historyDirty=false`
|
|
||||||
- [ ] renderRecentUploadsPanel: `sessionFilesData` auf letzten 2000 Einträge cappen, `_sessionFileKeys` parallel prunen
|
|
||||||
- [ ] removeFromQueueOnDone: O(N²) → splice via `_jobIndexById` (oder defer auf batch-done)
|
|
||||||
- [ ] Tests: Log-recovery, queue-prune, sort-throttle
|
|
||||||
- [ ] Release als 3.3.0
|
|
||||||
|
|
||||||
## Skip (Low Impact / kosmetisch)
|
## Loop-Notes
|
||||||
- CSS transition fix
|
- Cron-Job `01e33ae1` läuft alle 30min (:07/:37), Session-only.
|
||||||
- Module-level Sets clearing (genug Memory bisher)
|
- Pro Iteration: GENAU EIN Issue. Auto-Release bei grünen Tests. Boundary: keine Features, keine Major-Refactors.
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user