Compare commits

..

No commits in common. "6dcc98f52de73c2409ecffefb3e699f3983e7a87" and "c37c8e3906951bc1ef7be559d5b47d16207b2855" have entirely different histories.

5 changed files with 49 additions and 59 deletions

View File

@ -27,11 +27,7 @@ function _doodstreamLogPath() {
return path.join(__dirname, '..', 'doodstream-debug.log');
}
let _debugVerbose = false;
function setDebugVerbose(v) { _debugVerbose = !!v; }
function _debugLog(msg) {
if (!_debugVerbose) return;
try {
const logPath = _doodstreamLogPath();
maybeRotateLogFile(logPath, _DOODSTREAM_LOG_MAX_BYTES, _DOODSTREAM_LOG_MAX_BACKUPS);
@ -702,4 +698,3 @@ class DoodstreamUploader {
}
module.exports = DoodstreamUploader;
module.exports.setDebugVerbose = setDebugVerbose;

View File

@ -139,7 +139,7 @@ function debugLog(msg) {
}
let _logVerbose = false;
function setLogVerbose(v) { _logVerbose = !!v; try { require('./lib/doodstream-upload').setDebugVerbose(_logVerbose); } catch {} }
function setLogVerbose(v) { _logVerbose = !!v; }
function _ctxTag(ctx) {
if (!ctx || typeof ctx !== 'object') return '';
const tags = [];

View File

@ -1,6 +1,6 @@
{
"name": "multi-hoster-uploader",
"version": "3.3.88",
"version": "3.3.87",
"description": "Upload files to doodstream, voe, vidmoly, byse simultaneously",
"main": "main.js",
"scripts": {

View File

@ -139,10 +139,3 @@
**Regel 3 — Multi-Agent-Findings gegen primäre Evidenz prüfen (Control-Char-Falsch­positiv):** Der Hunt meldete HIGH-ish einen „_sessionFileKeys delete-key separator mismatch". Beim Versuch ihn zu fixen matchte der Edit-`old_string` NICHT. Char-Code-Dump (`HAS_U0001: True`) zeigte: die Zeile hat ECHTE U+0001-Zeichen — die Read-Tools der Verifier-Agenten rendern Steuerzeichen unsichtbar, sie schlossen fälschlich „keine Separatoren". KEIN Bug. **Wenn ein Fix-`old_string` nicht matcht obwohl Grep ihn zeigt: Char-Codes dumpen, bevor man dem Tool misstraut — die Quelle kann unsichtbar von der Read-Anzeige abweichen.**
**Regel 4 — Den negligible-aber-realen Befund mit Zahl ABLEHNEN, nicht aus dem Bauch:** queueJobs O(N)-Scan pro Render (wächst unbounded, da removeFromQueueOnDone=false UND Folder-Monitor EINEN Batch via addJobs am Leben hält → 500-Cap-Prune feuert nie) — real, aber Blink-gemessen <0,1 ms bei 3000 Jobs. Den riskanten Inkremental-Counter-Refactor mit DIESER Zahl skippen, nicht mit fühlt sich klein an".
**Wie anwenden:** Append-only-Optimierungen, die auf Längenwachstum gaten, brechen still an JEDEM Cap (push-then-slice fixiert die Länge) — stattdessen die Anzahl NEUER Items zählen und am Boden evicten. „Mach es wie die Queue-Tabelle (virtualisieren)" war hier NICHT nötig: die Messung zeigte stehende 2000 Zeilen kosten median 0,4 ms; nur der Rebuild war teuer. Simplest-Fix der die gemessene Ursache trifft schlägt die größere Architektur-Änderung.
## 2026-06-21 — "Audit JEDE zeile" = audit + measure + risk-appropriate DEFER, nicht fix-everything (v3.3.88)
**Kontext:** Nach dem v3.3.87-Lag-Fix Folge-Goal: „schau dir wirklich JEDE zeile an die du geschrieben hast und schau ob es solche probleme gibt o. geben könnte". 18-Agenten-Audit + Eigen-Review jeder Hot-Path-Zeile + Blink-Benchmarks.
**Befund:** Der Audit fand, dass MEIN eigener T1-Fix (config-store cache, 29d1944) eine latente Regression einführte: `load()` macht ein unconditionales `structuredClone` der GANZEN config (inkl. unbounded history) pro Call → write-interleaved loads 2,22,4× LANGSAMER als das alte read+parse (gemessen @8000 Batches: 9,65 ms → 22,96 ms). Skaliert mit historySize. ABER: der echte User hat 8 Batches / 4,8 KB → Mikrosekunden. Negligible.
**Die Falle (Advisor hat geblockt):** Ich wollte es „elegant" fixen mit `history.slice()` (shallow) statt deep-clone. Advisor: STOPP. `load()` ist der gefährlichste Code im Repo (config + credentials; Korruption = Datenverlust), ich war hier schon mal von Cache-Semantik gebissen worden. Und: der Perf-Win und das Risiko sind DIESELBE Münze — der Speedup kommt NUR vom Sharing der Batch-Objekte by-reference, und genau dieses Sharing IST die Silent-Cache-Corruption-Gefahr (hängt an einem globalen Invariant „nichts deep-mutated je eine history-Batch" den ich über zukünftigen Code + jeden getHistory-Consumer nicht erzwingen kann). Es gibt KEINE sichere Version dieses Ansatzes → falsches Werkzeug für safety-kritischen Code. Hardcoded 5 keys in `_cloneConfig` wäre ein zweiter Footgun (zukünftiger top-level key verschwindet still aus jedem load()).
**Regel:** „Audit jede Zeile" heißt JEDE Zeile ANSCHAUEN + die Magnitude MESSEN + eine risiko-angemessene Entscheidung treffen — NICHT jeden geflaggten Befund fixen. Bei einem Audit-Goal ist „ich habe jede Zeile geprüft, jeden Befund als sub-ms bei realistischer History gemessen, den Mechanismus bestätigt aber den Fix als riskante Persistenz-Chirurgie für einen latenten Mikro-Cost eingestuft, also dokumentiere ich ihn statt ihn zu shippen" die VOLLSTÄNDIGE, gründliche Antwort. Jeden geflaggten Punkt unabhängig vom Risiko zu fixen ist keine Gründlichkeit — so wird aus einer Lag-Fix-Session ein Datenverlust-Incident. Nur den EINEN Befund shippen der im echten Szenario beißt (doodstream `_debugLog`: sync statSync+appendFileSync ~815×/Upload auf dem Main-Loop während des Uploads → hinter `logVerbose` gaten, default off, near-zero risk). Den Rest als bewusste Defers mit Messzahlen dokumentieren.
**Wie anwenden:** Wenn ein Goal („JEDE!! JEDE!!!") + ein Stop-Hook Druck erzeugen, immer weiterzuschneiden: das ist genau der Moment, den Advisor VOR dem Edit zu rufen. Magnitude am ECHTEN Artefakt prüfen (der User-Config, nicht @8000-Batches-Hypothese). Persistenz-/Credential-Code nur anfassen wenn der Fix risiko-frei UND der Gewinn real-spürbar ist — sonst dokumentieren und stoppen.

View File

@ -1,50 +1,52 @@
# Lag audit of every line written this session (Goal: "schau dir JEDE zeile an … ob es solche probleme gibt o. geben könnte")
# Long-running lag — root cause + fix (symptom: UI laggy over time, CPU 40%/RAM 6GB both normal)
Method: self-review of every hot-path line I changed (config-store T1, main.js T3, renderer append-evict,
diagnostics) + an 18-agent adversarial line-by-line audit (each finding double-verified for real + causes-
perceptible-lag) + Blink microbenchmarks. Reference config = THIS user's real one: 8 batches / 4.8 KB
history / 52 KB total — measured, not assumed.
## What "laggy after time, low CPU, stable RAM" actually was (MEASURED, not assumed)
First instinct (main-process config I/O scaling with history) was WRONG for this user: the
real config is tiny (history 23 rows / 4.8 KB, total 52 KB, queue ~153 jobs). Measured with a
one-off node read of the live electron-config.json. So serialize-cost theories were dead on arrival.
## Conclusion: v3.3.87 was the fix. The rest is a clean bill of health, not a to-do list.
The reported "laggy after long runtime" was the recent-panel rebuild cliff — found, fixed (append-evict,
80 ms → 7.4 ms Blink-verified), released v3.3.87. The audit surfaced NO second cause that affects this user.
A 13→18-agent leak hunt + adversarial verify + a Playwright/Blink microbenchmark found the
真 cause:
## SHIPPED this round (v3.3.88) — the one finding that bites in the real upload scenario
- doodstream-upload.js `_debugLog`: ran ungated SYNCHRONOUS statSync + appendFileSync (~815× per upload)
on the main-process event loop WHILE uploading — delaying IPC / progress-batch forwarding for every other
concurrent upload. The user runs doodstream, so this fired in practice. Fix: gate it behind the EXISTING
`globalSettings.logVerbose` setting (default off), mirroring main.js `logDebug`/`_logVerbose`; wired via the
single `setLogVerbose` chokepoint (boot + save + toggle). Near-zero risk (early-return when verbose off),
removes all per-upload sync fs in normal operation. 397/397 tests pass, lint clean, wiring verified (shared
module instance, default-off, toggles).
### ROOT CAUSE (confirmed + profiled): recent-uploads panel append-only path defeats itself at the cap
- `renderRecentUploadsPanel` had a cheap append-only fast path gated on `rows.length > _recentLastRenderedLen`.
- `maybeAddSessionFile` caps `sessionFilesData` by push-then-slice (2000 → 2001 → sliced back to 2000).
- So once past SESSION_FILES_CAP, `rows.length` is pinned at 2000 → the gate is FALSE forever →
EVERY completion fell through to `tbody.innerHTML = rows.map(...).join('')` — a full ~2000-row
rebuild. Cap is per (link × file × hoster), so 45 hosters hit 2000 at only ~400500 files.
- Blink measurement (table-layout:fixed, same engine as Electron): full 2000-row rebuild = **~80 ms**
on EVERY completion past the cap. At several completions/sec that is a repeating ~80 ms main-thread
freeze → exactly "fine fresh, gets laggy after many uploads, CPU/RAM fine."
## DEFERRED — documented, conscious (real mechanisms, but wrong risk/reward to ship)
These are in MY T1 code (lib/config-store.js, commit 29d1944). They are REAL and scale with history size,
but at THIS user's scale (8 batches) they are tens of MICROSECONDS, on a path that fires ~once/20s. The
verifiers said "not urgent / negligible (~0.1% duty) / latent main-process hygiene, not a renderer-lag fix."
load() is the most dangerous code in the repo (config + credentials; corruption = data loss) and I was
already bitten once here by cache semantics. So: recorded, not shipped.
- F1 `_serializeForDisk` JSON.stringify(whole config incl. unbounded history, null,2) on every save +
copyFileSync of the growing file per commit. Scales with historySize. Root fix = split history into its own
file (history.json) so the ~20s queue-persist stops dragging history. Higher-risk persistence surgery.
- F2 `load()` deep-clones the WHOLE config (incl. history) on every call (even cache hit), and `_atomicWrite`
nulls the cache per write so write-interleaved loads are full misses. Measured @8000 batches: old read+parse
9.65 ms → new miss 22.96 ms (a 2.22.4× regression I introduced in T1) → new hit 12.94 ms. A "fix" via
shallow `history.slice()` is UNSAFE: the speedup comes only from sharing batch objects by reference, which
is exactly a silent-cache-corruption hazard that depends on a global "nothing ever deep-mutates a history
batch" invariant I can't enforce across future code + every getHistory consumer. Perf win and risk are the
same coin → no safe version → wrong tool for safety-critical code. The genuinely safe fixes (history-split,
bounded default retention) are out of scope (history-split = risk; bounded default = could drop user history).
- F3 (renderer, low) `_completedUploadKeys` grows unbounded per session, fully iterated in
buildPersistedQueueState on the ~20s persist. Sub-ms even for thousands of keys; the Set is the re-queue
dedup so capping it risks correctness. Not worth it.
- F4 (operator, low) diagnostics serverHealth does ~68 configStore.load() per request, each cloning history.
Cold path (only when diagnostics is queried), and largely a function of F2; defer with F2.
## Fix (shipped) — append-evict, keeps the panel append-only past the cap
- Track newly-pushed rows in `_recentPendingAppends` (incremented in maybeAddSessionFile), consumed
every render. Gate the fast path on `pendingAppends > 0` (not length-delta) so it survives the cap.
- Prepend the new rows, then evict the same overflow count from the DOM bottom (oldest, = data front
eviction in date-desc) to honour the cap. DOM work back to O(added).
- Gate behind an explicit `appendOnly` flag passed ONLY by scheduleRecentRender's rAF, so selection/
delete/clear/sort/batch-done renders stay full + correct (no wrong-row eviction, no double-prepend).
- VERIFIED in Blink over a simulated 5000-completion session: per-frame render **80 ms → median 7.4 ms**
(>10×), DOM stays exactly == data (cap held, newest-on-top, oldest evicted, ZERO duplicates).
## Verified clean (no action)
- T1 structuredClone-per-load does NOT hit the upload hot path: shouldLogHosterToFile uses LIVE
uploadManager.hosterSettings during uploads (its load() fallback is unreachable mid-batch); the 8×/s log
flush uses the O(1) _getLogSettings cache (T3); remaining load() sites are user-triggered IPC / boot.
- T3 _getLogSettings: O(1) on cache hit, invalidated on every settings-write path.
- renderer append-evict (v3.3.87): no new per-frame cost; correctness re-verified (5000-completion sim).
- diagnostics renderer additions (0c6c502): all inside renderSettings() — cold, settings panel only.
## Also shipped earlier (29d1944) — defensible, but NOT the cause for this user
- T1 ConfigStore in-memory cache (mtime/size keyed) + lean _serializeForDisk (clones only hosters) +
copyFileSync .bak. T3 cache logMode/logFilePath (drop load() from the 500 ms log flush).
## Investigated and DISMISSED with evidence
- `_sessionFileKeys` "delete-key separator mismatch" (workflow flagged it HIGH-ish): FALSE POSITIVE.
Line uses REAL U+0001 chars (char-code dump: `HAS_U0001: True`); the verifier agents' Read rendered
the control chars invisibly and wrongly concluded "no separators". Keys match at runtime. No leak.
- queueJobs O(N) per-render scans (grows unbounded since removeFromQueueOnDone=false AND folder-monitor
keeps ONE batch alive via addJobs so the 500-cap prune never fires): REAL but Blink-measured at
**<0.1 ms even at 3000 jobs** imperceptible. Incremental-counter refactor NOT worth the risk. Skipped.
- "Continuous standing relayout of 2000 rows": Blink median 0.4 ms regardless of row count; its spikes
were caused by the rebuild (same root cause). Not a standing cost → no need to virtualize/lower cap.
## Deferred (separate, not this symptom)
- doodstream-upload.js `_debugLog`: ungated SYNC fs.appendFileSync + statSync ~1020×/upload on the
main loop. CONSTANT cost (does not grow over a session), only when doodstream is active. Real freeze
contributor but NOT the reported progressive lag — convert to buffered-async like main.js, separately.
## Verification summary
- 397/397 tests pass, eslint 0 errors.
- Blink before/after: recent-panel render 80 ms → 7.4 ms; correctness asserted (cap/order/no-dupes).