Twitch-VOD-Manager/tasks/v5.0.0-plan-02-sqlite.md

Plan 02: SQLite-Foundation (Pillar 3)

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: SQLite-Infrastruktur ins Projekt einbauen + Migrator, der bestehende JSON-Daten aus C:\ProgramData\Twitch_VOD_Manager\*.json nach app.db ueberfuehrt. JSON-Pfade bleiben Source-of-Truth — SQLite laeuft als Shadow-Schreibziel. Cutover (SQLite wird Master) erfolgt in einem spaeteren Plan.

Architecture: better-sqlite3 (sync embedded). Schema-First via Inline-Konstante (schema-v5.ts) damit tsc keine SQL-Files kopieren muss. WAL-Mode, busy_timeout 5000ms, foreign_keys ON. Migrator ist idempotent (Marker in migrations_applied Tabelle). Bei Crash bleibt JSON-Pfad funktionsfaehig.

Tech Stack: better-sqlite3 + @types/better-sqlite3, vitest mit echten temp-DB-Files.

Verifikation pro Task: npm run build + npm run test:unit gruen.

---

Out of Scope fuer Plan 02

• Reads aus SQLite umstellen (Renderer/IPC bleibt auf JSON)
• electron-rebuild fuer better-sqlite3 (Workaround in Plan 02, voller Pipeline-Fix in spaeterem Plan)
• Schema-Migrationen v5 → v6
• Sub-only-Tabellen (kommen mit OAuth in Plan 03)

---

File Structure

Neu:

• src/main/infra/schema-v5.ts — Schema als TypeScript-String-Konstante
• src/main/infra/db.ts — better-sqlite3 Wrapper
• src/main/infra/db.test.ts
• src/main/domain/migrator.ts — JSON to SQLite
• src/main/domain/migrator.test.ts

Modifiziert:

• package.json
• src/main.ts — Migrator beim App-Start (fail-soft)
• CLAUDE.md
• tasks/v5.0.0-roadmap.md

---

Tasks

Task 1: better-sqlite3 installieren

• npm install --save better-sqlite3
• npm install --save-dev @types/better-sqlite3
• Verify: node -e "const D=require('better-sqlite3'); const d=new D(':memory:'); d.prepare('SELECT 1 AS x').get(); d.close(); console.log('ok')" → "ok"
• npm run build Exit 0
• Commit: build: add better-sqlite3 + @types

---

Task 2: schema-v5.ts (Schema als Inline-String)

• Datei src/main/infra/schema-v5.ts mit export const SCHEMA_V5_SQL = \...sql...`;`
• SQL-Inhalt umfasst Tabellen: schema_meta, config_kv, queue_items, downloaded_vods, streamers, archive_files, migrations_applied. Plus Indices.
• Alle Tabellen IF NOT EXISTS. INSERT OR IGNORE INTO schema_meta(key, value) VALUES ('schema_version', '5').
• Commit: feat(db): schema v5 inline (7 tables + indices)

Schema-Tabellen-Spec (Spalten):

schema_meta(key PK, value)
config_kv(key PK, value, updated_at)
queue_items(id PK, streamer_login, vod_id, clip_id, title, output_path,
            status, progress_pct, error_message, created_at, updated_at,
            completed_at, payload_json)
  idx_queue_status(status), idx_queue_streamer(streamer_login),
  idx_queue_created(created_at)
downloaded_vods(vod_id PK, downloaded_at default now)
streamers(login PK, auto_record, auto_vod_download, added_at default now)
  idx_streamers_autorec(auto_record), idx_streamers_autodl(auto_vod_download)
archive_files(path PK, streamer_login, size_bytes, duration_seconds,
              created_at, verified default 0)
  idx_archive_streamer(streamer_login)
migrations_applied(name PK, applied_at default now, payload)

---

Task 3: db.ts Wrapper + 5 Tests

Public DbHandle Interface:

• run(sql, params?) — single-statement execute
• get<T>(sql, params?) — first row or undefined
• all<T>(sql, params?) — alle Rows
• transaction<R>(fn) — atomares Bracket
• runBatch(sql) — multi-statement (intern via _db.exec(sql), fuer Schema-Bootstrap)
• close()
• raw getter

openDatabase(filePath: string): DbHandle

• Erstellt File (better-sqlite3 default-Verhalten)
• pragma journal_mode = WAL, pragma busy_timeout = 5000, pragma foreign_keys = ON
• Fuehrt Schema-Bootstrap aus: runBatch(SCHEMA_V5_SQL)
• Returnt das Handle-Objekt

Tests (src/main/infra/db.test.ts):

import { test, expect, describe, beforeEach, afterEach } from 'vitest';
import * as fs from 'fs';
import * as os from 'os';
import * as path from 'path';
import { openDatabase, type DbHandle } from './db';

let tmpDir: string;
let db: DbHandle | null = null;
beforeEach(() => {
    tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'db-test-'));
});
afterEach(() => {
    try { db?.close(); } catch { /* ignore */ }
    db = null;
    try { fs.rmSync(tmpDir, { recursive: true, force: true }); } catch { /* ignore */ }
});

describe('openDatabase', () => {
    test('creates a new file', () => {
        const target = path.join(tmpDir, 'a.db');
        db = openDatabase(target);
        expect(fs.existsSync(target)).toBe(true);
        expect(typeof db.run).toBe('function');
        expect(typeof db.get).toBe('function');
        expect(typeof db.all).toBe('function');
        expect(typeof db.close).toBe('function');
    });

    test('schema_meta row exists with schema_version=5', () => {
        db = openDatabase(path.join(tmpDir, 'b.db'));
        const row = db.get<{ value: string }>('SELECT value FROM schema_meta WHERE key = ?', ['schema_version']);
        expect(row?.value).toBe('5');
    });

    test('WAL mode active', () => {
        db = openDatabase(path.join(tmpDir, 'c.db'));
        const row = db.get<{ journal_mode: string }>('PRAGMA journal_mode');
        expect(row?.journal_mode).toBe('wal');
    });

    test('idempotent open: existing file keeps schema_version=5', () => {
        const target = path.join(tmpDir, 'd.db');
        db = openDatabase(target);
        db.close();
        db = openDatabase(target);
        const row = db.get<{ value: string }>('SELECT value FROM schema_meta WHERE key = ?', ['schema_version']);
        expect(row?.value).toBe('5');
    });

    test('run + get + all roundtrip', () => {
        db = openDatabase(path.join(tmpDir, 'e.db'));
        db.run('INSERT INTO downloaded_vods(vod_id) VALUES (?)', ['1234']);
        db.run('INSERT INTO downloaded_vods(vod_id) VALUES (?)', ['5678']);
        const one = db.get<{ vod_id: string }>('SELECT vod_id FROM downloaded_vods WHERE vod_id = ?', ['1234']);
        expect(one?.vod_id).toBe('1234');
        const all = db.all<{ vod_id: string }>('SELECT vod_id FROM downloaded_vods ORDER BY vod_id');
        expect(all.map(r => r.vod_id)).toEqual(['1234', '5678']);
    });
});

• Tests schreiben (sollen fehlschlagen — Modul fehlt)
• npm run test:unit zeigt 5 neue failures
• db.ts implementieren (better-sqlite3 wrappen, Interface oben einhalten)
• Tests passen — npm run test:unit Exit 0
• Build gruen
• Commit: feat(db): better-sqlite3 wrapper + schema bootstrap (5 tests)

---

Task 4: Migrator + 8 Tests

MigratorOptions:

• db: DbHandle
• appDataDir: string

MigrationResult:

• alreadyApplied: boolean
• configMigrated: boolean
• queueMigrated: boolean
• downloadedVodsCount: number
• streamersCount: number
• errors: MigrationError[] ({ source, message })

migrateJsonToSqlite(opts):

1. Check migrations_applied fuer name='v4-to-v5-jsons' → wenn vorhanden → return alreadyApplied: true
2. Lies appDataDir/config.json falls vorhanden:
   • Parse, in db.transaction(() => { ... }):
     • Whitelisted Keys (siehe Liste unten) → INSERT OR REPLACE INTO config_kv(key, value, updated_at) VALUES (?, ?, strftime('%s','now')) mit value = JSON.stringify(config[key])
     • downloaded_vod_ids (Array of String) → INSERT OR IGNORE INTO downloaded_vods(vod_id) VALUES (?)
     • auto_record_streamers → normalizeLogin + INSERT...ON CONFLICT DO UPDATE auto_record=1
     • auto_vod_download_streamers → normalizeLogin + INSERT...ON CONFLICT DO UPDATE auto_vod_download=1
   • Bei Erfolg: .v4-backup Copy schreiben (fs.copyFileSync)
   • Bei Fehler: error pushen, continue
3. Lies appDataDir/download_queue.json falls vorhanden:
   • Parse, fuer jedes Item (mit id-Feld): INSERT OR REPLACE INTO queue_items(...) VALUES (...)
   • Fields: id, streamer→normalizeLogin, vod_id, clip_id, title, output_path, status (default 'pending'), progress_pct, error_message, created_at, updated_at, completed_at, payload_json=JSON.stringify(item)
   • Bei Erfolg: .v4-backup Copy
4. Schreibe Marker: INSERT INTO migrations_applied(name, payload) VALUES ('v4-to-v5-jsons', JSON.stringify({summary}))

Whitelisted Config-Keys fuer config_kv (verbatim):

language, performance_mode, metadata_cache_minutes, streamlink_quality,
streamlink_disable_ads, download_chat_replay, capture_live_chat,
discord_webhook_url, discord_notify_live_start, discord_notify_live_end,
discord_notify_vod_complete, discord_notify_vod_auto_queued,
auto_cleanup_enabled, auto_cleanup_days, auto_cleanup_target,
auto_cleanup_action, log_stream_events, auto_vod_download_poll_minutes,
auto_vod_max_age_hours, auto_resume_live_recording,
auto_merge_resumed_parts, delete_parts_after_merge,
auto_record_poll_seconds, filename_template_vod, filename_template_parts,
filename_template_clip, smart_queue_scheduler, prevent_duplicate_downloads,
persist_queue_on_restart, auto_resume_queue_on_startup,
notify_on_each_completion

Tests (verbatim, src/main/domain/migrator.test.ts):

import { test, expect, describe, beforeEach, afterEach } from 'vitest';
import * as fs from 'fs';
import * as os from 'os';
import * as path from 'path';
import { openDatabase, type DbHandle } from '../infra/db';
import { migrateJsonToSqlite } from './migrator';

let tmpDir: string;
let appDataDir: string;
let db: DbHandle;
beforeEach(() => {
    tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'migrator-'));
    appDataDir = path.join(tmpDir, 'appdata');
    fs.mkdirSync(appDataDir, { recursive: true });
    db = openDatabase(path.join(tmpDir, 'app.db'));
});
afterEach(() => {
    db.close();
    try { fs.rmSync(tmpDir, { recursive: true, force: true }); } catch { /* ignore */ }
});

function writeJson(name: string, payload: unknown): string {
    const target = path.join(appDataDir, name);
    fs.writeFileSync(target, JSON.stringify(payload, null, 2), 'utf-8');
    return target;
}

describe('migrateJsonToSqlite', () => {
    test('no JSON files: writes migrations_applied marker', () => {
        const result = migrateJsonToSqlite({ db, appDataDir });
        expect(result.configMigrated).toBe(false);
        expect(result.queueMigrated).toBe(false);
        expect(result.downloadedVodsCount).toBe(0);
        expect(result.streamersCount).toBe(0);

        const marker = db.get<{ name: string }>('SELECT name FROM migrations_applied WHERE name = ?', ['v4-to-v5-jsons']);
        expect(marker?.name).toBe('v4-to-v5-jsons');
    });

    test('migrates config.json keys into config_kv', () => {
        writeJson('config.json', {
            language: 'de',
            performance_mode: 'speed',
            metadata_cache_minutes: 30,
            downloaded_vod_ids: ['1', '2', '3'],
            auto_record_streamers: ['foo', 'bar'],
        });
        const result = migrateJsonToSqlite({ db, appDataDir });
        expect(result.configMigrated).toBe(true);

        const lang = db.get<{ value: string }>('SELECT value FROM config_kv WHERE key = ?', ['language']);
        expect(JSON.parse(lang!.value)).toBe('de');

        const perf = db.get<{ value: string }>('SELECT value FROM config_kv WHERE key = ?', ['performance_mode']);
        expect(JSON.parse(perf!.value)).toBe('speed');
    });

    test('migrates downloaded_vod_ids', () => {
        writeJson('config.json', { downloaded_vod_ids: ['100', '200', '300'] });
        const result = migrateJsonToSqlite({ db, appDataDir });
        expect(result.downloadedVodsCount).toBe(3);
        const rows = db.all<{ vod_id: string }>('SELECT vod_id FROM downloaded_vods ORDER BY vod_id');
        expect(rows.map(r => r.vod_id)).toEqual(['100', '200', '300']);
    });

    test('migrates streamers from both lists', () => {
        writeJson('config.json', {
            auto_record_streamers: ['Alice', '@bob'],
            auto_vod_download_streamers: ['bob', 'carol'],
        });
        const result = migrateJsonToSqlite({ db, appDataDir });
        expect(result.streamersCount).toBeGreaterThanOrEqual(3);

        const alice = db.get<{ login: string; auto_record: number }>('SELECT login, auto_record FROM streamers WHERE login = ?', ['alice']);
        expect(alice?.auto_record).toBe(1);

        const bob = db.get<{ login: string; auto_record: number; auto_vod_download: number }>('SELECT login, auto_record, auto_vod_download FROM streamers WHERE login = ?', ['bob']);
        expect(bob?.auto_record).toBe(1);
        expect(bob?.auto_vod_download).toBe(1);

        const carol = db.get<{ login: string; auto_vod_download: number }>('SELECT login, auto_vod_download FROM streamers WHERE login = ?', ['carol']);
        expect(carol?.auto_vod_download).toBe(1);
    });

    test('migrates download_queue.json items', () => {
        writeJson('download_queue.json', [
            { id: 'q1', status: 'pending', streamer: 'foo', vod_id: 'v1', created_at: 1000, updated_at: 1000 },
            { id: 'q2', status: 'completed', streamer: 'bar', vod_id: 'v2', created_at: 2000, updated_at: 3000, completed_at: 3000 },
        ]);
        const result = migrateJsonToSqlite({ db, appDataDir });
        expect(result.queueMigrated).toBe(true);

        const all = db.all<{ id: string; status: string }>('SELECT id, status FROM queue_items ORDER BY id');
        expect(all).toHaveLength(2);
        expect(all[0].status).toBe('pending');
        expect(all[1].status).toBe('completed');
    });

    test('idempotent second run', () => {
        writeJson('config.json', { downloaded_vod_ids: ['1', '2'] });
        migrateJsonToSqlite({ db, appDataDir });
        const result2 = migrateJsonToSqlite({ db, appDataDir });
        expect(result2.alreadyApplied).toBe(true);
        const count = db.get<{ c: number }>('SELECT COUNT(*) AS c FROM downloaded_vods');
        expect(count?.c).toBe(2);
    });

    test('writes .v4-backup of source JSONs', () => {
        const configPath = writeJson('config.json', { language: 'en' });
        migrateJsonToSqlite({ db, appDataDir });
        expect(fs.existsSync(configPath + '.v4-backup')).toBe(true);
        expect(fs.readFileSync(configPath + '.v4-backup', 'utf-8')).toContain('"language": "en"');
    });

    test('malformed JSON is logged + skipped', () => {
        fs.writeFileSync(path.join(appDataDir, 'config.json'), '{ not valid json', 'utf-8');
        const result = migrateJsonToSqlite({ db, appDataDir });
        expect(result.configMigrated).toBe(false);
        expect(result.errors.length).toBeGreaterThan(0);
        expect(result.errors[0].source).toBe('config.json');
    });
});

• Tests schreiben (failing — Modul fehlt)
• Migrator implementieren (gemaess Interface + Step-Liste oben)
• Tests passen — npm run test:unit Exit 0
• Build gruen
• Commit: feat(db): JSON to SQLite migrator (idempotent, fail-soft, 8 tests)

---

Task 5: Migrator-Aufruf beim App-Start

main.ts: im app.whenReady().then(() => { ... }) Block, vor dem ersten createWindow()-Aufruf, einen try/catch-Block mit:

• require('./main/infra/db') und require('./main/domain/migrator') (lazy, damit Native-Fehler kein App-Start verhindern)

• openDatabase(path.join(APPDATA_DIR, 'app.db'))

• migrateJsonToSqlite({ db, appDataDir: APPDATA_DIR })

• Result ins debug.log loggen (falls debugLog Funktion vorhanden)

• db.close() im finally

• Catch: fail-soft, nur ins debug.log oder console.error

• Code adden

• Build gruen

• npm run test:e2e Exit 0 (Playwright startet App, Migrator laeuft auf leerer AppData)

• Commit: feat(db): wire migrator into app startup (fail-soft, lazy require)

---

Task 6: Full Verification + Version Bump 5.0.0-alpha.1

• npm run test:e2e:release Exit 0 (build + unit (mit jetzt >=104 Tests) + update-logic + 3 Playwright-Stages)
• npm version 5.0.0-alpha.1 --no-git-tag-version
• npm run build Exit 0
• Commit: release: 5.0.0-alpha.1 - SQLite migrator (shadow write, JSON stays master)
• Tag: git tag v5.0.0-alpha.1

---

Task 7: CLAUDE.md + Roadmap Update

CLAUDE.md Key Patterns / Persistence:

JSON files in C:\ProgramData\Twitch_VOD_Manager\ bleiben Source-of-Truth. Seit v5.0.0-alpha.1 spiegelt der Migrator (src/main/domain/migrator.ts) Konfiguration, Queue, downloaded_vod_ids und Streamer-Listen einmalig in app.db (SQLite, better-sqlite3). Schema: src/main/infra/schema-v5.ts. Cutover (SQLite wird Master) erfolgt in einem spaeteren Plan.

Roadmap: Plan 02 Status → DONE (v5.0.0-alpha.1), Plan 03 wird NEXT.

• CLAUDE.md aktualisieren
• Roadmap aktualisieren
• Commit: docs: SQLite migrator pattern + roadmap status update

---

Self-Review

• Tests echt (kein Skip / Stub)
• Idempotenz getestet
• Malformed-JSON getestet
• .v4-backup Copy getestet
• Fail-Soft Pfad in main.ts: App startet auch ohne native-build-success
• Spec-Coverage: Pillar 3 SQLite-Migrator + Schema v5 abgedeckt. Cutover bleibt fuer spaeter

Done-Definition

1. npm run test:unit Exit 0 mit >= 13 neuen Tests (5 db + 8 migrator)
2. npm run test:e2e:release Exit 0
3. SQLite app.db wird beim ersten Start angelegt
4. Migrator idempotent (zweimaliger Run)
5. .v4-backup Files existieren
6. Version 5.0.0-alpha.1 getaggt
7. CLAUDE.md + Roadmap aktualisiert

---

Execution Handoff

Inline-Execution via superpowers:executing-plans. Naechster Plan (tasks/v5.0.0-plan-03-oauth.md) nach Done-Definition.