Twitch-VOD-Manager/tasks/v5.0.0-plan-02-sqlite.md

# Plan 02: SQLite-Foundation (Pillar 3)

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** SQLite-Infrastruktur ins Projekt einbauen + Migrator, der bestehende JSON-Daten aus `C:\ProgramData\Twitch_VOD_Manager\*.json` nach `app.db` ueberfuehrt. JSON-Pfade bleiben Source-of-Truth — SQLite laeuft als Shadow-Schreibziel. Cutover (SQLite wird Master) erfolgt in einem spaeteren Plan.

**Architecture:** `better-sqlite3` (sync embedded). Schema-First via Inline-Konstante (`schema-v5.ts`) damit `tsc` keine SQL-Files kopieren muss. WAL-Mode, busy_timeout 5000ms, foreign_keys ON. Migrator ist idempotent (Marker in `migrations_applied` Tabelle). Bei Crash bleibt JSON-Pfad funktionsfaehig.

**Tech Stack:** better-sqlite3 + @types/better-sqlite3, vitest mit echten temp-DB-Files.

**Verifikation pro Task:** `npm run build` + `npm run test:unit` gruen.

---

## Out of Scope fuer Plan 02

- Reads aus SQLite umstellen (Renderer/IPC bleibt auf JSON)
- electron-rebuild fuer better-sqlite3 (Workaround in Plan 02, voller Pipeline-Fix in spaeterem Plan)
- Schema-Migrationen v5 → v6
- Sub-only-Tabellen (kommen mit OAuth in Plan 03)

---

## File Structure

**Neu:**
- `src/main/infra/schema-v5.ts` — Schema als TypeScript-String-Konstante
- `src/main/infra/db.ts` — better-sqlite3 Wrapper
- `src/main/infra/db.test.ts`
- `src/main/domain/migrator.ts` — JSON to SQLite
- `src/main/domain/migrator.test.ts`

**Modifiziert:**
- `package.json`
- `src/main.ts` — Migrator beim App-Start (fail-soft)
- `CLAUDE.md`
- `tasks/v5.0.0-roadmap.md`

---

## Tasks

### Task 1: better-sqlite3 installieren

- [ ] `npm install --save better-sqlite3`
- [ ] `npm install --save-dev @types/better-sqlite3`
- [ ] Verify: `node -e "const D=require('better-sqlite3'); const d=new D(':memory:'); d.prepare('SELECT 1 AS x').get(); d.close(); console.log('ok')"` → "ok"
- [ ] `npm run build` Exit 0
- [ ] Commit: `build: add better-sqlite3 + @types`

---

### Task 2: schema-v5.ts (Schema als Inline-String)

- [ ] Datei `src/main/infra/schema-v5.ts` mit `export const SCHEMA_V5_SQL = \`...sql...\`;`
- [ ] SQL-Inhalt umfasst Tabellen: `schema_meta`, `config_kv`, `queue_items`, `downloaded_vods`, `streamers`, `archive_files`, `migrations_applied`. Plus Indices.
- [ ] Alle Tabellen `IF NOT EXISTS`. `INSERT OR IGNORE INTO schema_meta(key, value) VALUES ('schema_version', '5')`.
- [ ] Commit: `feat(db): schema v5 inline (7 tables + indices)`

**Schema-Tabellen-Spec (Spalten):**

```
schema_meta(key PK, value)
config_kv(key PK, value, updated_at)
queue_items(id PK, streamer_login, vod_id, clip_id, title, output_path,
            status, progress_pct, error_message, created_at, updated_at,
            completed_at, payload_json)
  idx_queue_status(status), idx_queue_streamer(streamer_login),
  idx_queue_created(created_at)
downloaded_vods(vod_id PK, downloaded_at default now)
streamers(login PK, auto_record, auto_vod_download, added_at default now)
  idx_streamers_autorec(auto_record), idx_streamers_autodl(auto_vod_download)
archive_files(path PK, streamer_login, size_bytes, duration_seconds,
              created_at, verified default 0)
  idx_archive_streamer(streamer_login)
migrations_applied(name PK, applied_at default now, payload)
```

---

### Task 3: db.ts Wrapper + 5 Tests

**Public DbHandle Interface:**
- `run(sql, params?)` — single-statement execute
- `get<T>(sql, params?)` — first row or undefined
- `all<T>(sql, params?)` — alle Rows
- `transaction<R>(fn)` — atomares Bracket
- `runBatch(sql)` — multi-statement (intern via `_db.exec(sql)`, fuer Schema-Bootstrap)
- `close()`
- `raw` getter

**openDatabase(filePath: string): DbHandle**
- Erstellt File (better-sqlite3 default-Verhalten)
- `pragma journal_mode = WAL`, `pragma busy_timeout = 5000`, `pragma foreign_keys = ON`
- Fuehrt Schema-Bootstrap aus: `runBatch(SCHEMA_V5_SQL)`
- Returnt das Handle-Objekt

**Tests** (`src/main/infra/db.test.ts`):

```typescript
import { test, expect, describe, beforeEach, afterEach } from 'vitest';
import * as fs from 'fs';
import * as os from 'os';
import * as path from 'path';
import { openDatabase, type DbHandle } from './db';

let tmpDir: string;
let db: DbHandle | null = null;
beforeEach(() => {
    tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'db-test-'));
});
afterEach(() => {
    try { db?.close(); } catch { /* ignore */ }
    db = null;
    try { fs.rmSync(tmpDir, { recursive: true, force: true }); } catch { /* ignore */ }
});

describe('openDatabase', () => {
    test('creates a new file', () => {
        const target = path.join(tmpDir, 'a.db');
        db = openDatabase(target);
        expect(fs.existsSync(target)).toBe(true);
        expect(typeof db.run).toBe('function');
        expect(typeof db.get).toBe('function');
        expect(typeof db.all).toBe('function');
        expect(typeof db.close).toBe('function');
    });

    test('schema_meta row exists with schema_version=5', () => {
        db = openDatabase(path.join(tmpDir, 'b.db'));
        const row = db.get<{ value: string }>('SELECT value FROM schema_meta WHERE key = ?', ['schema_version']);
        expect(row?.value).toBe('5');
    });

    test('WAL mode active', () => {
        db = openDatabase(path.join(tmpDir, 'c.db'));
        const row = db.get<{ journal_mode: string }>('PRAGMA journal_mode');
        expect(row?.journal_mode).toBe('wal');
    });

    test('idempotent open: existing file keeps schema_version=5', () => {
        const target = path.join(tmpDir, 'd.db');
        db = openDatabase(target);
        db.close();
        db = openDatabase(target);
        const row = db.get<{ value: string }>('SELECT value FROM schema_meta WHERE key = ?', ['schema_version']);
        expect(row?.value).toBe('5');
    });

    test('run + get + all roundtrip', () => {
        db = openDatabase(path.join(tmpDir, 'e.db'));
        db.run('INSERT INTO downloaded_vods(vod_id) VALUES (?)', ['1234']);
        db.run('INSERT INTO downloaded_vods(vod_id) VALUES (?)', ['5678']);
        const one = db.get<{ vod_id: string }>('SELECT vod_id FROM downloaded_vods WHERE vod_id = ?', ['1234']);
        expect(one?.vod_id).toBe('1234');
        const all = db.all<{ vod_id: string }>('SELECT vod_id FROM downloaded_vods ORDER BY vod_id');
        expect(all.map(r => r.vod_id)).toEqual(['1234', '5678']);
    });
});
```

- [ ] Tests schreiben (sollen fehlschlagen — Modul fehlt)
- [ ] `npm run test:unit` zeigt 5 neue failures
- [ ] db.ts implementieren (better-sqlite3 wrappen, Interface oben einhalten)
- [ ] Tests passen — `npm run test:unit` Exit 0
- [ ] Build gruen
- [ ] Commit: `feat(db): better-sqlite3 wrapper + schema bootstrap (5 tests)`

---

### Task 4: Migrator + 8 Tests

**MigratorOptions:**
- `db: DbHandle`
- `appDataDir: string`

**MigrationResult:**
- `alreadyApplied: boolean`
- `configMigrated: boolean`
- `queueMigrated: boolean`
- `downloadedVodsCount: number`
- `streamersCount: number`
- `errors: MigrationError[]` (`{ source, message }`)

**migrateJsonToSqlite(opts):**
1. Check `migrations_applied` fuer `name='v4-to-v5-jsons'` → wenn vorhanden → return `alreadyApplied: true`
2. Lies `appDataDir/config.json` falls vorhanden:
   - Parse, in `db.transaction(() => { ... })`:
     - Whitelisted Keys (siehe Liste unten) → `INSERT OR REPLACE INTO config_kv(key, value, updated_at) VALUES (?, ?, strftime('%s','now'))` mit `value = JSON.stringify(config[key])`
     - `downloaded_vod_ids` (Array of String) → `INSERT OR IGNORE INTO downloaded_vods(vod_id) VALUES (?)`
     - `auto_record_streamers` → normalizeLogin + `INSERT...ON CONFLICT DO UPDATE auto_record=1`
     - `auto_vod_download_streamers` → normalizeLogin + `INSERT...ON CONFLICT DO UPDATE auto_vod_download=1`
   - Bei Erfolg: `.v4-backup` Copy schreiben (`fs.copyFileSync`)
   - Bei Fehler: error pushen, continue
3. Lies `appDataDir/download_queue.json` falls vorhanden:
   - Parse, fuer jedes Item (mit `id`-Feld): `INSERT OR REPLACE INTO queue_items(...) VALUES (...)`
   - Fields: id, streamer→normalizeLogin, vod_id, clip_id, title, output_path, status (default 'pending'), progress_pct, error_message, created_at, updated_at, completed_at, payload_json=JSON.stringify(item)
   - Bei Erfolg: `.v4-backup` Copy
4. Schreibe Marker: `INSERT INTO migrations_applied(name, payload) VALUES ('v4-to-v5-jsons', JSON.stringify({summary}))`

**Whitelisted Config-Keys fuer config_kv (verbatim):**

```
language, performance_mode, metadata_cache_minutes, streamlink_quality,
streamlink_disable_ads, download_chat_replay, capture_live_chat,
discord_webhook_url, discord_notify_live_start, discord_notify_live_end,
discord_notify_vod_complete, discord_notify_vod_auto_queued,
auto_cleanup_enabled, auto_cleanup_days, auto_cleanup_target,
auto_cleanup_action, log_stream_events, auto_vod_download_poll_minutes,
auto_vod_max_age_hours, auto_resume_live_recording,
auto_merge_resumed_parts, delete_parts_after_merge,
auto_record_poll_seconds, filename_template_vod, filename_template_parts,
filename_template_clip, smart_queue_scheduler, prevent_duplicate_downloads,
persist_queue_on_restart, auto_resume_queue_on_startup,
notify_on_each_completion
```

**Tests (verbatim, src/main/domain/migrator.test.ts):**

```typescript
import { test, expect, describe, beforeEach, afterEach } from 'vitest';
import * as fs from 'fs';
import * as os from 'os';
import * as path from 'path';
import { openDatabase, type DbHandle } from '../infra/db';
import { migrateJsonToSqlite } from './migrator';

let tmpDir: string;
let appDataDir: string;
let db: DbHandle;
beforeEach(() => {
    tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'migrator-'));
    appDataDir = path.join(tmpDir, 'appdata');
    fs.mkdirSync(appDataDir, { recursive: true });
    db = openDatabase(path.join(tmpDir, 'app.db'));
});
afterEach(() => {
    db.close();
    try { fs.rmSync(tmpDir, { recursive: true, force: true }); } catch { /* ignore */ }
});

function writeJson(name: string, payload: unknown): string {
    const target = path.join(appDataDir, name);
    fs.writeFileSync(target, JSON.stringify(payload, null, 2), 'utf-8');
    return target;
}

describe('migrateJsonToSqlite', () => {
    test('no JSON files: writes migrations_applied marker', () => {
        const result = migrateJsonToSqlite({ db, appDataDir });
        expect(result.configMigrated).toBe(false);
        expect(result.queueMigrated).toBe(false);
        expect(result.downloadedVodsCount).toBe(0);
        expect(result.streamersCount).toBe(0);

        const marker = db.get<{ name: string }>('SELECT name FROM migrations_applied WHERE name = ?', ['v4-to-v5-jsons']);
        expect(marker?.name).toBe('v4-to-v5-jsons');
    });

    test('migrates config.json keys into config_kv', () => {
        writeJson('config.json', {
            language: 'de',
            performance_mode: 'speed',
            metadata_cache_minutes: 30,
            downloaded_vod_ids: ['1', '2', '3'],
            auto_record_streamers: ['foo', 'bar'],
        });
        const result = migrateJsonToSqlite({ db, appDataDir });
        expect(result.configMigrated).toBe(true);

        const lang = db.get<{ value: string }>('SELECT value FROM config_kv WHERE key = ?', ['language']);
        expect(JSON.parse(lang!.value)).toBe('de');

        const perf = db.get<{ value: string }>('SELECT value FROM config_kv WHERE key = ?', ['performance_mode']);
        expect(JSON.parse(perf!.value)).toBe('speed');
    });

    test('migrates downloaded_vod_ids', () => {
        writeJson('config.json', { downloaded_vod_ids: ['100', '200', '300'] });
        const result = migrateJsonToSqlite({ db, appDataDir });
        expect(result.downloadedVodsCount).toBe(3);
        const rows = db.all<{ vod_id: string }>('SELECT vod_id FROM downloaded_vods ORDER BY vod_id');
        expect(rows.map(r => r.vod_id)).toEqual(['100', '200', '300']);
    });

    test('migrates streamers from both lists', () => {
        writeJson('config.json', {
            auto_record_streamers: ['Alice', '@bob'],
            auto_vod_download_streamers: ['bob', 'carol'],
        });
        const result = migrateJsonToSqlite({ db, appDataDir });
        expect(result.streamersCount).toBeGreaterThanOrEqual(3);

        const alice = db.get<{ login: string; auto_record: number }>('SELECT login, auto_record FROM streamers WHERE login = ?', ['alice']);
        expect(alice?.auto_record).toBe(1);

        const bob = db.get<{ login: string; auto_record: number; auto_vod_download: number }>('SELECT login, auto_record, auto_vod_download FROM streamers WHERE login = ?', ['bob']);
        expect(bob?.auto_record).toBe(1);
        expect(bob?.auto_vod_download).toBe(1);

        const carol = db.get<{ login: string; auto_vod_download: number }>('SELECT login, auto_vod_download FROM streamers WHERE login = ?', ['carol']);
        expect(carol?.auto_vod_download).toBe(1);
    });

    test('migrates download_queue.json items', () => {
        writeJson('download_queue.json', [
            { id: 'q1', status: 'pending', streamer: 'foo', vod_id: 'v1', created_at: 1000, updated_at: 1000 },
            { id: 'q2', status: 'completed', streamer: 'bar', vod_id: 'v2', created_at: 2000, updated_at: 3000, completed_at: 3000 },
        ]);
        const result = migrateJsonToSqlite({ db, appDataDir });
        expect(result.queueMigrated).toBe(true);

        const all = db.all<{ id: string; status: string }>('SELECT id, status FROM queue_items ORDER BY id');
        expect(all).toHaveLength(2);
        expect(all[0].status).toBe('pending');
        expect(all[1].status).toBe('completed');
    });

    test('idempotent second run', () => {
        writeJson('config.json', { downloaded_vod_ids: ['1', '2'] });
        migrateJsonToSqlite({ db, appDataDir });
        const result2 = migrateJsonToSqlite({ db, appDataDir });
        expect(result2.alreadyApplied).toBe(true);
        const count = db.get<{ c: number }>('SELECT COUNT(*) AS c FROM downloaded_vods');
        expect(count?.c).toBe(2);
    });

    test('writes .v4-backup of source JSONs', () => {
        const configPath = writeJson('config.json', { language: 'en' });
        migrateJsonToSqlite({ db, appDataDir });
        expect(fs.existsSync(configPath + '.v4-backup')).toBe(true);
        expect(fs.readFileSync(configPath + '.v4-backup', 'utf-8')).toContain('"language": "en"');
    });

    test('malformed JSON is logged + skipped', () => {
        fs.writeFileSync(path.join(appDataDir, 'config.json'), '{ not valid json', 'utf-8');
        const result = migrateJsonToSqlite({ db, appDataDir });
        expect(result.configMigrated).toBe(false);
        expect(result.errors.length).toBeGreaterThan(0);
        expect(result.errors[0].source).toBe('config.json');
    });
});
```

- [ ] Tests schreiben (failing — Modul fehlt)
- [ ] Migrator implementieren (gemaess Interface + Step-Liste oben)
- [ ] Tests passen — `npm run test:unit` Exit 0
- [ ] Build gruen
- [ ] Commit: `feat(db): JSON to SQLite migrator (idempotent, fail-soft, 8 tests)`

---

### Task 5: Migrator-Aufruf beim App-Start

**main.ts:** im `app.whenReady().then(() => { ... })` Block, vor dem ersten `createWindow()`-Aufruf, einen try/catch-Block mit:
- `require('./main/infra/db')` und `require('./main/domain/migrator')` (lazy, damit Native-Fehler kein App-Start verhindern)
- `openDatabase(path.join(APPDATA_DIR, 'app.db'))`
- `migrateJsonToSqlite({ db, appDataDir: APPDATA_DIR })`
- Result ins debug.log loggen (falls `debugLog` Funktion vorhanden)
- `db.close()` im `finally`
- Catch: fail-soft, nur ins debug.log oder console.error

- [ ] Code adden
- [ ] Build gruen
- [ ] `npm run test:e2e` Exit 0 (Playwright startet App, Migrator laeuft auf leerer AppData)
- [ ] Commit: `feat(db): wire migrator into app startup (fail-soft, lazy require)`

---

### Task 6: Full Verification + Version Bump 5.0.0-alpha.1

- [ ] `npm run test:e2e:release` Exit 0 (build + unit (mit jetzt >=104 Tests) + update-logic + 3 Playwright-Stages)
- [ ] `npm version 5.0.0-alpha.1 --no-git-tag-version`
- [ ] `npm run build` Exit 0
- [ ] Commit: `release: 5.0.0-alpha.1 - SQLite migrator (shadow write, JSON stays master)`
- [ ] Tag: `git tag v5.0.0-alpha.1`

---

### Task 7: CLAUDE.md + Roadmap Update

**CLAUDE.md Key Patterns / Persistence:**
> JSON files in `C:\ProgramData\Twitch_VOD_Manager\` bleiben Source-of-Truth. Seit v5.0.0-alpha.1 spiegelt der Migrator (`src/main/domain/migrator.ts`) Konfiguration, Queue, downloaded_vod_ids und Streamer-Listen einmalig in `app.db` (SQLite, better-sqlite3). Schema: `src/main/infra/schema-v5.ts`. Cutover (SQLite wird Master) erfolgt in einem spaeteren Plan.

**Roadmap:** Plan 02 Status → `DONE (v5.0.0-alpha.1)`, Plan 03 wird `NEXT`.

- [ ] CLAUDE.md aktualisieren
- [ ] Roadmap aktualisieren
- [ ] Commit: `docs: SQLite migrator pattern + roadmap status update`

---

## Self-Review

- [ ] Tests echt (kein Skip / Stub)
- [ ] Idempotenz getestet
- [ ] Malformed-JSON getestet
- [ ] `.v4-backup` Copy getestet
- [ ] Fail-Soft Pfad in main.ts: App startet auch ohne native-build-success
- [ ] Spec-Coverage: Pillar 3 SQLite-Migrator + Schema v5 abgedeckt. Cutover bleibt fuer spaeter

## Done-Definition

1. `npm run test:unit` Exit 0 mit >= 13 neuen Tests (5 db + 8 migrator)
2. `npm run test:e2e:release` Exit 0
3. SQLite `app.db` wird beim ersten Start angelegt
4. Migrator idempotent (zweimaliger Run)
5. `.v4-backup` Files existieren
6. Version 5.0.0-alpha.1 getaggt
7. CLAUDE.md + Roadmap aktualisiert

---

## Execution Handoff

Inline-Execution via `superpowers:executing-plans`. Naechster Plan (`tasks/v5.0.0-plan-03-oauth.md`) nach Done-Definition.