Harden proxy segment integrity verification
This commit is contained in:
@@ -4,6 +4,13 @@ All notable changes to Multi-Debrid Downloader are documented in this file.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Proxy segment integrity hotfix
|
||||
|
||||
- Read every completed proxy segment back from disk and compare its SHA-256 digest with the bytes received before accepting the segment.
|
||||
- Verify suspicious zero-filled ranges through a different proxy and reload the complete segment when the independent response differs.
|
||||
- Distinguish an actual manifest checksum pass from an integrity check skipped because no checksum was available.
|
||||
- Keep extraction and password handling unchanged.
|
||||
|
||||
## [2.0.78] - 2026-08-31
|
||||
|
||||
### Download presentation
|
||||
|
||||
+17
-3
@@ -10,10 +10,11 @@ Diese Datei hält den verifizierten technischen Arbeitsstand fest. Sie enthält
|
||||
|
||||
- Verifiziert am: 31. August 2026, Europe/Berlin
|
||||
- Lokaler Pfad: `C:\Users\Sascha\Desktop\Claude & ChatGPT Projekte\Multi-Debrid-Downloader`
|
||||
- Arbeitsbranch: `release/v2.0.78`
|
||||
- Quellbasis: `release/v2.0.77`
|
||||
- Arbeitsbranch: `hotfix/proxy-segment-readback-v2.0.79`
|
||||
- Quellbasis: `release/v2.0.78`
|
||||
- Release-Tag: `v2.0.78`
|
||||
- Baseline-Commit: `48c3677296f54d389f0982fc79edf2788a6d1191`
|
||||
- Hotfix-Basis: `5cee459d0bcc5c19d8d6483948a036a43c9d91b1`
|
||||
- Paketversion: `2.0.78`
|
||||
- Letztes Release: `Multi-Debrid-Downloader v2.0.78`, veröffentlicht am 31. August 2026 auf GitHub und Forgejo
|
||||
- Runtime-Voraussetzung: Node.js `>=20`; lokal verifiziert mit Node.js `24.19.0` und npm `11.17.0`
|
||||
@@ -145,6 +146,15 @@ Diese Datei hält den verifizierten technischen Arbeitsstand fest. Sie enthält
|
||||
- „Sitzung“ erhält den kumulativen Bytezähler bei großen laufenden Queues nun in jedem 750-Millisekunden-Live-Snapshot und aktualisiert sich damit genauso häufig wie „Verbleibend“. Der 1,5-Sekunden-Cache bleibt für die übrigen aufwendigeren Statistikdaten erhalten.
|
||||
- Die Entpacklogik und ihre Produktionspfade wurden nicht verändert. `v2.0.78` wurde nach ausdrücklicher Freigabe auf GitHub und Forgejo veröffentlicht; eine Installation oder ein produktiver Neustart ist nicht Bestandteil des Releases.
|
||||
|
||||
## Unveröffentlichter Proxy-Integritäts-Hotfix
|
||||
|
||||
- Drei unabhängig bereitgestellte Kopien von `p2p-libr7-S02E05.rar` wurden binär verglichen. JDownloader und der manuelle Real-Debrid-Browserdownload waren bei exakt 910.536.950 Byte SHA-256-identisch (`be0bd334a63d92611acae349effb0e23a4cf4bdd00feb2fad9f3d0edb11e36e9`) und ließen sich mit UnRAR fehlerfrei testen. Die MDD-Kopie hatte dieselbe Größe, aber SHA-256 `7a763e774ba8068d4a756a69c72ae72bbc18d649c3cb7a5c7dc2b1d6a6436a1e` und einen CRC-Fehler.
|
||||
- In der MDD-Kopie war ausschließlich der Bereich Byte 346.951.680 bis 346.955.311 durchgehend null. Die 3.632-Byte-Lücke lag innerhalb des 42. 8-MiB-Chunks; Segmentgrenzen und Endgröße waren korrekt. Damit ist ein still beschädigter Proxy-Segmentdownload nachgewiesen, nicht ein Fehler des Quellarchivs oder Extractors.
|
||||
- Jeder fertig geschriebene Proxy-Chunk wird nun direkt vom Datenträger zurückgelesen und sein SHA-256-Digest mit den tatsächlich empfangenen Bytes verglichen. Bei einer Abweichung wird der Fortschritt des Versuchs zurückgerechnet, der Proxy abgewertet und der vollständige Chunk über einen anderen Proxy neu geladen.
|
||||
- Nullfolgen ab 1.024 Byte werden zusätzlich stichprobenartig über einen anderen, gleichzeitig exklusiv reservierten Proxy erneut angefordert. Weicht die unabhängige Range-Antwort ab, wird ebenfalls der vollständige Chunk verworfen und neu geladen. Bestätigt die Gegenprobe echte Nullbytes, wird der Chunk normal akzeptiert.
|
||||
- Item-Logs melden den Grund `readback_mismatch` beziehungsweise `zero_run_mismatch` mit Range- und Chunkgrenzen. Die bisher irreführende Meldung „Integritätsprüfung bestanden“ bei fehlender `.sfv`-, `.md5`- oder `.sha1`-Prüfsumme wurde in „Integritätsprüfung nicht ausgeführt“ mit Begründung geändert.
|
||||
- Entpack-, Passwort- und Nachbearbeitungslogik wurden nicht verändert. Der Hotfix ist noch nicht veröffentlicht; Release, Installation und produktiver Neustart benötigen eine neue ausdrückliche Freigabe.
|
||||
|
||||
## Start-, Build- und Testbefehle
|
||||
|
||||
```powershell
|
||||
@@ -176,6 +186,9 @@ npm exec -- tsc --noEmit
|
||||
|
||||
## Verifizierungen vom 31. August 2026
|
||||
|
||||
- Unveröffentlichter Proxy-Integritäts-Hotfix: 15 von 15 Proxy-Segmenttests und 9 von 9 Manifest-Integritätstests erfolgreich. Die neuen End-to-End-Regressionen beweisen vollständiges Neuladen nach einer abweichenden 2-KiB-Nullantwort, Akzeptanz einer unabhängig bestätigten legitimen Nullfolge und Erkennung einer nach dem Netzwerkempfang in die Temp-Datei injizierten 1-KiB-Abweichung durch den Readback-Digest.
|
||||
- Vollständiger Client-Lauf des Hotfixes: 140 Testdateien erfolgreich, 1 optionale JVM-Testdatei übersprungen; 2.685 Tests erfolgreich und 4 übersprungen. Nur die zwei bekannten Symlink-Fixtures scheiterten vor ihrer Produktassertion mit Windows-`EPERM`.
|
||||
- Nach dem Hotfix erfolgreich: TypeScript, Main-Build, Renderer-Build, Node-Self-Check und 16 von 16 Backup-API-Tests. Die bekannte Vite-Warnung zum rund 576 KiB großen Renderer-Chunk bleibt bestehen.
|
||||
- Veröffentlichung `v2.0.78`: Der annotierte Tag und beide Release-Branches zeigen bei GitHub und Forgejo exakt auf `48c3677296f54d389f0982fc79edf2788a6d1191`. Beide öffentlichen Releases enthalten dieselben sechs Assets; alle zwölf erneut heruntergeladenen Dateien stimmen in Größe und SHA-256 exakt mit den lokalen Originalen überein. GitHub führt `v2.0.78` als Latest Release.
|
||||
- Release-Build `v2.0.78`: Installer und Portable-Datei wurden aus dem versionierten Quellstand neu erzeugt. Die Release-Prüfung bestätigte Paket- und Bundle-Version, Update-Metadaten, Artefaktnamen, SHA-512, Icon, Lizenzdateien sowie den entpackten Inhalt beider EXE-Archive. Von den 24 Public-Release-Metadatentests waren 22 erfolgreich; nur die zwei unter Windows ohne Symlink-Berechtigung nicht ausführbaren Fixtures endeten vor ihrer Produktassertion mit `EPERM`.
|
||||
- SHA-256 des `v2.0.78`-Setups: `caaef1ce83d533cf3095353f729b8ecedecb0adc6d9a372d4cfd5249551767a3`; SHA-256 der Portable-Datei: `92e12f210ac3160a4967fa5de491092dff932025293e20984a3ec9744e9d6f32`.
|
||||
@@ -247,10 +260,11 @@ npm exec -- tsc --noEmit
|
||||
- Der Proxy-Gesamtlimit-Hotfix ist als `v2.0.76` veröffentlicht. Eine Serverinstallation oder ein produktiver Neustart wurde nicht vorgenommen.
|
||||
- Der faire rollierende Proxy-Scheduler und der bytebasierte Paketfortschritt sind als `v2.0.77` veröffentlicht. Eine Serverinstallation oder ein produktiver Neustart wurde nicht vorgenommen.
|
||||
- Der 32/40-Proxybereich und der synchrone Live-Takt von „Sitzung“ und „Verbleibend“ sind als `v2.0.78` veröffentlicht. Eine Serverinstallation oder ein produktiver Neustart wurde nicht vorgenommen.
|
||||
- Der Proxy-Readback- und Nullbereich-Hotfix ist auf `hotfix/proxy-segment-readback-v2.0.79` vorbereitet, aber noch nicht veröffentlicht oder auf einem produktiven System installiert.
|
||||
|
||||
## Nächste sinnvolle Schritte
|
||||
|
||||
1. Nach einer getrennt freigegebenen Installation `v2.0.78` mit zwei echten parallelen Real-Debrid-Dateien zunächst bei 32 und anschließend optional bei 40 prüfen sowie 429/503, Zusatzverkehr, Festplattenlast, Netto-MB/s und den gemeinsamen Live-Takt von „Sitzung“ und „Verbleibend“ beobachten.
|
||||
1. Nach einer ausdrücklichen Release-Freigabe den Proxy-Integritäts-Hotfix als nächste Version vorbereiten, paketieren und veröffentlichen; anschließend auf dem Zielserver denselben problematischen Download erneut durchführen und Item-Log, SHA-256 sowie `UnRAR t` prüfen.
|
||||
2. Nach einer getrennt freigegebenen Serverinstallation den Ablauf Real-Debrid-Web-Download, Hauptfenster schließen und unmittelbar neu starten am echten Zielsystem verifizieren; vor jedem Eingriff Prozessbaum und Logtail sichern.
|
||||
3. Sicherheitsabhängigkeiten in einem separaten Upgrade-Branch aktualisieren, Electron-/Vite-/Vitest-Major-Wechsel einzeln testen und danach den vollständigen Windows-Paketpfad prüfen.
|
||||
4. Eine nichtdestruktive Strategie zur Bereinigung der divergierenden `main`-Branches abstimmen; kein Force-Push ohne ausdrückliche Freigabe.
|
||||
|
||||
@@ -10263,11 +10263,17 @@ export class DownloadManager extends EventEmitter {
|
||||
}
|
||||
throw new Error(`Integritätsprüfung fehlgeschlagen (${validation.message})`);
|
||||
}
|
||||
// Symmetry: a passed check was previously silent in the item log, so a
|
||||
// reader could not tell whether integrity ran and passed vs was skipped.
|
||||
this.logPackageForItem(item, "INFO", "Integritätsprüfung bestanden", {
|
||||
elapsedMs: integrityElapsedMs
|
||||
});
|
||||
if (validation.checked) {
|
||||
this.logPackageForItem(item, "INFO", "Integritätsprüfung bestanden", {
|
||||
result: validation.message,
|
||||
elapsedMs: integrityElapsedMs
|
||||
});
|
||||
} else {
|
||||
this.logPackageForItem(item, "INFO", "Integritätsprüfung nicht ausgeführt", {
|
||||
reason: validation.message,
|
||||
elapsedMs: integrityElapsedMs
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (active.abortController.signal.aborted) {
|
||||
@@ -11088,6 +11094,15 @@ export class DownloadManager extends EventEmitter {
|
||||
this.emitState();
|
||||
lastProxyUiEmitAt = nowTick;
|
||||
}
|
||||
},
|
||||
onValidationRetry: (event) => {
|
||||
logAttemptEvent("WARN", "Proxy-Segmentprüfung fehlgeschlagen, Segment wird neu geladen", {
|
||||
reason: event.reason,
|
||||
rangeStart: event.rangeStart,
|
||||
rangeEnd: event.rangeEnd,
|
||||
segmentStart: event.segmentStart,
|
||||
segmentEnd: event.segmentEnd
|
||||
});
|
||||
}
|
||||
});
|
||||
if (proxyResult.status === "completed") {
|
||||
|
||||
@@ -139,21 +139,24 @@ async function hashFile(filePath: string, algorithm: "crc32" | "md5" | "sha1"):
|
||||
});
|
||||
}
|
||||
|
||||
export async function validateFileAgainstManifest(filePath: string, packageDir: string): Promise<{ ok: boolean; message: string }> {
|
||||
export async function validateFileAgainstManifest(
|
||||
filePath: string,
|
||||
packageDir: string
|
||||
): Promise<{ ok: boolean; checked: boolean; message: string }> {
|
||||
const manifest = readHashManifest(packageDir);
|
||||
if (manifest.size === 0) {
|
||||
return { ok: true, message: "Kein Hash verfügbar" };
|
||||
return { ok: true, checked: false, message: "Kein Hash verfügbar" };
|
||||
}
|
||||
const keyByBaseName = normalizeManifestKey(path.basename(filePath));
|
||||
const keyByRelativePath = normalizeManifestKey(path.relative(packageDir, filePath));
|
||||
const entry = manifest.get(keyByRelativePath) || manifest.get(keyByBaseName);
|
||||
if (!entry) {
|
||||
return { ok: true, message: "Kein Hash für Datei" };
|
||||
return { ok: true, checked: false, message: "Kein Hash für Datei" };
|
||||
}
|
||||
|
||||
const actual = await hashFile(filePath, entry.algorithm);
|
||||
if (actual === entry.digest.toLowerCase()) {
|
||||
return { ok: true, message: `${entry.algorithm.toUpperCase()} ok` };
|
||||
return { ok: true, checked: true, message: `${entry.algorithm.toUpperCase()} ok` };
|
||||
}
|
||||
return { ok: false, message: `${entry.algorithm.toUpperCase()} mismatch` };
|
||||
return { ok: false, checked: true, message: `${entry.algorithm.toUpperCase()} mismatch` };
|
||||
}
|
||||
|
||||
@@ -2,7 +2,7 @@ import fs from "node:fs";
|
||||
import http from "node:http";
|
||||
import https from "node:https";
|
||||
import path from "node:path";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { createHash, randomUUID, timingSafeEqual } from "node:crypto";
|
||||
import type { IncomingMessage } from "node:http";
|
||||
import { HttpsProxyAgent } from "https-proxy-agent";
|
||||
|
||||
@@ -18,6 +18,10 @@ const PROXY_PROBE_END = 1;
|
||||
const MAX_REDIRECTS = 5;
|
||||
const MAX_SEGMENTS = 4_096;
|
||||
const MAX_PROXY_FILE_BYTES = 8 * 1024 * 1024;
|
||||
const SEGMENT_READBACK_BUFFER_BYTES = 256 * 1024;
|
||||
const SUSPICIOUS_ZERO_RUN_BYTES = 1024;
|
||||
const MAX_ZERO_RUN_VERIFICATIONS_PER_SEGMENT = 4;
|
||||
const ZERO_RUN_SENTINEL = Buffer.alloc(SUSPICIOUS_ZERO_RUN_BYTES);
|
||||
const DISK_ERROR_CODES = new Set(["ENOSPC", "EDQUOT", "EACCES", "EPERM", "EROFS", "EIO", "ENODEV"]);
|
||||
|
||||
export interface ProxyEndpoint {
|
||||
@@ -53,6 +57,19 @@ interface Segment {
|
||||
end: number;
|
||||
}
|
||||
|
||||
interface SegmentDownloadOutcome {
|
||||
bytes: number;
|
||||
suspiciousZeroRanges: Segment[];
|
||||
}
|
||||
|
||||
export interface ProxySegmentValidationRetry {
|
||||
reason: "readback_mismatch" | "zero_run_mismatch";
|
||||
rangeStart: number;
|
||||
rangeEnd: number;
|
||||
segmentStart: number;
|
||||
segmentEnd: number;
|
||||
}
|
||||
|
||||
export interface ProxySegmentedDownloadOptions {
|
||||
directUrl: string;
|
||||
targetPath: string;
|
||||
@@ -70,6 +87,7 @@ export interface ProxySegmentedDownloadOptions {
|
||||
waitWhilePaused?: () => Promise<void>;
|
||||
onTrafficBytes?: (bytes: number) => void;
|
||||
onProgress?: (deltaBytes: number, downloadedBytes: number, totalBytes: number) => void;
|
||||
onValidationRetry?: (event: ProxySegmentValidationRetry) => void;
|
||||
}
|
||||
|
||||
export type ProxySegmentedDownloadResult =
|
||||
@@ -103,6 +121,17 @@ class OriginHttpError extends Error {
|
||||
}
|
||||
}
|
||||
|
||||
class SegmentValidationError extends Error {
|
||||
public constructor(
|
||||
public readonly reason: ProxySegmentValidationRetry["reason"],
|
||||
public readonly rangeStart: number,
|
||||
public readonly rangeEnd: number,
|
||||
public readonly segment: Segment
|
||||
) {
|
||||
super(`proxy_segment_${reason}`);
|
||||
}
|
||||
}
|
||||
|
||||
interface ProxyLeaseOutcome {
|
||||
succeeded: boolean;
|
||||
transferredBytes?: number;
|
||||
@@ -185,6 +214,18 @@ class SharedProxyCoordinator {
|
||||
this.drain();
|
||||
}
|
||||
|
||||
public reportContentMismatch(proxy: ProxyEndpoint): void {
|
||||
const key = proxyEndpointKey(proxy);
|
||||
const previous = this.performance.get(key);
|
||||
this.cooldownUntil.set(key, Date.now() + 30_000);
|
||||
this.performance.set(key, {
|
||||
ewmaBytesPerSecond: (previous?.ewmaBytesPerSecond || 0) * 0.5,
|
||||
failures: (previous?.failures || 0) + 1,
|
||||
samples: previous?.samples || 0
|
||||
});
|
||||
this.drain();
|
||||
}
|
||||
|
||||
public acquire(
|
||||
poolKey: string,
|
||||
groupId: string,
|
||||
@@ -759,6 +800,149 @@ async function writeBufferAt(handle: fs.promises.FileHandle, buffer: Buffer, pos
|
||||
}
|
||||
}
|
||||
|
||||
function appendSuspiciousZeroRange(ranges: Segment[], start: number): void {
|
||||
if (ranges.length >= MAX_ZERO_RUN_VERIFICATIONS_PER_SEGMENT) {
|
||||
return;
|
||||
}
|
||||
const candidate = {
|
||||
index: ranges.length,
|
||||
start,
|
||||
end: start + SUSPICIOUS_ZERO_RUN_BYTES - 1
|
||||
};
|
||||
const previous = ranges[ranges.length - 1];
|
||||
if (previous && candidate.start <= previous.end + 1) {
|
||||
return;
|
||||
}
|
||||
ranges.push(candidate);
|
||||
}
|
||||
|
||||
function collectSuspiciousZeroRanges(
|
||||
chunk: Buffer,
|
||||
chunkStart: number,
|
||||
previousTrailingZeros: number,
|
||||
ranges: Segment[]
|
||||
): number {
|
||||
let leadingZeros = 0;
|
||||
while (leadingZeros < chunk.length && chunk[leadingZeros] === 0) {
|
||||
leadingZeros += 1;
|
||||
}
|
||||
if (previousTrailingZeros + leadingZeros >= SUSPICIOUS_ZERO_RUN_BYTES) {
|
||||
appendSuspiciousZeroRange(ranges, chunkStart - previousTrailingZeros);
|
||||
}
|
||||
|
||||
let searchAt = 0;
|
||||
while (ranges.length < MAX_ZERO_RUN_VERIFICATIONS_PER_SEGMENT) {
|
||||
const foundAt = chunk.indexOf(ZERO_RUN_SENTINEL, searchAt);
|
||||
if (foundAt < 0) {
|
||||
break;
|
||||
}
|
||||
appendSuspiciousZeroRange(ranges, chunkStart + foundAt);
|
||||
searchAt = foundAt + SUSPICIOUS_ZERO_RUN_BYTES;
|
||||
}
|
||||
|
||||
let trailingZeros = 0;
|
||||
for (let index = chunk.length - 1; index >= 0 && chunk[index] === 0; index -= 1) {
|
||||
trailingZeros += 1;
|
||||
if (trailingZeros >= SUSPICIOUS_ZERO_RUN_BYTES - 1) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
return trailingZeros;
|
||||
}
|
||||
|
||||
async function hashFileRange(
|
||||
filePath: string,
|
||||
start: number,
|
||||
length: number,
|
||||
signal: AbortSignal
|
||||
): Promise<Buffer> {
|
||||
const handle = await fs.promises.open(filePath, "r");
|
||||
const hash = createHash("sha256");
|
||||
const buffer = Buffer.allocUnsafe(Math.min(SEGMENT_READBACK_BUFFER_BYTES, Math.max(1, length)));
|
||||
let position = start;
|
||||
let remaining = length;
|
||||
try {
|
||||
while (remaining > 0) {
|
||||
if (signal.aborted) {
|
||||
throw abortError();
|
||||
}
|
||||
const requested = Math.min(buffer.length, remaining);
|
||||
const { bytesRead } = await handle.read(buffer, 0, requested, position);
|
||||
if (bytesRead <= 0) {
|
||||
throw new Error("proxy_segment_readback_underflow");
|
||||
}
|
||||
hash.update(buffer.subarray(0, bytesRead));
|
||||
position += bytesRead;
|
||||
remaining -= bytesRead;
|
||||
}
|
||||
return hash.digest();
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
}
|
||||
|
||||
async function readRangeOnce(
|
||||
target: URL,
|
||||
proxy: ProxyEndpoint,
|
||||
range: Segment,
|
||||
totalBytes: number,
|
||||
signal: AbortSignal,
|
||||
options: Required<Pick<ProxySegmentedDownloadOptions, "skipTlsVerify" | "connectTimeoutMs" | "idleTimeoutMs">>,
|
||||
onTrafficBytes?: (bytes: number) => void
|
||||
): Promise<Buffer> {
|
||||
const opened = await openRangeResponse(
|
||||
target,
|
||||
proxy,
|
||||
range.start,
|
||||
range.end,
|
||||
signal,
|
||||
options.skipTlsVerify,
|
||||
options.connectTimeoutMs,
|
||||
options.idleTimeoutMs
|
||||
);
|
||||
const response = opened.response;
|
||||
const status = response.statusCode || 0;
|
||||
if (status >= 400) {
|
||||
opened.dispose();
|
||||
throw new OriginHttpError(status);
|
||||
}
|
||||
const expectedBytes = range.end - range.start + 1;
|
||||
const contentRange = parseContentRange(response.headers["content-range"]);
|
||||
const contentLength = Number(response.headers["content-length"] || 0);
|
||||
if (status !== 206
|
||||
|| !contentRange
|
||||
|| contentRange.start !== range.start
|
||||
|| contentRange.end !== range.end
|
||||
|| contentRange.total !== totalBytes
|
||||
|| (contentLength > 0 && contentLength !== expectedBytes)) {
|
||||
opened.dispose();
|
||||
throw new Error("proxy_verification_range_mismatch");
|
||||
}
|
||||
|
||||
const chunks: Buffer[] = [];
|
||||
let received = 0;
|
||||
try {
|
||||
for await (const rawChunk of response) {
|
||||
if (signal.aborted) {
|
||||
throw abortError();
|
||||
}
|
||||
const chunk = Buffer.isBuffer(rawChunk) ? rawChunk : Buffer.from(rawChunk);
|
||||
onTrafficBytes?.(chunk.length);
|
||||
received += chunk.length;
|
||||
if (received > expectedBytes) {
|
||||
throw new Error("proxy_verification_range_overflow");
|
||||
}
|
||||
chunks.push(chunk);
|
||||
}
|
||||
if (received !== expectedBytes) {
|
||||
throw new Error("proxy_verification_range_underflow");
|
||||
}
|
||||
return Buffer.concat(chunks, received);
|
||||
} finally {
|
||||
opened.dispose();
|
||||
}
|
||||
}
|
||||
|
||||
async function downloadSegmentOnce(
|
||||
target: URL,
|
||||
tempPath: string,
|
||||
@@ -769,7 +953,7 @@ async function downloadSegmentOnce(
|
||||
callbacks: Pick<ProxySegmentedDownloadOptions, "waitWhilePaused" | "onTrafficBytes" | "onProgress">,
|
||||
totalBytes: number,
|
||||
currentProgress: () => number
|
||||
): Promise<number> {
|
||||
): Promise<SegmentDownloadOutcome> {
|
||||
const opened = await openRangeResponse(
|
||||
target,
|
||||
proxy,
|
||||
@@ -807,6 +991,9 @@ async function downloadSegmentOnce(
|
||||
throw error;
|
||||
}
|
||||
let received = 0;
|
||||
const receivedHash = createHash("sha256");
|
||||
const suspiciousZeroRanges: Segment[] = [];
|
||||
let trailingZeros = 0;
|
||||
try {
|
||||
for await (const rawChunk of response) {
|
||||
if (signal.aborted) {
|
||||
@@ -823,17 +1010,29 @@ async function downloadSegmentOnce(
|
||||
response.setTimeout(options.idleTimeoutMs);
|
||||
}
|
||||
await writeBufferAt(handle, chunk, segment.start + received);
|
||||
receivedHash.update(chunk);
|
||||
trailingZeros = collectSuspiciousZeroRanges(
|
||||
chunk,
|
||||
segment.start + received,
|
||||
trailingZeros,
|
||||
suspiciousZeroRanges
|
||||
);
|
||||
received += chunk.length;
|
||||
callbacks.onProgress?.(chunk.length, currentProgress() + chunk.length, totalBytes);
|
||||
}
|
||||
if (received !== expectedBytes) {
|
||||
throw new Error("proxy_segment_underflow");
|
||||
}
|
||||
return received;
|
||||
} finally {
|
||||
opened.dispose();
|
||||
await handle.close();
|
||||
}
|
||||
const writtenHash = await hashFileRange(tempPath, segment.start, expectedBytes, signal);
|
||||
const networkHash = receivedHash.digest();
|
||||
if (!timingSafeEqual(networkHash, writtenHash)) {
|
||||
throw new SegmentValidationError("readback_mismatch", segment.start, segment.end, segment);
|
||||
}
|
||||
return { bytes: received, suspiciousZeroRanges };
|
||||
}
|
||||
|
||||
export async function downloadWithProxySegments(options: ProxySegmentedDownloadOptions): Promise<ProxySegmentedDownloadResult> {
|
||||
@@ -943,19 +1142,20 @@ export async function downloadWithProxySegments(options: ProxySegmentedDownloadO
|
||||
if (signal.aborted) {
|
||||
throw abortError();
|
||||
}
|
||||
const lease = await sharedProxyCoordinator.acquire(poolKey, groupId, segmentProxies, excluded, signal);
|
||||
let lease = await sharedProxyCoordinator.acquire(poolKey, groupId, segmentProxies, excluded, signal);
|
||||
if (!lease) {
|
||||
break;
|
||||
}
|
||||
excluded.add(proxyEndpointKey(lease.proxy));
|
||||
const primaryProxy = lease.proxy;
|
||||
excluded.add(proxyEndpointKey(primaryProxy));
|
||||
let attemptProgress = 0;
|
||||
const startedAt = Date.now();
|
||||
try {
|
||||
const bytes = await downloadSegmentOnce(
|
||||
const outcome = await downloadSegmentOnce(
|
||||
target,
|
||||
tempPath,
|
||||
segment,
|
||||
lease.proxy,
|
||||
primaryProxy,
|
||||
signal,
|
||||
normalized,
|
||||
{
|
||||
@@ -971,13 +1171,60 @@ export async function downloadWithProxySegments(options: ProxySegmentedDownloadO
|
||||
);
|
||||
lease.release({
|
||||
succeeded: true,
|
||||
transferredBytes: bytes,
|
||||
transferredBytes: outcome.bytes,
|
||||
durationMs: Math.max(1, Date.now() - startedAt)
|
||||
});
|
||||
return bytes;
|
||||
lease = null;
|
||||
|
||||
if (outcome.suspiciousZeroRanges.length > 0) {
|
||||
lease = await sharedProxyCoordinator.acquire(poolKey, groupId, segmentProxies, excluded, signal);
|
||||
if (!lease) {
|
||||
throw new Error("proxy_zero_run_verification_unavailable");
|
||||
}
|
||||
const verificationStartedAt = Date.now();
|
||||
let verificationBytes = 0;
|
||||
for (const suspiciousRange of outcome.suspiciousZeroRanges) {
|
||||
const verification = await readRangeOnce(
|
||||
target,
|
||||
lease.proxy,
|
||||
suspiciousRange,
|
||||
probe.totalBytes,
|
||||
signal,
|
||||
normalized,
|
||||
(bytes) => {
|
||||
verificationBytes += bytes;
|
||||
options.onTrafficBytes?.(bytes);
|
||||
}
|
||||
);
|
||||
if (!timingSafeEqual(verification, ZERO_RUN_SENTINEL)) {
|
||||
lease.release({
|
||||
succeeded: true,
|
||||
transferredBytes: verificationBytes,
|
||||
durationMs: Math.max(1, Date.now() - verificationStartedAt)
|
||||
});
|
||||
lease = null;
|
||||
sharedProxyCoordinator.reportContentMismatch(primaryProxy);
|
||||
throw new SegmentValidationError(
|
||||
"zero_run_mismatch",
|
||||
suspiciousRange.start,
|
||||
suspiciousRange.end,
|
||||
segment
|
||||
);
|
||||
}
|
||||
}
|
||||
lease.release({
|
||||
succeeded: true,
|
||||
transferredBytes: verificationBytes,
|
||||
durationMs: Math.max(1, Date.now() - verificationStartedAt)
|
||||
});
|
||||
lease = null;
|
||||
}
|
||||
|
||||
return outcome.bytes;
|
||||
} catch (error) {
|
||||
const originError = error instanceof OriginHttpError;
|
||||
lease.release({ succeeded: originError || signal.aborted });
|
||||
lease?.release({ succeeded: originError || signal.aborted });
|
||||
lease = null;
|
||||
if (originError) {
|
||||
sharedProxyCoordinator.reportOriginStatus(error.statusCode);
|
||||
}
|
||||
@@ -987,6 +1234,18 @@ export async function downloadWithProxySegments(options: ProxySegmentedDownloadO
|
||||
if (options.signal.aborted || isDiskError(error)) {
|
||||
throw error;
|
||||
}
|
||||
if (error instanceof SegmentValidationError) {
|
||||
try {
|
||||
options.onValidationRetry?.({
|
||||
reason: error.reason,
|
||||
rangeStart: error.rangeStart,
|
||||
rangeEnd: error.rangeEnd,
|
||||
segmentStart: error.segment.start,
|
||||
segmentEnd: error.segment.end
|
||||
});
|
||||
} catch {
|
||||
}
|
||||
}
|
||||
lastError = error;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -28,6 +28,18 @@ describe("integrity", () => {
|
||||
fs.writeFileSync(path.join(dir, "hash.md5"), "5d41402abc4b2a76b9719d911017c592 movie.bin\n");
|
||||
const result = await validateFileAgainstManifest(filePath, dir);
|
||||
expect(result.ok).toBe(true);
|
||||
expect(result.checked).toBe(true);
|
||||
});
|
||||
|
||||
it("reports when no checksum was actually available", async () => {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), "rd-int-"));
|
||||
tempDirs.push(dir);
|
||||
const filePath = path.join(dir, "movie.bin");
|
||||
fs.writeFileSync(filePath, Buffer.from("hello"));
|
||||
|
||||
const result = await validateFileAgainstManifest(filePath, dir);
|
||||
|
||||
expect(result).toEqual({ ok: true, checked: false, message: "Kein Hash verfügbar" });
|
||||
});
|
||||
|
||||
it("skips manifest files larger than 5MB", () => {
|
||||
@@ -78,6 +90,7 @@ describe("integrity", () => {
|
||||
|
||||
const result = await validateFileAgainstManifest(filePath, dir);
|
||||
expect(result.ok).toBe(true);
|
||||
expect(result.checked).toBe(true);
|
||||
expect(result.message).toContain("MD5");
|
||||
});
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ import net from "node:net";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { Transform } from "node:stream";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { downloadWithProxySegments, normalizeProxyConnectionLimit, parseProxyList, selectFixedProxy } from "../src/main/proxy-segmented-download";
|
||||
|
||||
interface RunningServer {
|
||||
@@ -15,6 +15,7 @@ interface RunningServer {
|
||||
const cleanups: Array<() => Promise<void>> = [];
|
||||
|
||||
afterEach(async () => {
|
||||
vi.restoreAllMocks();
|
||||
while (cleanups.length > 0) {
|
||||
await cleanups.pop()?.();
|
||||
}
|
||||
@@ -236,6 +237,155 @@ describe("proxy segmented download", () => {
|
||||
expect(counters.some((counter) => counter.value > 0)).toBe(true);
|
||||
});
|
||||
|
||||
it("reloads a segment when another proxy disproves a suspicious zero-filled range", async () => {
|
||||
const content = Buffer.alloc(64 * 1024, 7);
|
||||
const ranges: string[] = [];
|
||||
let corruptedResponseSent = false;
|
||||
const server = http.createServer((request, response) => {
|
||||
const rangeHeader = String(request.headers.range || "");
|
||||
ranges.push(rangeHeader);
|
||||
const match = /^bytes=(\d+)-(\d+)$/.exec(rangeHeader);
|
||||
if (!match) {
|
||||
response.writeHead(416);
|
||||
response.end();
|
||||
return;
|
||||
}
|
||||
const start = Number(match[1]);
|
||||
const end = Math.min(Number(match[2]), content.length - 1);
|
||||
let body = content.subarray(start, end + 1);
|
||||
if (!corruptedResponseSent && rangeHeader === "bytes=0-4095") {
|
||||
body = Buffer.from(body);
|
||||
body.fill(0, 0, 2048);
|
||||
corruptedResponseSent = true;
|
||||
}
|
||||
response.writeHead(206, {
|
||||
"Content-Range": `bytes ${start}-${end}/${content.length}`,
|
||||
"Content-Length": body.length
|
||||
});
|
||||
response.end(body);
|
||||
});
|
||||
const targetServer = await listen(server);
|
||||
const counters = Array.from({ length: 3 }, () => ({ value: 0 }));
|
||||
const proxies = await Promise.all(counters.map((counter, index) => createConnectProxy(`zero${index}`, `pass${index}`, counter)));
|
||||
const directory = await createTempDirectory();
|
||||
const proxyFile = path.join(directory, "zero-retry-proxies.txt");
|
||||
const targetFile = path.join(directory, "zero-retry.bin");
|
||||
await fs.promises.writeFile(
|
||||
proxyFile,
|
||||
proxies.map((proxy, index) => `zero${index}:pass${index}@127.0.0.1:${proxy.port}`).join("\n")
|
||||
);
|
||||
const validationRetries: string[] = [];
|
||||
|
||||
const result = await downloadWithProxySegments({
|
||||
directUrl: `http://127.0.0.1:${targetServer.port}/zero-retry.bin`,
|
||||
targetPath: targetFile,
|
||||
proxyListPath: proxyFile,
|
||||
connections: 2,
|
||||
totalConnectionLimit: 2,
|
||||
signal: new AbortController().signal,
|
||||
minSegmentBytes: 1,
|
||||
onValidationRetry: (event) => validationRetries.push(event.reason)
|
||||
});
|
||||
|
||||
expect(result.status).toBe("completed");
|
||||
expect(await fs.promises.readFile(targetFile)).toEqual(content);
|
||||
expect(validationRetries).toContain("zero_run_mismatch");
|
||||
expect(ranges.filter((range) => range === "bytes=0-4095")).toHaveLength(2);
|
||||
expect(ranges).toContain("bytes=0-1023");
|
||||
});
|
||||
|
||||
it("accepts a legitimate zero-filled range after another proxy confirms it", async () => {
|
||||
const content = Buffer.alloc(64 * 1024, 9);
|
||||
content.fill(0, 8192, 10_240);
|
||||
const targetServer = await createRangeServer(content);
|
||||
const counters = Array.from({ length: 3 }, () => ({ value: 0 }));
|
||||
const proxies = await Promise.all(counters.map((counter, index) => createConnectProxy(`legit${index}`, `pass${index}`, counter)));
|
||||
const directory = await createTempDirectory();
|
||||
const proxyFile = path.join(directory, "legitimate-zero-proxies.txt");
|
||||
const targetFile = path.join(directory, "legitimate-zero.bin");
|
||||
await fs.promises.writeFile(
|
||||
proxyFile,
|
||||
proxies.map((proxy, index) => `legit${index}:pass${index}@127.0.0.1:${proxy.port}`).join("\n")
|
||||
);
|
||||
const validationRetries: string[] = [];
|
||||
|
||||
const result = await downloadWithProxySegments({
|
||||
directUrl: `http://127.0.0.1:${targetServer.port}/legitimate-zero.bin`,
|
||||
targetPath: targetFile,
|
||||
proxyListPath: proxyFile,
|
||||
connections: 2,
|
||||
totalConnectionLimit: 2,
|
||||
signal: new AbortController().signal,
|
||||
minSegmentBytes: 1,
|
||||
onValidationRetry: (event) => validationRetries.push(event.reason)
|
||||
});
|
||||
|
||||
expect(result.status).toBe("completed");
|
||||
expect(await fs.promises.readFile(targetFile)).toEqual(content);
|
||||
expect(validationRetries).toEqual([]);
|
||||
expect(targetServer.ranges).toContain("bytes=8192-9215");
|
||||
});
|
||||
|
||||
it("reloads a segment when the bytes read back from disk differ from the received bytes", async () => {
|
||||
const content = Buffer.alloc(64 * 1024, 13);
|
||||
const targetServer = await createRangeServer(content);
|
||||
const counters = Array.from({ length: 3 }, () => ({ value: 0 }));
|
||||
const proxies = await Promise.all(counters.map((counter, index) => createConnectProxy(`readback${index}`, `pass${index}`, counter)));
|
||||
const directory = await createTempDirectory();
|
||||
const proxyFile = path.join(directory, "readback-proxies.txt");
|
||||
const targetFile = path.join(directory, "readback.bin");
|
||||
await fs.promises.writeFile(
|
||||
proxyFile,
|
||||
proxies.map((proxy, index) => `readback${index}:pass${index}@127.0.0.1:${proxy.port}`).join("\n")
|
||||
);
|
||||
|
||||
const originalOpen = fs.promises.open.bind(fs.promises);
|
||||
let corruptionInjected = false;
|
||||
vi.spyOn(fs.promises, "open").mockImplementation((async (
|
||||
file: fs.PathLike,
|
||||
flags: fs.OpenMode,
|
||||
mode?: fs.Mode
|
||||
) => {
|
||||
const handle = await originalOpen(file, flags, mode);
|
||||
if (!corruptionInjected && flags === "r+" && String(file).includes(".proxy-")) {
|
||||
const originalWrite = handle.write.bind(handle) as (...args: unknown[]) => Promise<unknown>;
|
||||
const originalClose = handle.close.bind(handle);
|
||||
let firstPosition: number | null = null;
|
||||
handle.write = (async (...args: unknown[]) => {
|
||||
if (firstPosition === null && typeof args[3] === "number") {
|
||||
firstPosition = args[3];
|
||||
}
|
||||
return originalWrite(...args);
|
||||
}) as typeof handle.write;
|
||||
handle.close = (async () => {
|
||||
if (!corruptionInjected && firstPosition !== null) {
|
||||
corruptionInjected = true;
|
||||
await originalWrite(Buffer.alloc(1024), 0, 1024, firstPosition);
|
||||
}
|
||||
await originalClose();
|
||||
}) as typeof handle.close;
|
||||
}
|
||||
return handle;
|
||||
}) as typeof fs.promises.open);
|
||||
const validationRetries: string[] = [];
|
||||
|
||||
const result = await downloadWithProxySegments({
|
||||
directUrl: `http://127.0.0.1:${targetServer.port}/readback.bin`,
|
||||
targetPath: targetFile,
|
||||
proxyListPath: proxyFile,
|
||||
connections: 2,
|
||||
totalConnectionLimit: 2,
|
||||
signal: new AbortController().signal,
|
||||
minSegmentBytes: 1,
|
||||
onValidationRetry: (event) => validationRetries.push(event.reason)
|
||||
});
|
||||
|
||||
expect(result.status).toBe("completed");
|
||||
expect(corruptionInjected).toBe(true);
|
||||
expect(await fs.promises.readFile(targetFile)).toEqual(content);
|
||||
expect(validationRetries).toContain("readback_mismatch");
|
||||
});
|
||||
|
||||
it("lets fast proxies take over additional rolling chunks", async () => {
|
||||
const content = Buffer.allocUnsafe(512 * 1024);
|
||||
for (let index = 0; index < content.length; index += 1) content[index] = index % 233;
|
||||
|
||||
Reference in New Issue
Block a user