release: v3.3.77

fix(byse): treat 5xx/gateway upload failures as transient infra, not account faults
A single byse.sx gateway hiccup (HTTP 502 "<!doctype html>..." or a mid-upload ECONNRESET) was cascading through the whole failover chain and blacklisting every account for the batch. Root cause: the upload-POST throw sites threw PLAIN errors with no classification, so a 502 was treated as a GENERIC error -> mark-failed -> emit('account-failed') -> failover to the next account (which hits the same gateway) -> repeat until the chain is exhausted, then poison sibling files in the batch via the blacklist. The screenshots showed exactly this: "Primär ... Fallback #3" all failing with the same 502. byse did NOT change their API (the upload contract still matches their docs and was live-probed: GET /upload/server -> POST {key}+file -> files[0].filecode); these are transient infrastructure failures. A 5xx gateway error is not an account fault: every account hits the same gateway, so failing over is pointless and blacklisting is harmful. Fail open instead — retry the SAME account and, if byse stays down, fail the file cleanly without touching the account or the rotation cursor. - upload-manager: _isTransientNetworkError() now honors an explicit err.transientNetwork flag (checked before the empty-message guard) and, as a defensive fallback, matches /HTTP 5\d\d/, Bad Gateway, Service Unavailable, Gateway Time-out. The flag is made authoritative in _isFileRejectedError and _shouldSkipRetryOnAccountError (both return early when it is set) so a 5xx whose HTML body happens to contain a rejection/auth keyword can never be mis-binned as file/account. The post-rotation retry loop also breaks on a transient error (parity with the primary loop) to avoid burning the retry budget re-uploading on a fallback. - hosters: the upload POST throw sites tag err.transientNetwork when statusCode >= 500 (non-JSON body and non-2xx-with-JSON) and when the 2xx status-envelope carries status:500; 401/403/429 stay PLAIN so they remain account errors. The server-lookup path (apiGet/getUploadServer) is hardened symmetrically: a 5xx there tags transientNetwork and getUploadServer preserves the flag onto its wrapped error, so the heuristic shouldRetryServerLookup() can no longer be defeated by a 5xx body that contains an auth keyword. - hosters: fixed a separate real bug surfaced during investigation — _fetchByseFileList built the wrong URL https://api.byse.sx/api/file/list (the /api/ prefix is correct only for doodstream's doodapi.co host; byse already carries the api. subdomain). Live-probed: /api/file/list 302-redirects to the docs page, /file/list returns 200 JSON. The wrong path made the byse async-recovery poll silently dead (always []), removing the safety net that reclaims a large file that registered despite a 502. Now https://api.byse.sx/file/list. ECONNRESET was already transient and is unchanged. doodstream's 2xx empty-form stays hosterTransient (one attempt, no re-upload). vidmoly/clouddrop use their own uploaders and are unaffected; doodstream/voe apiKey uploads share uploadFile and correctly benefit from the same 5xx-is-infra logic. Deferred follow-up: poll-first-on-5xx dedup (route a 5xx through the now-working recovery poll before retrying, to reclaim a registered-but-502 file instead of re-uploading). It does not add new dupe risk vs the prior cascade and a pure 502 means the backend was never reached, so retry-same-account is dupe-safe for it. Investigated and reviewed by two multi-agent workflows (4-lens investigation with a synthesized fix spec; 4-lens adversarial review with per-finding verification): no API drift, 0 confirmed defects. Tests: classifier units (flag-above-message-guard, 5xx-transient, 4xx-stay-account), an end-to-end 502/503/500-envelope tag through uploadFile, and a 502-retries-same-account-no-cascade integration case. Suite 311/311. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 01:56:55 +02:00 · 2026-06-19 01:56:11 +02:00
6 changed files with 164 additions and 14 deletions
--- a/lib/hosters.js
+++ b/lib/hosters.js
@ -325,11 +325,15 @@ async function apiGet(url, signal) {
    try {
      data = JSON.parse(text);
    } catch {
-      throw new Error(`API-Antwort war kein JSON (HTTP ${res.status}): ${(text || '').slice(0, 200)}`);
+      const err = new Error(`API-Antwort war kein JSON (HTTP ${res.status}): ${(text || '').slice(0, 200)}`);
      if (res.status >= 500) err.transientNetwork = true;
      throw err;
    }
    if (data.status && [401, 403, 429, 500].includes(data.status)) {
-      throw new Error(data.msg || data.message || JSON.stringify(data));
+      const err = new Error(data.msg || data.message || JSON.stringify(data));
      if (data.status === 500) err.transientNetwork = true;
      throw err;
    }
    return data;
  } finally {
@ -342,6 +346,7 @@ async function apiGet(url, signal) {
 async function getUploadServer(hosterName, hosterConfig, apiKey, signal) {
  let lastMessage = '';
  let lastTransient = false;
  for (let attempt = 1; attempt <= SERVER_RETRY_ATTEMPTS; attempt++) {
    for (const endpoint of hosterConfig.serverEndpoints) {
@ -361,6 +366,7 @@ async function getUploadServer(hosterName, hosterConfig, apiKey, signal) {
      } catch (err) {
        if (err.name === 'AbortError') throw err;
        if (err.message) lastMessage = err.message;
        if (err.transientNetwork === true) lastTransient = true;
      }
    }
@ -394,6 +400,7 @@ async function getUploadServer(hosterName, hosterConfig, apiKey, signal) {
    // Genuine auth failures (invalid key / unauthorized / forbidden) make
    // shouldRetryServerLookup return false and stay classified as account errors.
    if (shouldRetryServerLookup(lastMessage)) e.hosterTransient = true;
    if (lastTransient) e.transientNetwork = true;
    throw e;
  }
  throw new Error('Kein Upload-Server erhalten. API-Key pruefen.');
@ -404,7 +411,7 @@ async function _fetchByseFileList(apiKey, signal) {
  // to match the upload we just did against what the server has. The API
  // shape is typical XFS: { status, msg, result: { files: [...] } } or
  // { status, msg, files: [...] }.
-  const url = `https://api.byse.sx/api/file/list?key=${encodeURIComponent(apiKey)}&per_page=100&sort=date&order=desc`;
+  const url = `https://api.byse.sx/file/list?key=${encodeURIComponent(apiKey)}&per_page=100&sort=date&order=desc`;
  try {
    const { body, statusCode } = await request(url, {
      method: 'GET', signal,
@ -573,9 +580,11 @@ async function uploadFile(hosterName, filePath, apiKey, onProgress, signal, thro
    payload = rawBody ? JSON.parse(rawBody) : {};
  } catch {
    const snippet = rawBody ? rawBody.slice(0, 240).replace(/\s+/g, ' ').trim() : '';
-    throw new Error(
+    const err = new Error(
      `Upload-Antwort von ${hosterName} war kein JSON (HTTP ${statusCode}${snippet ? `): ${snippet}` : ')'}`
    );
    if (statusCode >= 500) err.transientNetwork = true;
    throw err;
  }
  // Normalize valid-but-not-object JSON (JSON.parse('null') → null;
  // JSON.parse('"foo"') → string; JSON.parse('[1]') → array). Without this
@ -589,15 +598,19 @@ async function uploadFile(hosterName, filePath, apiKey, onProgress, signal, thro
  }
  if (statusCode < 200 || statusCode >= 300) {
-    throw new Error(
+    const err = new Error(
      payload.msg
      || payload.message
      || `Upload fehlgeschlagen (HTTP ${statusCode}${headers?.['content-type'] ? `, ${headers['content-type']}` : ''})`
    );
    if (statusCode >= 500) err.transientNetwork = true;
    throw err;
  }
  if (payload.status && [401, 403, 429, 500].includes(payload.status)) {
-    throw new Error(payload.msg || payload.message || JSON.stringify(payload));
+    const err = new Error(payload.msg || payload.message || JSON.stringify(payload));
    if (payload.status === 500) err.transientNetwork = true;
    throw err;
  }
  let result = null;
--- a/lib/upload-manager.js
+++ b/lib/upload-manager.js
@ -131,6 +131,7 @@ class UploadManager extends EventEmitter {
  // which takes priority in _shouldSkipRetryOnAccountError.
  _isFileRejectedError(err) {
    if (!err) return false;
    if (err.transientNetwork === true) return false;
    if (err.accountError === true) return false; // explicit account-level wins
    if (err.fileRejected === true) return true;
    if (!err.message) return false;
@ -161,7 +162,9 @@ class UploadManager extends EventEmitter {
  // out for this file without blacklisting the account, so other jobs in the
  // batch still get a fresh chance on it.
  _isTransientNetworkError(err) {
-    if (!err || !err.message) return false;
+    if (!err) return false;
    if (err.transientNetwork === true) return true;
    if (!err.message) return false;
    const m = String(err.message);
    const TRANSIENT = [
      /ENOTFOUND/i,
@ -177,7 +180,11 @@ class UploadManager extends EventEmitter {
      /dns (lookup|error|failed)/i,
      /getaddrinfo/i,
      /fetch failed/i,
-      /\bconnect (ETIMEDOUT|ECONN)/i
+      /\bconnect (ETIMEDOUT|ECONN)/i,
      /HTTP 5\d\d\b/i,
      /Bad Gateway/i,
      /Service Unavailable/i,
      /Gateway Time-?out/i
    ];
    return TRANSIENT.some(p => p.test(m));
  }
@ -188,6 +195,7 @@ class UploadManager extends EventEmitter {
  // or out of quota.
  _shouldSkipRetryOnAccountError(err) {
    if (!err) return false;
    if (err.transientNetwork === true) return false;
    // Explicit account-level flag from hoster parsers — highest priority.
    if (err.accountError === true) return true;
    if (!err.message) return false;
@ -982,6 +990,7 @@ class UploadManager extends EventEmitter {
            if (signal.aborted || this.stopAfterActive) break;
            if (this._isFileRejectedError(err)) break;
            if (this._isHosterTransientError(err)) break;
            if (this._isTransientNetworkError(err)) break;
            if (attempt >= maxAttempts) break;
          }
        }
--- a/package.json
+++ b/package.json
@ -1,6 +1,6 @@
 {
  "name": "multi-hoster-uploader",
-  "version": "3.3.76",
+  "version": "3.3.77",
  "description": "Upload files to doodstream, voe, vidmoly, byse simultaneously",
  "main": "main.js",
  "scripts": {
--- a/tests/byse-reject-recovery.test.js
+++ b/tests/byse-reject-recovery.test.js
@ -40,7 +40,7 @@ test('byse "Not video file format" (suspect) DOES poll recovery and claims the a
  let listCalls = 0;
  requestRouter = async (url, opts) => {
    const u = String(url);
-    if (/\/api\/file\/list/.test(u)) {
+    if (/\/file\/list/.test(u)) {
      listCalls++;
      const body = listCalls === 1
        ? '{"status":200,"result":{"files":[]}}'
@ -68,7 +68,7 @@ test('byse "Not video file format" with empty poll throws err.suspectReject so r
  let listCalls = 0;
  requestRouter = async (url, opts) => {
    const u = String(url);
-    if (/\/api\/file\/list/.test(u)) {
+    if (/\/file\/list/.test(u)) {
      listCalls++;
      if (listCalls >= 2) abort.abort();
      return { statusCode: 200, headers: {}, body: { text: async () => '{"status":200,"result":{"files":[]}}' } };
@ -95,7 +95,7 @@ test('byse "Not video file format" with probe-confirmed NON-video skips the reco
  let listCalls = 0;
  requestRouter = async (url, opts) => {
    const u = String(url);
-    if (/\/api\/file\/list/.test(u)) {
+    if (/\/file\/list/.test(u)) {
      listCalls++;
      return { statusCode: 200, headers: {}, body: { text: async () => '{"status":200,"result":{"files":[]}}' } };
    }
@ -122,7 +122,7 @@ test('byse explicit "Duplicate" rejection still throws fast WITHOUT recovery pol
  let listCalls = 0;
  requestRouter = async (url, opts) => {
    const u = String(url);
-    if (/\/api\/file\/list/.test(u)) {
+    if (/\/file\/list/.test(u)) {
      listCalls++;
      return { statusCode: 200, headers: {}, body: { text: async () => '{"status":200,"result":{"files":[]}}' } };
    }
@ -149,7 +149,7 @@ test('byse empty filecode WITHOUT explicit rejection still polls recovery', asyn
  let listCalls = 0;
  requestRouter = async (url, opts) => {
    const u = String(url);
-    if (/\/api\/file\/list/.test(u)) {
+    if (/\/file\/list/.test(u)) {
      listCalls++;
      const body = listCalls === 1
        ? '{"status":200,"result":{"files":[]}}'
@ -170,3 +170,65 @@ test('byse empty filecode WITHOUT explicit rejection still polls recovery', asyn
  assert.strictEqual(res.file_code, 'RECOVERED99');
  assert.ok(listCalls >= 2, 'recovery polling must run when there is no explicit rejection');
 });
 function stubBysePost(response) {
  requestRouter = async (url, opts) => {
    const u = String(url);
    if (/\/file\/list/.test(u)) {
      return { statusCode: 200, headers: {}, body: { text: async () => '{"status":200,"result":{"files":[]}}' } };
    }
    if (opts && opts.body && typeof opts.body[Symbol.asyncIterator] === 'function') {
      for await (const chunk of opts.body) { if (chunk && chunk.length === -1) break; }
    }
    return response();
  };
 }
 test('byse upload POST 502 (HTML gateway body) is tagged transientNetwork', async () => {
  stubByseUploadServer();
  stubBysePost(() => ({
    statusCode: 502,
    headers: { 'content-type': 'text/html' },
    body: { text: async () => '<!doctype html><html><head><title>502 Bad Gateway</title></head><body>502 Bad Gateway</body></html>' }
  }));
  await assert.rejects(
    () => uploadFile('byse.sx', tmpFile, 'VALIDKEY', null, null, null),
    (err) => err.transientNetwork === true && /kein JSON \(HTTP 502\)/.test(err.message)
  );
 });
 test('byse upload POST non-2xx JSON 503 is tagged transientNetwork', async () => {
  stubByseUploadServer();
  stubBysePost(() => ({
    statusCode: 503,
    headers: { 'content-type': 'application/json' },
    body: { text: async () => JSON.stringify({ status: 503, msg: 'Service Unavailable' }) }
  }));
  await assert.rejects(
    () => uploadFile('byse.sx', tmpFile, 'VALIDKEY', null, null, null),
    (err) => err.transientNetwork === true
  );
 });
 test('byse upload POST 2xx envelope {status:500} is transient; {status:403} stays account-level', async () => {
  stubByseUploadServer();
  stubBysePost(() => ({
    statusCode: 200,
    headers: { 'content-type': 'application/json' },
    body: { text: async () => JSON.stringify({ status: 500, msg: 'Internal Server Error' }) }
  }));
  await assert.rejects(
    () => uploadFile('byse.sx', tmpFile, 'VALIDKEY', null, null, null),
    (err) => err.transientNetwork === true
  );
  stubBysePost(() => ({
    statusCode: 200,
    headers: { 'content-type': 'application/json' },
    body: { text: async () => JSON.stringify({ status: 403, msg: 'Forbidden' }) }
  }));
  await assert.rejects(
    () => uploadFile('byse.sx', tmpFile, 'VALIDKEY', null, null, null),
    (err) => err.transientNetwork !== true
  );
 });
--- a/tests/suspect-reject-alternates.test.js
+++ b/tests/suspect-reject-alternates.test.js
@ -250,4 +250,31 @@ describe('suspect-reject alternate accounts', () => {
    assert.equal(summary.failed, 1);
    assert.equal(mockUploadFile.mock.calls.length, 1);
  });
  it('a transient 5xx (byse 502) retries the SAME account and fails clean — no blacklist, no failover cascade', async () => {
    const mgr = poolMgr([
      { id: 'acc1', apiKey: 'key1' },
      { id: 'acc2', apiKey: 'key2' },
      { id: 'acc3', apiKey: 'key3' }
    ], { retries: 2 });
    mgr._sleep = async () => {};
    mockUploadFile.mock.mockImplementation(async () => {
      const e = new Error('Upload-Antwort von byse.sx war kein JSON (HTTP 502): <!doctype html>');
      e.transientNetwork = true;
      throw e;
    });
    let accountFailed = 0;
    mgr.on('account-failed', () => { accountFailed++; });
    let summary = null;
    mgr.on('batch-done', (s) => { summary = s; });
    await mgr.startBatch([{ file: '/test/big.mkv', hoster: 'byse.sx', apiKey: 'key1', accountId: 'acc1' }]);
    assert.equal(summary.failed, 1);
    assert.equal(accountFailed, 0, 'a transient 502 must never emit account-failed');
    assert.equal(mgr.getFailedAccountKeys().length, 0, 'no account blacklisted on a transient 502');
    const keys = mockUploadFile.mock.calls.map(c => c.arguments[2]);
    assert.ok(keys.length >= 2, 'the 502 is retried on the same account');
    assert.ok(keys.every(k => k === 'key1'), 'every attempt stays on the primary — no cascade to key2/key3');
  });
 });
--- a/tests/upload-manager.test.js
+++ b/tests/upload-manager.test.js
@ -847,6 +847,45 @@ describe('UploadManager', () => {
      assert.equal(mgr._shouldSkipRetryOnAccountError(err), false);
    });
    it('transientNetwork flag is recognised even with an empty/absent message', () => {
      const mgr = new UploadManager({});
      const flagged = new Error('');
      flagged.transientNetwork = true;
      assert.equal(mgr._isTransientNetworkError(flagged), true, 'flag must win before the empty-message guard');
      assert.equal(mgr._isFileRejectedError(flagged), false);
      assert.equal(mgr._isHosterTransientError(flagged), false);
      assert.equal(mgr._shouldSkipRetryOnAccountError(flagged), false);
      const flaggedHtml = new Error('Upload-Antwort von byse.sx war kein JSON (HTTP 502): <!doctype html> forbidden duplicate');
      flaggedHtml.transientNetwork = true;
      assert.equal(mgr._isTransientNetworkError(flaggedHtml), true);
      assert.equal(mgr._shouldSkipRetryOnAccountError(flaggedHtml), false, 'flag overrides any account-keyword in the 502 HTML snippet');
      assert.equal(mgr._isFileRejectedError(flaggedHtml), false, 'flag overrides any rejection-keyword in the 502 HTML snippet');
    });
    it('5xx / gateway errors classify transient by message (defensive fallback), 4xx stay account-level', () => {
      const mgr = new UploadManager({});
      const transient = [
        'Upload-Antwort von byse.sx war kein JSON (HTTP 502): <!doctype html>',
        'Upload fehlgeschlagen (HTTP 503, text/html)',
        'HTTP 504 Gateway Time-out',
        'Bad Gateway',
        'Service Unavailable'
      ];
      for (const msg of transient) {
        assert.equal(mgr._isTransientNetworkError(new Error(msg)), true, `should be transient: ${msg}`);
      }
      const accountLevel = [
        'Upload fehlgeschlagen (HTTP 429, application/json)',
        'HTTP 403 Forbidden',
        'HTTP 401 Unauthorized'
      ];
      for (const msg of accountLevel) {
        assert.equal(mgr._isTransientNetworkError(new Error(msg)), false, `must NOT be transient: ${msg}`);
        assert.equal(mgr._shouldSkipRetryOnAccountError(new Error(msg)), true, `must stay account-level: ${msg}`);
      }
    });
    it('hoster-transient regex fallback catches wrapped doodstream empty-form errors', () => {
      const mgr = new UploadManager({});
      const cases = [