Skip to content

feat: add opt-in auto-sync scheduler for free-proxy sources (#7079)#7716

Merged
diegosouzapw merged 2 commits into
release/v3.8.49from
feat/7079-free-proxy-autosync
Jul 19, 2026
Merged

feat: add opt-in auto-sync scheduler for free-proxy sources (#7079)#7716
diegosouzapw merged 2 commits into
release/v3.8.49from
feat/7079-free-proxy-autosync

Conversation

@diegosouzapw

Copy link
Copy Markdown
Owner

Summary

Free-proxy provider sources (iplocate, proxifly, 1proxy, webshare) only ever ran sync() via a manual POST /api/settings/free-proxies/sync. There was no periodic re-fetch of new IPs — only proxyHealth/scheduler.ts periodically tested what was already in the pool. Since free-proxy lists rotate hourly, a manually-seeded pool went stale fast.

This adds a second, opt-in background scheduler, src/lib/freeProxyProviders/scheduler.ts, modeled on two existing scheduler idioms already proven in this codebase:

  • src/lib/proxyHealth/scheduler.tsglobalThis-guarded interval + isBuildProcess() / isBackgroundServicesDisabled() guards.

  • src/shared/services/providerLimitsSyncScheduler.tsisRunning reentrancy guard + elapsed-since-last-run initial delay + .unref()'d timers.

  • New src/lib/freeProxyProviders/syncCycle.ts extracts the per-provider sync loop (previously inline in sync/route.ts) into a shared runFreeProxySyncCycle() helper. Both the manual route and the new scheduler now go through this exact same code path, preserving the existing per-source error isolation ([BUG] Free proxy pool sync silently fails — 'Total: 0' with no error surfaced (Proxifly TLS fail, IPLocate 404) #5595) and the always-advancing sync timestamp ([BUG] Fix proxy pool UI sync bugs, missing bulk loaders, and constant Redis log spam #4878).

  • src/lib/freeProxyProviders/scheduler.ts — the new scheduler. Config: FREE_PROXY_AUTO_SYNC_ENABLED (default "false" — opt-in) and FREE_PROXY_AUTO_SYNC_INTERVAL_MS (default 30min, floor-clamped to 5min as outbound courtesy since 3 of the 4 sources have no in-module TTL guard). Skips scheduling entirely when no provider is enabled. Wired into src/instrumentation-node.ts right after the existing proxy-health scheduler import.

  • GET /api/settings/free-proxies/stats now also returns autoSync: { enabled, intervalMs }; lastSyncAt was already origin-agnostic so no change was needed there.

No DB migration, no dashboard UI changes (backend-only per the plan's open questions), no i18n keys (no new user-facing copy).

How it was validated (Hard Rule #18 — TDD)

Two new network-free test files (providers/timers are mocked/injected via test seams, matching the existing _setProvidersForTests idiom):

Also re-ran the existing regression suites that the extraction touches, to prove zero behavior change:

node --import tsx/esm --test tests/unit/free-proxy-auto-sync-scheduler.test.ts tests/unit/free-proxy-sync-cycle.test.ts tests/unit/proxy-pool-sync-4878.test.ts tests/unit/api/free-proxies-route.test.ts
# 26 pass, 0 fail

Gates run locally, all green: check-test-discovery.mjs, typecheck:core, typecheck:noimplicit:core (2 pre-existing unrelated errors in open-sse/services/combo.ts / open-sse/utils/usageTracking.ts / src/shared/services/cliRuntime.ts, confirmed present on the base branch), eslint (0 errors on changed files), check:complexity-ratchets (unchanged from baseline), check-file-size.mjs (no on touched files), check:cycles, check:docs-sync.

npm run test:coverage was intentionally not run locally (shared/loaded box) — this change is additive-only (2 new modules + 2 new test files; the only modified production files had a pure extraction and a 2-field additive JSON response change), so it should not regress the coverage ratchet; CI runs it authoritatively.

Closes #7079

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@diegosouzapw
diegosouzapw merged commit b6ba144 into release/v3.8.49 Jul 19, 2026
12 of 20 checks passed
@diegosouzapw
diegosouzapw deleted the feat/7079-free-proxy-autosync branch July 19, 2026 21:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(backend): scheduled auto-sync of free-proxy sources (dynamic self-refreshing pool)

1 participant