feat: add opt-in auto-sync scheduler for free-proxy sources (#7079)#7716
Merged
Conversation
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Free-proxy provider sources (
iplocate,proxifly,1proxy,webshare) only ever ransync()via a manualPOST /api/settings/free-proxies/sync. There was no periodic re-fetch of new IPs — onlyproxyHealth/scheduler.tsperiodically tested what was already in the pool. Since free-proxy lists rotate hourly, a manually-seeded pool went stale fast.This adds a second, opt-in background scheduler,
src/lib/freeProxyProviders/scheduler.ts, modeled on two existing scheduler idioms already proven in this codebase:src/lib/proxyHealth/scheduler.ts—globalThis-guarded interval +isBuildProcess()/isBackgroundServicesDisabled()guards.src/shared/services/providerLimitsSyncScheduler.ts—isRunningreentrancy guard + elapsed-since-last-run initial delay +.unref()'d timers.New
src/lib/freeProxyProviders/syncCycle.tsextracts the per-provider sync loop (previously inline insync/route.ts) into a sharedrunFreeProxySyncCycle()helper. Both the manual route and the new scheduler now go through this exact same code path, preserving the existing per-source error isolation ([BUG] Free proxy pool sync silently fails — 'Total: 0' with no error surfaced (Proxifly TLS fail, IPLocate 404) #5595) and the always-advancing sync timestamp ([BUG] Fix proxy pool UI sync bugs, missing bulk loaders, and constant Redis log spam #4878).src/lib/freeProxyProviders/scheduler.ts— the new scheduler. Config:FREE_PROXY_AUTO_SYNC_ENABLED(default"false"— opt-in) andFREE_PROXY_AUTO_SYNC_INTERVAL_MS(default 30min, floor-clamped to 5min as outbound courtesy since 3 of the 4 sources have no in-module TTL guard). Skips scheduling entirely when no provider is enabled. Wired intosrc/instrumentation-node.tsright after the existing proxy-health scheduler import.GET /api/settings/free-proxies/statsnow also returnsautoSync: { enabled, intervalMs };lastSyncAtwas already origin-agnostic so no change was needed there.No DB migration, no dashboard UI changes (backend-only per the plan's open questions), no i18n keys (no new user-facing copy).
How it was validated (Hard Rule #18 — TDD)
Two new network-free test files (providers/timers are mocked/injected via test seams, matching the existing
_setProvidersForTestsidiom):tests/unit/free-proxy-sync-cycle.test.ts— the extractedrunFreeProxySyncCycle()helper: per-source error isolation ([BUG] Free proxy pool sync silently fails — 'Total: 0' with no error surfaced (Proxifly TLS fail, IPLocate 404) #5595), timestamp always advances ([BUG] Fix proxy pool UI sync bugs, missing bulk loaders, and constant Redis log spam #4878), error clearing on a subsequent success,providersdefaulting togetEnabledProviders().tests/unit/free-proxy-auto-sync-scheduler.test.ts— the scheduler: disabled-by-default (no timer, sync never called), enabled-but-no-providers skips scheduling, interval floor-clamping,isBuildProcess()/OMNIROUTE_DISABLE_BACKGROUND_SERVICESguards, reentrancy guard (a second concurrent cycle never overlaps the first — asserted via a manually-gated promise), and the initial-delay computation (mock.timers) shortening the first tick when a recentlastSyncAtexists.Also re-ran the existing regression suites that the extraction touches, to prove zero behavior change:
Gates run locally, all green:
check-test-discovery.mjs,typecheck:core,typecheck:noimplicit:core(2 pre-existing unrelated errors inopen-sse/services/combo.ts/open-sse/utils/usageTracking.ts/src/shared/services/cliRuntime.ts, confirmed present on the base branch),eslint(0 errors on changed files),check:complexity-ratchets(unchanged from baseline),check-file-size.mjs(no✗on touched files),check:cycles,check:docs-sync.npm run test:coveragewas intentionally not run locally (shared/loaded box) — this change is additive-only (2 new modules + 2 new test files; the only modified production files had a pure extraction and a 2-field additive JSON response change), so it should not regress the coverage ratchet; CI runs it authoritatively.Closes #7079