Server side updates

Author: thibeaum

.github/workflows/update.yml

@@ -1,11 +1,13 @@
 name: Update proxy list
 
+# The scheduled cron was removed in favor of a server-side cron driving
+# scripts/update-and-push.sh (see README "Server cron setup"). GitHub
+# Actions' scheduler is too unreliable at the documented 5-minute minimum —
+# in practice we saw first-run delays of 30–60+ minutes and uneven cadence
+# afterward. The manual dispatch and push triggers remain as a fallback so
+# the workflow can still be invoked from the GitHub UI if the server cron
+# is down.
 on:
-  schedule:
-    # Every 5 minutes. GitHub may delay scheduled workflows under load — the
-    # real upstream cadence is "every minute" on the API, this mirror is a
-    # 5-minute snapshot.
-    - cron: "*/5 * * * *"
   workflow_dispatch:
   push:
     branches: [main]

README.md

@@ -197,14 +197,49 @@ curl 'https://api.proxyscrape.com/v4/free-proxy-list/get?request=display_proxies
 4. Shards by protocol and country and writes `data.{txt,json,csv}` for each
 5. Writes `proxies/stats.json` with the summary
 
-The [`.github/workflows/update.yml`](.github/workflows/update.yml) action runs the script on a 5-minute cron, commits any changes, and pushes. Run it locally too:
+A 5-minute cron on a ProxyScrape-controlled server drives [`scripts/update-and-push.sh`](scripts/update-and-push.sh), which pulls, runs the updater, commits, and pushes. The [GitHub Actions workflow](.github/workflows/update.yml) keeps `workflow_dispatch` and push triggers for manual / emergency runs, but the scheduled cron there is disabled — GitHub's scheduler is too unreliable at 5-minute cadence (we saw 30–60+ minute first-run delays and uneven cadence afterwards).
+
+Run the updater locally:
 
 ```bash
 python3 scripts/update.py
 ```
 
 No dependencies — Python 3.10+, stdlib only.
 
+### Server cron setup
+
+If you want to mirror this dataset yourself or run a fork, the wrapper is drop-in:
+
+```bash
+# 1. Clone the repo on the server
+git clone https://github.com/proxyscrape/free-proxy-list.git /srv/free-proxy-list
+cd /srv/free-proxy-list
+
+# 2. Verify the wrapper runs end-to-end (no commit if there are no changes)
+./scripts/update-and-push.sh
+
+# 3. Install the cron — every 5 minutes
+crontab -e
+```
+
+Add this line:
+
+```cron
+*/5 * * * * /srv/free-proxy-list/scripts/update-and-push.sh >> /var/log/free-proxy-list.log 2>&1
+```
+
+The wrapper:
+- Resolves the repo root from its own location, so `REPO_DIR` is only needed if you symlink the script elsewhere
+- Uses `flock` so a slow tick can't collide with the next one — overlapping crontab firings safely skip
+- Pulls with `--rebase --autostash` first, so concurrent edits (by humans or other servers) stay compatible
+- Only commits + pushes if `proxies/` actually changed
+- Exits non-zero on real failures so cron's `MAILTO` surfaces them
+
+Git push needs credentials available to the cron user. Two common patterns:
+- **Deploy key**: generate an SSH key, add the public half to the repo's Deploy keys (with write access), point the `origin` remote at `git@github.com:proxyscrape/free-proxy-list.git`
+- **Fine-grained PAT**: store in `~/.git-credentials` for the cron user, use HTTPS remote
+
 ## Other free tools from ProxyScrape
 
 If you're working with proxies, you probably want more than just a list. These are all free, no account required:

scripts/update-and-push.sh

@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+#
+# Cron-friendly wrapper around scripts/update.py:
+#   1. Pull any new remote commits (rebase, autostash safety net)
+#   2. Run the Python updater to refresh the proxies/ tree
+#   3. Commit + push only if anything actually changed
+#
+# Designed to be invoked from a crontab — see the "Server cron setup"
+# section in README.md for installation steps.
+#
+# Exits non-zero on any failure so cron's MAILTO surfaces it.
+
+set -euo pipefail
+
+# Resolve the repo root from this script's location so the wrapper works no
+# matter where you put the checkout. Override with REPO_DIR=... if you need
+# to point at a different working tree.
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_DIR="${REPO_DIR:-$(cd "$SCRIPT_DIR/.." && pwd)}"
+
+cd "$REPO_DIR"
+
+# Author for the bot commits. Override at the cron call site if you want a
+# different identity (e.g. a dedicated service account).
+GIT_USER_NAME="${GIT_USER_NAME:-proxyscrape-mirror-bot}"
+GIT_USER_EMAIL="${GIT_USER_EMAIL:-support@proxyscrape.com}"
+
+# Lock to prevent overlapping runs. A long network stall on one tick should
+# never collide with the next. flock returns 1 immediately when the lock is
+# held; that's the correct behavior for a 5-min cron — skip this tick.
+LOCKFILE="${TMPDIR:-/tmp}/free-proxy-list-mirror.lock"
+exec 9>"$LOCKFILE"
+if ! flock -n 9; then
+    echo "[$(date -u +'%Y-%m-%dT%H:%M:%SZ')] Another instance is running; skipping tick" >&2
+    exit 0
+fi
+
+log() {
+    echo "[$(date -u +'%Y-%m-%dT%H:%M:%SZ')] $*"
+}
+
+log "Starting update in $REPO_DIR"
+
+# Pull remote updates first — anyone editing the repo (humans, other servers)
+# stays compatible. --autostash protects against any stray local changes.
+git pull --rebase --autostash --quiet origin main
+
+# Run the updater. Failures here propagate (set -e).
+python3 scripts/update.py
+
+# Stage only the data we publish. The script also writes stats.json under
+# proxies/, so a single `git add proxies/` covers everything we care about.
+git add proxies/
+
+if git diff --cached --quiet; then
+    log "No changes — upstream list is unchanged this tick"
+    exit 0
+fi
+
+TS=$(date -u +"%Y-%m-%d %H:%M:%S UTC")
+git -c user.name="$GIT_USER_NAME" -c user.email="$GIT_USER_EMAIL" \
+    commit --quiet -m "chore(data): automatic update — $TS"
+
+# `--quiet` suppresses progress; stderr still surfaces auth/network errors.
+git push --quiet
+
+log "Pushed update"