update from Gluejar - #94
Open
eshellman wants to merge 1226 commits into
Open
Conversation
add openlibrary switch
check for empty images
fix image checking
deleted fixed doabs
simplify sitemap, remove expensive cover_image query
5% improvement
Maintenance 2024
fix downloads, add 3 cmp providers
Maintenance 2025
update MetadataReader for DOAB OAI feed
fix bugs in handle doab oai
A staging/test box whose database has been refreshed from a copy of production holds real users' real email addresses. Nothing today stops that box's normal mail-sending code (password resets, gift notices, campaign emails) from actually delivering to those real people -- found live on test.unglue.it 2026-08-31 while verifying #1237, and a milder version of the same class of problem already bit Eric once (regluit-provisioning#22, Feb 2026). Adds utils/safe_email_backend.py: AllowlistEmailBackend wraps the real EMAIL_BACKEND and, per outgoing message, either lets it through unchanged (every recipient is on an explicit allowlist) or redirects the whole message to a configured catch-all address, with the original recipients preserved in the subject/body so a tester can still see what would have gone out. Refuses to send (raises) rather than silently dropping or silently delivering to an unknown address when no redirect target is configured -- silent drops are exactly what made #1164 and regluit-provisioning#22 both take longer to diagnose than necessary. Wired in settings/common.py behind EMAIL_SAFE_MODE (env var, default off) so production and any environment that hasn't explicitly opted in are completely unaffected -- this only activates where something upstream (ansible group_vars, a hand-set env var) turns it on. This is a backstop, not the primary fix: RY's preference (2026-08-31, on #1238) is a synthetic/scrubbed test DB rather than a raw prod copy in the first place -- that's a bigger, separately-scoped follow-up (needs a full PII-field survey across core/libraryauth/payment). This backend stays valuable regardless, for whatever scrubbing misses or for a deliberate one-off raw-copy debugging session. Verified: `manage.py test libraryauth frontend utils` -- 63 tests, 3 failures + 2 errors, the same pre-existing baseline set as origin/master (diffed in #1237's session) -- zero net-new failures, all 13 new tests pass. 🤖 Generated with Claude Code Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015uv8CgcRqX4qjx9C74bVHj
CC review on PR #1239 found two real issues: 1. utils/tests.py's test_email_safe_mode_off_by_default asserted settings.EMAIL_BACKEND != AllowlistEmailBackend from inside a TestCase -- but Django's test runner (setup_test_environment) unconditionally overwrites EMAIL_BACKEND to the locmem backend before any test body runs, so that assertion could never fail regardless of whether the underlying logic was actually correct. 2. settings/common.py sets EMAIL_BACKEND at the end of the file, but any settings module doing `from .common import *` and then re-setting EMAIL_BACKEND afterward (settings/spike.py already does exactly this, for an unrelated reason) would silently defeat the safety net. Verified directly against the actual deploy template (regluit-provisioning/roles/regluit_prod/templates/prod.py.j2, what test.unglue.it/unglue.it really run as regluit.settings.prod) -- it does NOT re-set EMAIL_BACKEND, so the fix does reach the box it's meant to protect. That's an external file this repo doesn't control, though, so documented it as a live risk rather than something this commit can close for good. Fix for (1): pulled the EMAIL_SAFE_MODE decision out of settings/common.py into a plain function, resolve_email_backend() (utils/safe_email_backend.py), that settings/common.py now calls. Testing it directly -- no settings module, no test runner -- is what actually exercises the on/off/override logic; the previous test couldn't. Replaced the one vacuous test with four real ones covering off/on/override/os.environ-default. Verified: `manage.py test regluit.utils` -- 16 tests, all pass. `manage.py test libraryauth frontend utils` -- 66 tests, still the same 3 pre-existing failures + 2 pre-existing errors as origin/master, zero net-new. Also confirmed live: with EMAIL_SAFE_MODE=true in the environment, `settings.EMAIL_BACKEND` really does resolve to AllowlistEmailBackend and SAFE_EMAIL_REAL_BACKEND to the SMTP backend. 🤖 Generated with Claude Code Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015uv8CgcRqX4qjx9C74bVHj
…Backend Codex review on PR #1239 found a real security bypass plus several correctness/design gaps: - HIGH: _redirect_if_needed() checked message.to/.cc/.bcc directly, but Django's actual delivery target is message.recipients() -- a message subclass overriding recipients() could add a hidden real recipient that never appears in .to/.cc/.bcc, sailing through unredirected. Fixed by checking recipients() instead, plus a belt-and-suspenders post-rewrite check: if redirected.recipients() isn't exactly [redirect_to] after rewriting .to/.cc/.bcc (meaning the override defeated the rewrite too), refuse to send rather than guess it worked. - MEDIUM: AllowlistEmailBackend didn't inherit django.core.mail.backends.base.BaseEmailBackend, so `with backend:` (Django's documented connection-reuse pattern) raised TypeError. Now subclasses it properly. - MEDIUM: the "raise loudly" refusal-to-send path is invisible for Celery-queued mail, since core/tasks.py's send_mail_task() catches every exception without re-raising. Added an ERROR-level log call right before the raise, so it's at least visible in application logs even where the exception itself gets swallowed. - LOW: _is_allowed() didn't parse "Display Name <addr>" recipient strings, so a legitimately-allowlisted address in that form would fail-safe into an unnecessary redirect. Now uses email.utils.parseaddr to extract the bare address first. - Privacy: moved the real recipient list out of the redirected message's subject (subjects are far more likely than bodies to end up in SMTP logs / mailbox indexing) into the body only; subject is now a generic "[STAGING - redirected...]" marker. Also acknowledged, not fixed here (out of scope for this PR, per its own description): deploying this to an actual box -- setting EMAIL_SAFE_MODE=true + an allowlist/redirect address for both the web process and Celery -- is separate provisioning-side work. Test changes: fixed the mislabeled "cc_and_bcc" test (it only tested cc) into separate cc/bcc tests; added tests for the recipients() override bypass (now caught), context-manager support, display-name address parsing, and subject/body PII placement. Verified: `manage.py test regluit.utils` -- 21 tests, all pass. `manage.py test libraryauth frontend utils` -- 71 tests, still the same 3 pre-existing failures + 2 pre-existing errors as origin/master, zero net-new. 🤖 Generated with Claude Code Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015uv8CgcRqX4qjx9C74bVHj
…II in logs
Codex re-reviewed the fixes from the previous commit and verified two
of them (recipients() bypass, BaseEmailBackend inheritance) actually
work, but found three more real issues -- all confirmed live, not
just theoretical:
1. MEDIUM: the ERROR log call added to make a Celery-swallowed send
refusal visible was itself silently doing nothing.
settings/common.py's LOGGING has disable_existing_loggers=True, and
'regluit.utils.safe_email_backend' had no explicit entry in
LOGGING['loggers'] -- Django disables any logger not explicitly
listed there. Verified directly:
logging.getLogger('regluit.utils.safe_email_backend').disabled was
True before this fix, False after. Added an explicit loggers entry
(matching the existing regluit.downloads pattern).
2. MEDIUM: _is_allowed() had a fail-open bypass for malformed compound
values. parseaddr('real@example.com, ok@ebookfoundation.org')
returns ('', '') -- an unparseable *non-empty* input -- which the
old code treated identically to "genuinely empty recipient slot"
and let through unredirected. Verified live before/after. Now
distinguishes "nothing there" (harmless) from "something there
that couldn't be parsed" (fails closed).
3. LOW: the new ERROR log call and both RuntimeError messages embedded
the actual recipient addresses -- recreating the same PII-in-logs
exposure the subject/body split (previous commit) existed to avoid.
All three now report a count instead of the address list.
Verified: `manage.py test regluit.utils` -- 23 tests, all pass.
`manage.py test libraryauth frontend utils` -- 73 tests, still the
same 3 pre-existing failures + 2 pre-existing errors as origin/master,
zero net-new. Manually replicated Codex's exact live probes for all
three findings (logger.disabled, parseaddr on the malformed string)
before and after this fix to confirm each one.
🤖 Generated with Claude Code
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015uv8CgcRqX4qjx9C74bVHj
…sal log Round-2's fix dropped the recipient list from the log/exception text but added message.subject as a "safe" identifier instead -- Codex round 3 correctly pointed out a subject can itself carry PII (e.g. "Password reset for real@example.com"), recreating the exact exposure the subject/body split (two commits back) exists to avoid. Dropped the subject too; the log line is now just a bare recipient count. Added test_refusal_log_line_carries_no_pii, which uses assertLogs to capture the actual emitted log record (not just the exception text) and asserts neither the recipient address nor a PII-bearing subject appear in it. Not fixed, deliberately: round 3 also suggested _is_allowed() should do full RFC 5322 mailbox validation (parseaddr() accepts some oddly-formed-but-still-matching values, e.g. a quoted local-part or a trailing comma, that a stricter parser would reject). No concrete example was found where this lets a genuinely non-allowlisted address through -- every example given was an allowed address in an unusual form. Given three rounds have now closed every demonstrated bypass, treating this as a documented stopping point rather than open-ended hardening of a backstop feature. Verified: `manage.py test regluit.utils` -- 24 tests, all pass. `manage.py test libraryauth frontend utils` -- 74 tests, still the same 3 pre-existing failures + 2 pre-existing errors as origin/master, zero net-new. 🤖 Generated with Claude Code Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015uv8CgcRqX4qjx9C74bVHj
Add opt-in email allowlist/redirect backend for non-prod (backstop for #1238)
Password managers can auto-submit the login form right after autofill. When the user then clicks "Sign in with Password" themselves, the second POST carries the pre-login CSRF token (Django rotates the CSRF cookie on successful login), so the user lands on a bare 403 even though the first POST already logged them in. Confirmed live on prod 2026-08-31: an isTrusted:false synthetic click submitted the form 1.2s before the user's real click; every 302 login that day was paired with a 403 one second later. The guard is a document-level delegated submit listener in sitewide1.js, scoped to forms posting to /accounts/superlogin/: the first submission marks the form, later ones are prevented while it is in flight. It must live in sitewide JS rather than the form template because the sign-in lightbox is injected via jQuery .load(url + " #lightbox_content"), which strips inline <script> tags. Smoke tests pin the wiring: the login page loads sitewide1.js and the form still posts to the guarded action. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqzfARuryr6m6jzGscgAY6
…behavioral tests - Blocker: the 8s timed unlock could itself recreate the 403 (unlocking while a slow login navigation is still pending lets a resubmission carry the stale token). Removed; the lock now holds until navigation or bfcache restore. - Lock is module-global instead of per form node: standalone page + lightbox copies (or a reopened lightbox) are all locked by one login in flight. - Split into capture-phase blocker + bubble-phase acquirer, so a submission some other handler cancels never acquires the lock, and a locked submission is stopped before other handlers run. - Action matched by resolved pathname (URL()), not substring. - pageshow handler: a bfcache-restored page still holds the lock and a pre-login CSRF token, so reload it for fresh state. - Real behavioral coverage: dependency-free Node test with DOM fakes (static/js/tests/login_guard_test.js, 9 cases) extracting the guard between source markers; run from the Django suite when node exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqzfARuryr6m6jzGscgAY6
… rollback - Acquire the lock in the document CAPTURE listener (runs before any target/ancestor handler can stopPropagation the event away), with a deferred post-dispatch check that releases it if a later handler cancelled the submission -- fixes both the stuck-lock-on-late-cancel and the stopPropagation acquisition bypass. - Blocked submissions get stopImmediatePropagation() so downstream submit handlers cannot run side effects for them, and the blocked form's own button is disabled immediately (visible stuck-state even for a freshly reopened lightbox form). - isLoginForm now also requires same-origin, not just pathname. - Test harness models real propagation (doc capture -> target -> doc bubble, honoring preventDefault/stopPropagation/stopImmediate- Propagation, default action decided post-dispatch); 14 cases including one per round-2 finding. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqzfARuryr6m6jzGscgAY6
…tter-aware - Finding 1 (deferred defaultPrevented cannot prove "no POST" -- a handler can cancel-and-replace via form.submit(), and the canceled flag can mutate after the default-action decision): adopted Codex's recommended fix for this codebase -- the lock is now FAIL-CLOSED, held until navigation or bfcache restore, with no timer rollback. A synchronous defaultPrevented check still skips acquisition for events cancelled before the guard runs. - Finding 2 (disconnected form aborts submission, sticking the lock): subsumed by fail-closed; documented as an accepted limitation (no such handler exists; a stuck form is visible and reload fixes it). - Finding 3 (window-capture listener bypass): guard moved from document capture to WINDOW capture, the earliest propagation point. - Finding 4 (stopPropagation fallback weaker than claimed): fallback dropped; stopImmediatePropagation is universal in supported browsers. - Finding 5 (submitter/formaction): event.submitter preferred for the button to disable, querySelector fallback kept for implicit Enter-key submission. - Harness models window capture -> doc capture -> target -> doc bubble -> window bubble, plus pre-guard window-capture listeners; 16 cases. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqzfARuryr6m6jzGscgAY6
Guard login form against password-manager double-submit (403 CSRF on login) (#1240)
) Pay.__init__ used token-presence alone to decide "user is anonymous" and dropped transaction.user when calling make_account, even though a logged-in donor entering fresh card details always carries a token (no Account exists yet). Every such donation created a Stripe Customer mislabeled "anonymous user" with no Account linked back to the donor's profile (so saved-card reuse never worked either). Now passes user=transaction.user through (None for genuinely anonymous transactions, so that path is unchanged) -- make_account's existing if-user branch already does the right thing once given one. Reproduced and verified with the repo's mocked-StripeClient test pattern (no live Stripe calls): confirmed the new test_logged_in_donation_creates_identified_customer test fails against the pre-fix code and passes after; the anonymous-donation case is covered by a paired test and stays unchanged. payment+frontend suites hold at the pre-existing 3-failure/1-error/2-skipped baseline (re-verified against unmodified master), zero net-new failures. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MHjXjrfha4X6TTSzY5obUZ
Codex round-1 confirmed the fix is correct and the tests genuinely catch #1125, and flagged one out-of-scope finding: Pay.__init__ hands Execute the just-created account_id via transaction.preapproval_key, but Execute prefers a fresh transaction.user.profile.account DB lookup over it -- a narrow race (two concurrent submissions for the same user) could charge a different, concurrently-created Customer than the one this call just created/labeled. Filed as #1249 rather than folded into this narrowly-scoped fix: it's a pre-existing Execute account-resolution characteristic shared with delayed/queued campaign executions, not something #1125's Pay.__init__ change introduces. Took Codex's cheap, in-scope coverage suggestions instead: test_logged_in_donation_creates_identified_customer now uses a token-shaped (not raw-dict) value, asserts create_customer got card=<token>, asserts create_charge was called with customer=<the newly-created Customer's id> (so a post-Customer-creation failure wouldn't go unnoticed), and asserts the transaction actually reaches TRANSACTION_STATUS_COMPLETE with the right pay_key. payment+frontend suites still hold at the pre-existing 3F/1E/2S baseline. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MHjXjrfha4X6TTSzY5obUZ
Codex round-2 verdict: ship, no blocking issues -- confirmed the concurrency race deferral to #1249 is the right scope call, confirmed the fix is correct, confirmed the hardened test genuinely detects the regression (reran against pre-fix stripelib.py, failed as expected). Only optional polish suggested: assert_called_once() on the mocked create_customer/create_charge calls. Applied. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MHjXjrfha4X6TTSzY5obUZ
The AI-crawler rules added in the first commit had a precedence bug. A crawler obeys only the single most specific User-agent group that matches it and ignores "User-agent: *" entirely (RFC 9309). The ClaudeBot group carried only Crawl-delay, so declaring it would have *removed* ClaudeBot's existing Disallow rules for /accounts/, /feedback/, /socialauth/, /search/ and /googlebooks/ -- widening access to the faceted-search space whose URL explosion #1116 was cut to fix, rather than narrowing it. Every named group that is not a blanket "Disallow: /" now restates the baseline rules. ClaudeBot additionally excludes the listing and feed endpoints identified in #1189 as the expensive bot-hit paths (/free/, /bypub/, /pid/, /unglued/, /campaigns/, /api/) while leaving /work/ pages crawlable, so the throttle sheds load without costing discoverability. Verified Anthropic's published crawler documentation rather than relying on the earlier claim: ClaudeBot does support the non-standard Crawl-delay extension, so throttling it is meaningful. GPTBot and the other training crawlers document no such support and stay fully disallowed. Added Google-Extended and Applebot-Extended, which are AI-training opt-out tokens and carry no search-ranking effect. Googlebot and user-triggered agents (Claude-User, ChatGPT-User, OAI-SearchBot, Claude-SearchBot) remain unrestricted. Removed the inline "DECISION FOR REVIEW" comments now that the decisions are made, and dropped the traffic statistics from the served file -- this template is public output, so it carries policy, not operational detail. Tests: adds RobotsTxtTests (SimpleTestCase, no database) covering the production and non-production host branches, and a regression guard asserting that no named group omits the baseline rules. The guard was confirmed to fail against the previous version of this template and pass against this one. Full frontend suite: 40 tests, 3 failures, all three pre-existing on the branch without this change (RhPageTests, unrelated). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ANtyDhUBYCsvEytDfMXUJe
Codex round 1 caught a real error I carried over from the original draft without checking it: PerplexityBot was grouped with the training crawlers and given Disallow: /, which contradicts this PR's own stated policy of keeping AI *search* discovery enabled. Verified against Perplexity's published crawler documentation rather than assuming either way: PerplexityBot is their search-indexing crawler and is explicitly "not used to crawl content for AI foundation models"; the on-demand fetcher is Perplexity-User. Blocking it would have removed the collection from Perplexity search results while shedding no training load at all -- pure cost, no benefit. Removed the group so it falls through to User-agent: *. Checked Amazonbot the same way while here, since it was inherited from the same list: Amazon documents it as crawling that "may be used to train Amazon AI models," so it stays disallowed, correctly. Their separate Amzn-SearchBot does not crawl for generative training and already falls through untouched. Tests: extended the absence assertions from four agents to eight, so a future edit cannot quietly give a search or user-triggered agent its own group -- PerplexityBot, Perplexity-User, Amzn-SearchBot and Claude-SearchBot are now covered alongside the originals. This is the assertion that would have caught the PerplexityBot mistake. RobotsTxtTests: 3 tests, all passing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ANtyDhUBYCsvEytDfMXUJe
…C wording Three findings, all fair. Taken in order of consequence: 1. Removed Google-Extended and Applebot-Extended. Codex flagged that Disallow: / on Google-Extended also costs Gemini grounding, not only training. Following that through: neither token is a crawler at all -- Googlebot and Applebot do the fetching, and these are content-*usage* opt-outs. So they shed exactly zero crawl load while carrying a real AI-answer discoverability cost. That is the same "pure cost, no benefit" shape as the PerplexityBot entry removed in the previous commit, and it is a content-policy decision rather than a load question, so it does not belong in a load-shedding change. Dropped from this PR; worth deciding deliberately on its own terms. 2. Tests now assert the rules this PR actually exists for. The previous version checked Crawl-delay but never ClaudeBot's six expensive-path exclusions, and covered only three of the blocked agents -- deleting /api/ or the Bytespider stanza would have passed. Added CLAUDEBOT_EXTRA_DISALLOWS and BLOCKED_AGENTS constants, asserted every entry, and added an exhaustive equality check on the set of groups so a stanza cannot be dropped or slipped in unnoticed. Verified by deleting /api/ and Bytespider and confirming the suite fails. 3. Corrected the RFC 9309 explanation in the template comment and the test docstring. Multiple matching groups are combined rather than one winning; the accurate and load-bearing point is that "User-agent: *" applies only to crawlers matching no named group. Behavior was already correct -- the explanation could have misled a future edit. RobotsTxtTests: 3 tests, all passing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ANtyDhUBYCsvEytDfMXUJe
Four findings, all verified against primary sources before acting rather
than taken on the reviewer's word.
1. The tests were not actually database-free, and the docstring said they
were. Issuing the client request loads the root URLconf, which imports a
module that runs a query at import time. Under SimpleTestCase Django
creates no test database, so that query hit whatever database the
settings pointed at -- meaning the "isolated" test could read live
application data. Switched to TestCase so Django builds an isolated test
database, and removed the no-database claim. Kept the real URLconf rather
than substituting a minimal one, so the test still proves /robots.txt is
actually routed.
2. CCBot was blocked under a stated rationale that is false. Common Crawl
documents that CCBot honors Crawl-delay ("By increasing that number, you
will indicate to CCBot to slow down"), so the comment claiming these
crawlers offer nothing to throttle was wrong for it. Behavior is
unchanged -- CCBot is a real crawler and blocking it does shed load --
but the comment now says plainly that this is a deliberate opt-out from
the Common Crawl dataset rather than a technical necessity, and notes it
could be softened to a throttle. That is a content-policy call worth
making deliberately.
3. Removed cohere-ai. Cohere's own crawler documentation states they do not
operate crawlers for training generative models "at this time" and names
no active user agent; Coherebot appears only as a hypothetical example.
The token sheds no load and, pinned into an exhaustive test, implied a
protection that does not exist.
4. Fixed the test parser's group boundaries. It treated any non-User-agent
field as ending a run of user-agent lines, so an extension record such as
Crawl-delay between two consecutive User-agent lines would have split one
group into two -- and the precedence guard would then silently stop
covering the second agent. Now only Allow/Disallow close the run, per RFC
9309 2.2.4. Added a fixture covering exactly that shape.
RobotsTxtTests: 4 tests, all passing. Full frontend suite: 41 tests, 3
failures, the same three pre-existing RhPageTests failures unrelated to
this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ANtyDhUBYCsvEytDfMXUJe
Diffbot documents Crawl-delay support too, so the comment naming CCBot as the sole exception was wrong again -- the fourth round in a row to catch a factual error in a per-crawler capability claim. Rather than patch in "and Diffbot" and wait for the next round to find another, removed the class of claim entirely. The comment now states the policy (this is a content-usage decision to opt out of training-data collection), notes plainly that several of these crawlers do honor Crawl-delay and could be throttled instead, and tells the next editor to check the operator's current documentation before adding or moving an entry. That is accurate regardless of which agents support what, and it cannot rot the way the enumerated version kept doing. No behavior change: the same agents are disallowed. Same correction applied to the BLOCKED_AGENTS comment in the tests. RobotsTxtTests: 4 tests, all passing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ANtyDhUBYCsvEytDfMXUJe
Round 5 returned "no blocking defects found" with three P3s. Took two, declined one. Taken: 1. The precedence guard checked only that named groups restate the baseline Disallow rules, and discarded Allow rules entirely. RFC 9309 gives the longest match precedence, so a future "Allow: /accounts/public/" would re-open a path under the shorter "Disallow: /accounts/" while the guard still passed. The parser now records Allow rules and the guard asserts no named group allows anything beneath a baseline prefix. Verified by injecting exactly that rule and confirming the suite fails. 2. Corrected two comments that claimed more than is true. "Every training crawler is blocked" was wrong -- ClaudeBot trains and is deliberately throttled instead; it now reads "every crawler selected for blocking" and says why. The template's allowed-agents note implied those publishers collect no training data at all, which glosses over the Google-Extended grounding tradeoff; it now scopes the claim to crawling and states that training opt-outs are a separate content-usage decision not made here. Declined, and flagged for a separate change instead: 3. Host matching. Because the template compares raw HTTP_HOST, "unglue.it:443" and "Unglue.it" fall to the non-production branch and would serve "Disallow: /". That is real, but it is pre-existing behavior of the prod/non-prod gate rather than anything this PR introduces, it is orthogonal to AI crawlers, and getting it wrong in either direction is costly -- too strict and prod tells crawlers to go away, too loose and staging gets indexed. It deserves its own change with its own test, not a quiet edit inside a crawler-policy PR. The same applies to "www.unglue.it". RobotsTxtTests: 4 tests, all passing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ANtyDhUBYCsvEytDfMXUJe
robots.txt: throttle ClaudeBot, disallow training crawlers, keep search agents (refs #1172)
Pass transaction.user through to make_account for donation tokens (#1125)
…s (refs #1253) meta-webindexer was the largest single source of /free/ facet-browse requests on 2026-09-09 (48,318 of 86,287), and it fell through to the permissive "User-agent: *" group. Meta documents it as the crawler behind Meta AI search results, so it is treated like the other search-indexing crawlers here: not blocked. It gets a ClaudeBot-style group that restates the baseline rules and excludes the same expensive listing and feed endpoints, so work pages stay indexable. No Crawl-delay, since Meta does not document support for it. Test: RobotsTxtTests asserts the new group's rules and adds it to the exhaustive group set. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DjPGJQCBFvZ1D2fJgJQpe4
- Test asserts meta-webindexer's exact Disallow set, an empty Allow list and no extension records, so an added Allow, an over-broad Disallow, a Crawl-delay or a duplicate stanza now fails (each verified by mutation). - robots.txt comment: "not blocked outright", "work pages remain crawlable". - Fix an older test comment that claimed blocking a search crawler sheds no crawl load; #1253 shows one can. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DjPGJQCBFvZ1D2fJgJQpe4
- Assert there is no crawl-delay record for meta-webindexer, rather than that it has no extension records at all: a legitimate Sitemap line after the stanza must not fail the test (verified: passes; Crawl-delay still fails). - Comment: exact equality catches a duplicated copy of the rules, which is what it actually guarantees. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DjPGJQCBFvZ1D2fJgJQpe4
robots.txt: keep meta-webindexer out of /free/ and other listing pages (refs #1253)
Adds Crawl-delay: 10 to the meta-webindexer group as a labeled experiment. Meta documents no Crawl-delay support, but the crawler has run at a flat ~7,200 requests/hour since 2026-09-09, so production logs will show whether it honors the line. Also records the observation that Meta's crawlers ignore the "*" group while obeying a group that names them. RobotsTxtTests now requires exactly one Crawl-delay of 10 in that group (mutation-checked: missing, 5, and duplicated all fail). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DjPGJQCBFvZ1D2fJgJQpe4
- The named-group observation is stated for meta-externalagent only, and the /feedback/ observation without "for weeks". - The experiment note says a rate change would be assessed, not proven, and records the baseline start as 2026-09-09 12:00 UTC. Comment-only change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DjPGJQCBFvZ1D2fJgJQpe4
…l-delay robots.txt: Crawl-delay experiment for meta-webindexer (refs #1253)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.