Search: download version with rclone before indexing

[ericholscher]

index_build currently fetches every HTML page from storage one at a time. New Relic shows this is ~83% of the task's runtime — an average transaction spends 15.3s on 730 serial S3 round-trips (319 GETs + 411 HEADs) and only 73ms on Elasticsearch. Big versions take minutes per build, which held all the shared web/reindex worker slots and caused the recurring web-celery queue spikes (readthedocs/readthedocs-ops#1832 has the incident details).

This downloads the whole version once with rclone copy (parallel transfers, no per-file HEADs) into a temp dir and parses from the local copy. rclone_download_directory mirrors the existing rclone_sync_directory, so it works for both storage backends — and .com's storage class inherits it.

Things to double-check:

• Indexers read page contents lazily through HTMLFile.processed_json, so collect() now runs inside the temp dir context; the injected parser resolves paths relative to the local copy.
• rclone exit code 3 (directory not found) is treated like walking a missing storage path — indexers still run so they clean up previously indexed files. Any other rclone failure aborts the task so a transient storage error can't wipe a version's search index.
• Disk usage on web-celery instances: one version's HTML at a time in $TMPDIR, deleted on exit. Worth a sanity check on p99 version size vs instance disk.
• The search-marked tests (test_imported_file.py) exercise the new path end-to-end in CI; they need ES so they didn't run locally. Storage and parser tests pass locally on 3.14 with real rclone.

🤖 Generated with Claude Code