Better surface page retries to users while crawling

[Shrinks99]

Context

Currently errors list if a page can not be loaded, but with crawler 1.5.0, pages that could not be loaded are automatically retried. While crawling, users use the error logs as a record for what ends up in the crawl.

Additionally, reties are currently opaque to users. They occur after the URL list has been fully depleted, but users aren't given any feedback to indicate that this is the case. The crawl continues to run without any feedback about what it is doing or why.

Changes

• Log retries as a warning instead of an error unless the final retry fails to capture the page, then it gets logged as an error
• Add retries to the end of the queue so they are visible to users
• Live exclusions added after a URL has been added to the retry queue will apply to the URLs in the retry queue and they will be excluded accordingly