Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regular crawling should work when autodiscovery of sitemaps is turned off #1475

Open
tballison opened this issue Feb 21, 2025 · 0 comments
Open

Regular crawling should work when autodiscovery of sitemaps is turned off #1475

tballison opened this issue Feb 21, 2025 · 0 comments
Assignees
@tballison
Labels
bug
Milestone
3.3.0

Comments

@tballison
Copy link
Contributor

tballison commented Feb 21, 2025 •
edited
Loading

When I turned autodiscovery of sitemaps off (via sitemap.discovery: false), I noticed that sites with sitemaps were not getting crawled.

Hat tip and many thanks to @jnioche for confirming this as a bug and pointing to the source of the problem in the code.
@tballison tballison added the bug label Feb 21, 2025
@rzo1 rzo1 added this to the 3.3.0 milestone Feb 21, 2025
tballison added a commit that referenced this issue Feb 21, 2025
@tballison
#1475 -- if sitemap discovery is turned off, links should be extracted
8f61234
tballison added a commit that referenced this issue Feb 21, 2025
@tballison
#1475 -- cleanup
79f5188
tballison added a commit that referenced this issue Feb 21, 2025
@tballison
#1475 -- code style
7cbacda
tballison added a commit that referenced this issue Feb 24, 2025
@tballison
#1475 -- move logic to SitemapFilter
8895f97
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tballison tballison

Labels
bug
Projects
None yet
Milestone
3.3.0
Development

No branches or pull requests
2 participants
@tballison @rzo1