Feature: follow meta refresh redirects

[mrcnski]

Hi, I noticed that spider does not follow meta refresh redirects (without the chrome feature, at least). I also don't see an easy way to enable this behavior.

An example of what I mean, from view-source:https://docs.saltproject.io/:

<meta http-equiv="refresh" content="0; url=en/latest/contents.html" />

[mrcnski]

I tried something like this:

// Follow meta refresh redirects.
fn on_should_crawl_callback(page: &mut Page) -> bool {
    let redirect = get_meta_redirect_url(&page.get_html(), page.get_url());

    if let Some(redirect) = redirect {
        page.final_redirect_destination = Some(redirect);
    }

    true
}
website.on_should_crawl_callback = Some(on_should_crawl_callback);

However, page is an immutable &Page in the callback, so I cannot modify it in the callback.

There also doesn't seem to be anyway to get a reference to website.queue(...) inside the callback. The type signature of the callback restricts it to being either an fn or a closure which does not capture its environment.

[j-mendez]

Possible to handle http too. Will take a look at this soon. We do as much as we can extra to auto parse.