Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

urly.fi (cloudflare) always returning 404 #14

Open
hb9eue opened this issue May 17, 2019 · 1 comment
Open

urly.fi (cloudflare) always returning 404 #14

hb9eue opened this issue May 17, 2019 · 1 comment

Comments

@hb9eue
Copy link

hb9eue commented May 17, 2019

Hi, very usefull module.

At the moment I come acress a lot of spam using Cloudflare's shortener urly.fi

Esample from a spam mail: https://urly.fi/1cAF

When accessed manually thought openssl I get a result:

$ openssl s_client -connect urly.fi:443 -servername urly.fi
[...]
GET /1cAF HTTP/1.0
Host: urly.fi

HTTP/1.1 301 MOVED PERMANENTLY
Date: Fri, 17 May 2019 13:40:58 GMT
Content-Type: text/html; charset=utf-8
Connection: close
Set-Cookie: __cfduid=d2d87edbcdde7c2e41c88b24941f433d01558100458; expires=Sat, 16-May-20 13:40:58 GMT; path=/; domain=.urly.fi; HttpOnly
Location: https://besthotvalentines.com/qjqumernhzkmksy?t=goam
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 4d8601770ad0cc36-ZRH

But when I look at the debug output of SpamAssassin firing this module:

May 17 15:54:17.940 [11506] dbg: DecodeShortURLs: URL is not redirect: https://urly.fi/1cAF = 403 Forbidden

So I wonder where this 403 forbidden comes from. Does CloudFlare recognize the 'LWP' User agent and distrusts it? Is it a https problem?

-Benoît-

@MASHtm
Copy link

MASHtm commented Jun 13, 2019

I had the same effect with tiny.cc. I think they block some user-agents and "libwww" is on this list. I added
$self->{ua}->agent('Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0');
to the LWP::UserAgent initialization and then it works.

yitzhaq added a commit to yitzhaq/DecodeShortURLs that referenced this issue Aug 21, 2020
…and probably others behind Cloudflare, and maybe smfreegard#9)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants