fix: handle quoted charset and non-canonical content-type in title extraction

[krishna28238-arch]

httpx silently drops the title, or extracts it as mojibake, when the Content-Type response header deviates from the canonical lowercase form:

• Content-Type: text/html; charset="gbk" — the quoted charset value (legal per RFC 9110) is not matched by the charset=gbk substring check in DecodeData, so GBK/EUC-KR bodies are never transcoded and the title comes out as raw bytes
• Content-Type: TEXT/HTML or text/html ; charset=utf-8 — media types are case-insensitive and optional whitespace is allowed before parameters, but CanHaveTitleTag compares the value verbatim, so the title extraction is skipped entirely

$ httpx -u http://127.0.0.1:8000 -title
# content-type: text/html; charset="gbk"        -> []  (raw GBK bytes)
# content-type: TEXT/HTML                       -> (no title at all)

This strips quotes from the joined content-type before the charset matching, and normalizes the mime type (case + surrounding whitespace) in CanHaveTitleTag. Responses with canonical headers are unaffected. Regression tests cover both the quoted-charset and case/whitespace variants end-to-end (TestCanHaveTitleTag, TestExtractTitleDecodesCharset).

Summary by CodeRabbit

• Bug Fixes

  • Improved handling of quoted character-set values in HTTP content types, ensuring titles encoded with GBK and similar formats display correctly.
  • MIME type detection now handles capitalization differences and surrounding whitespace, improving recognition of HTML content.

• Tests

  • Added coverage for MIME type normalization and decoding titles from quoted and unquoted character-set declarations.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 2d34b188-9d4d-4a05-a805-255cebedab00

📥 Commits

Reviewing files that changed from the base of the PR and between bdc5c33 and be8322d.

📒 Files selected for processing (3)

• common/httpx/encodings.go
• common/httpx/title.go
• common/httpx/title_test.go

Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review.

---

Walkthrough

The change normalizes MIME types before title detection and removes quotes from content-type values before charset decoding. Tests cover MIME casing, whitespace, and GBK title decoding with quoted and unquoted charset parameters.

Changes

HTTP title extraction

Layer / File(s)	Summary
Normalize title MIME types common/httpx/title.go, common/httpx/title_test.go	CanHaveTitleTag trims and lowercases MIME types. Tests cover supported and unsupported values.
Decode quoted charset values common/httpx/encodings.go, common/httpx/title_test.go	DecodeData handles quoted charset parameters. Integration tests verify GBK title decoding.
Estimated code review effort: 2 (Simple)	~10 minutes

Merge Risk: ⚪ Minimal · up to be832

Title extraction now handles non-canonical Content-Type formatting and quoted GBK charset values while retaining canonical behavior. The covered changes present no remaining merge-readiness risk.

Poem

A rabbit checks the title line
And trims the spaces every time
GBK words bloom clear and bright
Quoted charsets now decode right
The little tests confirm the light

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 3 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main changes: support for quoted charset values and non-canonical Content-Type values during title extraction.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[ehsandeep]

@krishna28238-arch thanks for the PR, do you have test target to see before / after behavior?

[krishna28238-arch]

Yes — I've included end-to-end regression tests that
demonstrate the before/after behavior:

Test 1: TestCanHaveTitleTag (title_test.go)
Shows that non-canonical MIME types like "TEXT/HTML" and
" text/html ; charset=utf-8" are now correctly recognized
as HTML (previously skipped entirely, resulting in no
title extraction).

Test 2: TestExtractTitleDecodesCharset (title_test.go)
Shows that a GBK-encoded body with a quoted charset
declaration Content-Type: text/html; charset="gbk" now
transcodes correctly and extracts the title. Previously
the quoted value failed the charset= substring match, so
the body was never transcoded and the title came out as
raw bytes.

You can verify the before/after behavior by:

1. Check out the main branch and run:
   go test ./common/httpx/ -run TestCanHaveTitleTag -v
   go test ./common/httpx/ -run TestExtractTitleDecodesCharset -v

   (Tests will fail or show incorrect behavior)

2. Check out this PR branch and run the same commands:
   (Tests will pass, showing correct behavior)

Or run the full test suite to confirm no regressions:
go test ./common/httpx/ -v

[ehsandeep]

@krishna28238-arch I'm looking for real hosts to better understand the existing tool behavior, then test code to better understand the change. Can you share some?

[krishna28238-arch]

Thanks @ehsandeep, sure - here's what I found. The short version: the mainstream CJK web has moved to canonical UTF-8 headers, the legacy encodings are still out there, and the two shapes this PR fixes are straightforward to reproduce locally even though I couldn't find a live public host serving them today.

Real hosts I checked

I ran curl -s -D - over ~55 CN/KR/JP hosts (portals, provincial gov sites, university BBS front-ends, older reference sites). Every mainstream one is canonical now:

 $ curl -s -D - -o /dev/null https://www.baidu.com | grep -i ^content-type
Content-Type: text/html; charset=utf-8

 $ curl -s -D - -o /dev/null http://www.gov.cn | grep -i ^content-type
Content-Type: text/html

(baidu.com, qq.com, 163.com, sohu.com, sina.com.cn, gov.cn, people.com.cn, cctv.com, china.com.cn, naver.com, daum.net, yahoo.co.jp, etc)

One host that is still GB2312 today, in case you want to watch the existing behavior end to end:

 $ curl -s -D - -o /dev/null http://www.cnr.cn | grep -i ^content-type
Content-Type: text/html                        <- no charset in the header

 $ curl -s http://www.cnr.cn | grep -aoiE '<meta[^>]*charset[^>]*>' | head -1
<meta http-equiv="Content-Type" content="text/html; charset=gb2312">

 $ curl -s http://www.cnr.cn | file -bi -
text/html; charset=iso-8859-1                  <- body is GBK bytes

dev gets this one right - the <meta> fallback in DecodeData catches charset=gb2312":

 $ git checkout dev && go build -o /tmp/httpx-dev ./cmd/httpx
 $ echo http://www.cnr.cn | /tmp/httpx-dev -title -silent -no-color
https://www.cnr.cn [央广网·中央广播电视总台]

So the meta-declared path works today. What doesn't work is when the same value comes back in the HTTP header in non-canonical form:

1. quoted charset, text/html; charset="gbk" - DecodeData substring-matches charset=gbk, misses the quoted form, so the GBK body never gets transcoded and the title comes out as raw GBK bytes
2. MIME portion not exactly lowercase/trimmed - TEXT/HTML, or text/html coming from text/html ; charset=... - CanHaveTitleTag does an exact slices.Contains, so title extraction is just skipped

This same class of host has come up in the tracker a few times already: #37 (tnw.qq.com, GBK, 2020), #441 (euc-kr titles, 2021), #652 (GBK page, 2022), #1494 (103.45.249.230:7878, 2023). All four are migrated or offline now, so for the code-test part I set up local servers that emit the exact shapes:

# save as weird_ct.py
from http.server import HTTPServer, BaseHTTPRequestHandler
import threading, signal

class Handler(BaseHTTPRequestHandler):
    body = b''
    ctype = ''
    def do_GET(self):
        self.send_response(200)
        self.send_header('Content-Type', self.ctype)
        self.send_header('Content-Length', str(len(self.body)))
        self.end_headers()
        self.wfile.write(self.body)
    def log_message(self, fmt, *args):
        pass

class QuotedGBK(Handler):
    ctype = 'text/html; charset="gbk"'
    body = '<html><head><title>这是中文测试标题</title></head><body>Content</body></html>'.encode('gbk')

class UppercaseMIME(Handler):
    ctype = 'TEXT/HTML'
    body = b'<html><head><title>Uppercase MIME Type Test</title></head><body>Content</body></html>'

class Whitespace(Handler):
    ctype = 'text/html ; charset=utf-8'   # note the space before the semicolon
    body = b'<html><head><title>Whitespace Test</title></head><body>Content</body></html>'

for port, h in [(8001, QuotedGBK), (8002, UppercaseMIME), (8003, Whitespace)]:
    threading.Thread(target=HTTPServer(('127.0.0.1', port), h).serve_forever, daemon=True).start()
signal.pause()

 $ curl -s -D - -o /dev/null http://127.0.0.1:8001/ | grep -i ^content-type
Content-Type: text/html; charset="gbk"
 $ curl -s -D - -o /dev/null http://127.0.0.1:8002/ | grep -i ^content-type
Content-Type: TEXT/HTML
 $ curl -s -D - -o /dev/null http://127.0.0.1:8003/ | grep -i ^content-type
Content-Type: text/html ; charset=utf-8

dev branch:

 $ echo http://127.0.0.1:8001/ | /tmp/httpx-dev -title -silent -no-color
http://127.0.0.1:8001/ [IJԱ]  <- mojibake, raw GBK passed through, never transcoded

 $ echo http://127.0.0.1:8002/ | /tmp/httpx-dev -title -silent -no-color
http://127.0.0.1:8002/       <- no title at all, TEXT/HTML not recognized

 $ echo http://127.0.0.1:8003/ | /tmp/httpx-dev -title -silent -no-color
http://127.0.0.1:8003/       <- no title, "text/html " (trailing space) not recognized

and to confirm the first one is really just undecoded GBK (the same bytes the server sent):

 $ echo http://127.0.0.1:8001/ | /tmp/httpx-dev -title -silent -no-color | iconv -f gbk -t utf-8
http://127.0.0.1:8001/ [这是中文测试标题]

with this PR:

 $ git checkout fix-title-content-type-edge-cases && go build -o /tmp/httpx-fixed ./cmd/httpx
 $ echo http://127.0.0.1:8001/ | /tmp/httpx-fixed -title -silent -no-color
http://127.0.0.1:8001/ [这是中文测试标题]

 $ echo http://127.0.0.1:8002/ | /tmp/httpx-fixed -title -silent -no-color
http://127.0.0.1:8002/ [Uppercase MIME Type Test]

 $ echo http://127.0.0.1:8003/ | /tmp/httpx-fixed -title -silent -no-color
http://127.0.0.1:8003/ [Whitespace Test]

Regression check

I ran both builds over the 32 reachable hosts from the scan with -title -silent and the titles came out identical. The only differences were a redirect race on cctv.com/naver.com (each run caught a different side of the 302, in opposite directions) and one transient timeout on xhby.net. go test ./common/httpx/ passes on the branch, including the new TestCanHaveTitleTag / TestExtractTitleDecodesCharset.