getting IB A©MENT for IB PAYMENT - introduced using repair on open

[EdmundsEcho]

I have been working with pdfplumber for a few months now. I don't yet have a rigorous testing infrastructure up and running. I'm parsing 100's of different versions of bank statements. Anyway, somewhere along the way, this pdf that was working a few weeks ago, started to puch out a copy right glyph and some: IB PAYMENT TO -> IB A©MENT. Any ideas on where to problem solve would be greatly appreciated.

[jsvine]

Without access to the PDF, this will be difficult to debug. But as an initial foray: If you open the PDF in your browser, copy the relevant section of text, and paste it into a text editor, do you see IB PAYMENT TO or IB A©MENT?

[EdmundsEcho]

Great suggestion to help narrow down the issue. When I copy/paste I get the expected text:

Details
BALANCE BROUGHT FORWARD
IB PAYMENT TO
SJ BARNES
IB PAYMENT TO
NEDBANK CREDIT CA NEDC/CARD PN IB PAYMENT TO

... I will put together a min script that replicates the issue.

[EdmundsEcho]

In a min script I get what we expect:

{ Bounds: '(68, 398, 126, 406)', object_ids: '[]', Text: 'IB PAYMENT TO ' }

In my fully loaded script I get:

{ Bounds: '(68, 398, 126, 406)', object_ids: '[29]', Text: 'IB A\u00a9MENT TO '}

... weird. I'm only posting for ideas for how encoding might be an issue? regex filters?

[EdmundsEcho]

Found it: When I set repair=True, the encoding gets messed up. Likely unintended in the repair process using ghostscript?

with pdfplumber.open(self.pdf_path, repair=False, unicode_norm="NFC") as pdf:

If this is helpful, I can scrub the pdf of personal information and provide it here.