OCR of scanned invoice PDF format or invoice JPEG/PNG format is possible? - yes

[emshihab008]

Hi.
I would like to inform you that I need to process Scanned invoice PDF format or invoice JPEG/PNG format. In this regard, can I use any RPA python tool to extract the necessary information?

Regards,
Ekram

[fi-do]

Hola emshihab008,

I would suggest to transform your pdfs in pngs to extract via r.read() the content.

Hope it will help you,
fi-do

[emshihab008]

Hi @fi-do
According to your suggestion, I ran the code. But I didn't get the proper result. Is it possible with RPA Python? or I have to use python OCR module (opencv) for the accurate result.
And is there any possibility to snap specific region of png file for OCR ?

import rpa as r
r.init(True, True)
r.wait(2)
#online pdf to png convert
r.url("https://pdf2png.com/de/")
r.click("choose.png")
r.dclick("b.png")
r.wait(10)
r.click("all_files.png")
r.rclick("zip.png")
r.click('extract.png')
r.run('C:\Users\shihab\Downloads\pdf2png\books')
print(r.read(100,87,889,702))
r.close()

Output:

C:\Python\Python39\python.exe C:/PzcharmProjects/Practice.py
-y Paste T ��.-. F�. Move Copy Del EL �lwht J» 619.609 uon.
E] Pam mom� t°' t°' ' E' ��1� - j w 15 ist, wcnn das Geld nur �ir cincn Compute �.�
Clipboard Organize " l �'1� � , f' fu dzw � Sch" . r t
. �W Russo \on W n anz1g ulem rucht? man

• This PC > Downloads w [Lâ��Lâ��Ncm .. meint Peggy Ratsch, diC Spczialistin far lnfor-
  «. mWWu-chnik an der Schulbehordc des Bezirks Balti-
  55 vToday (9) �1;�. m, Ichrer bekommen das Geld �ir mehr Germ
  «hr lcicht, wcnn sic nachwcisen, dass sie eincn Computer
  mmmll einsctzen. Mit andcrcn Wonen: Besorge dir zu-
  mchst cincn Computer, dann wirst du spiter mehr be-
  is knmmcn. Das Zicl ist vcrmutlich, dass ieder Schiller einen
  �W's b°°" Computer crhélt. Ich bezwci�e allerdings, dass Peggy
  nts Ratsch dann aufhércn wiirde, noch mehr Geld zu fordem.
  ds V Yesterday (1) Technologen schcn in def Technologie nut die Lésuns V0�
  � Prohlcmcn, nic cin Problem. .
  ' W Présidcnt Clinton hat verkiindet, duos ein Compute: 131
  E jcdcm Klasscnzimmcr Ziel den Bildungsprosramm! �1
  . . nicht als Mincl f�r die Ambildung, Iondem :18 1hr 210"
  15mm HM" Was ist an dicscr Vorstdlung falsch? chn die Tedll'lo'
  'ISK(E:) V Earlierthis week (1) logic die Krénung des Unterrichts damellt, we�d� �Ch
  'ISK (E) natiirlich die Wcrtigkciten in den Klassenzimmem andern.
  'ISK(G:) a An dcr Poly High School im kalif0mischen 1-008 M
  war schon langc Zcit das DaCh undid�. �3d �3m
  PDF [,0 ft . . W. Was war
  P e das Regenwaw 1n die Ian!
  RECHW� Z� �"1? Zunichst mum die Com!"ltef W w Pl�.
  v Last week (17) DE� SChulleitcr neg dmchsagen, die Lchref kc.)nn�¢n nk�
  U Sickc m �mm. Kan :
  D m WM- _ 3�. v�. �."���.. _ . . _ L... .n. M '09

books.pdf

[kensoh]

Hi Ekram,

I believe RPA for Python's OCR (Tesseract) is similar to Python OCR since Tesseract is the industry standard. However, your scanned image is not a properly scanned image, it has deformation due to not being flat-scanned. Even using commercial paid RPA tools it is unlikely to get good accuracy.

Even if you scan it perfectly, to get the best results, I recommend using some 3rd party apps in below post -
https://www.linkedin.com/posts/kensoh_rpa-tagui-activity-6780711825156775938-Wyt4

These companies spent all their resources getting it right to convert scanned PDFs into digital form. Their app takes care of pre-processing like aligning orientation, color correction etc. So it is likely you get the best results from there (almost 200 languages supported).

To snap specific region, you can consider taking a snapshot of your PDF viewer, use an image editor to remove all dynamic content to make it transparent. Then save as .png. When you do r.read('frame.png'), OCR will be performed on their PDF viewer for you to do further processing. For eg, for macOS PDF view, you can do something like below (try downloading the image and you can see many parts are made transparent) -

[emshihab008]

Hello Kensoh,
Thank your for information. I know this document is quite unorganized. That's why I tried OCR using this document. If it is possible with that document, then every document can be processed. And thank you for your solution regarding snapping.
I have another question. Can I read document using r.read('apple.png')? or I have to provide coordinate?

import rpa as r
r.init(True, False)
k=r.read('apple.png')
Print(k)
r.close()

Output:

[RPA][ERROR] - TagUI process ended unexpectedly
[RPA][ERROR] - use init() before using send()

[kensoh]

I see.. Yes that makes sense your thinking. For your error I have not come across this before or raised by other users.

Can you attach the log file for SikuliX here? From the installation location below, there is TagUI folder, then src\tagui.sikuli*.log

For Windows, location of installation is %APPDATA%\tagui (%APPDATA% is usually C:\Users\Username\Appdata\Roaming)

[kensoh]

It is strange because your previous example using coordinates can capture but using image fails.

Other than the logs, can you also attach the apple.png file here? I'm assuming the apple.png is in the same folder as where you run the python script.

[emshihab008]

Hi Kensoh,
I ran the code today again and got the below output.

[RPA][ERROR] - cannot find apple.png
Process finished with exit code 0

I have attached log file and original apple png file. Please find.

tagui.log

[kensoh]

Hi Ekram,

When I run read() on an image that can't be found on the screen, I get similar errors too in the log like below -

[tagui] START  - listening for inputs

[tagui] INPUT  - [1] present /Users/kensoh/Desktop/apple.png
[tagui] ACTION - present /Users/kensoh/Desktop/apple.png
[tagui] OUTPUT - [1] ERROR

[tagui] INPUT  - [2] present /Users/kensoh/Desktop/apple.png
[tagui] ACTION - present /Users/kensoh/Desktop/apple.png
[tagui] OUTPUT - [2] ERROR

[tagui] INPUT  - [3] present /Users/kensoh/Desktop/apple.png
[tagui] ACTION - present /Users/kensoh/Desktop/apple.png
[tagui] OUTPUT - [3] ERROR

[tagui] INPUT  - [4] present /Users/kensoh/Desktop/apple.png
[tagui] ACTION - present /Users/kensoh/Desktop/apple.png
[tagui] OUTPUT - [4] ERROR

That is correct and expected. But in your case you are saying that the Python program crashes is it?

Can you create the following file and put ocr.tag and apple.png in ~/.tagui/src folder?

ocr.tag

read apple.png to ocr_text
echo `ocr_text`

Then try running the following to run the RPA using TagUI directly.

cd ~/.tagui/src
./tagui ocr.tag

Above steps will help to isolate the problem whether program is crashing at TagUI / SikuliX or something wrong in Python.

[emshihab008]

Hi Ken,
Initially, when I ran the code it was showing the following output:
[RPA][ERROR] - TagUI process ended unexpectedly

But last time, when i ran again, it showed:
[RPA][ERROR] - cannot find apple.png

According to your instruction, i created ocr.tag file and copy the code and put it recommended folder.
ocr.tag.txt

But I am confused about last line (cd ~/.tagui/src ./tagui ocr.tag). How should I write?

[kensoh]

Oh sorry I see you are on Windows. You can do the following from command prompt -

cd c:\Users\shihab\AppData\Roaming\tagui\src
tagui ocr.tag

If the same error happens, then issue is on TagUI side. See if you can join the weekly call at Thursday 4-5pm (UTC+8), that should be 10-11am time in Germany. Another possible issue is when the image resolution for Retina displays. Let me search that issue solution and paste here.

[kensoh]

See these 2 issues related to Retina display. The gist is that the image captured actually is a different density and much larger / smaller than actual screen. So users do some settings change to make capturing image work correctly.

aisingapore/TagUI#896
aisingapore/TagUI#818

[emshihab008]

Hi Ken,
Using command prompt, I tried to open ocr.tag. Bu it is showing cannot fine ocr.tag.

And i have also checked the following issue.
aisingapore/TagUI#896
aisingapore/TagUI#818
But these are all about retina display isuue. I am using windows, ASUS Laptop, LED display, pixel size (1366& * 768).

[kensoh]

I see.. This is a really strange issue. There doesn't seem to be anything else we can check, other than meeting on a Zoom call so that I can directly test on your computer. Will coming Thursday 10am Germany time be good for you to meet?

Since this issue is an upstream TagUI issue, we can meet on the weekly Zoom call to troubleshoot.

URL for the Zoom meeting is aisingapore/TagUI#914

By the way, just to be sure, on your screen there is really the image apple.png showing but it does not detect correct?