Big-endian: public suffix list logic doesn't work #3004
Description
Public suffix list logic (or, more likely, some low level code/data structures used by it) doesn't work on big-endian machines: category matches are completly wrong
You can reproduce the issue adding something like -L ../lists/public_suffix_list.dat in ndpiReader parameters in tests/do.sh.in and run the "ARM build" job on CI
See: https://github.com/ntop/nDPI/actions/runs/18685946721/job/53278483979
Run configuration "caches_cfg" [--cfg=lru.ookla.size,0 --cfg=lru.msteams.ttl,1]
ookla.pcap OK
teams.pcap ERROR
../../../example/ndpiReader --cfg=filename.config,../../../example/config.txt -A -p ../../../example/protos.txt -c ../../../example/categories.txt -r ../../../example/risky_domains.txt -j ../../../example/ja4_fingerprints.csv -S ../../../example/sha1_fingerprints.csv -G ../../../lists -L ../../../lists/public_suffix_list.dat -q -K JSON -k /dev/null -t -v 2 -i pcap/teams.pcap -w /tmp/reader.23306.out --cfg=lru.ookla.size,0 --cfg=lru.msteams.ttl,1 [old vs new]
16c16
< LRU cache fpc_dns: 15/41/28 (insert/search/found)
---
> LRU cache fpc_dns: 12/41/26 (insert/search/found)
50c50
< Acceptable 1330 611638 72
---
> Acceptable 1276 594145 71
51a52
> Unsafe 54 17493 1
58c59
< Collaborative 1150 568376 32
---
> Collaborative 1096 550883 31
59a61
> Malware 54 17493 1
73c75
< 8 TCP 192.168.1.6:60549 <-> 13.107.18.11:443 [proto: 91.219/TLS.Microsoft365][Stack: TLS.Microsoft365][IP: 21/Outlook][Encrypted][Confidence: DPI][FPC: 219/Microsoft365, Confidence: DNS][DPI packets: 10][cat: Collaborative/15][Breed: Acceptable][28 pkts/7696 bytes <-> 26 pkts/9797 bytes][Goodput ratio: 80/85][1.16 sec][Hostname/SNI: substrate.office.com][(Advertised) ALPNs: h2;http/1.1][(Negotiated) ALPN: h2][bytes ratio: -0.120 (Mixed)][IAT c2s/s2c min/avg/max/stddev: 0/0 47/23 539/167 115/43][Pkt Len c2s/s2c min/avg/max/stddev: 54/60 275/377 1494/1506 397/471][nDPI Fingerprint: f1f5d0bdab75c94bb51a63f8e074f0cd][TCP Fingerprint: 2_64_65535_15db81ff8b0d/Unknown][TLSv1.2][JA4: t12d1311h2_8b80da21ef18_77989cba1f4a][ServerNames: outlook.office.com,attachment.outlook.office.net,attachment.outlook.officeppe.net,bookings.office.com,delve.office.com,edge.outlook.office365.com,edgesdf.outlook.com,img.delve.office.com,outlook.live.com,outlook-sdf.live.com,outlook-sdf.office.com,sdfedge-pilot.outlook.com,substrate.office.com,substrate-sdf.office.com,afd-k-acdc-direct.office.com,beta-sdf.yammer.com,teams-sdf.yammer.com,beta.yammer.com,teams.yammer.com,attachments.office.net,attachments-sdf.office.net,afd-k.office.com,afd-k-sdf.office.com][JA3S: a66ea560599a2f5c89eec8c3a0d69cee][Issuer: C=US, O=DigiCert Inc, CN=DigiCert Cloud Services CA-1][Subject: C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Outlook.office.com][Certificate SHA-1: AA:D3:F5:66:06:48:AA:F8:8E:9B:79:D6:7F:1D:53:EA:3F:97:03:A2][Validity: 2019-07-12 00:00:00 - 2021-07-12 12:00:00][Cipher: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384][Plen Bins: 0,22,7,0,7,0,7,0,0,3,3,0,0,0,3,0,7,0,3,0,10,3,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,18,0,0]
---
> 8 TCP 192.168.1.6:60549 <-> 13.107.18.11:443 [proto: 91.219/TLS.Microsoft365][Stack: TLS.Microsoft365][IP: 21/Outlook][Encrypted][Confidence: DPI][FPC: 21/Outlook, Confidence: IP address][DPI packets: 10][cat: Malware/100][Breed: Unsafe][28 pkts/7696 bytes <-> 26 pkts/9797 bytes][Goodput ratio: 80/85][1.16 sec][Hostname/SNI: substrate.office.com][(Advertised) ALPNs: h2;http/1.1][(Negotiated) ALPN: h2][bytes ratio: -0.120 (Mixed)][IAT c2s/s2c min/avg/max/stddev: 0/0 47/23 539/167 115/43][Pkt Len c2s/s2c min/avg/max/stddev: 54/60 275/377 1494/1506 397/471][nDPI Fingerprint: f1f5d0bdab75c94bb51a63f8e074f0cd][TCP Fingerprint: 2_64_65535_15db81ff8b0d/Unknown][TLSv1.2][JA4: t12d1311h2_8b80da21ef18_77989cba1f4a][ServerNames: outlook.office.com,attachment.outlook.office.net,attachment.outlook.officeppe.net,bookings.office.com,delve.office.com,edge.outlook.office365.com,edgesdf.outlook.com,img.delve.office.com,outlook.live.com,outlook-sdf.live.com,outlook-sdf.office.com,sdfedge-pilot.outlook.com,substrate.office.com,substrate-sdf.office.com,afd-k-acdc-direct.office.com,beta-sdf.yammer.com,teams-sdf.yammer.com,beta.yammer.com,teams.yammer.com,attachments.office.net,attachments-sdf.office.net,afd-k.office.com,afd-k-sdf.office.com][JA3S: a66ea560599a2f5c89eec8c3a0d69cee][Issuer: C=US, O=DigiCert Inc, CN=DigiCert Cloud Services CA-1][Subject: C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Outlook.office.com][Certificate SHA-1: AA:D3:F5:66:06:48:AA:F8:8E:9B:79:D6:7F:1D:53:EA:3F:97:03:A2][Validity: 2019-07-12 00:00:00 - 2021-07-12 12:00:00][Cipher: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384][Plen Bins: 0,22,7,0,7,0,7,0,0,3,3,0,0,0,3,0,7,0,3,0,10,3,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,18,0,0]
80c82
< 15 TCP 192.168.1.6:60547 <-> 52.114.88.59:443 [proto: 91.250/TLS.Teams][Stack: TLS.Teams][IP: 276/Azure][Encrypted][Confidence: DPI][FPC: 219/Microsoft365, Confidence: DNS][DPI packets: 16][cat: Collaborative/15][Breed: Acceptable][20 pkts/3926 bytes <-> 15 pkts/8828 bytes][Goodput ratio: 66/89][0.32 sec][Hostname/SNI: chatsvcagg.teams.microsoft.com][(Advertised) ALPNs: h2;http/1.1][bytes ratio: -0.384 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 13/25 91/80 23/31][Pkt Len c2s/s2c min/avg/max/stddev: 66/66 196/589 1494/1506 320/612][nDPI Fingerprint: f1f5d0bdab75c94bb51a63f8e074f0cd][TCP Fingerprint: 2_64_65535_15db81ff8b0d/Unknown][TLSv1.2][JA4: t12d1311h2_8b80da21ef18_77989cba1f4a][Plen Bins: 0,21,10,5,0,5,10,5,0,0,0,5,0,0,0,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,5,0,0,0,0,0,0,0,0,0,0,5,21,0,0]
---
> 15 TCP 192.168.1.6:60547 <-> 52.114.88.59:443 [proto: 91.250/TLS.Teams][Stack: TLS.Teams][IP: 276/Azure][Encrypted][Confidence: DPI][FPC: 276/Azure, Confidence: IP address][DPI packets: 16][cat: Collaborative/15][Breed: Acceptable][20 pkts/3926 bytes <-> 15 pkts/8828 bytes][Goodput ratio: 66/89][0.32 sec][Hostname/SNI: chatsvcagg.teams.microsoft.com][(Advertised) ALPNs: h2;http/1.1][bytes ratio: -0.384 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 13/25 91/80 23/31][Pkt Len c2s/s2c min/avg/max/stddev: 66/66 196/589 1494/1506 320/612][nDPI Fingerprint: f1f5d0bdab75c94bb51a63f8e074f0cd][TCP Fingerprint: 2_64_65535_15db81ff8b0d/Unknown][TLSv1.2][JA4: t12d1311h2_8b80da21ef18_77989cba1f4a][Plen Bins: 0,21,10,5,0,5,10,5,0,0,0,5,0,0,0,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,5,0,0,0,0,0,0,0,0,0,0,5,21,0,0]
[-1330 611638 72-] {+1276 594145 71+}
[-1150 568376 32-] {+1096 550883 31+}
[-15/41/28-] {+12/41/26+}
[-219/Microsoft365,-] {+21/Outlook,+}
[-219/Microsoft365,-] {+276/Azure,+}
[-Collaborative/15][Breed: Acceptable][28-] {+Malware/100][Breed: Unsafe][28+}
[-DNS][DPI-] {+IP address][DPI+}
======================================================================
{+Malware 54 17493 1+}
{+Unsafe 54 17493 1+}
./do.sh: 1 pcap(s) failed
Failed: teams.pcap
Run configuration "caches_global" [--cfg=lru.ookla.scope,1 --cfg=lru.bittorrent.scope,1 --cfg=lru.stun.scope,1 --cfg=lru.tls_cert.scope,1 --cfg=lru.mining.scope,1 --cfg=lru.msteams.scope,1 --cfg=lru.fpc_dns.scope,1]
bittorrent.pcap OK
lru_ipv6_caches.pcapng ERROR
../../../example/ndpiReader --cfg=filename.config,../../../example/config.txt -A -p ../../../example/protos.txt -c ../../../example/categories.txt -r ../../../example/risky_domains.txt -j ../../../example/ja4_fingerprints.csv -S ../../../example/sha1_fingerprints.csv -G ../../../lists -L ../../../lists/public_suffix_list.dat -q -K JSON -k /dev/null -t -v 2 -i pcap/lru_ipv6_caches.pcapng -w /tmp/reader.23306.out --cfg=lru.ookla.scope,1 --cfg=lru.bittorrent.scope,1 --cfg=lru.stun.scope,1 --cfg=lru.tls_cert.scope,1 --cfg=lru.mining.scope,1 --cfg=lru.msteams.scope,1 --cfg=lru.fpc_dns.scope,1 [old vs new]
13,14c13,14
< Automa host: 18/0 (search/found)
< Automa domain: 9/0 (search/found)
---
> Automa host: 6/0 (search/found)
> Automa domain: 3/0 (search/found)
27c27
< Hash public domain suffix: 21/18 (search/found)
---
> Hash public domain suffix: 6/6 (search/found)
36c36,37
< Acceptable 88 20854 12
---
> Acceptable 82 14946 10
> Unsafe 6 5908 2
38c39
< Web 9 8862 3
---
> Web 3 2954 1
40a42
> Malware 6 5908 2
48,49c50,51
< 3 TCP [2001:db8:200::1]:443 -> [2001:db8:1::1]:44150 [proto: 91.220/TLS.Cloudflare][Stack: TLS.Cloudflare][IP: 0/Unknown][Encrypted][Confidence: DPI (cache)][FPC: 0/Unknown, Confidence: Unknown][DPI packets: 3][cat: Web/5][Breed: Acceptable][3 pkts/2954 bytes -> 0 pkts/0 bytes][Goodput ratio: 92/0][0.15 sec][(Negotiated) ALPN: h2][Risk: ** Unidirectional Traffic **][Risk Score: 10][Risk Info: No client to server traffic][TLSv1.2][ServerNames: *.bikroy.com,sni.cloudflaressl.com,bikroy.com][JA3S: 9ebc57def2efb523f25c77af13aa6d48][Issuer: C=US, O=Cloudflare, Inc., CN=Cloudflare Inc ECC CA-3][Subject: C=US, ST=California, L=San Francisco, O=Cloudflare, Inc., CN=sni.cloudflaressl.com][Certificate SHA-1: FA:93:76:9C:39:4D:08:97:FA:8F:CE:80:E4:7A:8F:8E:CF:71:30:A0][Validity: 2021-06-29 00:00:00 - 2022-06-28 23:59:59][Cipher: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256][Plen Bins: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,100,0,0,0,0,0]
< 4 TCP [2001:db8:200::1]:443 -> [2001:db8:1::1]:44192 [proto: 91.220/TLS.Cloudflare][Stack: TLS.Cloudflare][IP: 0/Unknown][Encrypted][Confidence: DPI (cache)][FPC: 0/Unknown, Confidence: Unknown][DPI packets: 3][cat: Web/5][Breed: Acceptable][3 pkts/2954 bytes -> 0 pkts/0 bytes][Goodput ratio: 92/0][0.15 sec][(Negotiated) ALPN: h2][Risk: ** Unidirectional Traffic **][Risk Score: 10][Risk Info: No client to server traffic][TLSv1.2][ServerNames: *.bikroy.com,sni.cloudflaressl.com,bikroy.com][JA3S: 9ebc57def2efb523f25c77af13aa6d48][Issuer: C=US, O=Cloudflare, Inc., CN=Cloudflare Inc ECC CA-3][Subject: C=US, ST=California, L=San Francisco, O=Cloudflare, Inc., CN=sni.cloudflaressl.com][Certificate SHA-1: FA:93:76:9C:39:4D:08:97:FA:8F:CE:80:E4:7A:8F:8E:CF:71:30:A0][Validity: 2021-06-29 00:00:00 - 2022-06-28 23:59:59][Cipher: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256][Plen Bins: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,100,0,0,0,0,0]
---
> 3 TCP [2001:db8:200::1]:443 -> [2001:db8:1::1]:44150 [proto: 91.220/TLS.Cloudflare][Stack: TLS.Cloudflare][IP: 0/Unknown][Encrypted][Confidence: DPI (cache)][FPC: 0/Unknown, Confidence: Unknown][DPI packets: 3][cat: Malware/100][Breed: Unsafe][3 pkts/2954 bytes -> 0 pkts/0 bytes][Goodput ratio: 92/0][0.15 sec][(Negotiated) ALPN: h2][Risk: ** Unidirectional Traffic **][Risk Score: 10][Risk Info: No client to server traffic][TLSv1.2][ServerNames: *.bikroy.com,sni.cloudflaressl.com,bikroy.com][JA3S: 9ebc57def2efb523f25c77af13aa6d48][Issuer: C=US, O=Cloudflare, Inc., CN=Cloudflare Inc ECC CA-3][Subject: C=US, ST=California, L=San Francisco, O=Cloudflare, Inc., CN=sni.cloudflaressl.com][Certificate SHA-1: FA:93:76:9C:39:4D:08:97:FA:8F:CE:80:E4:7A:8F:8E:CF:71:30:A0][Validity: 2021-06-29 00:00:00 - 2022-06-28 23:59:59][Cipher: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256][Plen Bins: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,100,0,0,0,0,0]
> 4 TCP [2001:db8:200::1]:443 -> [2001:db8:1::1]:44192 [proto: 91.220/TLS.Cloudflare][Stack: TLS.Cloudflare][IP: 0/Unknown][Encrypted][Confidence: DPI (cache)][FPC: 0/Unknown, Confidence: Unknown][DPI packets: 3][cat: Malware/100][Breed: Unsafe][3 pkts/2954 bytes -> 0 pkts/0 bytes][Goodput ratio: 92/0][0.15 sec][(Negotiated) ALPN: h2][Risk: ** Unidirectional Traffic **][Risk Score: 10][Risk Info: No client to server traffic][TLSv1.2][ServerNames: *.bikroy.com,sni.cloudflaressl.com,bikroy.com][JA3S: 9ebc57def2efb523f25c77af13aa6d48][Issuer: C=US, O=Cloudflare, Inc., CN=Cloudflare Inc ECC CA-3][Subject: C=US, ST=California, L=San Francisco, O=Cloudflare, Inc., CN=sni.cloudflaressl.com][Certificate SHA-1: FA:93:76:9C:39:4D:08:97:FA:8F:CE:80:E4:7A:8F:8E:CF:71:30:A0][Validity: 2021-06-29 00:00:00 - 2022-06-28 23:59:59][Cipher: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256][Plen Bins: 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,100,0,0,0,0,0]