Errors for segments with size > 2147483647 bytes #16
Comments
|
Under the hood, Daffodli uses a Java array to store hex binary bytes which has a maximum size of 231-1 bytes (i.e. 2 GB). If hexBinary larger than 2GB is needed, it's probably better to use the Daffodil Binary Large Object extension. To use this, you can change the type from |
|
The large hexBinary fields in NITF should be replaced by arrays with large maxOccurs containing smaller hexBinary elements. This would enable them to represent blobs of data much larger than a single DFDL element can represent without hitting any int-size limits or JVM array maximum limits. E.g., something like (untested): This change would affect the shape of the infoset, so it's a major new version and many of the test expected infosets would need to be adapted. |
|
That would work for my scenario
From: Mike Beckerle ***@***.***>
Sent: Friday, January 10, 2025 10:59 AM
To: DFDLSchemas/NITF ***@***.***>
Cc: Lara Blatchford ***@***.***>; Author ***@***.***>
Subject: Re: [DFDLSchemas/NITF] Errors for segments with size > 2147483647 bytes (Issue #16)
The large hexBinary fields in NITF should be replaced by arrays with large maxOccurs containing smaller hexBinary elements. This would enable them to represent blobs of data much larger than a single DFDL element can represent without hitting any int-size limits or JVM array maximum limits.
E.g., something like (untested):
<element name="whatever" dfdl:lengthKind="explicit" dfdl:length='{ ...
the big blob length ...}'>
<complexType>
<sequence>
<element name="blob" dfdl:lengthKind="implicit">
<!-- this blob element allows a dfdl:outputValueCalc='{
dfdl:contentLength(..../pixels/blob) }' to work to capture the length
when unparsing -->
<complexType>
<sequence>
<!-- Avoid giant lines. This is XML. Users *may* want
to open it in a text editor.
Note max size of blob is 10000000100. That's ~10TB
…-->
<element name="a" type="xs:hexBinary" minOccurs="0"
maxOccurs="100000000" dfdl:lengthKind="explicit" dfdl:length="100"
dfdl:occursCountKind="implicit"/>
<element name="last" type="xs:hexBinary"
minOccurs="0" maxOccurs="1" dfdl:lengthKind="delimited"/>
</sequence>
</complexType>
</element>
</sequence>
</complexType>
</element>
-
Reply to this email directly, view it on GitHub<#16 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AQ3OWUWJXB2HHWCLPGOKQBD2J7U4FAVCNFSM6AAAAABU6VHSU6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKOBTGA2TEMJQG4>.
You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>
|
When attempting to parse a NITF 2.1 file with a large image segment the following error occurred "Parse Error: Failed to populate ImageSegment[1]. Cause Parse Error: Cannot convert '2516582400' from Long type to Int (Value '2516582400' is out of range for xs:int"
The PayloadLength is xs:int. When the schema file is changed to make PayloadLength xs:unsignedLong, another error occurs:
"Failed to convert: Parse Error: Failed to populate ImageSegment[1]. Cause: Parse Error: Choice dispatch branch failed: List( Parse Error: Length for xs:hexBinary exceeds maximum of 2147483647 bytes: 2516582400".
I believe the xs:hexBinary type should have an unbounded maximum, so the reason for this error is unclear.
The test data would have to be approved for release by DISA. The test data request form and information about the files to be requested can be provided upon request.
The text was updated successfully, but these errors were encountered: