Skip to content

Commit 24fa040

Browse files
Larhzutorvalds
Larhzu
authored and
torvalds
committed
decompressors: add XZ decompressor module
In userspace, the .lzma format has become mostly a legacy file format that got superseded by the .xz format. Similarly, LZMA Utils was superseded by XZ Utils. These patches add support for XZ decompression into the kernel. Most of the code is as is from XZ Embedded <http://tukaani.org/xz/embedded.html>. It was written for the Linux kernel but is usable in other projects too. Advantages of XZ over the current LZMA code in the kernel: - Nice API that can be used by other kernel modules; it's not limited to kernel, initramfs, and initrd decompression. - Integrity check support (CRC32) - BCJ filters improve compression of executable code on certain architectures. These together with LZMA2 can produce a few percent smaller kernel or Squashfs images than plain LZMA without making the decompression slower. This patch: Add the main decompression code (xz_dec), testing module (xz_dec_test), wrapper script (xz_wrap.sh) for the xz command line tool, and documentation. The xz_dec module is enough to have a usable XZ decompressor e.g. for Squashfs. Signed-off-by: Lasse Collin <[email protected]> Cc: "H. Peter Anvin" <[email protected]> Cc: Alain Knaff <[email protected]> Cc: Albin Tonnerre <[email protected]> Cc: Phillip Lougher <[email protected]> Signed-off-by: Andrew Morton <[email protected]> Signed-off-by: Linus Torvalds <[email protected]>
1 parent fb7fa58 commit 24fa040

17 files changed

+3783
-0
lines changed

‎Documentation/xz.txt

+121
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
2+
XZ data compression in Linux
3+
============================
4+
5+
Introduction
6+
7+
XZ is a general purpose data compression format with high compression
8+
ratio and relatively fast decompression. The primary compression
9+
algorithm (filter) is LZMA2. Additional filters can be used to improve
10+
compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters
11+
improve compression ratio of executable data.
12+
13+
The XZ decompressor in Linux is called XZ Embedded. It supports
14+
the LZMA2 filter and optionally also BCJ filters. CRC32 is supported
15+
for integrity checking. The home page of XZ Embedded is at
16+
<http://tukaani.org/xz/embedded.html>, where you can find the
17+
latest version and also information about using the code outside
18+
the Linux kernel.
19+
20+
For userspace, XZ Utils provide a zlib-like compression library
21+
and a gzip-like command line tool. XZ Utils can be downloaded from
22+
<http://tukaani.org/xz/>.
23+
24+
XZ related components in the kernel
25+
26+
The xz_dec module provides XZ decompressor with single-call (buffer
27+
to buffer) and multi-call (stateful) APIs. The usage of the xz_dec
28+
module is documented in include/linux/xz.h.
29+
30+
The xz_dec_test module is for testing xz_dec. xz_dec_test is not
31+
useful unless you are hacking the XZ decompressor. xz_dec_test
32+
allocates a char device major dynamically to which one can write
33+
.xz files from userspace. The decompressed output is thrown away.
34+
Keep an eye on dmesg to see diagnostics printed by xz_dec_test.
35+
See the xz_dec_test source code for the details.
36+
37+
For decompressing the kernel image, initramfs, and initrd, there
38+
is a wrapper function in lib/decompress_unxz.c. Its API is the
39+
same as in other decompress_*.c files, which is defined in
40+
include/linux/decompress/generic.h.
41+
42+
scripts/xz_wrap.sh is a wrapper for the xz command line tool found
43+
from XZ Utils. The wrapper sets compression options to values suitable
44+
for compressing the kernel image.
45+
46+
For kernel makefiles, two commands are provided for use with
47+
$(call if_needed). The kernel image should be compressed with
48+
$(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2
49+
dictionary. It will also append a four-byte trailer containing the
50+
uncompressed size of the file, which is needed by the boot code.
51+
Other things should be compressed with $(call if_needed,xzmisc)
52+
which will use no BCJ filter and 1 MiB LZMA2 dictionary.
53+
54+
Notes on compression options
55+
56+
Since the XZ Embedded supports only streams with no integrity check or
57+
CRC32, make sure that you don't use some other integrity check type
58+
when encoding files that are supposed to be decoded by the kernel. With
59+
liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32
60+
when encoding. With the xz command line tool, use --check=none or
61+
--check=crc32.
62+
63+
Using CRC32 is strongly recommended unless there is some other layer
64+
which will verify the integrity of the uncompressed data anyway.
65+
Double checking the integrity would probably be waste of CPU cycles.
66+
Note that the headers will always have a CRC32 which will be validated
67+
by the decoder; you can only change the integrity check type (or
68+
disable it) for the actual uncompressed data.
69+
70+
In userspace, LZMA2 is typically used with dictionary sizes of several
71+
megabytes. The decoder needs to have the dictionary in RAM, thus big
72+
dictionaries cannot be used for files that are intended to be decoded
73+
by the kernel. 1 MiB is probably the maximum reasonable dictionary
74+
size for in-kernel use (maybe more is OK for initramfs). The presets
75+
in XZ Utils may not be optimal when creating files for the kernel,
76+
so don't hesitate to use custom settings. Example:
77+
78+
xz --check=crc32 --lzma2=dict=512KiB inputfile
79+
80+
An exception to above dictionary size limitation is when the decoder
81+
is used in single-call mode. Decompressing the kernel itself is an
82+
example of this situation. In single-call mode, the memory usage
83+
doesn't depend on the dictionary size, and it is perfectly fine to
84+
use a big dictionary: for maximum compression, the dictionary should
85+
be at least as big as the uncompressed data itself.
86+
87+
Future plans
88+
89+
Creating a limited XZ encoder may be considered if people think it is
90+
useful. LZMA2 is slower to compress than e.g. Deflate or LZO even at
91+
the fastest settings, so it isn't clear if LZMA2 encoder is wanted
92+
into the kernel.
93+
94+
Support for limited random-access reading is planned for the
95+
decompression code. I don't know if it could have any use in the
96+
kernel, but I know that it would be useful in some embedded projects
97+
outside the Linux kernel.
98+
99+
Conformance to the .xz file format specification
100+
101+
There are a couple of corner cases where things have been simplified
102+
at expense of detecting errors as early as possible. These should not
103+
matter in practice all, since they don't cause security issues. But
104+
it is good to know this if testing the code e.g. with the test files
105+
from XZ Utils.
106+
107+
Reporting bugs
108+
109+
Before reporting a bug, please check that it's not fixed already
110+
at upstream. See <http://tukaani.org/xz/embedded.html> to get the
111+
latest code.
112+
113+
Report bugs to <[email protected]> or visit #tukaani on
114+
Freenode and talk to Larhzu. I don't actively read LKML or other
115+
kernel-related mailing lists, so if there's something I should know,
116+
you should email to me personally or use IRC.
117+
118+
Don't bother Igor Pavlov with questions about the XZ implementation
119+
in the kernel or about XZ Utils. While these two implementations
120+
include essential code that is directly based on Igor Pavlov's code,
121+
these implementations aren't maintained nor supported by him.

‎include/linux/xz.h

+264
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,264 @@
1+
/*
2+
* XZ decompressor
3+
*
4+
* Authors: Lasse Collin <[email protected]>
5+
* Igor Pavlov <http://7-zip.org/>
6+
*
7+
* This file has been put into the public domain.
8+
* You can do whatever you want with this file.
9+
*/
10+
11+
#ifndef XZ_H
12+
#define XZ_H
13+
14+
#ifdef __KERNEL__
15+
# include <linux/stddef.h>
16+
# include <linux/types.h>
17+
#else
18+
# include <stddef.h>
19+
# include <stdint.h>
20+
#endif
21+
22+
/* In Linux, this is used to make extern functions static when needed. */
23+
#ifndef XZ_EXTERN
24+
# define XZ_EXTERN extern
25+
#endif
26+
27+
/**
28+
* enum xz_mode - Operation mode
29+
*
30+
* @XZ_SINGLE: Single-call mode. This uses less RAM than
31+
* than multi-call modes, because the LZMA2
32+
* dictionary doesn't need to be allocated as
33+
* part of the decoder state. All required data
34+
* structures are allocated at initialization,
35+
* so xz_dec_run() cannot return XZ_MEM_ERROR.
36+
* @XZ_PREALLOC: Multi-call mode with preallocated LZMA2
37+
* dictionary buffer. All data structures are
38+
* allocated at initialization, so xz_dec_run()
39+
* cannot return XZ_MEM_ERROR.
40+
* @XZ_DYNALLOC: Multi-call mode. The LZMA2 dictionary is
41+
* allocated once the required size has been
42+
* parsed from the stream headers. If the
43+
* allocation fails, xz_dec_run() will return
44+
* XZ_MEM_ERROR.
45+
*
46+
* It is possible to enable support only for a subset of the above
47+
* modes at compile time by defining XZ_DEC_SINGLE, XZ_DEC_PREALLOC,
48+
* or XZ_DEC_DYNALLOC. The xz_dec kernel module is always compiled
49+
* with support for all operation modes, but the preboot code may
50+
* be built with fewer features to minimize code size.
51+
*/
52+
enum xz_mode {
53+
XZ_SINGLE,
54+
XZ_PREALLOC,
55+
XZ_DYNALLOC
56+
};
57+
58+
/**
59+
* enum xz_ret - Return codes
60+
* @XZ_OK: Everything is OK so far. More input or more
61+
* output space is required to continue. This
62+
* return code is possible only in multi-call mode
63+
* (XZ_PREALLOC or XZ_DYNALLOC).
64+
* @XZ_STREAM_END: Operation finished successfully.
65+
* @XZ_UNSUPPORTED_CHECK: Integrity check type is not supported. Decoding
66+
* is still possible in multi-call mode by simply
67+
* calling xz_dec_run() again.
68+
* Note that this return value is used only if
69+
* XZ_DEC_ANY_CHECK was defined at build time,
70+
* which is not used in the kernel. Unsupported
71+
* check types return XZ_OPTIONS_ERROR if
72+
* XZ_DEC_ANY_CHECK was not defined at build time.
73+
* @XZ_MEM_ERROR: Allocating memory failed. This return code is
74+
* possible only if the decoder was initialized
75+
* with XZ_DYNALLOC. The amount of memory that was
76+
* tried to be allocated was no more than the
77+
* dict_max argument given to xz_dec_init().
78+
* @XZ_MEMLIMIT_ERROR: A bigger LZMA2 dictionary would be needed than
79+
* allowed by the dict_max argument given to
80+
* xz_dec_init(). This return value is possible
81+
* only in multi-call mode (XZ_PREALLOC or
82+
* XZ_DYNALLOC); the single-call mode (XZ_SINGLE)
83+
* ignores the dict_max argument.
84+
* @XZ_FORMAT_ERROR: File format was not recognized (wrong magic
85+
* bytes).
86+
* @XZ_OPTIONS_ERROR: This implementation doesn't support the requested
87+
* compression options. In the decoder this means
88+
* that the header CRC32 matches, but the header
89+
* itself specifies something that we don't support.
90+
* @XZ_DATA_ERROR: Compressed data is corrupt.
91+
* @XZ_BUF_ERROR: Cannot make any progress. Details are slightly
92+
* different between multi-call and single-call
93+
* mode; more information below.
94+
*
95+
* In multi-call mode, XZ_BUF_ERROR is returned when two consecutive calls
96+
* to XZ code cannot consume any input and cannot produce any new output.
97+
* This happens when there is no new input available, or the output buffer
98+
* is full while at least one output byte is still pending. Assuming your
99+
* code is not buggy, you can get this error only when decoding a compressed
100+
* stream that is truncated or otherwise corrupt.
101+
*
102+
* In single-call mode, XZ_BUF_ERROR is returned only when the output buffer
103+
* is too small or the compressed input is corrupt in a way that makes the
104+
* decoder produce more output than the caller expected. When it is
105+
* (relatively) clear that the compressed input is truncated, XZ_DATA_ERROR
106+
* is used instead of XZ_BUF_ERROR.
107+
*/
108+
enum xz_ret {
109+
XZ_OK,
110+
XZ_STREAM_END,
111+
XZ_UNSUPPORTED_CHECK,
112+
XZ_MEM_ERROR,
113+
XZ_MEMLIMIT_ERROR,
114+
XZ_FORMAT_ERROR,
115+
XZ_OPTIONS_ERROR,
116+
XZ_DATA_ERROR,
117+
XZ_BUF_ERROR
118+
};
119+
120+
/**
121+
* struct xz_buf - Passing input and output buffers to XZ code
122+
* @in: Beginning of the input buffer. This may be NULL if and only
123+
* if in_pos is equal to in_size.
124+
* @in_pos: Current position in the input buffer. This must not exceed
125+
* in_size.
126+
* @in_size: Size of the input buffer
127+
* @out: Beginning of the output buffer. This may be NULL if and only
128+
* if out_pos is equal to out_size.
129+
* @out_pos: Current position in the output buffer. This must not exceed
130+
* out_size.
131+
* @out_size: Size of the output buffer
132+
*
133+
* Only the contents of the output buffer from out[out_pos] onward, and
134+
* the variables in_pos and out_pos are modified by the XZ code.
135+
*/
136+
struct xz_buf {
137+
const uint8_t *in;
138+
size_t in_pos;
139+
size_t in_size;
140+
141+
uint8_t *out;
142+
size_t out_pos;
143+
size_t out_size;
144+
};
145+
146+
/**
147+
* struct xz_dec - Opaque type to hold the XZ decoder state
148+
*/
149+
struct xz_dec;
150+
151+
/**
152+
* xz_dec_init() - Allocate and initialize a XZ decoder state
153+
* @mode: Operation mode
154+
* @dict_max: Maximum size of the LZMA2 dictionary (history buffer) for
155+
* multi-call decoding. This is ignored in single-call mode
156+
* (mode == XZ_SINGLE). LZMA2 dictionary is always 2^n bytes
157+
* or 2^n + 2^(n-1) bytes (the latter sizes are less common
158+
* in practice), so other values for dict_max don't make sense.
159+
* In the kernel, dictionary sizes of 64 KiB, 128 KiB, 256 KiB,
160+
* 512 KiB, and 1 MiB are probably the only reasonable values,
161+
* except for kernel and initramfs images where a bigger
162+
* dictionary can be fine and useful.
163+
*
164+
* Single-call mode (XZ_SINGLE): xz_dec_run() decodes the whole stream at
165+
* once. The caller must provide enough output space or the decoding will
166+
* fail. The output space is used as the dictionary buffer, which is why
167+
* there is no need to allocate the dictionary as part of the decoder's
168+
* internal state.
169+
*
170+
* Because the output buffer is used as the workspace, streams encoded using
171+
* a big dictionary are not a problem in single-call mode. It is enough that
172+
* the output buffer is big enough to hold the actual uncompressed data; it
173+
* can be smaller than the dictionary size stored in the stream headers.
174+
*
175+
* Multi-call mode with preallocated dictionary (XZ_PREALLOC): dict_max bytes
176+
* of memory is preallocated for the LZMA2 dictionary. This way there is no
177+
* risk that xz_dec_run() could run out of memory, since xz_dec_run() will
178+
* never allocate any memory. Instead, if the preallocated dictionary is too
179+
* small for decoding the given input stream, xz_dec_run() will return
180+
* XZ_MEMLIMIT_ERROR. Thus, it is important to know what kind of data will be
181+
* decoded to avoid allocating excessive amount of memory for the dictionary.
182+
*
183+
* Multi-call mode with dynamically allocated dictionary (XZ_DYNALLOC):
184+
* dict_max specifies the maximum allowed dictionary size that xz_dec_run()
185+
* may allocate once it has parsed the dictionary size from the stream
186+
* headers. This way excessive allocations can be avoided while still
187+
* limiting the maximum memory usage to a sane value to prevent running the
188+
* system out of memory when decompressing streams from untrusted sources.
189+
*
190+
* On success, xz_dec_init() returns a pointer to struct xz_dec, which is
191+
* ready to be used with xz_dec_run(). If memory allocation fails,
192+
* xz_dec_init() returns NULL.
193+
*/
194+
XZ_EXTERN struct xz_dec *xz_dec_init(enum xz_mode mode, uint32_t dict_max);
195+
196+
/**
197+
* xz_dec_run() - Run the XZ decoder
198+
* @s: Decoder state allocated using xz_dec_init()
199+
* @b: Input and output buffers
200+
*
201+
* The possible return values depend on build options and operation mode.
202+
* See enum xz_ret for details.
203+
*
204+
* Note that if an error occurs in single-call mode (return value is not
205+
* XZ_STREAM_END), b->in_pos and b->out_pos are not modified and the
206+
* contents of the output buffer from b->out[b->out_pos] onward are
207+
* undefined. This is true even after XZ_BUF_ERROR, because with some filter
208+
* chains, there may be a second pass over the output buffer, and this pass
209+
* cannot be properly done if the output buffer is truncated. Thus, you
210+
* cannot give the single-call decoder a too small buffer and then expect to
211+
* get that amount valid data from the beginning of the stream. You must use
212+
* the multi-call decoder if you don't want to uncompress the whole stream.
213+
*/
214+
XZ_EXTERN enum xz_ret xz_dec_run(struct xz_dec *s, struct xz_buf *b);
215+
216+
/**
217+
* xz_dec_reset() - Reset an already allocated decoder state
218+
* @s: Decoder state allocated using xz_dec_init()
219+
*
220+
* This function can be used to reset the multi-call decoder state without
221+
* freeing and reallocating memory with xz_dec_end() and xz_dec_init().
222+
*
223+
* In single-call mode, xz_dec_reset() is always called in the beginning of
224+
* xz_dec_run(). Thus, explicit call to xz_dec_reset() is useful only in
225+
* multi-call mode.
226+
*/
227+
XZ_EXTERN void xz_dec_reset(struct xz_dec *s);
228+
229+
/**
230+
* xz_dec_end() - Free the memory allocated for the decoder state
231+
* @s: Decoder state allocated using xz_dec_init(). If s is NULL,
232+
* this function does nothing.
233+
*/
234+
XZ_EXTERN void xz_dec_end(struct xz_dec *s);
235+
236+
/*
237+
* Standalone build (userspace build or in-kernel build for boot time use)
238+
* needs a CRC32 implementation. For normal in-kernel use, kernel's own
239+
* CRC32 module is used instead, and users of this module don't need to
240+
* care about the functions below.
241+
*/
242+
#ifndef XZ_INTERNAL_CRC32
243+
# ifdef __KERNEL__
244+
# define XZ_INTERNAL_CRC32 0
245+
# else
246+
# define XZ_INTERNAL_CRC32 1
247+
# endif
248+
#endif
249+
250+
#if XZ_INTERNAL_CRC32
251+
/*
252+
* This must be called before any other xz_* function to initialize
253+
* the CRC32 lookup table.
254+
*/
255+
XZ_EXTERN void xz_crc32_init(void);
256+
257+
/*
258+
* Update CRC32 value using the polynomial from IEEE-802.3. To start a new
259+
* calculation, the third argument must be zero. To continue the calculation,
260+
* the previously returned value is passed as the third argument.
261+
*/
262+
XZ_EXTERN uint32_t xz_crc32(const uint8_t *buf, size_t size, uint32_t crc);
263+
#endif
264+
#endif

‎lib/Kconfig

+2
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,8 @@ config LZO_COMPRESS
106106
config LZO_DECOMPRESS
107107
tristate
108108

109+
source "lib/xz/Kconfig"
110+
109111
#
110112
# These all provide a common interface (hence the apparent duplication with
111113
# ZLIB_INFLATE; DECOMPRESS_GZIP is just a wrapper.)

0 commit comments

Comments
 (0)