many memory safety bugs in stb_image and stb_vorbis

[boergeson]

I did look into stb_image.h v2.30 and stb_vorbis.c v1.22 and I found a of integer overflows, out of bounds accesses, and logic errors .. I have PoCs for many of them and submitted a PR with fixes for the ones that are easy fix and you can REPRODUCE them if you check the fork

abt the PoCs

there is working PoCs for bugs 1-12 in a fork: https://github.com/boergeson/stb/tree/fix/memory-safety-issues/poc to check them out

palette leak (bug 5) is the most visual -- you can see the stack data in the output pixel values. The GIF two_back (bug 3) crashes immediately under ASAN

PR

I submitted a fix PR for bugs 1, 2, 3, 5, 8, and 12: #1927

the bugs

stb_image.h

1. stbi__convert_format16 missing overflow check (line 1820)

The 8-bit format converter was hardened with stbi__malloc_mad3:

// line 1763 -- safe
good = (unsigned char *) stbi__malloc_mad3(req_comp, x, y, 0);

But the 16-bit version still uses a raw malloc:

// line 1820 -- no overflow check
good = (stbi__uint16 *) stbi__malloc(req_comp * x * y * 2);

With large dimensions (e.g. 32768x32768 RGBA), req_comp * x * y * 2 overflows to 0. The subsequent conversion loop writes 8 GB into a zero-size buffer.

Reachable via 16-bit PNG, PSD, or PNM with a requested component count different from source

2. stbi__convert_16_to_8 / stbi__convert_8_to_16 (lines 1193, 1209)

It's same pattern: w * h * channels is computed as a plain int multiplication with no overflow check, then passed to stbi__malloc. The convert_8_to_16 version also multiplies by 2, doubling the overflow risk.

3. Animated GIF two_back pointer underflow (line 7023)

two_back = out - 2 * stride;

out is start of the allocated buffer. out - 2 * stride points before heap allocation. When frame useing disposal method 3, line 6818 reads from this pointer:

memcpy( &g->out[pi * 4], &two_back[pi * 4], 4 );

crashes Under AddressSanitizer,, correct formula should be out + (layers - 2) * stride.

PoC: a 3-frame animated GIF with disposal method 3 on the third frame

4. Animated GIF layers*stride overflow (lines 6991-7021)

layers incrementeing on each frame, layers * stride is used for realloc and memcpy offset calculations with NO overflow check .. A 400x400 RGBA GIF overflows after about 3356 frames.

5. PNG palette index out-of-bounds (line 4988 )

stbi__expand_png_palette receivies the palette length but explicitily ignores it:

STBI_NOTUSED(len);

Pixel values can be 0-255 no matter palette size. the palette array is 1024 bytes on stack, only partially initialised. any pixel index beyond actual palette size reads uninitialiszed stack memory into output image.

PoC: paletted PNG with 1 palette entry and pixel values 0-255. 161 out of 255 not-zero pixels containes stack data

6. Invalid PNG color type + bit depth accepted (line 5112)

The PNG spec restricts color types 2, 4, 6 to bit depths 8 and 16 only. stb_image accepts them at depth 1, 2, and 4. The bit-unpacking code was designed for singlechannel images at sub-8-bit depths. With img_n=3 or 4, the stride assumptions are wrong !!

PoC: a PNG with color type 2 (RGB) and depth 4 loads successfully when it should be rejected.

7. PNG IDAT idata_limit doubling overflow (line 5188)

while (ioff + c.length > idata_limit)
    idata_limit *= 2;

After collecting many large IDAT chunks, idata_limit can overflow uint32. It wraps to 0, STBI_REALLOC_SIZED allocates 0 bytes, and stbi__getn writes up to 1 GB into it ..............

8. JPEG marker failure returns success (line 3440)

if (!stbi__process_marker(j, m)) return 1;

When stbi__process_marker returns 0 (failure), the function returns 1 (success) ...should be return 0!!! bad markers betw progressive scans are just accepted "in silence"..

9. progressive JPEG uninitialized heap leak (lines 3346, 3085)

Coeefficient buffers are allocated with malloc (not calloc!) stbi__jpeg_finish processeing all components no matter whether they appeared in any SOS scan .. A progressive JPEG that defines 3 components but only scans 2 leaks heap data through IDCT into output image

10. BMP palette uninitialized read (line 5601)

this is actuallty similar to the PNG palette issue. 8bpp BMP with small palette allows pixel indices beyond palette size to read from the uninnintialized pal[256][4] stack array.

11. stbi__vertical_flip_slices overflow (line 1250)

slice_size = w * h * bytes_per_pixel with no overflow check.!!!! Used in pointer arithmetic in the animated GIF flip path.

stb_vorbis.c

12. Bounds clamp sign error (line 1891)

// comment says offset is (p_inter*ch)+(c_inter)
effective = len*ch - (p_inter*ch - c_inter);
//                                ^ should be + c_inter

subtracting c_inter instead of adding it. clamp does allow 2 * c_inter extra samples past buffer end

13. Codebook entries*dimensions overflow (line 3860)

For lookup_type 2, c->lookup_values = c->entries * c->dimensions with no overflow check. entries can be up to 2^24-1, dimensions up to 2^16-1. The product wraps uint32. The subsequent mults allocation is undersized

14. Multiplicands Allocation Overflow (lines 3878, 3880)

sizeof(float) * entries * dimensions overflows. small alloc but large write loop..

15. Unchecked comment_list_length (line 3660)

32-bit value from stream, multiplied by sizeof(char*) for the allocation. Overflowing on 32-bit platforms.!!

again there is PoC for 1-12 in the fork: https://github.com/boergeson/stb/tree/fix/memory-safety-issues/poc

and fix PR for bugs 1, 2, 3, 5, 8, and 12: #1927