picotls has infinite recursion in the minicrypto ASN.1 decoder

Summary

Picotls is a TLS protocol library that allows users select different crypto backends based on their use case. Picotls implements its own ASN.1 validation helper, which is used by the minicrypto backend while parsing local PKCS#8 private keys. Prior to commit c14231d801407640bc42c2dcf92783409ea6a7c7, the validator recursively descends into constructed ASN.1 elements without enforcing a maximum nesting depth. If an application loads an attacker-supplied private-key file through ptls_minicrypto_load_private_key(), or otherwise calls the public ASN.1 validation API on untrusted DER, a crafted deeply nested ASN.1 structure can exhaust the process stack and crash the application. Note that the libcrypto (OpenSSL) backend does not use the ASN.1 validation helper of picotls, and therefore is immune to this vulnerability. The vulnerability has been addressed in commit c14231d801407640bc42c2dcf92783409ea6a7c7.