Junos OS: PTX Series: Denial of Service in packet processing due to heavy route churn when J-Flow sampling is enabled

Summary

A Data Processing vulnerability in the Multi-Service process (multi-svcs) on the FPC of Juniper Networks Junos OS on the PTX Series routers may lead to the process becoming unresponsive, ultimately affecting traffic forwarding, allowing an attacker to cause a Denial of Service (DoS) condition . The Multi-Service Process running on the FPC is responsible for handling sampling-related operations when a J-Flow configuration is activated. This can occur during periods of heavy route churn, causing the Multi-Service Process to stop processing updates, without consuming any further updates from kernel. This back pressure towards the kernel affects further dynamic updates from other processes in the system, including RPD, causing a KRT-STUCK condition and traffic forwarding issues. An administrator can monitor the following command to check if there is the KRT queue is stuck: user@device > show krt state ... Number of async queue entries: 65007 <--- this value keep on increasing. The following logs/alarms will be observed when this condition exists: user@junos> show chassis alarms 2 alarms currently active Alarm time Class Description 2020-10-11 04:33:45 PDT Minor Potential slow peers are: MSP(FPC1-PIC0) MSP(FPC3-PIC0) MSP(FPC4-PIC0) Logs: Oct 11 04:33:44.672 2020 test /kernel: rts_peer_cp_recv_timeout : Bit set for msp8 as it is stuck Oct 11 04:35:56.000 2020 test-lab fpc4 user.err gldfpc-multi-svcs.elf: Error in parsing composite nexthop Oct 11 04:35:56.000 2020 test-lab fpc4 user.err gldfpc-multi-svcs.elf: composite nexthop parsing error Oct 11 04:43:05 2020 test /kernel: rt_pfe_veto: Possible slowest client is msp38. States processed - 65865741. States to be processed - 0 Oct 11 04:55:55 2020 test /kernel: rt_pfe_veto: Memory usage of M_RTNEXTHOP type = (0) Max size possible for M_RTNEXTHOP type = (8311787520) Current delayed unref = (60000), Current unique delayed unref = (10896), Max delayed unref on this platform = (40000) Current delayed weight unref = (71426) Max delayed weight unref on this platform= (400000) curproc = rpd Oct 11 04:56:00 2020 test /kernel: rt_pfe_veto: Too many delayed route/nexthop unrefs. Op 2 err 55, rtsm_id 5:-1, msg type 2 This issue only affects PTX Series devices. No other products or platforms are affected by this vulnerability. This issue affects Juniper Networks Junos OS on PTX Series: 18.2 versions prior to 18.2R3-S7; 18.3 versions prior to 18.3R3-S4; 18.4 versions prior to 18.4R2-S8, 18.4R3-S7; 19.1 versions prior to 19.1R3-S4; 19.2 versions prior to 19.2R3-S1; 19.3 versions prior to 19.3R3-S1; 19.4 versions prior to 19.4R2-S4, 19.4R3-S1; 20.1 versions prior to 20.1R2; 20.2 versions prior to 20.2R2; 20.3 versions prior to 20.3R1-S2, 20.3R2. This issue does not affect Juniper Networks Junos OS versions prior to 18.2R1.