CVE 2023-44487 and How To Avoid an HTTP/2 Rapid Reset Attack
CVE-2023-44487, known as Rapid Reset, is a high severity denial-of-service (DoS) vulnerability in the HTTP/2 protocol. The damage from exploits of CVE 2023-44487 has been substantial as W3Techs reports that more than a third (35%) of the websites worldwide use HTTP/2.
In this blog, our expert explains how CVE 2023-44487 works and can be exploited, the impact to users, and mitigation steps you should take right away to avoid Rapid Reset attacks.
What Is CVE-2023-44487?
Back to topCVE-2023-44487, known as Rapid Reset, is a denial-of-service (DoS) vulnerability in the HTTP/2 protocol. It was first noticed in August 2023 and on October 10, the National Vulnerability Database assigned it a score of 7.5, indicating high severity. The attack takes advantage of an underlying weakness in HTTP/2, which leads to massive server resource consumption and impacts customer traffic.
Who Does CVE-2023-44487 Impact?
The CVE 2023-44487 Rapid Reset vulnerability is believed to impact anyone who has implemented HTTP/2, including all modern web servers. Vendors using DDoS mitigation services were able to absorb major parts of the attack, but given its unprecedented scale, a number of clients were still affected.
Rapid Reset attacks can have significant impact on services of any scale, and large-scale implementers of HTTP/2, including Google, AWS, and Cloudflare, were all affected.
Currently there is no evidence that CVE 2023-44487 affects HTTP/3, due to the large differences between the two protocols. Despite that, vendors using HTTP/3 capable servers are advised to proactively implement mechanisms similar to the mitigation steps listed further down in this blog.
Back to topHow Does CVE-2023-44487 Work?
Understanding how CVE 2023-44487 works first requires a basic understanding of how HTTP/2 functions as a web protocol.
What Is HTTP/2?
HTTP/2 is a major revision of the HTTP network protocol, derived from an earlier experimental SPDY protocol developed by Google. HTTP/2 is the first new version of HTTP since HTTP/1.1, which was standardized in 1997. HTTP/2 was presented for consideration in December 2014, and approved to publish in February 2015. It was updated in February 2020 in regard to TLS 1.3.
HTTP/2 uses streams to transmit messages, called "frames", between the endpoints. Stream multiplexing allows multiple streams to be opened over a single TCP connection. This enables better utilization of each connection, as clients can have multiple requests without managing multiple individual connections.
The HTTP/2 protocol allows clients to cancel a stream by sending an RST_STREAM frame to the server. This is a one-sided process because HTTP/2 does not require the client and server to coordinate the cancellation. The cancellation takes effect immediately after the server receives the RST_STREAM frame, before any other data from that TCP connection is processed.
The attack relies on the ability to send an RST_STREAM frame immediately after sending the request frame, which instructs the server to start working on the request, and then rapidly resets it. The request is canceled, but the HTTP/2 connection remains open and new streams can be created. The server still needs to do significant amounts of work for each of these canceled requests, such as allocating new stream data structures, parsing the query and decompressing the header, and mapping the URL to a resource.
This ability to cancel streams immediately allows each connection to have an indefinite number of these canceled requests. Since the requests are explicitly canceled, the attacker never reaches the limit on the number of concurrent open streams. This effectively drains the server resources and leads to negative impact on customer traffic, and subsequently, to denial of service.
An additional advantage for the attacker is that the server doesn't send a response to the canceled requests, which reduces the load on the attacker's endpoint.
In the aftermath of the initial attacks, two new Rapid Reset variants were observed. In the first one, the streams are not canceled immediately. Instead, a large batch of streams is opened, then after some time these streams are canceled, and a new batch is immediately opened. The second variant abandons the request cancellation entirely, and instead focuses on trying to open more concurrent streams than the server advertised.
Back to topCVE-2023-44487 Exploit Consequences
The attacks were significantly larger than any previous DDoS attacks, with Google reporting a peak of nearly 400 million requests per second. Meanwhile, Cloudflare revealed that the attack was almost three times bigger than their previous biggest DDoS attack on record, peaking at over 201 million requests per second.
It was subsequently discovered that the attacker was able to generate this massive attack using a botnet of only 20 thousand machines, which is a small number compared to modern botnets, made up of hundreds of thousands or even millions of machines. Given that the entire web typically generates 1 to 3 billion requests per second, using this method with a larger botnet could easily focus an entire web’s worth of requests on a single target.
Rapid stream generation and cancellation results in additional load on the server, which could lead to denial of service. The clients experience an increased number of "502 Bad Gateway" and "499 client closed request" errors as the servers struggle to process all the requests. Couldflare reports that about 12% of their client requests were impacted during the peak of the attack on August 29.
Major vendors utilizing HTTP/2 have since taken steps to mitigate the attack, effectively reducing the impact on their servers to 0.
Back to topCVE-2023-44487 Mitigation Steps
Blocking individual requests is not considered a viable mitigation against this class of attacks, as the number of requests is too high. Instead, the entire connection needs to be closed.
HTTP/2 provides built-in support for closing connections using the GOAWAY frame. This involves sending an informational GOAWAY frame that does not set a limit on opening new streams, and one round trip later sending another GOAWAY that forbids opening new streams. However, this process leaves the connection open to abuse for too long, so as a mitigation response, the GOAWAY needs to be set to limit the request creation immediately. In order to do this, the abusive connections need to be identified and distinguished from the legitimate ones. It's important to remember that the cancellation of requests is not always abusive and exists in HTTP/2 for better management of the request processing.
The mitigation of Rapid Reset mostly centers around tracking connection statistics and using various signals and business logic to determine how useful each connection is. For example, if a connection has a large number of requests with more than half of them canceled, it could be a candidate for a mitigation response. The response depends on the severity of the attack and the risk to each individual platform, and can range from a forceful GOAWAY frame to immediately terminating the connection.
To mitigate against the non-canceling variant of this attack, HTTP/2 servers should close connections that exceed the concurrent stream limit. Again, the response depends on the severity and the connection can be closed either immediately, or after a certain number of repeated offenses.
Customers who use HTTP/2 capable web servers are advised to verify with their web server vendor to determine if they are affected and, if so, install the latest patches from their respective vendors to address this issue.
CVE 2023-44487 Recommendations for Apache and Nginx
Typical Apache deployments are not affected by Rapid Reset, as reported by The Apache HTTP Server Project. The measures to limit excessive load that Apache already has are effective against the attack. It could cause extra resource consumption on the Apache processes, but would not impact any of the backends. As an extra mitigation measure, an upgrade of the libnghttp2 dependency of mod_http2 to at least version 1.57.0 is recommended.
Similarly, Nginx reports that their web server effectively prevents the Rapid Reset attack by using the default keepalive limit. However, if Nginx is configured with a keepalive that is substantially higher than the default and recommended setting, the attack may cause excessive resource consumption. A patch to prevent this has been released, and users who need to configure Nginx beyond the recommended specifications are advised to rebuild the binaries from the latest codebase.
Back to topFinal Thoughts
Since it was disclosed, CVE 2023-44487 has caused considerable damage, with widespread attacks resulting in frustrated customers getting DoS errors. These kinds of cyberattacks can negatively impact revenue (interfering with e-commerce transactions) and reputations of the companies whose websites are targeted. If you have implemented HTTP/2, your system may be vulnerable, and you should take immediate action to prevent Rapid Reset attacks before it's too late.
Need Technical Support for Your Web Infrastructure?
Companies around the world trust OpenLogic to support their open source software. Our experts can help optimize and troubleshoot all the OSS in your stack, including web infrastructure technologies like NGINX, Tomcat, and Apache Server. Contact us today to get a quote!
Additional Resources
- Guide - Overview of Open Source Web Infrastructure
- Blog - Best Practices for Web Server Security
- Blog - Understanding CVEs and CVSS Scores
- Blog - Debunking Open Source Software Security Myths
- On-Demand Webinar - Open Source Security and Compliance
- Video - Open Source Security
