TCP Packet Capture on Slow Connections: Tracing a 200ms Nginx Request Body Delay

A travel-industry customer found that some web API requests were taking an extra 200 milliseconds for no apparent reason. OpenResty XRay ran a targeted TCP packet capture on only the slow connections — without touching any code or hurting production performance — and traced the nginx request body read latency to the client app itself: their Android software was deliberately delaying request body sends by 200ms after the headers.

Below, we walk through the full analysis and show how OpenResty XRay uses its “smart packet capturing” feature to capture packets on slow (or other erroneous) TCP connections automatically.

The Problem

Some of the API requests take way too long, about 200 milliseconds, even though the application servers should handle them very fast.

The long latency issue can have many different causes, such as slow disk I/O, packet loss in the kernel network stack or other network devices, high CPU usage that slows down the applications or web servers, or just poor network links and connectivity. With so many possible suspects, dynamic tracing tools like OpenResty XRay can make it much easier to solve these tricky problems.

Analysis: TCP Packet Capture on Slow Connections

Response latency distribution

OpenResty XRay starts by sampling the requests that the web server processes (in this case, it is an OpenResty or Nginx web server) in the customer’s production environment. One of the great things about OpenResty XRay is that it can safely analyze online applications and servers without any changes in the running user processes.

It then automatically creates the following logarithmic distribution chart for the response latencies.

We can notice that 5 requests with a long latency fall into the range of 131ms ~ 262ms.

The slowest requests

OpenResty XRay also automatically collected the URI and other info about those slowest requests.

We changed the Host column name so that we can share the sample here.

We can tell from the table above that all the slow requests are for the URI /view/shopcart/.

Smart Nginx Packet Capture

OpenResty XRay has the unique feature of capturing and analyzing TCP packets only on slow connections that we are interested in. In this case, we capture packets on TCP connections with a request latency longer than 100 milliseconds, for example.

OpenResty XRay automatically identified those slow TCP connections and captured the TCP packets on them. One of the samples looks like this in the web console of OpenResty XRay.

On this chart, the small circles represent the ingress packets for the server, while the squares represent the egress packets.

One packet stands out in the chart with a strangely long latency, about 200ms. It is the ACK+PUSH packet in a small circle, which means it is an incoming or ingress packet sent from the client.

At this point, we can rule out the application server or the gateway web server as the cause. It must be the client or the network links between the client and the server that add this extra 200ms latency. (For a similar per-request packet capture diagnosis on a Java Tomcat stack, see Java Tomcat Slow Response Time: Pinpoint the Exact Slow Request with PCAP.)

Root Cause: Client-Side Request Body Delay

It is simple to check that the network link between the client and the server is flawless. And our customer then turns their attention to the client applications for their Android mobile apps.

And soon they discover that the client software deliberately adds a 200ms delay between sending the request header and the request body — the root cause of the nginx request body read latency. Mystery solved!

Latency After the Fix

After the customer fixed the extra delay in their client application, the 200ms-long latency issue is gone, as evidenced by the new OpenResty XRay request latency distribution chart below.

FAQ

How do you capture TCP packets on only the slow connections?

OpenResty XRay’s smart packet capture filters at the connection level — you set a latency threshold (for example, 100ms), and it captures TCP packets only on connections whose request latency exceeds that threshold. This selective approach avoids capturing packets on every connection, so it does not affect production server performance.

What causes nginx request body read latency?

In this case, the nginx request body read latency was caused by the client application deliberately delaying request body sends by 200ms after sending the headers. The long latency can also have other causes, such as slow disk I/O, packet loss in the kernel network stack or other network devices, high CPU usage, or poor network links and connectivity. A targeted TCP packet capture on the slow connections can help narrow down which factor is responsible.

How do you tell if the delay is on the client side from a packet capture?

In the packet capture timeline, check which side sends the delayed packet. If an ingress packet from the client — such as an ACK+PUSH carrying the request body — arrives after a large time gap, the delay is on the client or network side. In this case, the 200ms gap appeared on an incoming ACK+PUSH packet, and the network link was verified to be flawless, so the delay pointed directly to the client application.

Does smart packet capture affect production server performance?

No. OpenResty XRay captures packets only on connections whose request latency exceeds the configured threshold, so the vast majority of connections are untouched. The article’s customer ran this in their production environment without any performance impact.

Fully-automated analysis

OpenResty XRay can automatically choose the right analyzers to sample the target process, automatically analyze the generated flame graph, and finally present the findings in a human-friendly form in the automatic analysis reports. This makes it much easier for our users, who do not have to monitor their servers, or manually run the suitable analyzer, or even understand the results sampled by the analyzers themselves.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.