P99 Latency: Why Averages Lie, and How to Diagnose the Slowest 1%

P99 latency is the value at the 99th percentile once you sort every request by response time from fastest to slowest — 99% of requests are faster than this value, and only the slowest 1% exceed it. In systems at scale, that “slowest 1%” means the experience of hundreds or thousands of real users every second. Average latency can hide these spikes; P99 does not.

You own a production service. Support forwards a complaint: “Payments occasionally hang, and a retry fixes it.” You open your dashboards: average latency is flat, the error rate is zero, CPU is low — everything is green. You check the usual suspects out of experience — slow queries, external dependencies, concurrency settings — all normal. You try to reproduce it under load; it won’t reproduce. Meanwhile the complaints keep trickling in, upstream and downstream teams come knocking with their SLAs, and you don’t have a single metric that proves the problem exists.

This is the classic predicament of a tail-latency problem. What makes it so stubborn is not that the technology is arcane, but that it hits four blind spots at once: monitoring reports aggregates, so slow requests are diluted by the average; it is intermittent, so it doesn’t exist in your test environment; the root cause often only triggers under real production load — an O(n) query only explodes at real scale, a regex only backtracks on specific input; and production is precisely where you can least afford to change code, add instrumentation, or restart the service. The first metric you should look at here is P99. This article starts from the definition and unfolds the full root-cause spectrum and diagnostic path — every number in the cases below comes from a real production environment.

What is P99 latency?

P99 latency (99th percentile latency) characterizes the response-time boundary of the slowest 1% of a batch of requests. What does it look like in the data? Suppose you collected the response times of 100 requests and sorted them from fastest to slowest:

#1  … #98 :  ≤ 10ms
#99       :  300ms   ← P99 = 300ms
#100      :  450ms

The first 98 requests all complete within 10ms, but the 99th is 300ms, so P99 is 300ms. This isn’t just a theoretical construct — it’s exactly the shape we saw in a fintech gateway case, which we’ll walk through in full below.

How it’s computed

Computing P99 involves no averaging. It’s pure rank-and-select:

1. Collect the response times of a batch of requests
2. Sort them from smallest to largest
3. Take the value at the 99th-percentile position

Note: P99 does no averaging — whatever value sits at the 99th-percentile position is the answer. That’s why it isn’t diluted, the way an average is, by the large mass of “normal” requests in the middle; it reflects the tail experience directly.

P50 vs P95 vs P99: what each percentile tells you

Different percentile latencies answer different questions. The table below helps you pick the right metric:

For example, suppose the same batch of 1,000 requests yields three readings: P50 = 8ms, P95 = 40ms, P99 = 300ms. All three numbers are true, but only P99 exposes the tail — watch only the first two and you’d think the system is healthy.

P50 tells you the “normal case,” P95 tells you the “upper bound most of the time,” and P99 tells you the “worst real experience.” In a high-traffic system, 1% is not a number you can ignore — if your service handles 10,000 requests per second, P99 describes what 100 users experience every second.

Which percentile you choose as an SLO depends on the business context. Payment gateways and trading systems typically hold themselves to P99 or even P99.9; internal tools may only need to watch P95. The key principle: if you only look at P50, your understanding of the system is optimistic; if you only look at the average, your understanding may be wrong.

Why average latency lies

Here’s an analogy: airport security averages 30 seconds per person, but the passenger in front of you who got pulled aside for a bag search took 10 minutes. The number “30 seconds on average” isn’t lying, but it has zero explanatory power for your queueing experience right now — latency metrics work the same way.

Latency distributions have a mathematical property: they have a lower bound (they can’t go below 0ms) but no upper bound (a single GC pause or lock contention can push one request’s latency arbitrarily high). This right-skewed distribution means a few extremely slow requests barely move the average, yet they push P99 to alarming heights.

Two real cases show how insidious this is:

• Everything green, SLA breached. A fintech gateway peaking at 500,000 QPS had a P50 steadily within 10ms and dashboards that looked entirely normal, yet deep sampling with OpenResty XRay found that the slowest 1% of requests exceeded 300ms — already threatening the SLA on a critical transaction path, and one step away from HTTP 504 timeouts at the gateway. What the root cause was, we’ll cover in the spectrum in the next section.
• Error rate, throughput, and average all normal; P99 alone spikes. A D-language order service had entirely healthy routine monitoring, yet P99 shot from a 120ms baseline to 350ms. The team investigated the database, external dependencies, and concurrency settings in turn — all dead ends. OpenResty XRay flame graphs ultimately pinned it to two root causes inside the service itself — also cataloged in the next section’s spectrum.

Both cases make the same point: average latency is proof that “most people are fine,” not proof that “no one has a problem.” If your alerting only watches the average, you’ll find out about the tail after your users complain.

Why is P99 latency high? A root-cause spectrum

Before you investigate P99, ask yourself one question: when the spike happens, is the CPU busy?

This question splits P99 root causes along a single axis into two broad classes: the CPU is busy doing things it shouldn’t (on-CPU causes) and the CPU is waiting (off-CPU causes). This classification isn’t academic — it directly determines which kind of tool you should reach for.

The CPU is busy on the wrong things (on-CPU causes)

When CPU utilization is high and P99 is high, time is being wasted on useless computation. All six categories below come from root causes we’ve pinpointed in real production environments:

① Regex backtracking. Nested-quantifier patterns (shapes like (a+)+$) can degenerate into exponential backtracking on specific input. The 300ms spike in the gateway case above was caused by exactly this: a string-matching function that took 244.64ms on a single call on specific input.

② Recompilation on every request. A regex in the log phase was recompiled on every single request, so this supposedly “low-overhead” phase was instead eating 26.5% of CPU. Enabling a compilation cache freed that CPU directly. (Same case)

③ Build defects. The root cause isn’t in the code — it’s in the build pipeline: in one case the base image omitted the --with-pcre-jit build flag, so PCRE JIT acceleration was never in effect across the entire cluster (full case); in another, a debug-only -O0 compile flag was left in by mistake, measurably costing 10% of throughput (keepalive case).

④ Connection storms. An upstream block missing a single keepalive line turns every request into a full TCP handshake and teardown:

upstream backend {
    server 10.0.0.2:8080;
    # Missing the keepalive directive here; connect()/close() will show up as two wide frames on the CPU flame graph
}

The kernel spends its time setting up and tearing down connections instead of processing data. After adding a single keepalive 64; line, throughput recovered from 6,301 QPS to 21,923 QPS — a 3.48x improvement.

⑤ GC pauses. D’s conservative GC can’t distinguish pointers from “integers that look like pointers,” so it must scan the entire heap word by word; under high-frequency allocation this forms a positive feedback loop — the bigger the heap, the longer the scan, the longer the pause. GC took 26.4% of CPU in the flame graph. (D-language case)

⑥ An O(n) algorithm exploding at scale. An order query walked the full array with a linear filter — no sign of trouble under light load, but 59.4% of CPU under real traffic. Code review can’t catch it; only a flame graph under production load can see it. (Same case)

The method for pinpointing on-CPU root causes is sampling plus CPU flame graphs. For how to read flame graphs and hands-on techniques, see our flame graphs explained guide.

The CPU is waiting (off-CPU causes)

If your P99 is high but CPU utilization is low, the time is going to waiting rather than computing: lock contention, disk/network I/O blocking, slow upstream/downstream dependencies, scheduling queueing (a thread is ready but can’t grab a core). Problems like these don’t reveal their cause in top, and there’s no prominent wide block on a CPU flame graph either — the time a thread spends off the CPU is invisible to on-CPU sampling; no amount of sampling captures the “waiting” itself.

This kind of root cause is especially lethal to tail latency on event-driven servers: when a single worker blocks, all concurrent requests on that worker wait idle at once. In one production OpenResty application, 99.8% of the process’s off-CPU time was traced to an io.popen call and its pipe read in the Lua code — synchronous I/O had stalled the entire event loop, and switching to a non-blocking API raised single-core throughput from 126 RPS to 18,537 RPS. Another case quantified the severity of the blocking: a single iteration of the event loop blocked for as long as 75 milliseconds — and those 75 milliseconds became a latency spike for every concurrent request on that worker.

To diagnose this kind of root cause you need off-CPU analysis — it measures the time and call stacks during which a thread is off the CPU, and it complements on-CPU flame graphs exactly: the latter answers “what is the CPU busy with,” the former answers “what is the process waiting for.” For the specific methods, tools, and real root-cause chains across runtimes like Perl, Go, Python, and Nginx, see our off-CPU analysis guide.

Uneven request distribution (configuration causes)

⑦ reuseport not enabled. When an Nginx listening port doesn’t have reuseport enabled, requests can’t be distributed evenly across worker processes — some workers are saturated while others sit nearly idle, and the saturated ones produce tail latency. This kind of root cause hides in the operating system’s request-distribution layer, and the total CPU usage in top shows nothing unusual. After tuning the configuration, overall performance improved 20–30%.

The root cause isn’t on your server (the client)

Sometimes the culprit behind P99 isn’t on the server side at all. A travel-industry customer found that some API requests carried an extra 200ms of latency; OpenResty XRay’s intelligent packet capture captured only on the slow TCP connections, in a targeted way, and ultimately found the latency came from the client — their Android app deliberately delayed 200ms after sending the request headers before sending the request body. No amount of server-side optimization would ever solve this.

How to diagnose high P99 latency

Back to that predicament at the start: complaints in hand, dashboards all green, reproduction failing. What you need here isn’t more guesswork — it’s to first figure out where the latency is going. Optimizing without diagnosing is leaving it to luck. The D-language case above is a ready-made argument: the team investigated the database, external dependencies, and concurrency settings in turn — all dead ends — because the root cause was in the service’s own runtime behavior, while every piece of “conventional wisdom” pointed outward.

Here is a diagnostic path validated across multiple production cases:

Step 1: Confirm the tail is real

Under low traffic, P99 can be unreliable — the sample is too small, and one or two stray slow requests can push P99 high. Confirm that the P99 jitter you’re observing recurs persistently at a sufficient request volume, rather than being statistical noise.

Step 2: Catch the slow request itself, not the aggregate metrics

Aggregate metrics (average, P95, error rate) tell you “there’s a problem,” but not “where the problem is.” You need to capture the specific slow requests: which URI did they hit? Which code path did they take? Which step took the most time? In the client-latency case, it was precisely the selective packet capture targeting slow connections that attributed the 200ms of latency precisely to the client’s sending behavior, not the server’s processing.

Step 3: Triage — busy or waiting?

This step decides which kind of tool you reach for next:

Step 4: Pinpoint to the function and line level in production

At this step, you run into the hardest of the four blind spots from the start: the root cause only appears in production under real load, so you need function-level, even line-level, performance data — yet you can’t change code, can’t add instrumentation, and can’t restart the service. For financial core systems in particular, this is a line you cannot cross. Traditional methodology reaches its limit here.

OpenResty XRay solves this with non-invasive dynamic tracing: it samples the running process directly, with no code changes, no instrumentation, and no restarts, automatically generating flame graphs and pinpointing specific functions. In the two cases above, the vague “P99 occasionally over budget” was restored, exactly this way, into specific functions and specific percentages within minutes.

How to reduce P99 latency: fix what you found

After diagnosis, the direction of the fix is dictated by the root cause — you don’t work down a checklist item by item, you fix directly what you saw in the flame graph or packet capture. The table below summarizes the root causes we pinpointed in real cases, the corresponding fixes, and the measured gains:

The table above is only the conclusions. How each root cause was pinpointed step by step while every dashboard stayed green, and how the fix was verified to work afterward — how to read the flame graph, how the evidence adds up — the full process is in each case’s original article, and that part is the reusable method.

Note one key point: you must re-sample after every optimization step. A flame graph is a snapshot under a specific load, and optimizing changes the system’s hotspot distribution — in the D-language case, only after GC pressure dropped did the sampling proportions of the other hotspots, previously “inflated” by GC pauses, return to their true weight. Planning all subsequent steps off the first flame graph is a common mistake.

Finding the root cause is expensive; the fix is often cheap — a single keepalive directive, one build flag, one cache switch. That is exactly why the diagnostic path above exists.

Summary

Compressing the whole article into four sentences:

• P99 is the yardstick of the tail experience; it quantifies what the slowest 1% of requests actually go through.
• The average can’t prove anything about the tail: the latency distribution has a lower bound but no upper bound, and spikes don’t move the average.
• Tail root causes split along a single axis into two classes: the CPU is busy on the wrong things, or the CPU is waiting.
• Diagnosis comes before optimization: see the root cause first with a flame graph or packet capture, then fix the one you saw.

Frequently Asked Questions

What is a good P99 latency?

There’s no single number that fits every scenario. The target value for P99 depends on your SLO and business context: user-facing critical transaction paths (like a payment gateway) are typically far stricter, while internal batch jobs are far more lenient. The right way to judge is: at your current traffic, does P99 meet the promise you’ve made about end-user experience or your upstream/downstream SLAs? If it does, and it’s stable, then it’s “good.”

Why is P99 high when average latency looks normal?

Because the latency distribution has a lower bound but no upper bound — a few slow requests barely move the average, but they push P99 high. The fintech gateway above is a textbook example: P50 under 10ms, dashboards all green, yet the slowest 1% exceeded 300ms. The average diluted spikes like these into statistical noise.

Why is P99 latency high when CPU utilization is low?

Low CPU with high latency means the time is going to waiting rather than computing — lock contention, disk/network I/O blocking, slow upstream/downstream dependencies, and scheduling queueing all fall into this category. Time like this is invisible to a CPU flame graph: the thread has already left the CPU, so on-CPU sampling naturally can’t capture it. Diagnosing this kind of problem requires off-CPU analysis — measuring the time and call stacks during which the thread is off the CPU. For the specific methods, see our off-CPU analysis guide.

What’s the difference between P99 latency and tail latency?

Tail latency is a general term for the tail of the latency distribution — the slowest batch of requests. P99 is one way to quantify tail latency: it takes the 99th percentile as the yardstick for the tail. P99.9 and P99.99 are more extreme tail-latency metrics. In practice, “P99 latency” and “tail latency” are often used interchangeably, but strictly speaking P99 is a precise statistic, while tail latency is a qualitative description of the shape of the distribution.

How do you measure P99 latency in production?

Two common approaches: one is to collect the response time of each request at the application or gateway layer and compute percentiles (most APM and monitoring systems support this); the other is to sample the running process directly with a non-invasive dynamic-tracing tool (such as OpenResty XRay), with no code changes or instrumentation. The key difference: the former tells you what P99 is, the latter can also tell you why P99 is high — it restores latency into a function-level flame graph, letting you see exactly which function, which line of code, the slow request’s time went to.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.