OpenResty XRay is a non-intrusive diagnostic system built on dynamic tracing technology. Unlike traditional APM agents, it performs no code injection or modification on the target process by default. The OpenResty XRay Agent collects data only when a user initiates an analysis session — on demand and at low sampling frequency. This on-demand sampling philosophy is what fundamentally ensures minimal overhead and safe operation in production environments.

When profiling a live Go application in production, the primary concern is the performance overhead introduced by the profiling tool itself. This article uses real measurement data to quantify the actual impact of the OpenResty XRay Agent — running in production mode — on system CPU, memory, and load, as well as on application throughput and request latency.

The data shows that even while the analyzer is actively running, the impact on maximum application throughput is just 1.5%, and the effect on average request latency is a mere 6 microseconds. When the OpenResty XRay Agent is idle — not executing any analysis task — it incurs zero overhead.

Test Environment and Performance Baseline

To establish a baseline for comparison, we captured system metrics using the top command before starting the analyzer. As shown below, the target gin-helloworld process (a Go application) was consuming approximately 43% CPU. The system’s one-minute load average stood at 0.62, CPU idle was around 84%, and available memory was approximately 1,546 MB.

During Analysis: Observed System Resource Impact

To simulate a real-world diagnostic scenario, we used the OpenResty XRay console to launch a 300-second (5-minute) “High CPU usage” analysis in production mode against the target Go process (via: Guided Analysis → High CPU usage → select target process).

Choosing production mode is critical: it is specifically designed for live environments and minimizes performance impact through techniques such as low-frequency sampling. The trade-off is that analysis sessions may take longer to complete.

During the analysis run, we observed the following changes across system metrics:

• Target process CPU usage: rose to about ~47%, up approximately 4 percentage points from the baseline.
• System one-minute load average: rose to 0.63, a negligible increase from the baseline of 0.62.
• CPU idle: held steady around ~85%, essentially unchanged from the baseline of 84%.
• Available memory: rose to ~1,622 MB, up 76 MB from the baseline.

In summary, the OpenResty XRay analyzer’s impact on system-level resources — CPU, memory, and load — during active sampling is measurable but minimal, posing no meaningful risk to system stability.

Key Metrics: Quantifying the Impact on Throughput and Latency

For production services, throughput and latency are the two metrics that matter most. We tested each across three scenarios to establish a clear comparison.

1. Maximum Throughput

We used a load testing tool to measure peak server throughput across three configurations:

• Without the OpenResty XRay Agent installed: maximum throughput was approximately 24,500 RPS.
• With the Agent installed but the analyzer idle: maximum throughput remained unchanged.
• With the analyzer actively sampling: maximum throughput was approximately 24,100 RPS — just 1.5% lower than the no-sampling baseline.

These results confirm that running the analyzer has a negligible impact on the target process’s maximum throughput.

2. Average Request Latency

We measured the impact of sampling on request latency across the same three scenarios:

• Without the OpenResty XRay Agent installed: average request latency was 406 microseconds.
• With the Agent installed but the analyzer idle: average request latency was unchanged.
• With the analyzer actively running: request latency rose to 412 microseconds — an increase of just 6 microseconds.

This demonstrates that running the analyzer has an equally negligible impact on request latency.

It is worth noting that automatic analysis on the “Insights” and “Dashboard” pages carries a similarly negligible performance overhead.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.