OpenResty XRay is a non-intrusive diagnostic system built on dynamic tracing technology. Unlike traditional APM agents, it performs no code injection or modification on the target process by default. The OpenResty XRay Agent collects data only when a user initiates an analysis — on-demand and at low sampling frequency. This on-demand sampling philosophy is what fundamentally ensures minimal overhead and strong safety guarantees in production environments.

When profiling a live PHP application in production, the primary concern is always the overhead introduced by the profiling tool itself. This article uses real measurement data to quantify the actual impact of the OpenResty XRay Agent on system CPU, memory, and load, as well as on application throughput and request latency, while performing dynamic tracing analysis on a Perl application in production mode.

The data shows that even when the profiler is running at full tilt, the impact on maximum application throughput is just 3.7%, and the impact on average request latency is only 0.22 milliseconds. When no analysis task is running, the OpenResty XRay Agent incurs zero performance overhead.

Testing Environment and Performance Baseline

To establish a comparison baseline, we captured system performance metrics using the top command before starting the profiler. As shown below, the target php process was consuming approximately 50% CPU, the system’s 1-minute load average was 0.31, the CPU idle percentage was approximately 85%, and available memory was approximately 2,433 MB.

During Analysis: Observed System-Level Resource Changes

To simulate a real-world diagnostic scenario, we used the OpenResty XRay console to launch a 300-second (5-minute) “High CPU usage” analysis against the PHP process in production mode.

Selecting “production mode” is important here: it is specifically designed for live environments and minimizes performance impact through techniques such as low-frequency sampling. The trade-off is that analysis runs may take longer to complete.

During profiler operation, we observed the following subtle changes across system metrics:

• Target process CPU usage: Increased to ~53%, up approximately 3 percentage points from the baseline.
• System 1-minute load average: Increased to 0.41, a modest rise from the baseline of 0.31.
• CPU idle percentage: Increased to ~85.9%, essentially unchanged from the baseline of 85%.
• Available system memory: Held steady at ~2,433 MB, with no meaningful change.

In summary, the OpenResty XRay profiler does introduce some measurable impact on system-level resources (CPU, memory, and load) during sampling, but the magnitude is small enough to pose no threat to system stability.

Key Metrics: Precise Measurement of Throughput and Latency Impact

For production services, throughput and latency are the most critical indicators of performance. We ran three sets of comparative benchmarks covering these two core metrics.

1. Maximum Throughput

We used a load-testing tool to measure peak server throughput under different conditions.

• Without the Agent: 369 req/s

• Agent installed, profiler not running: 369 req/s (no change)

• Profiler active: 355 req/s

The results show that with the profiler actively sampling, peak throughput is approximately 355 requests per second — only 3.7% lower than the no-sampling baseline.

2. Average Request Latency

We measured the impact on request latency during profiler sampling.

• Without the Agent: 5.36 ms
• Agent installed, profiler not running: 5.36 ms (no change)
• Profiler active: 5.58 ms

The results show that running the profiler added only 0.22 ms to average request latency.

Summary

Through comprehensive measurement of system resources, application throughput, and request latency, we can conclude that the dynamic tracing architecture of OpenResty XRay delivers measurable, predictable overhead when diagnosing production PHP applications in real time, with minimal impact on core business metrics. This demonstrates that it is a profiling tool that can be safely and confidently deployed as a permanent fixture in production environments.

The overhead of automatic analysis on the Insights and Dashboard pages is equally negligible.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.