OpenResty XRay is a non-intrusive diagnostic system built on dynamic tracing technology. Unlike traditional APM agents, it performs no code injection or modification on the target process by default. Only when a user initiates an analysis does the OpenResty XRay Agent collect the necessary data — on demand, at a low sampling frequency. This on-demand sampling approach fundamentally ensures minimal overhead and strong safety guarantees in production environments.

When profiling a live Perl application in production, the primary concern is the performance cost introduced by the profiling tool itself. This article uses real measurement data to quantify the actual impact of the OpenResty XRay Agent — running in production mode against a Perl process — on system CPU, memory, and load, as well as on application throughput and request latency.

The data shows that even while the profiler is running at full tilt, the impact on maximum application throughput is less than 0.3%, and the impact on average request latency is only 20 microseconds. Furthermore, when the OpenResty XRay Agent is not actively running an analysis task, it incurs zero performance overhead.

Test Environment and Performance Baseline

To establish a baseline for comparison, we used the top command to capture system metrics before starting the profiler. As shown below, the target perl process was consuming approximately 49% CPU, the system’s 1-minute load average was 0.87, CPU idle was approximately 87%, and available memory was approximately 2730 MB.

Observed System-Level Resource Changes During Analysis

To simulate a real-world diagnostic scenario, we used the OpenResty XRay console to launch a 300-second (5-minute) “High CPU usage” analysis against the Perl process in production mode.

Choosing “production mode” is critical: it is specifically designed for live environments and minimizes performance impact through optimizations such as low-frequency sampling.

The figure below shows the analysis task running in progress.

While the profiler was running, we observed the following subtle changes across system metrics:

• Target process CPU usage: Rose to ~50%, up approximately 1 percentage point from the baseline.
• System 1-minute load average: Rose to 1.04, a modest increase from the baseline of 0.87.
• CPU idle: Dropped to ~85%, barely changed from the baseline of 87%.
• Available memory: Dropped to ~2725 MB, a reduction of approximately 5 MB — well within normal variance.

Overall, the OpenResty XRay profiler’s impact on system-level resources (CPU, memory, and load) during the sampling period was measurable but minimal, with no meaningful impact on system stability.

Key Metrics: Precise Measurement of Throughput and Latency Impact

For production services, throughput and latency are the two most critical performance indicators. We ran three sets of comparative benchmarks to evaluate the impact on each.

1. Maximum Throughput

We used a load testing tool to measure the maximum throughput of the server process under three different conditions.

• Without Agent: 1024 req/s

• With Agent installed but profiler inactive: 1024 req/s (no change)

• With profiler active: 1021 req/s

The results show that with the profiler actively sampling, maximum throughput drops to approximately 1021 requests per second — just 0.3% below the baseline.

2. Average Request Latency

We measured the average request latency experienced by clients under high concurrency.

• Without Agent: 1250 μs

• With Agent installed but profiler inactive: 1250 μs (no change)

• With profiler active: 1270 μs

The results show that the profiler introduced only 20 μs of additional average request latency.

Conclusion

Through comprehensive measurement of system resources, application throughput, and request latency, we can conclude that the dynamic tracing architecture of OpenResty XRay delivers measurable and predictable performance overhead when conducting real-time diagnostics on production Perl applications — with negligible impact on core business metrics. This establishes it as a profiling tool that can be deployed with confidence as a permanent fixture in production environments.

The overhead introduced by automated analysis on the Insights and Dashboard pages is negligible as well.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.