OpenResty XRay is a non-intrusive diagnostic system built on dynamic tracing technology. Unlike traditional APM agents, it performs no code injection or modification on the target process by default. The OpenResty XRay Agent only collects data when a user explicitly initiates an analysis — on demand, at low sampling frequency. This on-demand sampling approach fundamentally ensures minimal overhead and safe operation in production environments.

When profiling a live Python application in production, the primary concern is the performance overhead introduced by the profiling tool itself. This article presents real benchmark data to precisely quantify the impact of the OpenResty XRay Agent — running in production mode — on system CPU, memory, and load, as well as on application throughput and request latency.

The data shows that even during active analysis, the impact on maximum application throughput is just 3.4%, and the impact on average request latency is only 0.16 milliseconds. When the OpenResty XRay Agent is not performing any analysis, its performance overhead is exactly zero.

Test Environment and Performance Baseline

To establish a baseline for comparison, we captured system metrics using the top command before starting the analyzer. As shown below, the target gunicorn process (a Python application) had a CPU usage of approximately 46.5%. The system’s one-minute load average was 0.7, the CPU idle percentage was approximately 87.2%, and available memory was approximately 2632 MB.

System-Level Resource Impact During Analysis

To replicate a real-world diagnostic scenario, we used the OpenResty XRay console to launch a 300-second (5-minute) “High CPU usage” analysis against the Python process in production mode (path: Guided Analysis → High CPU usage → select target process).

Selecting “production mode” is critical, as it is specifically designed for live production environments. Through optimizations such as low-frequency sampling, it minimizes performance impact — though this also means analysis runs may take longer.

During the analysis, we observed the following minor changes across system metrics:

• Target process CPU usage: Increased to ~48%, up approximately 1.5 percentage points from the baseline.
• One-minute load average: Rose to 0.84, an increase of 0.14 over the baseline of 0.7.
• CPU idle percentage: Dropped to ~86.7%, nearly identical to the baseline of 87.2%.
• Available memory: Held steady at ~2631 MB, a decrease of roughly 1 MB — no meaningful change.

In summary, while OpenResty XRay does introduce some measurable impact on system-level resources (CPU, memory, and load) during active profiling, the magnitude is negligible and poses no threat to system stability.

Key Metrics: Precise Measurement of Throughput and Latency Impact

For production services, throughput and latency are the defining measures of performance. We ran three sets of comparative benchmarks covering both of these core metrics.

1. Maximum Throughput

We used a load testing tool to measure the server’s maximum throughput across different configurations.

• With the OpenResty XRay Agent not installed, maximum throughput is approximately 2,300 requests per second.
• With the Agent installed but the profiler not running, maximum throughput remains unchanged.
• With the profiler actively sampling, maximum throughput is approximately 2,220 RPS — only 3.4% lower than the baseline without sampling.

To summarize: with the profiler running, maximum throughput sits at approximately 2,220 requests per second, just 3.4% below the no-sampling baseline.

2. Average Request Latency

We measured how profiler sampling affects request latency.

• With the OpenResty XRay Agent not installed, average request latency is 4.32 ms.
• With the Agent installed but the profiler not running, average request latency is unchanged.
• With the profiler running, request latency increases to 4.48 ms — an added overhead of just 0.16 ms.

Conclusion

Across all three dimensions — system resource consumption, application throughput, and request latency — the data confirms that OpenResty XRay’s dynamic tracing architecture delivers measurable, predictable overhead when performing real-time diagnostics on production Python applications, with negligible impact on core business metrics. This makes it a profiling tool that can be safely and continuously deployed in production environments without hesitation.

The overhead introduced by automatic analysis on the Insights and Dashboard pages is equally minimal.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.