OpenResty XRay is a non-intrusive diagnostic system built on dynamic tracing technology. Unlike traditional APM agents, it performs no code injection or modification on the target process by default. The OpenResty XRay Agent only collects data when a user explicitly initiates an analysis session, doing so on-demand and at low sampling frequencies. This on-demand sampling principle fundamentally ensures minimal overhead and safe operation in production environments.

When profiling a live Rust application in production, the primary concern is the performance overhead introduced by the profiling tool itself. This article presents empirical measurements that precisely quantify the real-world impact of the OpenResty XRay Agent on system CPU, memory, and load, as well as on application throughput and request latency, while performing dynamic tracing analysis on a Rust application in production mode.

The data shows that even while the profiler is actively running, the impact on maximum application throughput is just 2.2%, and the impact on average request latency is only 1.12 microseconds. Furthermore, when no analysis task is running, the OpenResty XRay Agent incurs exactly zero overhead.

Test Environment and Performance Baseline

To establish a baseline for comparison, we recorded the system’s performance metrics using the top command before starting the profiler. As shown below, the target rocket-server process (a Rust application) was consuming approximately 73% CPU. The system’s 1-minute load average was 0.86, the CPU idle percentage was approximately 74.3%, and available memory was approximately 1,566 MB.

System-Level Resource Changes During Analysis

To replicate a real-world diagnostic scenario, we used the OpenResty XRay console to launch a 300-second (5-minute) “High CPU usage” analysis session against the Rust process in production mode (path: Guided Analysis → High CPU usage → select target process).

Selecting “production mode” is critical, as it is specifically designed for live environments and minimizes performance impact via optimizations such as low-frequency sampling. The trade-off is that analysis sessions may take longer to complete.

During the profiler run, we observed the following subtle changes across system metrics:

• Target process CPU usage: increased to ~74%, less than 1 percentage point above the baseline.
• System 1-minute load average: rose to 0.92, up 0.06 from the baseline of 0.86.
• CPU idle percentage: held steady at ~74.5%, nearly identical to the baseline of 74.3%.
• Available memory: dropped to ~1,564 MB, a decrease of only ~2 MB — effectively unchanged.

In summary, while the OpenResty XRay profiler does introduce some overhead on system-level resources (CPU, memory, and load) during sampling, the impact is minimal and poses no threat to system stability.

Key Metrics: Precise Measurement of Throughput and Latency Impact

For production services, throughput and latency are the defining measures of performance. We ran a three-part comparative benchmark covering both of these critical metrics.

1. Maximum Throughput

We used a load testing tool to measure the server’s maximum throughput across three different conditions.

• Without the OpenResty XRay Agent installed, the maximum throughput was approximately 56,600 requests per second.
• With the Agent installed but the profiler not running, the maximum throughput remained unchanged.
• With the profiler actively sampling, the maximum throughput was approximately 55,300 RPS — only 2.2% lower than the baseline without sampling.

These results demonstrate that running the profiler has a negligible effect on the target process’s maximum throughput.

2. Average Request Latency

We measured the impact on request latency while the profiler was sampling.

• Without the OpenResty XRay Agent installed, the average request latency was 37.79 microseconds.
• With the Agent installed but the profiler not running, the average request latency remained unchanged.
• With the profiler running, the request latency rose to 38.91 microseconds — an increase of just 1.12 microseconds.

This confirms that running the profiler also has a negligible effect on the target process’s request latency.

Conclusion

Through comprehensive measurement of system resources, application throughput, and request latency, we can conclude that OpenResty XRay’s dynamic tracing architecture ensures that the performance overhead of real-time diagnostics on production Rust applications is both quantifiable and predictable, with a negligible impact on core business metrics. This makes it a profiling tool that can be safely and confidently deployed in production environments on an ongoing basis.

The overhead of automatic analysis on the Insights and Dashboard pages is equally minimal.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.