In this tutorial, we’ll show the actual performance impact of OpenResty XRay when tracing Perl applications. We will watch CPU, memory, load average, traffic throughput, latency, etc. when OpenResty XRay is actively sampling and analyzing the target system. OpenResty XRay is a non-invasive diagnostic system based on our own dynamic tracing technology. It has minimal performance overhead. It can analyze running applications in real-time to help pinpoint the root causes of various issues.

Process Performance Before Running Analysis

How will OpenResty XRay’s Agent affect a target Perl application’s performance? Let’s find out in this video.

First, let’s run the top command to check the process list.

Right now, the Agent of OpenResty XRay is not running any analysis. So its performance impact on the target process are strictly zero. Thanks to dynamic-tracing technologies. It never modifies the target processes in any way.

Look at this process named ‘perl’.

Its CPU utilization fluctuates around 49%.

The last minute load average of the whole system is 0.87.

The current CPU idle percentage is about 87%.

The current available memory is about 2730MB.

Next, let’s instruct OpenResty XRay to actively analyze that Perl process. Then we can see its performance impact on the target process and target system.

Performance Impact When the Analyzer is Running

Open the OpenResty XRay web console in the web browser.

Make sure it is the right machine you are watching.

The production mode is for analyzing online production environments. It tries hard to minimize the performance impact on the target applications and servers. It may take longer to analyze though.

Go to the “Guided Analysis” page.

Let’s select “High CPU usage”.

Click on “Next”.

Select the Perl application.

Select the target process that we saw previously in top.

Let’s start analyzing.

The system will keep performing different rounds of analysis.

It will keep sampling for 300 seconds, or 5 minutes.

Let’s return to the previous terminal window.

We can see the CPU utilization is around 50%. It’s merely 1% higher than what we’ve seen before. So there is no measurable impact on the target process’s CPU usage.

And the last minute load average of the whole process is 1.04, a little higher than the previous value, 0.87.

The CPU idle percentage is about 85%, there is no significant change from the previous value, 87%.

The current available memory is about 2725MB, a decrease of 5MB from before, it’s still within the normal fluctuation range.

Impact Measurement on Throughput and Latency

We measured the maximum throughput of server processes in different cases. Here is what we found.

Without installing the OpenResty XRay Agent, the maximum throughput is about 1024 requests per second.

When the OpenResty XRay agent is installed but not running the analyzers, the request rate remains the same value.

When analyzers are sampling, the request rate is about 1021 requests per second, merely 0.3% lower than when not doing analysis.

It can be seen that running the analyzers for performance analysis has minimal impact on the maximum throughput of the target process.

Now, let’s check out the benchmark showing the impact on request latency during the sampling process. This is what we found in comparison.

Without installing the agent, the average request latency is about 1250 microseconds.

When the agent is installed but the analyzers are not running, the average latency remains unchanged.

When the analyzers are sampling, the latency is about 1270 microseconds. An increase of 20 microseconds.

So, running the analyzers for performance analysis has a minimal impact on the latency as well.

Let’s go back to the web console.

The overhead of automatic analysis in the “Insights” and “Dashboard” page is similarly small and tiny.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.