In this tutorial, we’ll show the actual performance impact of OpenResty XRay when tracing Python applications. We will watch CPU, memory, load average, traffic throughput, latency, etc. when OpenResty XRay is actively sampling and analyzing the target system. OpenResty XRay is a non-invasive diagnostic system based on our own dynamic tracing technology. It has minimal performance overhead. It can analyze running applications in real-time to help pinpoint the root causes of various issues.

Process Performance Before Running Analysis

How will OpenResty XRay’s Agent impact the performance of a target Python application? Let’s see in this video.

Let’s run the top command to check the process list.

Right now, the Agent of is not running any analysis. So its performance impact on the target processes is strictly zero. Thanks to dynamic-tracing technologies. It never modifies the target processes in any way.

Look at this gunicorn process. We already know it is written in Python.

Its CPU utilization is about 46.5%.

The load average of the last minute is 0.7.

The CPU idle percentage is 87.2%.

The current available memory is about 2632MB.

Let’s instruct to actively analyze that Python process. Then we can see its performance impact on the target.

Performance Impact When the Analyzer is Running

Open the OpenResty XRay web console in the web browser.

Make sure it is the right machine you are watching.

The production mode is for analyzing online production environments. It tries hard to minimize the performance impact on the target applications and servers. It may take longer to analyze though.

Go to the “Guided Analysis” page.

Let’s select “High CPU usage”.

Click on “Next”.

Select the Python application.

Select the process that consumes 47% of the CPU resources. This is what we saw previously in top.

Let’s start analyzing. The system will keep performing different rounds of analysis.

It will keep sampling for 300 seconds, or 5 minutes.

Let’s return to the terminal window. We can see that CPU utilization is around 48%. It went up about 1.5%.

The load average of the last minute is 0.84 now, increased by 0.14 from 0.7.

The CPU idle percentage is 86.7%, nearly the same as the previous value of 87.2%.

The currently available memory is 2631MB, a decrease of about 1MB from before. There is no significant change from before.

Impact Measurement on Throughput and Latency

We measured the maximum server throughput in different cases. Here’s what we found.

Without installing the Agent, the maximum throughput is about 2300 requests per second.

When the Agent is installed but not running the analyzers, the request rate remains the same.

When the analyzers are sampling, the request rate is around 2220 RPS, only 3.4% lower than when not doing analysis.

It can be seen that running the analyzers for performance analysis has minimal impact on the maximum throughput of the target process.

Here is the benchmark showing the impact on request latency during the sampling process. And this is what we found in comparison.

Without installing the Agent, the average request latency is 4.32 milliseconds.

When the Agent is installed but the analyzers are not running, the average latency remains unchanged.

When the analyzers are running, the latency becomes 4.48 milliseconds. An increase of merely 0.16 milliseconds.

It is evident that running the analyzers for performance analysis has a minimal impact on the latency of the target process.

The overhead of automatic analysis in the “Insights” page and “Dashboard” page is similarly small and tiny.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.