Today I’m going to show you how to use OpenResty XRay to quickly pinpoint the blocking Python code paths. These are the code paths that might slow down your processes and prevent them from using the full CPU power.

Problem: low CPU usage

Let’s run the top command to check the CPU usage of each process.

Look at this gunicorn python process. Its CPU usage is very low, only 12%. And it doesn’t go up even though there are many requests coming in.

Let’s run the ps command to see more details about this process.

Here we can see that this process is using the standard Python 3 binary executable that comes with the Linux distribution.

Let’s take a look at the access log file for this Python web application.

As you can see, there are a lot of client requests coming in, but the CPU usage is still low. This means that there is something blocking the Python code from running efficiently. But what is it? How can we find out?

Use the guidede analysis feature of OpenResty XRay to spot the blocking Python code paths

Let’s use OpenResty XRay to check out this unmodified process. We can analyze it in real time and figure out what’s going on.

Let’s open the OpenResty XRay web console in the web browser.

Make sure it is the right machine you are watching.

Go to the Guided Analysis page.

Here you can see different types of problems that you can diagnose.

Let’s select “Low CPU usage and cannot go up”.

Click on “Next”.

Select the Python application.

Select the process that consumes 10% of the CPU resources. This is what we saw previously in top.

Make sure that the application type is right.

Usually the default should be correct.

OpenResty XRay can analyze multiple language levels at the same time. We’ll keep both Python and C selected.

We can also set the maximum analyzing time. We’ll leave it as 300 seconds, which is the default value.

Let’s start analyzing.

The system will keep performing different rounds of analysis. Now it’s executing the first round.

The first round is done and it’s on to the second one already. That’s enough for this case.

Let’s stop analyzing now.

We can see it automatically creates a report.

This is the C code path that is blocking the CPU from running efficiently.

The first function is the poll system call. It means that the Python interpreter is waiting for I/O events. But what kind of I/O events? To find out, we need to look at more context here.

The _PyEval_EvalFrameDefault function indicates that it is currently running some Python code.

Then we look at the #1 off-CPU Python code path.

Look at this Python run function defined in the standard subprocess.py module file. This means that the Python code is running a subprocess command and waiting for its output.

The function “handle_by_script” is in our business-level Python code base.

Click to see more details.

The most significant blocking code path was automatically derived from this Python-land off-CPU flame graph. It shows the big picture.

Below are more detailed explanations and suggestions regarding the current issue.

It talks about the handle_by_script function we saw earlier.

It mentions that this function runs a subprocess.

Hover the mouse over the green box for the Python function named handle_by_script.

We can see the Python source file of this function. And its full path for the Processor.py file in the tooltip.

And the source line number is 12.

Click the icon to copy the full Python source file path for this function.

Use the vim editor and paste the code path we just copied to look at the corresponding business python code. You can use any editor you like.

Go to line 12, as OpenResty XRay suggested.

We see that this Python code line is indeed calling the function subprocess.run.

It is also in the function handle_by_script shown in the earlier report.

Automatic analysis and reports

OpenResty XRay can also monitor online processes automatically and show analysis reports.

Go to the Insights page.

You can find the reports in the Insights page for daily and weekly periods.

For this reason, you don’t have to use the Guided Analysis feature. Guided analysis is useful for application development and demonstration purposes.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.