Python Process Stuck at 8% CPU: Off-CPU Analysis Traces a Blocking subprocess.run Call

When a Python process keeps receiving requests but its CPU usage stays low — a gunicorn worker stuck around 8% in our case — some code path is blocking the OS thread instead of doing work on the CPU. This tutorial shows how to find the exact blocking Python code path with OpenResty XRay’s off-CPU analysis: it traced 92.4% of the blocked time to a subprocess.run call on line 12 of a business processor.py file — with no code changes and no process restarts.

The Symptom: Python Process CPU Usage Stays Low Under Heavy Requests

Let’s run the top command to check the CPU usage of each process.

Look at this gunicorn python process. Its CPU usage is very low, only around 8%. And it doesn’t go up even though there are many requests coming in.

The ps command shows that this process is using the standard Python 3 binary executable that comes with the Linux distribution, and the access log file of this Python web application keeps growing with new client requests while the CPU usage stays low. This means that there is something blocking the Python code from running efficiently. But what is it? How can we find out?

Find the Blocking Python Code Paths with OpenResty XRay’s Guided Analysis

Let’s use OpenResty XRay to check out this unmodified process. We can analyze it in real time and figure out what’s going on.

Open the OpenResty XRay web console in the web browser, make sure you are on the right machine, and go to the “Guided Analysis” page. Among the problem types you can diagnose here, select “Low CPU usage and cannot go up”. Then follow the wizard: select the Python application, pick the gunicorn child process consuming 10% of the CPU resources (the one we saw earlier in top), and keep the defaults for the remaining steps — the application type, both Python and C language levels, and the 300-second maximum analysis time.

Start analyzing. The system will keep performing different rounds of analysis; two rounds are enough for this case, so we stop there and it automatically creates a report.

The report shows the #1 C-land code path blocking the CPU from running efficiently, accounting for 96% of the off-CPU time.

The first function is the poll system call. It means that the Python interpreter is waiting for I/O events. But what kind of I/O events? To find out, we need to look at more context here.

The _PyEval_EvalFrameDefault function indicates that it is currently running some Python code.

Then we look at the #1 off-CPU Python code path, which accounts for 92.4% of the blocked time.

Look at this Python run function defined in the standard subprocess.py module file. This means that the Python code is running a subprocess command and waiting for its output.

The function “handle_by_script” is in our business-level Python code base.

Click to see more details.

The most significant blocking code path was automatically derived from this Python-land off-CPU flame graph. It shows the big picture. Off-CPU analysis like this is also the standard way to diagnose latency problems — see how we pinpointed a 244ms latency spike in a 500k QPS OpenResty gateway.

Below are more detailed explanations and suggestions regarding the current issue.

It talks about the handle_by_script function we saw earlier.

It mentions that this function runs a subprocess.

Hover the mouse over the green box for the Python function named handle_by_script.

We can see the Python source file of this function. And its full path for the processor.py file in the tooltip.

And the source line number is 12.

Click the icon to copy the full Python source file path for this function.

Use the vim editor and paste the code path we just copied to look at the corresponding business python code. You can use any editor you like.

Go to line 12, as OpenResty XRay suggested.

We see that this Python code line is indeed calling the function subprocess.run, running an external bash script and waiting for its output.

It is also in the function handle_by_script shown in the earlier report. The same off-CPU approach also works for Perl and Go processes.

Automatic Analysis and Reports in the Insights Page

OpenResty XRay can also monitor online processes automatically and generate analysis reports for daily and weekly periods in the “Insights” page — so you don’t have to use the “Guided Analysis” feature by hand, though it remains useful for application development and demonstration purposes.

FAQ

Why is my Python process not using full CPU even under heavy load?

Because some code path is blocking the OS thread instead of doing work on the CPU. In the case above, the gunicorn worker stayed around 8% CPU under heavy requests because the Python code was running a subprocess command and waiting for its output, sitting blocked in the poll system call.

How do I find which Python code is blocking, without modifying the code?

Run OpenResty XRay’s “Guided Analysis” against the unmodified running process and choose “Low CPU usage and cannot go up”. The generated report automatically infers the most significant blocking code path, and hovering over a function in the off-CPU flame graph reveals its exact source file and line number.

Why does subprocess.run block my Python web application?

subprocess.run starts an external command and waits for it to finish. In the case above, the handle_by_script business function called subprocess.run on line 12 of processor.py to run a bash script, so the gunicorn worker spent 92.4% of its blocked time waiting for the subprocess’s output instead of serving requests.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.