Perl Process Stuck at 13% CPU: Off-CPU Flame Graphs Pinpoint a Blocking HTTP Call

When a Perl process keeps receiving requests but its CPU usage stays low — around 13% in our case — some code path is blocking the OS thread, typically synchronous I/O such as an HTTP call waiting on select. This tutorial shows how to find the exact blocking Perl code path, down to the source file and line number, using OpenResty XRay’s off-CPU flame graphs — with no code changes and no process restarts.

The Symptom: Perl Process CPU Usage Stays Low Under Heavy Requests

Let’s run the top command to check the CPU usage of each process.

Take a look at this perl process. The CPU usage is only around 13%. The usage does not increase even as many requests continue to come in.

Run the ps command to check all Perl processes.

We can see it is the standard perl binary executable shipped with the Linux distribution.

Let’s take a look at the access log file for this Perl application.

We can see many client request are coming in, but the CPU usage remains low. This means that there is something blocking the Perl code from running efficiently. But we don’t know what it is. How can we find out?

Find the Blocking Perl Code Paths with OpenResty XRay’s Off-CPU Flame Graphs

Let’s use OpenResty XRay to check this unmodified process. You can analyze it in real-time and figure out what is happening.

Open the OpenResty XRay web console in the web browser, make sure you are on the right machine, and go to the “Guided Analysis” page.

Among the problem types you can diagnose here, select “Low CPU usage and cannot go up”.

Then follow the wizard: select the Perl application, pick the process consuming 14% of the CPU resources (the one we saw earlier in top), and keep the defaults for the remaining steps — the application type, both Perl and C/C++ language levels, and the 300-second maximum analysis time.

Start analyzing. The system will keep performing multiple rounds of analysis; the first round is enough for this case, so we stop there.

It automatically generated a report.

This is the problem type we’re analyzing, “off-CPU”.

This is the C code path that blocks operating system threads the most.

The first function is select, a system call.

Perl_pp_select is a built-in function that handles Perl’s select function. It is a part of Perl’s internal for monitoring and waiting for I/O events on sockets and other files.

We can see from this C function that Perl code is currently executing.

Next, we look at the hottest Perl code path blocking the OS thread from using the CPU.

The top C function, select, is the blocking point in the C code path we just saw.

The can_read Perl function of the Net::HTTP::Methods module waits until the socket has received new data for reading.

Along the caller chain, we can see that it is blocked when reading the response headers in this read_response_headers function.

remote_fetch is a function in our business-level code, it belongs to our own Perl module, Service::Processor.

The most critical blocking code path was automatically inferred from this Perl-land off-CPU flame graph. Off-CPU analysis like this is also the standard way to diagnose latency problems — see how we pinpointed a 244ms latency spike in a 500k QPS OpenResty gateway.

Below are more explanations and suggestions about the issue.

It refers to the select function we saw earlier.

It mentions that this function uses the select call to proceed to the next step based on the data.

This is also a reference to the remote_fetch function we saw earlier.

Let’s go back to the code path.

Hover the mouse over the green box for the remote_fetch Perl function.

You can see the source file and line number of this function in the tooltip.

Click the icon to copy the perl source file path.

On the terminal, use the vim editor to open the source file. Paste the file path we just copied. You can use any editor you like.

Go to line 66, as OpenResty XRay suggested.

We can see it is sending out an HTTP GET request and waiting for its response.

To avoid blocking HTTP requests in Perl, you can consider use nonblocking frameworks like Coro. The same off-CPU approach also works for Go and Python processes.

Automatic Analysis and Reports in the Insights Page

OpenResty XRay can also monitor online processes automatically and generate analysis reports for daily and weekly periods in the “Insights” page — so you don’t have to use the “Guided Analysis” feature by hand, though it remains useful for application development and demonstration purposes.

OpenResty XRay is a non-invasive diagnostic system based on our own dynamic-tracing technology. It can monitor and scan performance problems, behavioral issues, and security vulnerabilities in real-time.

FAQ

Why is my Perl process not using full CPU even under heavy load?

Because some code path is blocking the OS thread instead of doing work on the CPU. In the case above, the can_read function of the Net::HTTP::Methods module waits in the select system call until the socket receives new data, so the process sits mostly idle while client requests keep piling up.

How do I find which Perl code is blocking, without modifying the code?

Run OpenResty XRay’s “Guided Analysis” against the unmodified running process and choose “Low CPU usage and cannot go up”. The generated report automatically infers the most critical blocking code path, and hovering over a function in the off-CPU flame graph reveals its exact source file and line number.

What does the off-CPU flame graph show in this case?

It shows the code paths where the OS thread spends its time blocked rather than running on the CPU. In this case, the Perl-land off-CPU flame graph points directly at our remote_fetch business function, which blocks in read_response_headers on the select call while waiting for an HTTP response.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.