Today I’m going to show you how to use OpenResty XRay to quickly pinpoint the blocking Perl code paths. These are the code paths that might slow down your processes and prevent them from using the full CPU power.

Problem: low CPU usage

Let’s run the top command to check the CPU usage of each process.

Take a look at this perl process. The CPU usage is only around 13%. The usage does not increase even as many requests continue to come in.

Run the ps command to check all Perl processes.

We can see it is the standard perl binary executable shipped with the Linux distribution.

Let’s take a look at the access log file for this Perl application.

We can see many client request are coming in, but the CPU usage remains low. This means that there is something blocking the Perl code from running efficiently. But we don’t know what it is. How can we find out?

Use the guided analysis feature of OpenResty XRay to spot the blocking Perl code paths

Let’s use OpenResty XRay to check this unmodified process. You can analyze it in real-time and figure out what is happening.

Open the OpenResty XRay web console in the web browser.

Make sure you are on the right machine.

You can choose the right machine from the list below if the current one is not correct.

Go to the “Guided Analysis” page.

Here you can see several types of problem that you can diagnose.

Let’s select “Low CPU usage and cannot go up”.

Click on “Next”.

Select the Perl application.

Select the process that consumes 14% of the CPU resources. This is what we saw earlier in top.

Make sure the application type is right. The default should be usually correct.

OpenResty XRay can analyze multiple language levels at the same time. We’ll keep both Perl and C/C++ selected.

We can set the maximum time to analyze. We’ll leave it at 300 seconds here, just as the default.

Let‘s start analyzing.

The system will keep performing multiple rounds of analysis. It is now executing the first round.

The first round is done and it’s already on the second one. That’s enough for this case.

Let’s stop analyzing.

It automatically generated a report.

This is the problem type we’re analyzing, “off-CPU”.

This is the C code path that blocks operating system threads the most.

The first function is select, a system call.

Perl_pp_select is a built-in function that handles Perl’s select function. It is a part of Perl’s internal for monitoring and waiting for I/O events on sockets and other files.

We can see from this C function that Perl code is currently executing.

Next, we look at the hottest Perl code path blocking the OS thread from using the CPU.

The top C function, select, is the blocking point in the C code path we just saw.

The can_read Perl function of the Net::HTTP::Methods module waits until the socket has received new data for reading.

Along the caller chain, we can see that it is blocked when reading the response headers in this read_response_headers function.

remote_fetch is a function in our business-level code, it belongs to our own Perl module, Service::Processor.

The most critical blocking code path was automatically inferred from this Perl-land off-CPU flame graph.

Below are more explanations and suggestions about the issue.

It refers to the select function we saw earlier.

It mentions that this function uses the select call to proceed to the next step based on the data.

This is also a reference to the remote_fetch function we saw earlier.

Let’s go back to the code path.

Hover the mouse over the green box for the remote_fetch Perl function.

You can see the source file and line number of this function in the tooltip.

Click the icon to copy the perl source file path.

On the terminal, use the vim editor to open the source file. Paste the file path we just copied. You can use any editor you like.

Go to line 66, as OpenResty XRay suggested.

We can see it is sending out an HTTP GET request and waiting for its response.

To avoid blocking HTTP requests in Perl, you can consider use nonblocking frameworks like Coro.

Automatic analysis and reports

OpenResty XRay can also monitor online processes automatically and show analysis reports.

Go to the “Insights” page.

You can find the reports in the insights page for daily and weekly periods. For this reason, you don’t have to use the “Guided Analysis” feature.

Though “Guided Analysis” is useful for application development and demonstration purposes.

OpenResty XRay is a non-invasive diagnostic system based on our own dynamic-tracing technology. It can monitor and scan performance problems, behavioral issues, and security vulnerabilities in real-time.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.