Tody I’m going to show you another step-by-step guide for OpenResty XRay. We’ll quickly pinpoint the hottest Perl code paths in an already running Perl process. These code paths might take most of the CPU time of your applications.

Problem: high CPU usage

Let’s start by running the top command to check the CPU usage.

As you can see, a Perl process is taking up 100% of a CPU core.

Run the ps command to see the full command line for this process.

We can see it is the standard Perl binary executable shipped with the Linux distribution.

Use the guidede analysis feature of OpenResty XRay to spot the hottest Perl code paths

Let’s use OpenResty XRay to check out this unmodified process. We can analyze it in real time and figure out what’s going on.

Let’s open the OpenResty XRay web console in the web browser.

Make sure it is the right machine you are watching.

Go to the Guided Analysis page.

Here you can see different types of problems that you can diagnose.

Let’s select High CPU Usage.

Click on “Next”.

Select the application.

Select the process that consumes 99% of the CPU resources. This is what we saw previously in top.

Make sure that the application type is right.

Usually the default should be correct.

OpenResty XRay can analyze multiple language levels at the same time. We’ll keep both Perl and C selected.

We can also set the maximum analyzing time. We’ll leave it as 300 seconds, which is the default value.

Let’s start analyzing.

The system will keep performing different rounds of analysis. Now it’s executing the first round.

The first round is done and it’s on to the second one already. That’s enough for this case.

Let’s stop analyzing now.

We can see it automatically creates a report.

This is the hottest C code path for CPU.

This is a C function named Perl_fbm_instr in the Perl VM runtime. It uses the FBM algorithm to search strings for regular expression matching.

The C function here indicates that it is indeed performing regular expression matching.

The perl_run function means that the Perl VM is currently executing Perl code.

After checking the hottest C code path, we then look at the hottest Perl code paths.

This C function is still Perl_fbm_instr we just discussed.

This is a Perl function named scan. It is part of the business logic and it’s in the Services::Processor package.

These two caller Perl functions give more context in the business-level code base.

And this Perl function belongs to the Dancer2 Perl web framework.

This is the second hottest Perl code path.

The only difference is this Glibc function, memchr. It searches a character in a memory buffer using the Intel AVX2 instruction set. is also part of the regular expression matching operation. The Perl function call chain is exactly the same.

Click “More” to see details about the #1 hottest Perl code path.

The code path was automatically derived from this Perl-land CPU flame graph.

Below are more detailed explanations and suggestions regarding the current issue.

It talks about the Perl_fbm_instr C function we saw earlier.

It mentions that this function uses the “Fast Boyer-Moore” string search alrogithm.

And it also covers the Perl function scan.

It mentions that this function is in the Services:Processor package.

Hover the mouse over the green box for the Perl function named scan.

We can see the Perl source file of the scan function. And its full path for the Processor.pm file in the tooltip.

The Perl source line number is 14.

Click the icon to copy the full Perl source file path for this function.

Use the vim editor and paste the code path we just copied to look at the corresponding business Perl code. You can use any editor you like.

Go to line 14 as shown in the report tooltip.

We see that this Perl code line is indeed doing regular expression matching.

It is also in the function scan as shown in the report.

Automatic analysis and reports

OpenResty XRay can also monitor online processes automatically and show analysis reports.

Go to the Insights page.

You can find the reports in the Insights page for daily and weekly periods.

For this reason, you don’t have to use the Guided Analysis feature. Guided analysis is useful for application development and demonstration purposes.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.