In this tutorial, you will learn how to quickly identify the largest Perl memory objects or values in your applications. Thanks to the power of OpenResty XRay; we can pinpoint these Perl memory objects via detailed data reference paths. We’ll also demonstrate the Perl object memory distribution flame graph invented by ourselves. Memory leaks or large memory footprints are no longer a myth in your Perl code!

Problem: high memory usage

Let’s run the top command to check the memory usage of each process. As shown, there is a Perl process using the most memory.

It’s more than 180MB.

Let’s run the ps command to see more details about this process. We can see it is the standard Perl binary executable shipped with the Linux distribution.

Use the guided analysis feature of OpenResty XRay to find the largest Perl objects or values taking the most RAM

Let’s use OpenResty XRay to check out this unmodified process. We can analyze it in real time and figure out what’s going on. Open the OpenResty XRay web console in the web browser.

Make sure it is the right machine you are watching.

You can choose the right machine from the list below if the current one is not correct.

Go to the “Guided Analysis” page.

Here you can see different types of problems that you can diagnose.

Let’s select “High memory usage”.

Click on “Next”.

Select the Perl application.

Select the process that consumes more than 180MB of the memory resources. This is what we saw previously in top.

Make sure that the application type is right. Usually, the default should be correct.

OpenResty XRay can analyze multiple language levels at the same time. We’ll keep both Perl and C/C++ selected.

We can also set the maximum analyzing time. We’ll leave it as 300 seconds, which is the default value.

Let’s start analyzing.

The system will keep performing different rounds of analysis. Now it’s executing the first round.

The first round is done and it’s on to the second one already. That’s enough for this case.

Let’s stop analyzing.

We can see it automatically creates an analysis report.

This is the type of problem we are going to diagnose. It’s memory.

We can see most of the memory is allocated by the libc allocator. It’s more than 160MB.

This is the Perl memory object reference path occupying the most memory. Obviously, this Perl VM uses most of the memory allocated by Libc.

It is in the Services::Preprocessor Perl module.

In the module’s symbol table, there is a variable named order_name_cache.

And this variable is a Perl hash table.

Click to see more details.

This object reference path is automatically inferred from this Perl-land GC objects memory distribution flame graph.

Below are more detailed explanations and suggestions regarding the current issue. It explains the objects in the data reference paths one by one.

The module name Services::PreProcessor corresponds to the Perl source file path. So we can find the source files through them.

On the terminal, switch the current working directory to the Perl application’s source tree.

Use the find command to find the source file. Copy the full file path. Use the vim editor to open the source file. You can use any editor you like.

Let’s go back to the data reference path. Click the icon to copy the variable name for the Perl hash table.

Paste what we just copied. We can see the hash table is defined here.

And where this hash table variable is used.

Automatic analysis and reports

OpenResty XRay can also monitor online processes automatically and show analysis reports.

You can find the reports on the Insights page for daily and weekly periods. For this reason, you don’t have to use the “Guided Analysis” feature.

Though “Guided analysis” is useful for application development and demonstration purposes.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.