Today, I will show you another step-by-step guide for analyzing Erlang/OTP applications with OpenResty XRay. We’ll quickly pinpoint the hottest Erlang code paths in an already-running Erlang process. These code paths might take most of the CPU time of your applications, and they are all automatically inferred from the flame graphs design and developed by ourselves. OpenResty XRay can analyze both the Erlang language level and the underlying C language level of the Beam VM. OpenResty XRay is a truly non-intrusive dynamic analysis tool that requires no installation of any special modules or plugins in the target application, no recompilation of the target application, and no need to restart the running processes.

Problem: high CPU usage

Run the top command to check the CPU usage.

As shown, this process consumes more than 200% of CPU cores.

It is called reabr3. rebar3 is a tool for managing and building Erlang projects. Here, we use it to start the project.

Let’s run the ps command to see more details about this process.

We can see it is the standard rebar3 binary executable shipped with the Linux distribution. Naturally, this program is also compiled using the Erlang, which comes with the standard distribution.

Spot the CPU-hottest Erlang code paths

Let’s use OpenResty XRay to check out this unmodified process. We can analyze it in real time and figure out what’s going on.

Open the OpenResty XRay Web console in the web browser.

Make sure it is the right machine you are watching.

You can choose the right machine from the list below if the current one is not correct.

Go to the “Guided Analysis” page.

Here, you can see different types of problems that you can diagnose.

Let’s select “High CPU Usage”.

Click on “Next”.

Select the Erlang application.

Select the process that consumes almost 200% of CPU cores. This is what we saw previously in top.

Make sure that the application type is right. Usually, the default should be correct.

OpenResty XRay can analyze multiple language levels at the same time. We’ll keep both Erlang and C/C++ selected.

We can also set the maximum analyzing time. We’ll leave it as 300 seconds, which is the default value.

Let’s start analyzing.

The system will keep performing different rounds of analysis. Now, it’s executing the first round.

The first round is done, and it’s already on the second one. That’s enough for this case.

Let’s stop analyzing.

It automatically generated an analysis report.

This is the type of problem we diagnose. It’s CPU.

Look at the #1 hottest Erlang code path.

check_resp_contentfunction is a business function used to check the content of the response.

It calls the filter function of the lists module to filter the list.

The filter condition is an anonymous function that uses regular expression matching within itself. These calls all occur within the check_resp_content function.

Click to see more.

This code path is automatically inferred from this Erlang-land CPU flame graph.

Here are more detailed explanations of the earlier code path.

Go back to the previous hot code path. Hover the mouse over the green box for the first function. You can see the source file of this function in the tooltip.

The Erlang source line number is 15.

Click the icon to copy the source file path.

Use the vim editor to open this Erlang source file. You can use any editors you like.

Go to line 15, as OpenResty XRay suggested.

This is the run function of the re module.

This is the lists:filter function call we saw earlier in the report.

This code path is indeed inside the check_resp_content function. You can optimize this regex to avoid costly backtracking operations by the regex engine. Or consider using a non-backtracking regex engine.

These two code paths are similar, and both of them execute regular matches.

The third Erlang code path is calculating the length of the string.

iolist is the input string to be matched. And erts_iolist_size is calculating the length of the string to be matched.

The Content is the input string to be matched.

Switch back to the web console. Take a look at the C++ code path that takes up the most CPU time. The C++ language level analysis shows performance hotspots at the level of Erlang’s Beam VM.

The function match in PCRE performs regular matching.

The second function erts_pcre_exec is Erlang runtime’s wrapper for the pcre_exec function in the PCRE library.

re_run function in the re module of Erlang is used for executing regular expressions.

This hexadecimal address indicates that this code path is executed in JIT-compiled Erlang code.

Automatic analysis and reports

OpenResty XRay can also monitor online processes automatically and generate analysis reports. Go to the “Insights” page.

You can find the reports on the Insights page for daily and weekly periods. For this reason, you don’t have to use the “Guided Analysis” feature.

Though “Guided analysis” is useful for application development and demonstration purposes.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.