In this tutorial, I’m going to show you how to use OpenResty XRay to quickly pinpoint the hottest Lua code paths. These code paths might take up most of the CPU time in your OpenResty or Nginx applications.

Problem: high CPU usage

Let’s start by running the top command to check the CPU usage on the target server. As you can see, an Nginx worker process is taking up almost 100% of a CPU core.

Use the guidede analysis feature of OpenResty XRay to spot the hottest Lua code paths

Let’s use OpenResty XRay to check out this unmodified process. We can analyze it in real time and figure out what’s going on.

Let’s open the OpenResty XRay web console in the web browser.

You need to sign in first if you haven’t.

Enter your account name, which is your email address.

Enter your password.

Click on “Sign in”.

Make sure it is the right server you are watching.

Go to the Guided Analysis page.

Here you can see different types of problems that you can diagnose.

Let’s select High CPU.

Click on “Next”.

Select the application.

Select the process that consumes 97% of the CPU resources.

Make sure that the application type is correct. Usually the default should be right.

OpenResty XRay can analyze multiple language levels at the same time. We’ll keep both Lua and C selected.

We can also set the maximum analyzing time. We’ll leave it as 300 seconds, which is the default value.

Let’s start analyzing.

The system will keep performing different rounds of analysis. Now it’s executing the first round.

The first round is done and it’s on to the second one already. That’s enough for this case.

Let’s stop analyzing now.

We can see it automatically creates a report.

The report shows the hottest Lua code path that takes up the most CPU time.

Click here to see more details.

There is also a CPU flame graph with the hot code path marked in red.

The hot Lua code path involves the MD5 computation function call.

And the caller functions along the path are from the business-level code of the target application.

Let’s find the exact source line in the application business code, starting with the gen_order_md5 Lua function.

When we hover over the function’s green box, we can see the full path of the Lua source file, processor.lua, in the tooltip.

And the source line number is 29.

Click the icon to copy the full Lua source file path for this function.

Use the VI editor and paste the code path we just copied to look at the corresponding business Lua code. You can use any editor you like.

From the previous report, we know it is line 29.

We can see this Lua source line indeed containes the md5 computation in a loop.

It is also in the function gen_order_md5 shown in the report.

Automatic analysis and reports

OpenResty XRay can also monitor online processes automatically and show analysis reports.

Go to the Insights page.

You can find the reports in the Insights page for daily and weekly periods.

For this reason, you don’t have to use the Guided Analysis feature. Guided analysis is useful for applicaton development and demonstration purposes.

OpenResty XRay is a non-invasive diagnostic system based on our own dynamic-tracing technology. It can monitor and scan the performance problems, behavioral issues, and security vulnerabilities in real-time.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.