We will show how to analyze your OpenResty or Nginx applications without debug symbols. Even in the absence of any debug information, it can analyze executables through machine learning, reconstruct debug symbols, and perform in-depth and comprehensive analysis through dynamic tracing. OpenResty XRay has the superpower of generating debug symbols from stripped executables, which makes it possible to get detailed information about C and Lua functions, source files, and source lines even without debug symbols.

Problem: Debug Symbols are missing

Run the ps command to see the full command line for an OpenResty or Nginx process.

We can see the path of the executable file for this process. This OpenResty is compiled from a source by the user herself.

Use the file command to inspect this executable. Copy the file path.

The file is stripped and lacks the any debug symbols or symbol table. We cannot find the corresponding debug symbol installation packages for this program either.

Then use the file command to check the LuaJIT Dynamic Linked Library.

Once again, the debug symbols for the LuaJIT file are also missing.

Automatically Analyze and Rebuild Debug Symbols

Fortunately, OpenResty XRay has the superpower to generate debug symbols from stripped executable files. It makes it possible to do in-depth and full analysis via dynamic tracing.

Open the OpenResty XRay Web console in the web browser.

Go to the “Guided Analysis” page.

Here, you can see different types of problems that you can diagnose.

Let’s select these four types of problems.

Click on “Next”.

Select the OpenResty application.

Select the worker process.

OpenResty XRay can analyze multiple language levels at the same time. We’ll keep both Lua and C/C++ selected.

Let’s start analyzing.

OpenResty XRay will check if any executable file is missing debug symbols and then try to rebuild them automatically. The symbol rebuild task is now running.

After the symbols have been successfully rebuilt, it starts executing all the relevant analyzers as usual.

Let’s stop analyzing.

We can see it automatically created an analysis report.

The first section of the report is for the CPU problem type.

First, check out Lua’s performance hotspots.

Click to see more details.

The hot code path was automatically inferred from this C-land CPU flame graph.

Click to enlarge the flame graph.

We can see all the C function names, including inlined functions.

It even shows the C source file name.

And the source code line number! It looks like magic, doesn’t it?

Click to see more.

This is the #1 hottest Lua code path using the most CPU time.

The close function is called by the generate function.

This is a Lua function called generate. It is defined in the Lua source file lua/report.lua.

The generate function was called by the generate_monthly_report defined in the lua/invoice.lua file.

Hover the mouse over the green box for the generate function. We can see the full path for the Lua source file in the tooltip.

The Lua source line number is 11.

Click the icon to copy the source file path.

Use the vim editor to open the source file. And look at the Lua code in this file. You can use any editors you like.

Go to line 11, as OpenResty XRay suggested. You can see the close function in the report. Even without the debug symbols, we can still get details about the hot Lua function and source line.

The generate function is used to query the database and generate reports.

Now, let’s check out the memory report.

On the Lua land, a good portion of the memory is taken up by dead Lua objects not yet released by the LuaJIT garbage collector.

Click to enlarge it.

Zoom in this data reference path.

In this path, the table is the data structure of all Lua modules loaded by the current Lua application. Apparently, these loaded Lua modules also take up a lot of memory.

Automatic analysis and reports

OpenResty XRay can also monitor online processes automatically and generate analysis reports.

Go to the “Insights” page.

You can find the automatic reports on the “Insights” page for daily and weekly periods. For this reason, you don’t have to use the “Guided Analysis” feature.

Though “Guided analysis” is useful for application development and demonstration purposes.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.