Have you ever found yourself staring at a pile of Perl call stacks and memory allocation logs, unable to make sense of their relationships? Or when troubleshooting performance issues in OpenResty applications, you know there’s something wrong with memory management, but can’t pinpoint which piece of code is causing the problem?

The limitations of traditional debugging tools often only let us see the symptoms of a problem—a crash point, an error log, or a static memory snapshot. It’s like fumbling in the dark, only able to touch a small part of the issue.

Today, we’re sharing a breakthrough debugging experience: through the powerful combination of UDB and OpenResty XRay, we not only tracked down the exact call locations of functions in Perl code but also revealed their complete execution paths and contextual relationships, leaving nowhere for memory management issues to hide. This is no longer static analysis, but an exploratory journey through the timeline of program execution.

What is UDB?

UDB is a revolutionary Time-Travel Debugger developed by Undo, specifically designed to address the pain points developers face when debugging complex programs. For Perl developers, UDB offers an unprecedented debugging experience:

• Time-Travel Debugging: No more repeatedly restarting applications to reproduce issues. UDB allows you to move backward or forward through your program’s execution history, like a time machine, precisely pinpointing the moment when problems occur.
• Precise Root Cause Identification: Through conditional breakpoints and data watchpoints, UDB helps you jump directly to the triggering conditions of issues, saving significant troubleshooting time.
• Deep Program State Analysis: Real-time viewing and modification of Perl variable values and memory states, providing deep insights into the internal workings of your program to quickly identify abnormal behavior.
• Enhanced Perl Debugging: UDB is fully compatible with the GDB interface, providing Perl developers with a powerful debugging toolchain that supports comprehensive analysis of Perl applications.

Powerful Combination with OpenResty XRay for an Unparalleled Debugging Experience

OpenResty XRay is an industry-leading dynamic tracing product that automatically analyzes running applications to precisely identify performance bottlenecks, abnormal behaviors, and security vulnerabilities. When UDB is used in conjunction with OpenResty XRay, you gain:

• Comprehensive Problem Diagnosis: Complete analytical perspective from macro performance to micro call stacks
• Precise Identification of Perl Application Bottlenecks: Quickly identify and resolve performance issues
• Non-Intrusive Analysis: Obtain deep runtime information without modifying code
• Actionable Optimization Recommendations: Specific improvement strategies based on actual runtime data

By combining UDB’s time-travel debugging capabilities with OpenResty XRay’s deep analysis features, Perl developers can significantly improve their diagnostic efficiency, reducing troubleshooting work that might typically take days down to hours or even minutes. This substantially lowers development and maintenance costs while accelerating product iteration.

Hands-on: Analyzing Perl Application Call Stacks with OpenResty XRay and UDB

Let’s walk through a real-world example demonstrating how to use UDB to analyze call stacks in Perl applications:

Step 1: Record Application Execution Traces and Replay Recorded Samples

First, use UDB’s Live Record tool to capture the execution process of your Perl application:

1. Use the Live Record tool to record a sample of a running Perl application. 1.1 Select ‘Live Recording’ in the OpenResty XRay console. 1.2 Choose the target application and process, then click ‘Start Recording’. 1.3 Click the ‘Generate Recording File’ icon to create a recording file. 1.4 After generation is complete, click ‘Stop Recording’ to end the recording process. 1.5 Download the recording file for analysis.

2. Compile the relevant tools on OpenResty XRay and download the compiled tools to your local environment.

3. Use the UDB tool to load the recorded sample and set up the debugging environment:

udb --sessions=no -ex "set pagination off" -ex "set python print-stack full" perl.rec

Step 2: Analyzing Recorded Samples

This article demonstrates how to examine Perl execution call stacks in udb, using Perl memory allocation analysis as an example. We’ll set a breakpoint at the malloc function.

0% 45,793> b malloc
Breakpoint 1 at 0x7ffff7d39350 (19 locations)
0% 45,793> c
Continuing.

Breakpoint 2.1, 0x00007ffff7d39350 in malloc () from /tmp/undodb.589172.1748251515.02232.13d6176da97764fb/debuggee-1-2qlyi70u/symbol-files/lib64/libc.so.6

Step 3: Analyzing the Underlying C Call Stack

4. Use the bt command to view the current C-level call stack:

12% 792,793> bt
#0  0x00007ffff7d39350 in malloc () from /tmp/undodb.645183.1748253655.3532953.516ae1d6b3245a9c/debuggee-1-8z7p5sbz/symbol-files/lib64/libc.so.6
#1  0x000000000049af33 in Perl_safesysmalloc (size=<optimized out>, size@entry=81) at util.c:153
#2  0x00000000004cf8a8 in Perl_sv_grow (my_perl=my_perl@entry=0x5e22a0, sv=sv@entry=0x1c37570, newlen=81, newlen@entry=80) at sv.c:1605
#3  0x00000000004beee0 in Perl_do_readline (my_perl=0x5e22a0) at pp_hot.c:1993
#4  0x00000000004b9c36 in Perl_runops_standard (my_perl=0x5e22a0) at run.c:41
#5  0x0000000000442d4b in S_run_body (oldscope=<optimized out>, my_perl=<optimized out>) at perl.c:2478
#6  perl_run (my_perl=0x5e22a0) at perl.c:2406
#7  0x000000000041c19a in main (argc=<optimized out>, argv=<optimized out>, env=<optimized out>) at perlmain.c:116

Looking at the C call stack, we can see that the system’s low-level malloc function is being triggered, but this only shows information at the C language level and doesn’t provide the complete call path of the Perl business code.

Since GDB’s built-in bt command only provides the call stack at the C language level, it’s not helpful for analyzing Perl language-level code. Therefore, we need to leverage OpenResty XRay to analyze the call stack at the Perl code level.

Step 4: Analyzing the Complete Call Stack of Perl Code

5. Load the Perl call stack analysis tool provided by OpenResty XRay:

0% 7,955> source perl-on-cpu.y.py

6. Use the perl_on_cpu command to get the complete Perl call stack:

0% 48,437> perl_on_cpu 
C:__libc_malloc
@/usr/local/openresty-perl/lib/site_perl/5.24.4/Net/Server/PreFork.pm:303
Net::Server::PreFork::run_parent
@/usr/local/openresty-perl/lib/site_perl/5.24.4/Starman/Server.pm:164
Starman::Server::run_parent
@/usr/local/openresty-perl/lib/site_perl/5.24.4/Net/Server/PreFork.pm:109
Net::Server::PreFork::loop
@/usr/local/openresty-perl/lib/site_perl/5.24.4/Net/Server.pm:58
Net::Server::run
@/usr/local/openresty-perl/lib/site_perl/5.24.4/Starman/Server.pm:106
Starman::Server::run
@/usr/local/openresty-perl/lib/site_perl/5.24.4/Plack/Handler/Starman.pm:25
Plack::Handler::Starman::run
@/usr/local/openresty-perl/lib/site_perl/5.24.4/Plack/Loader.pm:84
Plack::Loader::run
@/usr/local/openresty-perl/lib/site_perl/5.24.4/Plack/Runner.pm:279
Plack::Runner::run
@/usr/local/openresty-perl/bin/starman:38
main::(eval)

Through this complete Perl call stack, we can clearly see memory allocation behavior at line 303 in PreFork.pm.

Why More Developers Are Choosing UDB?

UDB’s time-travel debugging capability is one of its most powerful features. With this functionality, we can freely move forward or backward through the recorded execution trace, precisely locating the position we’re interested in.

By adding specific breakpoints, we can stop the process exactly where we need to. For example, if we want to analyze an upcoming memory violation, we can set a breakpoint on the free function.

0% 51,725> b free
Breakpoint 3 at 0x7ffff7d398c0 (41 locations)
0% 51,725> c
Continuing.

Breakpoint 3.1, 0x00007ffff7d398c0 in free () from /tmp/undodb.589172.1748251515.02232.13d6176da97764fb/debuggee-1-2qlyi70u/symbol-files/lib64/libc.so.6
0% 52,039> perl_on_cpu

C:__libc_free
@/usr/local/openresty-perl/lib/site_perl/5.24.4/Net/Server/PreFork.pm:309
Net::Server::PreFork::run_parent
@/usr/local/openresty-perl/lib/site_perl/5.24.4/Starman/Server.pm:164
Starman::Server::run_parent
@/usr/local/openresty-perl/lib/site_perl/5.24.4/Net/Server/PreFork.pm:109
Net::Server::PreFork::loop
@/usr/local/openresty-perl/lib/site_perl/5.24.4/Net/Server.pm:58
Net::Server::run
@/usr/local/openresty-perl/lib/site_perl/5.24.4/Starman/Server.pm:106
Starman::Server::run
@/usr/local/openresty-perl/lib/site_perl/5.24.4/Plack/Handler/Starman.pm:25
Plack::Handler::Starman::run
@/usr/local/openresty-perl/lib/site_perl/5.24.4/Plack/Loader.pm:84
Plack::Loader::run
@/usr/local/openresty-perl/lib/site_perl/5.24.4/Plack/Runner.pm:279
Plack::Runner::run
@/usr/local/openresty-perl/bin/starman:38
main::(eval)

We recently captured an interesting phenomenon using UDB: the Perl code call stacks for free breakpoints and malloc breakpoints are almost identical, with the only difference being in the final line of code.

This call stack analysis, precise down to the code line level, gives us unprecedented clarity into program execution logic. Imagine facing a complex memory management issue and being able to see the complete context of each allocation and deallocation with such precision—this is the revolution that time-travel debugging brings.

Traditional debugging approaches always leave us “feeling around in the dark”:

• GDB Core Dumps only show a “snapshot” of the moment a program crashes
• Log analysis only provides scattered “fragments of evidence”
• Problem reproduction often requires repeatedly restarting applications, which is extremely inefficient

But when OpenResty XRay is combined with UDB, everything changes:

1. Complete execution history playback: Even after the original process has terminated, you can freely navigate through the program’s entire execution history
2. Precise inspection at any point in time: Want to see the program state at a specific moment? Pause and analyze anytime
3. Dynamic context restoration: No more static snapshots—experience a vivid recreation of the execution process

Summary

Through this deep practice, we’ve witnessed a new paradigm in debugging. What does this combination bring to developers?

1. Panoramic Execution Context Not just seeing the underlying C call stack, but precisely tracking every step of Perl business code execution. From system calls to business logic, everything is in plain sight.

2. Freedom to Navigate the Time Dimension Say goodbye to the “restart method”! Go wherever you want in the program’s execution history—intermittent issues and complex scenarios are no longer nightmares.

3. Precision Targeting of Performance Bottlenecks Combined with OpenResty XRay’s analytical capabilities, memory allocation patterns, hotspot functions, performance bottlenecks—nothing can hide.

4. Post-mortem Analysis Superpowers Program crashed? Process exited? No problem. Through recorded samples, you can still perform complete post-mortem analysis—something traditional Core Dumps can’t even approach.

When your application involves complex memory management or cross-language calls, traditional debugging methods often drive you crazy. The UDB + OpenResty XRay combination is precisely the tool that allows you to fundamentally understand program behavior. If you want to leave behind the limitations of traditional debugging and experience this “god’s-eye view” debugging approach, UDB and OpenResty XRay are waiting for you. After all, in this fast-paced development world, who wouldn’t want a tool that makes debugging elegant and efficient?

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.