Have you ever found yourself drowning in log files while troubleshooting an HTTP timeout issue? Or perhaps you’ve attempted to trace call stacks using Core Dumps, only to end up frustrated because you couldn’t capture the actual context of the problem?

In this deep dive, we leveraged UDB, a powerful time travel debugger, alongside OpenResty XRay, to completely reconstruct a Python network request from initiation to response. This approach allowed us to pinpoint the root cause that traditional debugging methods couldn’t reveal.

The Unique Value of UDB in Python Debugging

UDB (Undo Debugger) is a time-travel debugger that brings a revolutionary debugging experience to Python developers. Compared to traditional debugging tools like pdb or PyCharm, UDB’s greatest advantage lies in its ability to completely record program execution, allowing developers to freely navigate through execution history.

For an interpreted language like Python, UDB’s value is particularly significant:

• Complete Execution Trace Capture: Records every step of the Python interpreter’s execution, including calls to underlying C extension modules
• Unlimited Problem Reproduction: The same bug can be repeatedly analyzed without restarting the program
• Seamless Cross-Language Debugging: Simultaneously observe execution flows in both Python code and underlying C/C++ extensions
• Precise Performance Analysis: Leverage time-travel capabilities to accurately pinpoint performance bottlenecks
• Flexible Extension Capabilities: Built on GDB, fully compatible with GDB’s Python API, allowing debugger functionality to be extended through Python scripts

Enhancing Debugging Capabilities with OpenResty XRay

OpenResty XRay is a powerful dynamic tracing tool that automatically analyzes running applications to quickly identify performance bottlenecks, anomalous behaviors, and security vulnerabilities while providing actionable optimization recommendations. Under the hood, OpenResty XRay is powered by our proprietary Y language, enabling seamless support across multiple runtime environments including Stap+, eBPF+, GDB, and ODB.

UDB allows loading GDB backend Python extensions, enabling direct access to XRay’s advanced analysis tools within the UDB environment. By combining UDB’s time-travel debugging with OpenResty XRay’s deep performance analysis capabilities, developers gain comprehensive insights into application behavior—from macro-level performance metrics to micro-level call stacks—significantly accelerating problem diagnosis and resolution.

Real-World Example: Analyzing Python Network Request Call Stacks

Let’s dive into a practical example to demonstrate how UDB can be used to deeply analyze the execution flow of network requests in Python applications.

Step 1: Recording Application Execution Traces

First, we need to use UDB’s Live Record tool to capture the execution of a Python networking application:

1. We’ll select a Python application that uses the requests library to send HTTP requests as our sample, and record its execution with the Live Record tool to capture comprehensive execution details.

2. Then we’ll load this recording sample using the UDB tool:

udb -ex "set pagination off" -ex "set python print-stack full" python.rec

Step 2: Setting Up Key Network Request Breakpoints

When working in a UDB environment, we need to establish strategic breakpoints to capture critical phases of network requests. For HTTP requests, the most fundamental system calls to monitor are connect (establishing connection) and recv (receiving data):

start 1> break connect
Breakpoint 1 at 0x7ffff750f590: file ../sysdeps/unix/sysv/linux/connect.c, line 24.
start 1> break recv
Breakpoint 2 at 0x7ffff750f700: file ../sysdeps/unix/sysv/linux/recv.c, line 24.
start 1> c
Continuing.
[New Thread 1259245.1261346]
[Switching to Thread 1259245.1261346]

Thread 2 "python3" hit Breakpoint 1, __libc_connect (fd=fd@entry=3, addr=addr@entry=..., len=len@entry=16) at ../sysdeps/unix/sysv/linux/connect.c:24

When your program execution hits the connect system call, UDB will pause execution and display the current position. This indicates that your Python application is attempting to establish a network connection - the first crucial step in any HTTP request. By setting breakpoints at these key points, we can precisely capture and analyze each stage of network operations in real-time.

Step 3: Analyzing the Underlying C Call Stack

Once a breakpoint is triggered, the first diagnostic step is to use the bt command to examine the current C-level call stack:

5% 181,380> bt
#0  __libc_connect (fd=fd@entry=3, addr=addr@entry=..., len=len@entry=16) at ../sysdeps/unix/sysv/linux/connect.c:24
#1  0x00007ffff74f5b3f in try_connect (family=<optimized out>, addrlen=16, addr=0x7fffe40042a0, source_addrp=0x7fffe8d47848, afp=<synthetic pointer>, fdp=<synthetic pointer>)
    at ../sysdeps/posix/getaddrinfo.c:2272
#2  __GI_getaddrinfo (name=<optimized out>, name@entry=0x7fffe8d9a900 "localhost", service=<optimized out>, service@entry=0x7fffe8d99bb8 "8888", hints=<optimized out>, hints@entry=0x7fffe8d48070,
    pai=pai@entry=0x7fffe8d48058) at ../sysdeps/posix/getaddrinfo.c:2493
#3  0x00007ffff768e5b2 in socket_getaddrinfo (self=0x7fffe9cd7920, args=<optimized out>, kwargs=<optimized out>) at ./Modules/socketmodule.c:6763
#4  0x00007ffff79c98e8 in cfunction_call (func=0x7fffe9cd7f60, args=0x7fffe8d80400, kwargs=0x0) at Objects/methodobject.c:537
...省略部分输出...
#29 0x00007ffff7b667f8 in thread_run (boot_raw=0x7fffe4005aa0) at ./Modules/_threadmodule.c:1114
#30 0x00007ffff7aef967 in pythread_wrapper (arg=<optimized out>) at Python/thread_pthread.h:237
#31 0x00007ffff7489d22 in start_thread (arg=<optimized out>) at pthread_create.c:443
#32 0x00007ffff750ed40 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

This C call stack reveals some crucial information about a network connection issue:

1. From the bottom of the stack (#32-#30), we can see that this code is executing within a Python thread
2. At #2-#0, the system is executing getaddrinfo and connect functions, indicating the program is resolving a hostname and attempting to establish a connection
3. From #3, we can see this call originated from Python’s socket module

However, this C call stack primarily shows low-level implementation details, which provide limited help in understanding the Python application logic. We can’t directly determine which Python function or module initiated this network request. For that, we need to leverage tools provided by OpenResty XRay to analyze the Python code path.

Step 4: Analyzing the Python Call Stack

While the C call stack is detailed, it’s not intuitive for Python developers. To obtain more meaningful information at the Python level, we need to use specialized tools provided by OpenResty XRay:

1. First, load the Python call stack analysis tool provided by OpenResty XRay:

5% 181,380> source python-udb.y.py

This is a specialized Python extension developed by OpenResty XRay specifically for UDB environments. It’s designed to parse Python’s internal structures and extract complete Python call stacks with precision.

2. Next, let’s jump to the heart of the Python interpreter - the _PyEval_EvalFrameDefault function, which is responsible for executing Python code:

5% 181,380> b _PyEval_EvalFrameDefault
Breakpoint 3 at 0x7ffff790c100: file ./Include/internal/pycore_pystate.h, line 107.
5% 181,406> c
Continuing.

Thread 2 "python3" hit Breakpoint 3, _PyEval_EvalFrameDefault (tstate=0x93cb00, frame=0x7ffff73de000, throwflag=0) at ./Include/internal/pycore_pystate.h:107

This function serves as the core engine of the Python interpreter, where bytecode execution happens. Setting a breakpoint here allows us to capture the execution context of Python code in real-time, giving you unprecedented visibility into your application’s behavior.

3. Now we can leverage the powerful python_bt command to obtain a complete Python call stack:

5% 182,479> python_bt
C:_PyEval_EvalFrameDefault
@/usr/local/openresty-python3/lib/python3.12/socket.py:106
_intenum_converter
@/usr/local/openresty-python3/lib/python3.12/socket.py:978
getaddrinfo
@/usr/local/openresty-python3/lib/python3.12/site-packages/urllib3/util/connection.py:60
create_connection
@/usr/local/openresty-python3/lib/python3.12/site-packages/urllib3/connection.py:199
_new_conn
@/usr/local/openresty-python3/lib/python3.12/site-packages/urllib3/connection.py:279
connect
@/usr/local/openresty-python3/lib/python3.12/http/client.py:1035
send
@/usr/local/openresty-python3/lib/python3.12/http/client.py:1091
_send_output
@/usr/local/openresty-python3/lib/python3.12/http/client.py:1331
endheaders
@/usr/local/openresty-python3/lib/python3.12/site-packages/urllib3/connection.py:441
request
@/usr/local/openresty-python3/lib/python3.12/site-packages/urllib3/connectionpool.py:495
_make_request
@/usr/local/openresty-python3/lib/python3.12/site-packages/urllib3/connectionpool.py:789
urlopen
@/usr/local/openresty-python3/lib/python3.12/site-packages/requests/adapters.py:667
send
@/usr/local/openresty-python3/lib/python3.12/site-packages/requests/sessions.py:703
send
@/usr/local/openresty-python3/lib/python3.12/site-packages/requests/sessions.py:589
request
@/usr/local/openresty-python3/lib/python3.12/site-packages/requests/api.py:59
request
@/usr/local/openresty-python3/lib/python3.12/site-packages/requests/api.py:73
get
@/opt/app/processor.py:10
make_request
@/usr/local/openresty-python3/lib/python3.12/threading.py:1012
run
@/usr/local/openresty-python3/lib/python3.12/threading.py:1075
_bootstrap_inner
@/usr/local/openresty-python3/lib/python3.12/threading.py:1032
_bootstrap

This Python call stack clearly illustrates the entire request call chain:

• At the lowest level is the Python thread’s startup function, _bootstrap
• In the middle, we see various layers of calls through the requests library
• Finally, in our business code at /opt/app/processor.py line 10, the make_request function calls the requests.get method

This is much more intuitive than a C call stack, giving us a complete view of the call path from business code to low-level network operations.

4. Next, continue executing the program until the recv breakpoint is triggered, and analyze the call stack for receiving response data:

5% 182,479> delete breakpoint 3
5% 182,479> c
Continuing.

Thread 4 "python3" hit Breakpoint 2, __libc_recv (fd=3, buf=buf@entry=0x7fffe400a210, len=len@entry=8192, flags=flags@entry=0) at ../sysdeps/unix/sysv/linux/recv.c:24
5% 185,730> b _PyEval_EvalFrameDefault
Breakpoint 4 at 0x7ffff790c100: file ./Include/internal/pycore_pystate.h, line 107.
5% 185,730> c
Continuing.

Thread 2 "python3" hit Breakpoint 4, _PyEval_EvalFrameDefault (tstate=0x93cb00, frame=0x7ffff73ddea0, throwflag=0) at ./Include/internal/pycore_pystate.h:107
5% 186,005> python_bt
C:_PyEval_EvalFrameDefault
@/usr/local/openresty-python3/lib/python3.12/site-packages/urllib3/connection.py:504
getresponse
@/usr/local/openresty-python3/lib/python3.12/site-packages/urllib3/connectionpool.py:536
_make_request
@/usr/local/openresty-python3/lib/python3.12/site-packages/urllib3/connectionpool.py:789
urlopen
@/usr/local/openresty-python3/lib/python3.12/site-packages/requests/adapters.py:667
send
@/usr/local/openresty-python3/lib/python3.12/site-packages/requests/sessions.py:703
send
@/usr/local/openresty-python3/lib/python3.12/site-packages/requests/sessions.py:589
request
@/usr/local/openresty-python3/lib/python3.12/site-packages/requests/api.py:59
request
@/usr/local/openresty-python3/lib/python3.12/site-packages/requests/api.py:73
get
@/opt/app/processor.py:10
make_request
@/usr/local/openresty-python3/lib/python3.12/threading.py:1012
run
@/usr/local/openresty-python3/lib/python3.12/threading.py:1075
_bootstrap_inner
@/usr/local/openresty-python3/lib/python3.12/threading.py:1032
_bootstrap

This call stack reveals the process of receiving and processing HTTP responses. On the surface, this call path appears similar to sending requests, but the details are significantly different—the key function appears on line 504 of urllib3/connection.py, executing code related to getresponse.

Thanks to OpenResty XRay’s fully automated analysis capabilities, combined with UDB’s “time-travel” functionality, we’ve not only reconstructed the complete lifecycle of network requests but also precisely drilled down to code-level details at every stage—from connection establishment to data transmission and response reception. This depth of analysis is simply impossible with traditional debugging tools and Core Dumps.

Why is this a revolutionary approach to debugging in the “time dimension”? Compared to traditional methods, UDB provides truly dynamic, comprehensive contextual information:

• Core Dumps only capture the static state at the moment of program failure
• UDB can replay the entire request process, even after the original process has terminated
• You can set breakpoints at any stage of the request process to observe variables and function calls
• It even crosses the boundaries between Python and underlying C extensions, tracking the complete system behavior chain

For example, when facing an HTTP timeout issue, Core Dumps typically only show you “where it ultimately failed”; but with UDB, you can traverse the execution path step by step from the moment the request was initiated, precisely identifying which second experienced a delay and which call slowed down the response.

Summary

Through this practical case study, we’ve demonstrated how UDB combined with OpenResty XRay can help developers deeply analyze the execution process of Python network requests. UDB’s time-travel debugging capability provides Python developers with unprecedented insights, allowing us to:

1. Track the entire execution path: Capturing every step from high-level Python code to low-level system calls without missing anything
2. Precisely locate key points: Setting breakpoints and analyzing at various stages of network requests (connection establishment, data transmission, response reception)
3. Seamless cross-language debugging: Simultaneously examining the execution of Python code and underlying C extensions
4. Free historical analysis: Moving freely through execution history, unrestricted by traditional debuggers’ limitations

If you’re struggling with network issues, performance bottlenecks, or complex call stacks, consider trying the new “time-travel debugging” approach. Let UDB and OpenResty XRay help you escape debugging quagmires and see every frame of your program’s execution with crystal clarity.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.