How to Trace Rust Panics in Production Without Restarting

OpenResty XRay traces Rust panics in live production applications without restarting or modifying the process, detecting both crashing and silently caught panics. It automatically interprets Rust-level Exception Flame Graphs to pinpoint the exact code path behind each panic, minimizing performance overhead in latency-sensitive production environments.

This tutorial walks through using XRay’s Guided Analysis to trace a panic inside a real Rust application, from launching the scan to reading the automatically generated report.

Trace Rust Panics with XRay’s Guided Analysis

This video will demonstrate detecting and analyzing exceptions or panics inside live Rust applications.

Configure and run Guided Analysis for Rust panics

Switch to the OpenResty XRay Web console. Make sure that the application type is right. Usually, the default should be correct.

The language level here is just “Rust”.

We can also set the maximum analyzing time. We’ll leave it as 300 seconds, which is the default value.

Let’s start analyzing.

The system will keep performing different rounds of analysis. Now, it’s executing the first round.

The first round is done and it’s already on the second one. That’s enough for this case.

Let’s stop analyzing.

Read the panic report and hot code path

It automatically generated an analysis report.

This is the type of problem we diagnose. It’s Errors & Exceptions.

The report shows the code path that throws out the most Rust exceptions or panics.

The function internal_constructor is used to construct PanicInfo structure. It contains information about the location of the panic and the payload associated with the panic.

The panic_fmt function is a low-level function that formats a panic message and triggers a panic.

The panic_bounds_check function throws a panic if an array or slice-bound check fails. It’s a Rust safeguard to avoid out-of-bound memory accesses at runtime.

The stat_client_usage function belongs to the business-level code of the target application.

Click “More” to see details.

The hot code path was automatically inferred from this Rust-Land Panic Flame Graph.

Below are more detailed explanations and suggestions regarding the current issue.

It mentions the PanicInfo structure we saw earlier.

It also explains panic was caused by bounds check failure.

Trace the hot code path to the exact Rust source line

Go back to the hot code path. Hover the mouse over the green box for the function.

You can see the source file and the full path of this function in the tooltip.

The source line number is 7.

Click the icon to copy the source file path.

On the terminal, use the vim editor to open the source file. Paste the file path we just copied. You can use any editor you like.

Go to line 7, as OpenResty XRay suggested.

The array subscript client_id may exceed the array CLIENTS. Based on the panic_bounds_check function we saw earlier, it is clear that there is a data subscript out-of-bounds access here.

This line is inside the stat_client_usage function, as shown in the report. This Rust exception is actually caught by the application itself. And did not cause the Rust application to crash. Even so, OpenResty XRay can still detect and track it in real time.

Automatic Rust Panic Reports (No Guided Analysis Needed)

Our product can also monitor online processes automatically and show analysis reports. You can find the reports on the “Insights” page for daily and weekly periods. For this reason, you don’t have to use the “Guided Analysis” feature.

Though “Guided analysis” is useful for application development and demonstration purposes.

FAQ

Q: How does OpenResty XRay detect Rust panics without restarting the application? A: XRay uses non-intrusive dynamic tracing — powered by its Y language targeting runtimes like Stap+, eBPF+, and GDB — to observe a running Rust process from the outside, so it can capture panics without touching the process’s code or lifecycle.

Q: Can OpenResty XRay catch Rust panics that the application already handles? A: Yes. Even when a panic is caught by the application itself and never crashes the process, XRay still detects it and tracks it in real time, exposing the code path that most logging-based approaches miss.

Q: What does the panic_bounds_check function tell you about a Rust panic? A: panic_bounds_check throws a panic when an array or slice-bounds check fails — Rust’s runtime safeguard against out-of-bounds memory access. Seeing it in the Exception Flame Graph points straight to a data-subscript out-of-bounds bug in the code.

Q: Where can I see Rust panic reports without running Guided Analysis? A: The “Insights” page in the XRay console generates daily and weekly reports automatically, so you don’t need to run a Guided Analysis session each time you want to check for new panics.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.