In this tutorial, we will walk through how to use OpenResty XRay to analyze the exceptions (or panics) in the Rust applications online. OpenResty XRay provides a powerful tool for developers to pinpoint and resolve exceptions in their Rust applications. The code paths is obtained by OpenResty XRay automatically analyzing and interpreting the Rust language-level Exception Flame Graphs. It will minimize the performance impact. This makes it ideal for production environments sensitive to performance overhead and latency.

How to detect and analyze exceptions inside live Rust applications

This video will demonstrate detecting and analyzing exceptions or panics inside live Rust applications.

Switch to the OpenResty XRay Web console. Make sure that the application type is right. Usually, the default should be correct.

The language level here is just “Rust”.

We can also set the maximum analyzing time. We’ll leave it as 300 seconds, which is the default value.

Let’s start analyzing.

The system will keep performing different rounds of analysis. Now, it’s executing the first round.

The first round is done and it’s already on the second one. That’s enough for this case.

Let’s stop analyzing.

It automatically generated an analysis report.

This is the type of problem we diagnose. It’s Errors & Exceptions.

The report shows the code path that throws out the most Rust exceptions or panics.

The function internal_constructor is used to construct PanicInfo structure. It contains information about the location of the panic and the payload associated with the panic.

The panic_fmt function is a low-level function that formats a panic message and triggers a panic.

The panic_bounds_check function throws a panic if an array or slice-bound check fails. It’s a Rust safeguard to avoid out-of-bound memory accesses at runtime.

The stat_client_usage function belongs to the business-level code of the target application.

Click “More” to see details.

The hot code path was automatically inferred from this Rust-Land Panic Flame Graph.

Below are more detailed explanations and suggestions regarding the current issue.

It mentions the PanicInfo structure we saw earlier.

It also explains panic was caused by bounds check failure.

Go back to the hot code path. Hover the mouse over the green box for the function.

You can see the source file and the full path of this function in the tooltip.

The source line number is 7.

Click the icon to copy the source file path.

On the terminal, use the vim editor to open the source file. Paste the file path we just copied. You can use any editor you like.

Go to line 7, as OpenResty XRay suggested.

The array subscript client_id may exceed the array CLIENTS. Based on the panic_bounds_check function we saw earlier, it is clear that there is a data subscript out-of-bounds access here.

This line is inside the stat_client_usage function, as shown in the report. This Rust exception is actually caught by the application itself. And did not cause the Rust application to crash. Even so, OpenResty XRay can still detect and track it in real time.

Automatic analysis and reports

Our product can also monitor online processes automatically and show analysis reports. You can find the reports on the “Insights” page for daily and weekly periods. For this reason, you don’t have to use the “Guided Analysis” feature.

Though “Guided analysis” is useful for application development and demonstration purposes.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.