In this tutorial, we will walk through how to use OpenResty XRay to detect and analyze the exceptions (or panics) in the Go (golang) applications online. OpenResty XRay offers the capability to detect and analyze exceptions or panics in online Go processes, including caught ones that do not lead to process crashes. The code paths are obtained by OpenResty XRay automatically analyzing and interpreting the Go code-level Exception Flame Graphs. It will minimize the performance impact. This makes it ideal for production environments sensitive to performance overhead and latency.

How to detect and analyze exceptions inside live Go applications

This video demonstrates how to analyze exceptions, or panics, in an arbitrary Go application running online.

We are using an unmodified Go compiler coming with the Linux distribution.

The target Go program was compiled using the normal go build command. No special options or changes are needed.

Open the OpenResty XRay web console in the web browser.

Make sure it is the right machine you are watching.

You can choose the right machine from the list below if the current one is not correct.

Go to the “Guided Analysis” page.

Here you can see different types of problems that you can diagnose.

Let’s select “Errors & exceptions”.

Click on “Next”.

Select the Go application.

Select the target process.

Make sure that the application type is right. Usually, the default should be correct.

The language level here is just “Go”.

We can also set the maximum analyzing time. We’ll leave it as 300 seconds, which is the default value.

Let’s start analyzing.

The system will keep performing different rounds of analysis. Now it’s executing the first round.

The first round is done and it’s already on the second one. That’s enough for this case.

Let’s stop analyzing.

It automatically generated an analysis report.

This is the type of problem we diagnose. It’s Errors & Exceptions.

Look at the #1 Go code path that throws out exceptions.

Go eventually throws exceptions through the gopanic function of the Go runtime.

In this code path, gopanic function is called by panicdivide function of the runtime. It indicates that this is a division by zero error.

It’s the calculate function in the caller context. This function is on the business code level.

handleHTTPRequest function is responsible for handling incoming HTTP requests in the Gin web framework for Go.

Click on “More”.

This hot code path is automatically inferred from this Go-land exception flame graph.

Below are more detailed explanations regarding the current issue. It mentions the exceptions above caused by a division by zero error.

Here’s a code example that can fix this error.

Let’s go back to the original hot code path.

Hover the mouse over the green box for the calculate function.

We can see the source file of this function in the tooltip.

The source line number is 25.

Click the icon to copy the source file path.

Use the vim editor to open the source file. Paste the file path we just copied. You can use any editor you like.

Go to line 25, as OpenResty XRay suggested.

This line is inside the calculate function, as we saw earlier in the code path. We should have checked if the divisor, b, is zero first.

Next, check the second code path that throws out exceptions.

Here is the gopanic function as before.

This exception was triggered by the goPanicIndex function of the runtime in this code path.

It could be an array or slice index out of bounds here in this weekday.Handler function. And this Go function is on the business code level.

Hover the mouse over this function. Here is the source file of this function.

The source line number is 17.

Let’s copy the source file path.

Let’s open the source file on the terminal. Paste what we copied.

Go to line 17. There is indeed an array element access via indexing.

The array variable is named weekDays.

This function fails to check the value range of the index variable i.

And this i variable might take any integer value from the client request.

Automatic analysis and reports

OpenResty XRay can also monitor online processes automatically and generate analysis reports. Go to the “Insights” page.

You can find the reports on the “Insights” page for daily and weekly periods. For this reason, you don’t have to use the “Guided Analysis” feature.

Though “Guided analysis” is useful for application development and demonstration purposes.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.