How to Trace Go Panics in Production Without Restarting

OpenResty XRay traces Go panics in live production applications without restarting or recompiling the process, detecting both crashing and silently caught panics. It automatically interprets Go-level Exception Flame Graphs to pinpoint the exact code path behind each panic, minimizing performance overhead in latency-sensitive production environments.

This tutorial walks through using XRay’s Guided Analysis to trace a panic inside a real Go application, from launching the scan to reading the automatically generated report.

Trace Go Panics with XRay’s Guided Analysis

Open the XRay console and run Guided Analysis for Go panics

This video demonstrates how to analyze exceptions, or panics, in an arbitrary Go application running online.

We are using an unmodified Go compiler coming with the Linux distribution.

The target Go program was compiled using the normal go build command. No special options or changes are needed.

Open the OpenResty XRay web console in the web browser.

Make sure it is the right machine you are watching.

You can choose the right machine from the list below if the current one is not correct.

Go to the “Guided Analysis” page.

Here you can see different types of problems that you can diagnose.

Let’s select “Errors & exceptions”.

Click on “Next”.

Select the Go application.

Select the target process.

Make sure that the application type is right. Usually, the default should be correct.

The language level here is just “Go”.

We can also set the maximum analyzing time. We’ll leave it as 300 seconds, which is the default value.

Let’s start analyzing.

The system will keep performing different rounds of analysis. Now it’s executing the first round.

The first round is done and it’s already on the second one. That’s enough for this case.

Let’s stop analyzing.

Trace the first Go panic: a division-by-zero in gopanic

It automatically generated an analysis report.

This is the type of problem we diagnose. It’s Errors & Exceptions.

Look at the #1 Go code path that throws out exceptions.

Go eventually throws exceptions through the gopanic function of the Go runtime.

In this code path, gopanic function is called by panicdivide function of the runtime. It indicates that this is a division by zero error.

It’s the calculate function in the caller context. This function is on the business code level.

handleHTTPRequest function is responsible for handling incoming HTTP requests in the Gin web framework for Go.

Click on “More”.

This hot code path is automatically inferred from this Go-land exception flame graph.

Below are more detailed explanations regarding the current issue. It mentions the exceptions above caused by a division by zero error.

Here’s a code example that can fix this error.

Let’s go back to the original hot code path.

Hover the mouse over the green box for the calculate function.

We can see the source file of this function in the tooltip.

The source line number is 25.

Click the icon to copy the source file path.

Use the vim editor to open the source file. Paste the file path we just copied. You can use any editor you like.

Go to line 25, as OpenResty XRay suggested.

This line is inside the calculate function, as we saw earlier in the code path. We should have checked if the divisor, b, is zero first.

Trace the second Go panic: an out-of-bounds index in goPanicIndex

Next, check the second code path that throws out exceptions.

Here is the gopanic function as before.

This exception was triggered by the goPanicIndex function of the runtime in this code path.

It could be an array or slice index out of bounds here in this weekday.Handler function. And this Go function is on the business code level.

Hover the mouse over this function. Here is the source file of this function.

The source line number is 17.

Let’s copy the source file path.

Let’s open the source file on the terminal. Paste what we copied.

Go to line 17. There is indeed an array element access via indexing.

The array variable is named weekDays.

This function fails to check the value range of the index variable i.

And this i variable might take any integer value from the client request.

Automatic Go Panic Reports (No Guided Analysis Needed)

OpenResty XRay can also monitor online processes automatically and generate analysis reports. Go to the “Insights” page.

You can find the reports on the “Insights” page for daily and weekly periods. For this reason, you don’t have to use the “Guided Analysis” feature.

Though “Guided analysis” is useful for application development and demonstration purposes.

FAQ

Q: How does OpenResty XRay detect Go panics without restarting the application? A: XRay uses non-intrusive dynamic tracing — powered by its Y language targeting runtimes like Stap+, eBPF+, and GDB — to observe a running, unmodified Go process from the outside. There’s no special build flags or recompiling required; a normal go build binary works as-is.

Q: Can OpenResty XRay catch Go panics that the application already handles? A: Yes. Even when a panic is caught and never crashes the process, XRay still detects it, exposing the code path that most logging-based approaches miss.

Q: What is gopanic and how does it help trace a Go panic? A: gopanic is the Go runtime function through which every panic is eventually thrown. Its caller — such as panicdivide for a division-by-zero error or goPanicIndex for an out-of-bounds index — tells you exactly what kind of panic occurred before you even look at the business code.

Q: Where can I see Go panic reports without running Guided Analysis? A: The “Insights” page in the XRay console generates daily and weekly reports automatically, so you don’t need to run a Guided Analysis session each time you want to check for new panics.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.