Today I’m going to show you how to use OpenResty XRay to quickly pinpoint the Go code paths that blocking operating system threads. These are the code paths that might slow down your processes and prevent them from using the full CPU power.

Problem: low CPU usage

Let’s run the top command to check the CPU usage of each process.

Take a look at this chat-service golang process. We already know it is written in Go. The CPU usage is very low, only 2%. The usage does not increase even when many requests continue to come in.

Let’s take a look at the access log file for this golang application.

We can see many client requests are flooding in. However, the CPU usage remains low. This means that there is something blocking the golang code from running efficiently. But we don’t know what it is yet. How can we find out?

Use the guidede analysis feature of OpenResty XRay to spot the blocking Go code paths

Let’s use OpenResty XRay to check out this unmodified process. You can analyze it in real-time and figure out what is happening.

Let’s open the OpenResty XRay XRay web console in the web browser.

Make sure we are looking at the right machine.

Go to the “Guided Analysis” page.

Here you can see different types of problems that you can diagnose.

Let’s select “Low CPU usage and cannot go up”.

Click on “Next”.

Select the Go application.

Select the process that consumes 2% of the CPU resources. This is what we saw earlier in top.

Make sure that the application type is right. Usually the default should be correct.

The language level here is just “Go”.

We can set the maximum time to analyze. We’ll leave it at 300 seconds here, just as the default.

Let’s start analyzing.

The system will keep performing different rounds of analyses. It is now executing the first round.

The first round is done and it’s already on the second one. That’s enough for this case.

Let’s stop analyzing.

It automatically generated an analysis report.

This is the type of problem we are going to diagnose. It’s off-CPU.

This Go-land code path is blocking efficient execution.

The first Go function is Syscall6, a system call taking 6 arguments. To identify what kind of syscall it is, we need to check out more context info.

Based on the Go function name, Process.wait, the underlying syscall should be waitpid.

Look at this golang function exec.Cmd.run defined in the standard os module. This is for running system shell commands and waiting for them to complete.

The chat.RateLimit Go function is in our business-level code base.

Click “More” to see details.

The most significant blocking code path was automatically inferred from this Go-land off-CPU flame graph.

Below are more explanations and suggestions about the issue.

It refers to the chat.RateLimit function we saw earlier.

It mentions that this function runs a system command and waits for it.

Let’s go back to the original hot code path.

Hover the mouse over the green box for the chat.RateLimit golang function.

You can see the golang source file of this function. And its path to the Processor.go file in the tooltip.

The source line number is 46.

Click the icon to copy the golang source file path.

On the terminal, the current working directory is already in the source code directory of this Go application. Use the find command to find the source file.

Paste the file path we just copied.

Copy the full file path. Use the vim editor and paste the code path we just copied to look at the corresponding business Go code. You can use any editor you like.

Go to line 46, as OpenResty XRay suggested.

We see that this golang code line calls the function cmd.Run.

Also in the function RateLimit shown in the earlier report. Now it’s easy to optimize away this bottleneck!

Automatic analysis and reports

OpenResty XRay can also monitor online processes automatically and show analysis reports.

Go to the Insights page.

You can find the reports in the Insights page for daily and weekly periods.

For this reason, you don’t have to use the Guided Analysis feature. Guided analysis is useful for application development and demonstration purposes.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.