Go Off-CPU Analysis: Tracing a Running Process Stuck at 2% CPU to a Blocking Shell Call

Golang off-CPU analysis pinpoints where a Go program sits blocked instead of executing — the reason CPU usage stays low while requests keep flooding in. In this walkthrough, we use OpenResty XRay to analyze a live, unmodified Go service stuck at 2% CPU and trace it to a blocking shell command at a single source line — no code changes or redeploy.

The Symptom: Go Service CPU Usage Stays Low Under Load

Let’s run the top command to check the CPU usage of each process.

Take a look at this chat-service golang process. We already know it is written in Go. The CPU usage is very low, only 2%. The usage does not increase even when many requests continue to come in. (If you have the opposite symptom — CPU usage too high — see finding the hottest Go code paths instead.)

Let’s take a look at the access log file for this golang application.

We can see many client requests are flooding in. However, the CPU usage remains low. This means that there is something blocking the golang code from running efficiently. But we don’t know what it is yet. How can we find out?

Running Off-CPU Analysis on the Live Go Process with OpenResty XRay

Let’s point OpenResty XRay at this unmodified process and analyze it in real time — no instrumentation, no restart. Open the OpenResty XRay web console, go to the Guided Analysis page, and select the problem type “Low CPU usage and cannot go up”.

Step through the wizard to pick the target Go process (the one at 2% CPU we saw in top), keep the default language level “Go” and the default 300-second analysis window, and start the analysis. After one round of sampling, stop it — for this case that is enough. OpenResty XRay automatically generates a report, and the problem type it identifies is off-CPU: the Go service is spending its time blocked, not running.

The report reads the blocking path from the bottom up. The first Go function is Syscall6, a system call taking six arguments — the frame alone doesn’t tell you which syscall, so you follow the surrounding frames for context.

The neighboring Process.wait frame indicates the underlying syscall is waitpid. Above it, exec.Cmd.Run (from the standard library) means the program is running a system shell command and waiting for it to finish. Following the path up, the culprit reaches our own business code: the chat.RateLimit function.

This most significant blocking code path was automatically inferred from the Go-land off-CPU flame graph below. The same off-CPU analysis approach also works for Perl and Python processes.

The report also spells out the explanation and suggestions: Syscall6 performs a six-argument system call, the path traces up through exec.Cmd.Run to chat.RateLimit, and that business function runs a system command and waits for it.

To jump straight to the offending source, hover over the chat.RateLimit frame. The tooltip shows the exact source file and line — chat/processor.go, line 46.

Open that file at line 46, and the bottleneck is right there: RateLimit calls exec.Command("/usr/bin/sleep", "0.01") and then cmd.Run(), launching a shell command and blocking on it. Now it’s easy to optimize away.

Automatic Analysis and Reports, Without the Guided Steps

You don’t have to run Guided Analysis by hand. OpenResty XRay can also monitor your online processes automatically and publish daily and weekly reports on the Insights page — the same off-CPU code paths and CPU usage summary, generated for you. Guided Analysis is then mainly for development and demonstration.

Frequently Asked Questions

How do you do off-CPU analysis on a Go program?

Run it against the live, unmodified process. In OpenResty XRay’s Guided Analysis, select “Low CPU usage and cannot go up”, pick the target Go process, and let it sample for a round. It automatically generates a report with a Go-land off-CPU flame graph and infers the most significant blocking code path — no instrumentation, code changes, or redeploy.

Why does my Go service’s CPU usage stay low even under heavy load?

Because its operating system threads are blocked waiting instead of executing Go code. In the case above, the business function chat.RateLimit used exec.Cmd.Run to run a shell command and waited for it to finish (a waitpid system call underneath), which kept CPU usage at 2% no matter how many requests came in.

What does Syscall6 mean in a Go off-CPU flame graph?

Syscall6 is the Go runtime function that makes a system call taking six arguments. The frame alone does not tell you which system call it is — you need the surrounding context. In our flame graph, the neighboring Process.wait frame indicated that the underlying syscall was waitpid.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.