etcd High CPU Usage: Find the Hottest Go Code Paths with Flame Graphs

When an etcd server burns high CPU, the fastest way to find the cause is to profile the live process and read its Go-level CPU flame graph. In this tutorial, OpenResty XRay analyzes an unmodified etcd binary consuming over 70% of a CPU core — with no code changes and no restart — and pinpoints the hottest Go code paths: gRPC request handling (processUnaryRPC), the key-value range and put handlers, and Go runtime stack growth (runtime.newstack).

In this tutorial, I will demonstrate how the CPU time is spent qualitatively inside Go’s etcd server with OpenResty XRay. I will show the most CPU-intensive Go code paths in it. OpenResty XRay will automatically analyze the Go (golang) language level CPU flame graph.

Symptom: etcd Consuming Over 70% of a CPU Core

Running top shows the etcd process consuming more than 70% of a CPU core (74.3% here):

The ps command confirms this is a standard etcd binary (/usr/local/bin/etcd) started with ordinary cluster flags — no custom build, no profiling instrumentation, and no changes of any kind:

Find etcd’s Hottest Go Code Paths with Guided Analysis

With OpenResty XRay, we can analyze this unmodified, running process in real time. The “Guided Analysis” wizard walks through selecting a diagnosis type, a target, and an application type. We pick High CPU usage as the problem to diagnose:

Then we select the live etcd process as the analysis target. OpenResty XRay detects it as a Go application (PID 3638889, ~76% CPU), and we keep the default Go language level and the default 300-second maximum running time before starting the analysis:

After a couple of sampling rounds, OpenResty XRay generates a report. It ranks the hottest Go-land code paths for CPU time, alongside how much CPU the read and write system calls consume:

This is the #1 hottest Go-land code path for CPU time, taking 17.6% of it:

The processUnaryRPC function is in the Go gRPC library. It’s responsible for handling the simplest type of gRPC messages — a single request answered by a single response:

This function is called by the handleStream function:

Click “More” to see details about this code path:

Reading the Go CPU Flame Graph

The code path above was automatically derived from this Go-land CPU flame graph. In the flame graph, the width of each frame is proportional to its share of CPU time:

Click the icon to enlarge the flame graph:

Continue to zoom in:

The function _KV_Range_Handler gets the keys in the range from the key-value store:

The function _KV_Put_Handler puts the given key into the key-value store:

The Range function is used to query the key-value data stored in etcd by range:

It calls runtime.newobject to create a large number of golang GC objects:

The function runtime.newstack has a high CPU overhead when writing data to etcd. This function is an internal function of the Go language runtime. It is used to create a new stack space for a goroutine:

Below are more detailed explanations and suggestions regarding the current issue, such as reducing the data sent over the network, using connection pooling, and tuning gRPC settings:

It mentions the function processUnaryRPC:

And it processes the unary RPC:

Jumping to the Source Line

Let’s go back to the code path. Hover the mouse over the green box for the first function:

We can see the source file of this function, and the full path for the server.go file in the tooltip:

The source line number is 1024:

Click the icon to copy the full Go source file path for this function:

Use the find command to find the source file:

Paste the file path we just copied:

Copy the full file path. Use the vim editor to open the source file and look at the golang code in this file. You can use any editors you like:

Go to line 1024, as OpenResty XRay suggested:

The function md.Handler calls different message handlers depending on the type of gRPC message. The _KV_Range_Handler and _KV_Put_Handler we saw earlier are two examples of such md.Handler callbacks:

On the status bar you can see that this source line is indeed inside the processUnaryRPC function, as the report mentioned:

The Other Hot Code Paths

The second hottest code path consumes about 12% of the CPU time:

From this function, we know it is writing to a network socket:

This is invoking the write system call:

This function sends response data to the network sockets via the HTTP/2 protocol:

The third hottest code path consumes about 11% of the CPU time:

Here, runtime.mcall mainly executes the scheduling of goroutines:

Now we see the fourth hottest Go code path, taking about 10.5% of the CPU time:

This is for logging every unary gRPC call. We might want to skip such logging to save CPU time:

Automatic etcd CPU Monitoring and Reports

Beyond on-demand Guided Analysis, OpenResty XRay can also monitor online processes automatically and produce daily and weekly reports on the “Insights” page. For this etcd instance, the daily report shows CPU usage averaging 51.33% (peaking at 164%), with the write system call and Go GC object allocation among the top CPU consumers:

Because these reports are generated automatically, you don’t have to run Guided Analysis yourself — Guided Analysis is most useful for application development and demonstration.

FAQ

Why is my etcd server using so much CPU?

In this case, most of etcd’s CPU time went to handling gRPC requests. The single hottest Go code path was processUnaryRPC in the Go gRPC library (17.6% of CPU time), which dispatches to etcd’s _KV_Range_Handler and _KV_Put_Handler — the functions that read key ranges from, and write keys to, the key-value store. Sending response data over HTTP/2 and Go runtime work (runtime.newobject, runtime.newstack, and goroutine scheduling via runtime.mcall) accounted for most of the rest. The same techniques apply to any Go program with high CPU usage; see how to find the hottest Go code paths.

How can I profile etcd CPU usage without changing or restarting it?

OpenResty XRay analyzes the live, unmodified etcd process directly — no code changes, no rebuild, and no restart. Its “Guided Analysis” feature samples the running process and automatically derives the Go-level CPU flame graph and the hottest code paths.

What are the hottest code paths in etcd under high CPU?

For this workload the top paths were: (1) processUnaryRPC handling unary gRPC calls into the KV range and put handlers (17.6%); (2) writing response data to network sockets over HTTP/2 (11.7%); (3) runtime.mcall scheduling goroutines (11.1%); and (4) logging every unary gRPC call (10.5%) — which you may be able to skip to save CPU time.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.