Try OpenResty XRay to analyze your running applications now

Our in-house cache service built on Sled, an embedded key-value database written in Rust, was consuming more than 100% CPU. Rust CPU profiling with OpenResty XRay traced the load to two hot code paths — sled::tree::Tree::insert and get_inner (almost 40% of the CPU time) — down to the exact source line, without changing any code or recompiling.

In this tutorial, we walk through the whole analysis step by step. The hot code paths shown below were obtained by OpenResty XRay automatically analyzing and interpreting Rust language-level CPU flame graphs.

The Problem: A Sled-Based Cache Service Using Over 100% CPU

Sled is an embedded KV database written in Rust, and our in-house cache service is built on top of it. In top, the process’s CPU usage stayed above 100%.

top output showing the Rust Sled cache service process consuming more than 100% CPU

Profile the Running Rust Process with Guided Analysis

Let’s use OpenResty XRay to check this unmodified process. You can analyze it in real time and figure out what is happening, with no code changes and no recompilation. The runtime overhead of this kind of analysis is very low, as we measured in OpenResty XRay’s overhead on Rust applications.

Open the OpenResty XRay web console, make sure you are watching the right machine, and go to the “Guided Analysis” page. Here you can see the different types of problems that you can diagnose.

Guided Analysis page in OpenResty XRay listing diagnosable problem types including High CPU Usage

Select “High CPU Usage”, then select the Sled application and the process that consumes more than 100% of the CPU resources — the one we saw previously in top.

Process selection step in Guided Analysis showing the Sled process with over 100% CPU usage

The application type defaults to Rust, and the language level here is just “Rust”. We leave the maximum analyzing time at its default of 300 seconds and start analyzing. The system keeps performing different rounds of analysis; once the first round is done, that’s already enough for this case, so we stop the analysis and it automatically generates an analysis report.

Automatically generated Guided Analysis report for the high CPU usage problem in the Rust Sled process

Read the Rust CPU Flame Graph: The #1 Hottest Code Path

The report shows the #1 hottest Rust-land code path for the CPU time.

The number one hottest Rust code path reported by OpenResty XRay, topped by sled::tree::Tree::insert

The first function, sled::tree::Tree::insert, is used in Sled for data insertion.

The report labeling sled::tree::Tree::insert as the number one hottest function, used to insert data into the Sled tree

Click “More” to see the details.

Clicking the More button on the report entry to expand the detailed analysis of this hot path

The hot code path above was automatically inferred from the Rust-land CPU flame graph below.

Rust-land CPU flame graph from which OpenResty XRay automatically inferred the hot insert code path

Below are the more detailed explanations and suggestions that the report gives for the problem: the Explanation section describes insert and each function along its call chain, while the Suggestions section points to optimizations such as batched writes (sled::Batch), concurrency, and parameter tuning.

The report’s Explanation and Suggestions sections describing the insert call chain and advising batched writes, concurrency, and tuning

Click this icon to enlarge the flame graph.

Clicking the icon to enlarge the Rust CPU flame graph for a closer look at the calls

Click the insert function frame to see more details.

Clicking the insert function frame in the enlarged flame graph to inspect its internal calls

On the left side, the view_for_key function takes a large portion — this is the function in the Sled library that gets a snapshot view for a given key.

Zoomed flame graph view showing the view_for_key function taking a large portion of the insert path

On the right, pagecache is a component of Sled used to manage data on a page-by-page basis. Writes are first stored in the pagecache’s memory page, then persisted by flushing to disk when the batch is full.

Flame graph on the right showing Sled’s pagecache component managing writes on a page-by-page basis

Keep zooming in.

Zooming further into the flame graph to see the calls beneath pagecache

You can see the realloc function in glibc — libc’s memory allocation functions are hot in this workload.

Flame graph showing glibc’s realloc memory allocation function as a hot spot in this workload

Jump from the Flame Graph to Sled’s Source Code

On the terminal, use the find command to locate the Sled library source code directory in the cargo cache.

Using the find command in the terminal to locate the Sled library source directory in the cargo cache

Copy the directory found and enter the Sled source code directory.

Copying the path found by find and entering the Sled source code directory

Back in the flame graph, hover the mouse over the green box for the insert function: the tooltip shows the source file of this function.

Flame graph tooltip revealing the source file path for the insert function

The source line number shown in the tooltip is 164.

Tooltip showing the source line number 164 for the insert function

Click this icon to copy the source file path of the function.

Clicking the icon to copy the source file path of the insert function

Open the source file in the editor you like and paste the path you just copied (here we use vim).

Opening the pasted Sled source file path in vim

Go to line 164, as OpenResty XRay suggested.

Jumping to line 164 in the editor as suggested

This line is inside the insert function.

Sled source code opened in an editor at line 164, inside the insert function body

The #2 Hottest Code Path: get_inner and view_for_key

Next, check the second code path. The #2 hottest code path consumes almost 40% of the CPU time.

The number two hottest Rust code path consuming almost 40% of CPU time, topped by get_inner

The top function call, get_inner, is the function in Sled that looks up the data.

The report labeling get_inner as the top function of the second hottest path, responsible for looking up data in Sled

The get function is the interface that the library exposes to the user to get data, and it calls get_inner internally.

get is the interface Sled exposes to users to fetch data, calling get_inner internally

Click “More” to see the details.

Clicking More to expand the detailed analysis of the get_inner hot path

Enlarge the flame graph to see the details of the get_inner function call.

Enlarging the flame graph to inspect the get_inner call details

Zoom in further on get_inner.

Zooming further into the get_inner function’s child calls in the flame graph

You can see that the get_inner function is taken up mostly by the same view_for_key function mentioned earlier.

Flame graph showing most of get_inner’s CPU time taken by the view_for_key function

The sled::lru::Lru::accessed function is used in Rust’s Sled library to update the accessed state of an item in the LRU cache, and to return a list of page IDs that need to be evicted.

Flame graph detail showing sled::lru::Lru::accessed updating LRU cache state inside the get_inner path

Boost your application's performance by OpenResty XRay for free

Automatic Rust CPU Analysis Reports

OpenResty XRay can also monitor online processes automatically and show analysis reports.

OpenResty XRay automatically monitoring online processes and generating analysis reports

Switch to the “Insights” page.

Switching to the Insights page in the OpenResty XRay console

You can find the reports on the Insights page for daily and weekly periods, so you don’t have to use the “Guided Analysis” feature at all.

Insights page in OpenResty XRay showing automatic daily and weekly CPU analysis reports

That said, “Guided Analysis” is still useful for application development and demonstration purposes. In the Insights daily report you can see the process’s CPU usage (min 102%, avg 107%, max 113%), along with the hottest Rust code paths ranked by percentage.

Insights daily report showing the sled-cache process CPU min 102%/avg 107%/max 113% and the hottest Rust code paths ranked by percentage

Beyond CPU time, OpenResty XRay can also trace panics in Rust programs and diagnose high disk I/O in Rust applications in the same non-invasive way.

Frequently Asked Questions

How do I profile CPU usage of a Rust program without changing its code?

Use a non-invasive profiler based on dynamic tracing. OpenResty XRay analyzes the running, unmodified Rust process directly: you select the process in its Guided Analysis feature, and it samples the process and generates Rust language-level CPU flame graphs — no recompilation and no instrumentation of the target process.

How do I find which Rust function is consuming the most CPU?

Sample the process and get a Rust-land CPU flame graph, then let OpenResty XRay automatically infer the hottest code paths from it — no manual flame graph reading required. In our Sled case, it reported sled::tree::Tree::insert as the #1 hottest path and even pointed to the exact source line (line 164).

Why was our Sled-based service using more than 100% CPU?

In this case, the CPU time concentrated in two paths: data insertion via sled::tree::Tree::insert — where view_for_key and pagecache writes dominated, with glibc’s realloc allocation function running hot — and data lookups via get_inner, which consumed almost 40% of the CPU time, again mostly inside view_for_key.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

Try out OpenResty XRay for free today

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.