Why Your Rust Process Has High Disk I/O (and How to Find the Cause)

In this tutorial, OpenResty XRay diagnoses high disk I/O in a Rust application named sled-service (based on Sled, an embedded KV database written in Rust). Using Rust-level disk write count, latency, and throughput flame graphs, it identifies the code path with the most disk activity and drills down to the exact source file and line number.

Identifying the Rust process with high disk I/O

First, run the ps command to check this application.

We can see a Rust application named sled-service. It is based on Sled, an embedded KV database written in Rust.

Let’s use OpenResty XRay to check out the operation of the hard disk in this application.

Tracing the hottest Rust disk write code paths with flame graphs

Switch to the OpenResty XRay Web console. Make sure it is the right machine you are watching.

Go to the “Guided Analysis” page.

Here, you can see different types of problems that you can diagnose.

Let’s select “High disk IO”.

Click on “Next”.

Select the Rust application named sled-service.

Select the rust process.

Make sure that the application type is right. Usually, the default should be correct.

The language level is just “Rust”.

We can also set the maximum analyzing time. We’ll leave it as 300 seconds, which is the default value.

Let’s start analyzing.

The system will keep performing different rounds of analysis. Now, it’s executing the first round.

The first round is done, and it’s on to the second one already. That’s enough for this case.

Stop analyzing.

We can see it automatically creates an analysis report.

This is the type of problem we diagnose. It’s “Disk I/O”.

This is the analysis of hard disk write count.

This code path has the maximum disk write operations.

These two __libc_pwrite64 functions are C functions used to write data at specific locations in a file.

In the Rust standard library, write_at and write_at_all are functions that write data at specific locations in a file.

The pwrite_all function is responsible for supporting parallel writing of whole pages of data to a file.

The write_to_log function is used to write data to a log file for persistent storage or other operations.

Click to see more details.

This code path was automatically derived from this Rust-land disk write count flame graph.

Below are more detailed explanations and suggestions regarding the current issue.

It mentions the function pwrite64 we saw earlier.

Let’s go back to the code path. Hover the mouse over the green box for this function.

We can see its source file’s full path in the tooltip.

The source line number is 730.

Click the icon to copy the full Rust source file path for this function.

Use the vim editor to open the source file. And look at the Rust code in this file. You can use any editors you like.

Check line 730, as OpenResty XRay suggested.

The pwrite_all function is used in the Sled library to write data to a log file. It serves to atomically write a block of data to a specific offset in the file.

This is the analysis of disk write latency.

This code path is writing to the database data file.

It is almost the entire source of hard disk write latency.

This is the analysis of the disk write throughput.

It is the same code path that writes to the database data file.

The write rate is close to 12 MB/s.

These are files with the most frequent disk writes.

This file has the highest volume of writing data. It is the database file for the sled application.

The writing rate is about 6.4 MB/s.

These are files with the most accumulated write latency.

It is the same file with the highest writing data latency.

The proportion is around 40%.

Automatic disk I/O analysis and reports

OpenResty XRay can also monitor online processes automatically and generate analysis reports. Go to the “Insights” page.

You can find the automatic reports on the “Insights” page for daily and weekly periods. For this reason, you don’t have to use the Guided Analysis feature.

Though guided analysis is useful for application development and demonstration purposes.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

Frequently Asked Questions

Why does my Rust process have high disk I/O?

Usually because one or a few code paths write far more often or in larger volumes than the rest of the application. In this example, the Sled-based sled-service process concentrated nearly all its disk activity in a single code path — write_to_log calling pwrite_all — that wrote to the database’s data file, which OpenResty XRay’s flame graphs isolated directly.

How do I find which Rust function is causing excessive disk writes?

Run OpenResty XRay’s Guided Analysis, select “High disk IO,” and pick the target process. It generates a disk write count flame graph showing which code path writes the most, then lets you hover over the hot function to reveal its exact source file and line number.

What’s the difference between disk write count, latency, and throughput analysis?

Write count shows how many write operations a code path performs; latency shows how much time those writes spend blocked on the disk; throughput shows the data volume written per second. In this Sled example, all three analyses pointed to the same code path writing to the database’s data file, at a write rate close to 12 MB/s, and at the file level the single busiest file alone accounted for about 40% of accumulated write latency at roughly 6.4 MB/s.

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.