In this tutorial, you will learn how to use OpenResty XRay to quantitatively analyze high disk I/O issues in an online Rust application. With the Rust-level disk write count, latency, and throughput flame graphs generated by OpenResty XRay, you can identify the Rust code paths with the most significant disk latency and write volume. Plus, it offers precise line-by-line code insights to locate the problem’s root cause swiftly and guide optimizations.

Problem: high disk I/O

First, run the ps command to check this application.

We can see a Rust application named sled-service. It is based on Sled, an embedded KV database written in Rust.

Let’s use OpenResty XRay to check out the operation of the hard disk in this application.

Spot the problematic Rust code paths

Switch to the OpenResty XRay Web console. Make sure it is the right machine you are watching.

Go to the “Guided Analysis” page.

Here, you can see different types of problems that you can diagnose.

Let’s select “High disk IO”.

Click on “Next”.

Select the Rust application named sled-service.

Select the rust process.

Make sure that the application type is right. Usually, the default should be correct.

The language level is just “Rust”.

We can also set the maximum analyzing time. We’ll leave it as 300 seconds, which is the default value.

Let’s start analyzing.

The system will keep performing different rounds of analysis. Now, it’s executing the first round.

The first round is done, and it’s on to the second one already. That’s enough for this case.

Stop analyzing.

We can see it automatically creates an analysis report.

This is the type of problem we diagnose. It’s “Disk I/O”.

This is the analysis of hard disk write count.

This code path has the maximum disk write operations.

These two __libc_pwrite64 functions are C functions used to write data at specific locations in a file.

In the Rust standard library, write_at and write_at_all are functions that write data at specific locations in a file.

The pwrite_all function is responsible for supporting parallel writing of whole pages of data to a file.

The write_to_log function is used to write data to a log file for persistent storage or other operations.

Click to see more details.

This code path was automatically derived from this Rust-land disk write count flame graph.

Below are more detailed explanations and suggestions regarding the current issue.

It mentions the function pwrite64 we saw earlier.

Let’s go back to the code path. Hover the mouse over the green box for this function.

We can see its source file’s full path in the tooltip.

The source line number is 730.

Click the icon to copy the full Rust source file path for this function.

Use the vim editor to open the source file. And look at the Rust code in this file. You can use any editors you like.

Check line 730, as OpenResty XRay suggested.

The pwrite_all function is used in the Sled library to write data to a log file. It serves to atomically write a block of data to a specific offset in the file.

This is the analysis of disk write latency.

This code path is writing to the database data file.

It is almost the entire source of hard disk write latency.

This is the analysis of the disk write throughput.

It is the same code path that writes to the database data file.

The write rate is close to 12 MB/s.

These are files with the most frequent disk writes.

This file has the highest volume of writing data. It is the database file for the sled application.

The writing rate is about 6.4 MB/s.

These are files with the most accumulated write latency.

It is the same file with the highest writing data latency.

The proportion is around 40%.

Automatic analysis and reports

OpenResty XRay can also monitor online processes automatically and generate analysis reports. Go to the “Insights” page.

You can find the automatic reports on the “Insights” page for daily and weekly periods. For this reason, you don’t have to use the Guided Analysis feature.

Though guided analysis is useful for application development and demonstration purposes.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.