OpenResty XRay is a dynamic tracing product that allows you to analyze online applications without modifying them. It’s highly efficient and safe to use. You can gain deep insights into any running process at any time, just like performing an X-ray on your software.

Unlike traditional profiling tools, OpenResty XRay doesn’t rely on the JVM’s safepoint mechanism, meaning it can profile without affecting Java application performance. This unique approach allows XRay to provide more accurate and comprehensive performance data while minimizing interference with the analyzed application.

The core technology of OpenResty XRay is based on our independently improved dynamic tracing technologies, such as eBPF+ and Stap+. These advanced technologies enable XRay to provide deeper and more precise analysis capabilities while maintaining extremely low system overhead. It’s worth noting that OpenResty XRay supports both the old 3.10 kernel of RHEL/CentOS Linux 7 and the new 6.x kernels, offering users broad system compatibility.

OpenResty XRay supports Java as one of its applications. Unlike traditional JVM agent methods, OpenResty XRay uses advanced dynamic tracing technology to offer you the following unique advantages:

1. Zero intrusion: No need to modify your Java application code or configuration, nor add new plugins or libraries.
2. No special JVM options required: You can use existing OpenJDK 8/11/17 without adding any special JVM startup options.
3. Wide system compatibility: Supports a variety of operating systems from modern Linux distributions to older systems like CentOS 7.
4. Extremely low-performance overhead: The additional burden on Java applications during sampling is almost negligible, and there is no impact at all when not sampling.

These features make OpenResty XRay a truly “plug-and-play” solution, allowing you to perform in-depth analysis of Java applications at any time and in any environment, just like performing an X-ray on your software.

In this article, we’ll show you how to use OpenResty XRay to analyze CPU and disk I/O performance issues in Java applications.

High CPU Usage Issues in Java Applications

When a Java application’s CPU usage is abnormally high, it usually means that certain code paths are consuming excessive computational resources. Guided analysis with OpenResty XRay can help you quickly locate these hotspot codes.

For example, here’s a CPU flame graph generated by OpenResty XRay for a Java application under load-testing (in this example, it’s a Tomcat HTTP application, but it applies to any unmodified Java application):

In this flame graph, each bar represents a function call, and the width of the bar indicates the proportion of CPU time consumed by that function. The wider the bar, the more CPU time the function consumes.

Analyzing this graph carefully, we can clearly see that the Java function sun.nio.ch.FileDispatcherImpl:write0 occupies the largest share of CPU time. This is a native method used internally by the Java Virtual Machine, mainly for writing data to files or network sockets. This finding aligns well with our load testing scenario, as a large amount of CPU resources are used to return response data during the test. It’s worth noting that OpenResty XRay can precisely locate specific .java source files and line numbers, allowing you to directly find the exact location of performance hotspots, greatly improving the efficiency of problem identification and optimization.

Based on these in-depth analysis results, you can optimize these performance hotspots in a targeted manner. Possible optimization strategies include but are not limited to: improving algorithm efficiency, reducing unnecessary calculations, introducing appropriate caching mechanisms, or optimizing I/O operations. Through these targeted optimizations, you can effectively reduce the CPU usage of the application and improve overall performance.

CPU Blocking Issues in Java Applications

Sometimes, low CPU usage in Java applications can also be a problem, usually caused by I/O blocking, waiting for locks, or other resource contention. OpenResty XRay can help you find these bottlenecks.

For example, here’s an off-CPU flame graph generated by OpenResty XRay for a Java application with low CPU usage:

In this flame graph, each bar represents a function call, and the width of the bar indicates the time the function blocks the CPU. The wider the bar, the more time the function blocks the CPU.

Carefully observing the widest part of the flame graph, we can find that the java.util.concurrent.locks.ReentrantLock:lock function is called on line 76 of BankSystem.java to acquire a lock. The wide rectangle here indicates that multiple threads are waiting for this lock, preventing CPU usage from increasing.

By analyzing this detailed information, you can quickly identify the root cause of low CPU utilization. Depending on the specific situation, you can take appropriate optimization measures, such as improving lock strategies, optimizing I/O operations, or introducing asynchronous programming techniques, thereby comprehensively improving the application’s performance.

High Disk I/O Issues in Java Applications

Excessive disk I/O not only significantly affects application performance but can also cause the entire system to run slowly. OpenResty XRay is a powerful tool that can help you precisely locate code paths causing high disk I/O in Java applications.

The following figure shows a flame graph of a Java application in a high disk I/O scenario:

Carefully analyzing the widest part of the flame graph, we can clearly see that the write function java.io.Writer:write called on line 103 of the SaveCache.java file is the main source of disk I/O pressure.

Based on this precise information, you can optimize disk I/O intensive code in a targeted manner. For example, you can consider using buffering techniques, implementing asynchronous I/O operations, or adopting batch processing strategies to effectively reduce disk I/O pressure, thereby improving overall system performance.

Through the in-depth analysis of OpenResty XRay, you can more efficiently identify and resolve performance bottlenecks, ensuring that your Java application maintains excellent performance under various load conditions.

Supported Java Versions and Operating Systems

OpenResty XRay supports mainstream OpenJDK Java versions, including:

• Java 8
• Java 11
• Java 17

At the same time, OpenResty XRay also supports various Linux operating systems, from the latest distributions to older systems like CentOS 7, providing you with maximum flexibility and compatibility.

Automated Sampling and Expert Analysis

OpenResty XRay not only provides powerful analysis capabilities but also features automated sampling and report generation capabilities. This brings more convenience to your performance optimization work:

1. Automatic online on-demand sampling: XRay can automatically trigger sampling based on preset conditions, without manual intervention. This means you can automatically start collecting data when specific load or performance metrics reach a certain threshold, ensuring that the most critical performance issues are captured.

2. Automatic generation of analysis reports: Based on the collected data, XRay can automatically generate detailed analysis reports. These reports contain key information such as performance hotspots, resource usage, potential optimization suggestions, etc., allowing you to quickly understand the performance status of the application.

3. Expert team support: In addition to automated features, our expert team is always ready to provide you with deeper support:

   • We can customize new analyzers for you based on your specific needs to meet specific performance monitoring requirements.
   • Our experts will assist you in interpreting automatically generated reports, providing professional insights and optimization suggestions.
   • In case of complex performance issues, our team can provide more in-depth analysis and solutions.

Here’s a sample report generated by OpenResty XRay:

By combining automated tools and expert support, OpenResty XRay provides you with a comprehensive performance optimization solution, helping your Java application always maintain optimal performance.

Performance Impact and Overhead

OpenResty XRay is designed to be non-intrusive and lightweight. During sampling, the additional overhead added to the Java application process is usually very small and negligible. When not sampling, the process runs at full speed without any impact.

This low-overhead feature makes OpenResty XRay an ideal production environment monitoring tool, allowing you to perform performance analysis anytime, anywhere, without worrying about significant impact on your application.

Future Plans

We haven’t stopped here. There are more plans to make OpenResty XRay even more powerful and useful for you. Some features we’re developing include:

• Analyzing the latency of network requests in Java applications, intelligently capturing data packets of problematic requests.
• Supporting other technology stacks and open-source software. We hope to make OpenResty XRay a universal tool that can analyze any online application, regardless of the underlying technology. Some targets we’re considering include Nginx modules, Envoy extensions, PostgreSQL extensions, Perl/Python/Ruby modules and libraries, and more.

If you have any suggestions or needs for more metrics or features, please let us know. We’re always listening to your feedback and continuously improving our products to meet your needs.

Conclusion

In this article, we’ve shown you how to use OpenResty XRay to analyze CPU and disk I/O usage in Java applications. We’ve also shown you some examples of the results and explained what they mean.

By using OpenResty XRay, you can easily find out which code paths are consuming CPU and disk resources, and how much they’re impacting overall performance. You can then use this information to optimize your Java application to run faster and smoother.

If you’d like to learn more about the OpenResty XRay product and how it can help you handle other aspects of online applications, please visit our website or contact us for more details.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

About the Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.