In today’s complex software system environments, production issues are often as elusive as a black box. OpenResty XRay as a next-generation dynamic tracing product, can transform online software systems from black boxes into white boxes, allowing potential problems to be discovered before they escalate, providing enterprises with continuous insight and control over their systems.

Traditional offline simulation testing methods, regardless of how many resources are invested, have extremely limited coverage and always maintain a gap with real production environments. Meanwhile, traditional white-box analysis methods require constantly adding monitoring metrics and debug logs, which is both cumbersome and inefficient. We need a more intelligent and efficient solution.

OpenResty XRay: A Dynamic Tracing Tool That Turns Black Boxes into White Boxes

OpenResty XRay fundamentally solves system transparency issues. Through it, development teams can gain deep insights into internal system operations and promptly identify and resolve potential problems. It’s not only suitable for production environments; many customers also use it in development environments for white-box analysis. Regardless of the environment, white-box analysis is more efficient and convenient than black-box analysis.

The ultimate goal of OpenResty XRay is not just to solve production issues, but to help customers quickly understand complex systems and improve them at low cost. We call this “turning stone into gold”—transforming complex, difficult system problems into clear, controllable solutions, which is the most valuable and challenging work.

Comprehensive Multi-dimensional Analysis Capabilities

OpenResty XRay provides comprehensive multi-dimensional dynamic tracing and analysis capabilities, covering all aspects of system operation:

1. Thread blocking and lock issue analysis: Precisely locate thread contention and deadlock situations
2. CPU-related problem diagnosis: Resolve high CPU usage or inefficient CPU resource utilization issues
3. Memory issue troubleshooting: Analyze the root causes of excessive memory consumption or memory leaks
4. Disk IO analysis: Identify disk read/write bottlenecks
5. Network and request analysis: Perform intelligent packet capture and automatic analysis for application-level request latency, long-tail requests, and abnormal connections
6. Exception capture and analysis: Comprehensively capture and analyze various exceptions (including exceptions caught by the application itself)
7. Crash analysis: In-depth parsing of core dump files and process crash causes
8. Memory corruption diagnosis: Detect heap corruption, stack corruption, and general buffer overflow issues

Since OpenResty XRay can turn the entire software stack into a white box, the boundaries of its analytical capabilities are limited only by our and our customers’ imagination. As we promise: “You wish it, we make it happen!”

Real-World Case Studies

Case One: Infinite Loop Issue in Bilibili’s Gateway System

Bilibili’s internal gateway system, developed based on OpenResty, once encountered a severe issue where all servers were at 100% CPU utilization but unable to process requests. Using OpenResty XRay’s C and Lua language-level CPU flame graphs, the root cause was quickly identified: a string type weight value “0” in the configuration metadata mismatched with the numeric type expected by the lua-resty-balancer library, resulting in an infinite loop. Once discovered, the issue was immediately resolved, and the system returned to normal.

For more information, you can see the detailed analysis process in the article Resolving Bilibili’s major site incident with OpenResty XRay.

Case Two: Performance Optimization for Kong Servers

A customer’s Kong servers were consuming abnormally high CPU resources even under low traffic conditions. Using OpenResty XRay’s automatic sampling feature, it was discovered that the string.lower standard function was throwing Lua exceptions, which led to identifying a bug in the customer’s custom auth plugin. After fixing it, the Kong server’s performance improved significantly.

For more details about this case, you can refer to How we solved a CPU bottleneck caused by Lua exceptions in a custom Kong plugin.

Case Three: Discovery of Hardware-Level Bug

In another case, OpenResty XRay helped discover a hardware bug in the Xeon CPU used by a customer in production. Initially, the customer attempted to resolve it through a kernel patch, but OpenResty XRay further identified issues with the patch, ultimately confirming a defect at the CPU hardware level. This fully demonstrates that XRay can not only analyze software problems but even detect hardware-level vulnerabilities.

Product Vision

The vision of OpenResty XRay is to transform applications across all technology stacks and programming languages from black boxes into white boxes, ensuring that no mysterious issues remain in production environments. We are committed to building the most excellent bug hunters, helping customers delve deep into system internals to discover and resolve the most deeply hidden problems.

No matter how complex the system or how elusive the issue, OpenResty XRay can help you turn black boxes into white boxes, making system problems impossible to hide. We look forward to collaborating with more customers to achieve stable and efficient system operations, freeing technical teams from tedious troubleshooting so they can focus on innovation and creating business value.

With the support of OpenResty XRay, we believe every complex system can become transparent and controllable, and every technical team can more confidently face challenges. This is the driving force and mission behind our continuous progress.

OpenResty Inc. also provides powerful private library services covering technical requirements across various industries. These private libraries offer significant advantages in performance optimization, security protection, and data processing, helping enterprises quickly build efficient, secure, and reliable application systems. Whether in finance, e-commerce, travel, or media industries, OpenResty’s private library services can provide targeted solutions.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this sharing, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.