In today’s rapidly evolving internet landscape, system performance has become a critical competitive factor for businesses. However, as system complexity continues to increase, traditional performance analysis and troubleshooting methods often fall short. OpenResty XRay has emerged as a next-generation dynamic tracing technology, providing developers and operations teams with a powerful “eagle eye” that leaves no performance bottleneck or hidden issue undetected.

Try out OpenResty XRay for free today

Dynamic Tracing: A Powerful Tool for Solving Complex System Issues

Before delving into OpenResty XRay, we need to understand the core value of dynamic tracing technology. As an advanced diagnostic method, dynamic tracing can collect critical data while systems are running without causing significant interference. This capability is particularly valuable in today’s complex distributed environments, offering a fresh perspective for observing and understanding system behavior.

1. Production Environment Diagnostics

In production environments, system issues often appear and disappear without a trace, making them difficult to capture. Traditional methods might require service restarts or code modifications, which are nearly unacceptable for high-availability systems. Dynamic tracing technology allows us to:

  • Identify high-latency functions, memory leaks, and other issues without restarting services
  • Observe function call frequency and execution time under real user requests in real-time
  • Perform deep diagnostics without impacting the production environment

2. The Revolution in Performance Analysis

Compared to static analysis, dynamic tracing provides performance data that more closely reflects actual operational states:

  • Precisely identifies “cold paths” and “hot paths” in systems
  • Uncovers overlooked performance bottlenecks in code
  • Offers data-driven optimization directions, rather than relying on experience-based guesswork

3. Deep System-Level Tracing

Dynamic tracing isn’t limited to the application level; it can penetrate deep into system layers:

  • Tracks kernel events and system calls, revealing the complete picture of application-system interactions
  • Automatically records variable value changes, making complex logic debugging simple and intuitive
  • Provides a full-stack view from application to system

Based on these powerful dynamic tracing capabilities, OpenResty XRay has built a comprehensive solution that transforms these technical advantages into practical tools, helping developers and operations teams solve real-world problems.

Why Choose OpenResty XRay?

1. Powered by Next-Generation Dynamic Tracing Technology, 100% Non-Intrusive

OpenResty XRay employs innovative dynamic tracing technology that requires absolutely no source code modifications or application recompilation. This means you can analyze any running system at any time without causing disruption.

2. Innovative Automatic Sampling Method, Efficient and Lightweight with Low Overhead

OpenResty XRay utilizes breakthrough intelligent sampling algorithms that thoroughly resolve the pain point of high performance overhead found in traditional dynamic tracing tools. XRay offers 3 sampling modes, precisely controlling sampling frequency and depth to provide in-depth analysis while minimizing system performance impact.

According to our empirical test data, OpenResty XRay demonstrates extremely low performance overhead when tracing applications in various programming languages:

This low-overhead characteristic makes XRay a truly suitable dynamic tracing tool for continuous operation in production environments. You can confidently deploy XRay on mission-critical systems without concerns about negative impacts on user experience or system stability.

3. Full-Stack, Comprehensive Analysis Without Blind Spots

XRay can delve into system calls, CPU utilization, memory usage, disk I/O operations, and other aspects, providing a comprehensive performance view that leaves no potential performance issues hidden.

4. Application Analysis Without Debug Symbols

Even in production environments lacking debug symbols, OpenResty XRay can still automatically analyze executable files through machine learning algorithms, reconstruct debug symbols, and deliver valuable analytical results.

The case study Analyze OpenResty/Nginx Applications without Debug Symbols demonstrates OpenResty XRay’s symbol reconstruction capabilities.

5. Container Transparency Capabilities

In containerized environments, OpenResty XRay can transparently analyze applications within containers without special configurations or privileges, providing powerful diagnostic capabilities for cloud-native applications.

6. Automated Security Analysis

Beyond performance analysis, OpenResty XRay can automatically detect potential security vulnerabilities and abnormal behaviors, offering additional protection for system security.

Product Focus: XRay Applications in Real-World Scenarios

Case 1: Using OpenResty XRay to Analyze Memory Issues, Quickly Implementing Fixes

A customer’s core business system faced serious memory consumption issues, with system memory usage continuously increasing, eventually leading to service instability. Traditional tools failed to effectively identify the root cause, but after using OpenResty XRay, the team precisely located the problematic code in just a few hours—a seemingly harmless Lua closure function causing memory leaks under specific conditions.

After the fix, system memory usage immediately decreased by 60%, significantly improving service stability. More importantly, the entire troubleshooting and repair process required no service restarts, resulting in almost zero impact on the production environment.

For details on the analysis process, see: Memory Reduced by 60%, OpenResty XRay Precisely Identifies Problem Code, Quickly Implements Fixes.

Case 2: OpenResty XRay Analyzes and Resolves a Major Bilibili Production Incident

Bilibili once experienced a severe production incident where their core API gateway generated numerous 499 errors, affecting normal access for millions of users. Traditional monitoring tools could only identify the symptoms, not the root cause.

Using OpenResty XRay’s dynamic tracing capabilities, Bilibili’s technical team quickly discovered the issue was in an edge case within OpenResty’s cosocket connection pool implementation, causing connections to close prematurely under high concurrency. XRay not only helped them locate the problem but also provided detailed execution paths and variable states, making the fix straightforward and efficient.

The case Resolving Bilibili’s major site incident with OpenResty XRayt demonstrates XRay’s value in emergency troubleshooting, especially in complex distributed system environments where it can quickly pinpoint subtle issues in underlying components.

Case Study 3: Resolving a CPU Bottleneck Caused by a Lua Exception in a Custom Kong Plugin

A company using Kong API Gateway for their core business traffic experienced abnormally high CPU usage, which severely impacted service performance. Initial analysis indicated the problem was in a custom Kong plugin, but the specific cause was difficult to determine.

Using OpenResty XRay’s CPU profiling functionality, the team discovered that the issue was related to a frequently called string.lower() function. Further analysis revealed that this function was triggering Lua exceptions when processing specific non-ASCII characters, causing the exception handling path to be frequently executed and consuming significant CPU resources.

After fixing this issue, system CPU usage immediately decreased by 40%, and service response time reduced by 30%. This case demonstrates XRay’s unique advantage in identifying seemingly simple but high-impact performance issues, especially in complex Lua/OpenResty application scenarios.

For a detailed analysis process, please refer to: How we solved a CPU bottleneck caused by Lua exceptions in a custom Kong plugin.

Common Misconceptions and FAQs: What You Might Want to Know About Dynamic Tracing

“Will it significantly impact performance?”

No. OpenResty XRay has developed unique sampling techniques specifically for production environments, with runtime performance overhead that is virtually imperceptible in most scenarios. You can confidently use it in production environments.

“Do I need to modify my business code?”

Not at all. OpenResty XRay is 100% non-intrusive, requiring no code modifications, recompilation, or application restarts to begin analysis.

“Is deployment complicated?”

OpenResty XRay’s deployment process is carefully designed to be simple and efficient. Whether you’re using physical servers, virtual machines, or containerized environments, XRay provides appropriate installation solutions. Based on your system environment, you can choose the most suitable installation method:

  • Container environments: In Kubernetes clusters, simply apply a few YAML configuration files to complete Agent deployment
  • Mainstream Linux distributions: APT/RPM repository support is available for one-click installation via package managers
  • Specific environments: Pre-compiled Bundle packages are provided, ready to use without complex dependencies

After installation, XRay automatically detects the system environment and configures appropriately, eliminating the need for complex manual adjustments. For large-scale deployments, it also supports integration with automation scripts and configuration management tools.

For XRay installation tutorials, please refer to:

Conclusion

Traditional debugging tools typically pause program execution, making them unsuitable for production environments; whereas OpenResty XRay can perform real-time analysis without interrupting services. Compared to traditional APM tools, XRay doesn’t require pre-configured instrumentation, can dynamically determine analysis targets, and provides deeper performance insights. In contrast to eBPF, XRay offers higher-level abstractions and a more user-friendly interface, lowering the barrier to entry while maintaining powerful analytical capabilities.

OpenResty XRay is transforming how developers and operations teams resolve performance issues. Whether troubleshooting complex production environment problems or conducting routine performance optimization, XRay delivers unprecedented insights, helping you build faster, more stable, and more secure systems.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

Try OpenResty XRay to analyze your running applications now

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.