In large systems within the financial industry, continuous and stable operation is a critical requirement. Unexpected service issues can not only introduce performance risks but also lead to immeasurable business consequences. Recently, one of our major financial clients encountered a typical challenge: a 24x7 service, after being deployed and running for a few days, experienced a sudden memory usage spike to several gigabytes, a problem that was never reproducible in their test environment.

This is precisely where OpenResty XRay’s value shines. As a software diagnostic tool that provides deep visibility into live processes, it empowers teams to address such high-risk, hard-to-locate, and elusive problems. This article will walk you through this client’s real-world case, demonstrating how XRay quickly identified the root cause of the issue using a flame graph in a high-pressure financial services environment.

Pinpointing Memory Leak Root Causes with a Flame Graph

We utilized OpenResty XRay to analyze our most memory-intensive production process, generating a flame graph visualizing the memory distribution of Perl GC objects:

In this flame graph, each layer represents a call stack, and its width indicates the proportion of memory consumed.

This Perl GC object memory distribution flame graph immediately revealed the problem: a significant memory leak within a cached data structure. This was an entirely unsuspected location, explaining why conventional debugging methods had failed to identify the issue.

Beyond the primary leak, the flame graph also highlighted several other areas where memory usage could be optimized, insights that traditional performance analysis tools struggle to uncover with such efficiency and comprehensiveness.

Significant Optimization Achievements

Leveraging the analysis results from OpenResty XRay, we swiftly identified and resolved issues within the code. Following targeted fixes and optimizations, the system’s memory usage has seen remarkable improvement:

Now, even with the system running continuously for several weeks, the memory footprint of each process consistently remains at a low level, just over 60MB, completely eliminating the memory leak issue.

Further Reading

The case study of our financial industry client is just one example of the value OpenResty XRay delivers in complex systems. In fact, we have encountered similar challenging problems in various real-world scenarios and have consistently found rapid solutions using XRay.

Interested readers can refer to the following technical case studies:

1. How OpenResty XRay Thoroughly Analyzes Nginx Memory Corruption Issues
2. How UDB and OpenResty XRay Let You See Through the Entire Perl Code Execution Process
3. Case Study: How to Trace a Memory Leak Caused by an LRU Cache Using OpenResty XRay
4. Memory and CPU usage statistics among Kong plugins online

These case studies will further demonstrate how OpenResty XRay not only addresses individual problems but also provides deep observability and diagnostic capabilities for diverse technology stacks and business systems.

Summary

In complex systems, memory leaks are often deeply hidden. From multi-gigabyte memory leaks to stable operation at 60MB, OpenResty XRay has once again proven its powerful capabilities in resolving complex production issues.

• Real-time Insight: No downtime required, enabling precise, surgical diagnosis of the production environment and preventing business interruption.
• Granular Precision: Fine-grained analysis at the GC object level, delving deep into the code to accurately pinpoint the root cause of problems.
• Intuitive Visualization: Flame graphs make complex problems instantly clear, significantly lowering the barrier to analysis.

In this real-world application, we not only resolved the memory leak but also uncovered other performance optimization opportunities within the system, thereby improving overall code quality.

If your team is also grappling with intractable problems such as performance bottlenecks, memory leaks, or high CPU usage, and is tired of endless “firefighting” and “guessing games,” then it’s time to embrace a new approach. Contact us to usher in a new era of efficient operations and maintenance!

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.