Our business extensively uses the OpenResty ecosystem. Occasionally, we encounter some peculiar issues that are difficult to locate, such as memory and CPU hotspots. With OpenResty XRay, we can quickly identify the root cause and promptly resolve these issues.
OpenResty XRay is simple to deploy, non-intrusive to business operations, and has an intuitive interface that provides us with many optimization guidelines. It has become our powerful tool.
Wishing OpenResty XRay continued success!

OpenResty XRay is an advanced Application Performance Management (APM) product developed by OpenResty Inc. It performs real-time X-ray diagnostics and optimization of application systems in a non-intrusive manner. Since opening applications for trial use in the first half of this year, it has successfully helped numerous clients and received widespread acclaim.

Shanghai YUNDUN Information Technology Co., Ltd. (YUNDUN) is an innovative enterprise focused on providing next-generation security products and services. Within YUNDUN’s product line, open-source OpenResty is extensively used. The high performance and flexible customization capabilities of open-source OpenResty provide powerful support for YUNDUN’s products. Meanwhile, during the use of open-source OpenResty, they have also encountered some challenging issues.

High Memory Usage by Worker Processes

This was YUNDUN’s first use case for XRay. At that time, the OpenResty worker processes in the production environment were consuming more memory than expected, and the memory usage would not decrease even after traffic was diverted. After an extensive investigation, they initially determined that high memory consumption by LuaJIT was the cause, but they couldn’t locate more specific problem areas. They hoped XRay could help analyze and diagnose the issue.

Analysis Process

First, we deployed OpenResty XRay directly in the production environment. Yes, XRay doesn’t require application cooperation and analyzes applications in a non-intrusive way, making it suitable for direct deployment in production environments.

For memory-related issues, OpenResty XRay has integrated a complete analysis solution. For specific background knowledge, you can refer to these blog posts:

1. How OpenResty and Nginx Allocate and Manage Memory
2. How OpenResty and Nginx Shared Memory Zones Consume Physical Memory

We first focused on the memory usage of OpenResty worker processes at the application level.

We can see that Stream LuaJIT occupied most of the memory. Clicking on this block reveals more details.

We can see that most of the memory is managed by the LuaJIT GC in the stream subsystem. Clicking on this block for further analysis.

We can see that there are millions of table objects occupying most of the memory. We continue to examine the GC object reference relationship flamegraph.

In the Lua world, a Lua GC object is considered “alive” if it’s referenced by a GC root; otherwise, it’s “dead” and will eventually be cleaned up and released as the GC works.

The GC object reference relationship flamegraph displays reference chains in a flamegraph format, with references shown from bottom to top, and the width representing memory usage size.

In our specific flamegraph, we can see that objects referenced by ngx.ctx occupy a significant amount of memory and are quite scattered. Let’s select a column in the flamegraph:

Here we can see that ngx.ctx.game_conf.tcp references 66 tables, occupying over 1MB of memory. Examining other columns reveals similar reference patterns. With this critical information, the Yundun engineering team quickly completed optimization improvements.

From starting to use XRay, to analyzing and locating the problem, optimizing improvements, and finally verifying effectiveness in production, the entire process took only half a day. Without XRay’s precise problem-locating capabilities, this would have been unimaginable.

Let’s use XRay again to check the actual situation after optimization. Overall memory usage decreased by over 60%.

Now we can see that Stream LuaJIT usage is no longer high, having decreased by over 80% compared to pre-optimization, indicating that the optimization work has been successfully completed.

Questions about Worker Process Memory Not Being Released

However, there’s still a question: why didn’t memory usage significantly decrease after traffic was diverted? Let’s look at the analysis after traffic was diverted, as shown below:

We can see that in the memory managed by the LuaJIT memory allocator, most of it remains in the free memory pool and hasn’t been released to the operating system. This is because the LuaJIT memory allocator has a free memory pool mechanism, and it only releases memory to the operating system when there are continuous segments of free memory.

We can also see the complete process of memory usage changes through the trend chart, as shown below:

From the memory trend chart, it’s evident that during the traffic diversion process, the memory in use gradually decreased, and free memory gradually increased. When traffic was routed back, the free memory began to be utilized again. The clouds have cleared, and the mystery is solved.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.