A memory leak occurs when a program allocates memory, keeps a reference to it, and never releases it—so over time, the process’s resident memory (RSS) only goes up and never comes down. Tracking down a leak in production comes with far tighter constraints than in development: you cannot restart the service, you cannot attach GDB or run valgrind, and you cannot change the code and redeploy.

This article lays out a detection methodology that works under exactly these constraints. First, we distinguish real leaks from “false leaks”—a large but stable memory footprint, or an allocator’s free pool that never returns memory to the operating system, is not a leak. Then we map real leaks to four recurring patterns, from unbounded caches in garbage-collected languages to ownership bugs in C/C++ code. Finally, we apply top-down layered attribution to trace an ever-climbing RSS curve all the way down to a specific object name and source line—as with the leaking cache in a financial Perl service that pushed memory to several gigabytes; once fixed, the process stabilized at 60MB.

The Symptom: A Memory Curve That Only Goes Up

On a monitoring dashboard, a memory leak exhibits a classic triad of symptoms:

  1. RSS keeps climbing even under steady traffic—the process consumes more and more memory, regardless of request volume.
  2. Restarting is the only “painkiller”—memory drops sharply after a restart, then climbs again, producing the familiar periodic sawtooth curve on monitoring graphs: memory climbs to nearly 100%, the process is forced to restart, and the climb begins all over again.
  3. top and pmap only show totals—they tell you “this process uses 1GB”, but they cannot attribute that memory to a specific object, module, or line of code.

The key to locating a memory leak is not in top’s numbers, but in object-level reference path analysis: finding out who is holding the memory, through which reference path, and why it is never released.

Memory Leak or Just High Memory Usage?

Not every case of “high memory” is a leak. The decisive criterion is: a large but stable memory footprint vs. unbounded growth.

If the biggest memory consumer in a PHP process is a nearly 40MB HTML string, that is not necessarily a leak—it may simply be that line 40 of ProdController.php read an entire web page into memory in one shot via file_get_contents, creating a large memory footprint. The fix for this class of problems is to optimize the memory usage pattern (for example, switching to streamed responses), not to hunt for a leak.

If your process’s memory is high but no longer growing, you have a memory footprint problem, not a leak. Each of the following four tutorials shows how to find the largest memory objects in a live process—without changing code or restarting:

But if memory keeps growing and never stops—that is a leak. Read on.

Four Leak Patterns with Real Root-Cause Chains

Memory leak mechanisms fall into four broad categories. For each one, we describe the general mechanism and present one or more real root-cause chains—traced from the object reference path all the way down to the source code.

1. Unbounded Caches in Garbage-Collected Languages

Mechanism: A garbage collector (GC) only reclaims objects that have no references. If a cache structure holds references to objects and never evicts them, then from the GC’s point of view those objects are forever “live”—their memory will never be reclaimed.

Real root-cause chains:

In a Python gunicorn process, OpenResty XRay’s GC object flame graph traced roughly 570MB of memory to the order_name_cache dictionary in the order_service.service.order.prev_processor module: the handle() function wrote a new record for every order, but no code ever deleted old records—textbook unbounded growth.

In a Perl service at a financial firm, memory ballooned to several gigabytes within days. A Perl GC object memory distribution flame graph exposed the problem immediately: a cache data structure was accumulating memory in a completely unexpected place. After the fix, memory dropped from roughly 100MB at startup to about 60MB, staying stable at 60MB+ over long-term operation—a reduction of over 95%.

In YUNDUN’s production environment, OpenResty worker processes were using far more memory than expected. OpenResty XRay’s LuaJIT GC object reference flame graph found that ngx.ctx.game_conf.tcp referenced 66 tables occupying over 1MB, with millions of table objects in the process. The entire process, from deploying the analysis tool to verifying the fix in production, took only half a day. After the fix, total memory dropped by more than 60%, with the Stream LuaJIT portion dropping by more than 80%.

2. High-Level Objects Pinning Low-Level Native Memory

Mechanism: A small object at the language level can hold a reference to underlying C/C++ structures. The memory for those C structures is attributed to the Glibc allocator, while the language-level GC sees only a reference a few dozen bytes in size—the disparity can be extreme.

Real root-cause chains:

In an OpenResty application, worker process memory kept growing linearly. OpenResty XRay’s memory analysis report showed the Glibc allocator holding about 93% of the memory, while the LuaJIT allocator held only about 2.4%. The leak appeared to be in the C layer—but the real root cause was in the Lua layer: a lua-resty-lrucache object named _LOADED.dynamic_cert.cert_cache was caching SSL certificates parsed via ssl.parse_pem_cert and ssl.parse_pem_priv_key, and the LRU cache’s capacity was set so large that it almost never evicted anything. Every time a certificate for a new domain was parsed, its underlying OpenSSL structures took up permanent residence in the Glibc arenas.

For Lua-level memory leaks, OpenResty XRay’s lj-gco-ref analyzer can directly display the complete reference path of GC objects: GC roots => registry => ._LOADED => <module name> => <table name>—tracing from the GC roots all the way to the specific leaking table, with no code changes and no restarts.

3. Ownership Bugs in C/C++ Code

Mechanism: The allocating side and the freeing side disagree about who owns a piece of memory—the allocating code assumes downstream code will free it, the downstream code assumes upstream or some other module will, and in the end nobody does.

Real root-cause chain:

In an Nginx C++ module memory leak case, OpenResty XRay’s memory leak flame graph pointed directly at the ngx_dubbo_hessian2_encode_payload_map function. Digging into the source revealed that line 96 of ngx_dubbo_util.cpp allocated a std::basic_string object via new. The object was wrapped and handed to a downstream module for processing, but because of the unusual way it was created, it was mislabeled as “not managed by this module”—the downstream automatic cleanup mechanism received the signal “this memory is someone else’s responsibility” and silently skipped it. In reality, no other party was responsible for freeing it. Every new request allocated another such block, and none of them were ever released.

4. Server Memory Pool Leaks

Mechanism: High-performance servers like Nginx manage memory allocation through a memory pool architecture. Pooled allocation makes individual allocations invisible to external tools—valgrind sees only the pool’s bulk allocations and has no way to understand the lifecycles inside the pool. When the pool’s own lifecycle management goes wrong, memory leaks.

Real root-cause chain:

In a high-concurrency API gateway built on Nginx, a single worker process’s memory kept climbing from hundreds of megabytes to over 1GB. OpenResty XRay first confirmed at the system level that the memory was mostly held by the Glibc allocator, then drilled into the Nginx memory pool layer—finding that memory consumption was concentrated in pools created by Tengine’s dynamic upstream module. A further analysis of Glibc’s memory block size distribution found 2,240 blocks in the 256k–512k range—a count that deviated sharply from the process’s normal runtime profile, and these blocks were being held long-term without release. Finally, a C-level memory leak flame graph mapped the allocation behavior back to the complete C call chain, forming a verifiable root-cause chain from the allocation site to the end of the memory’s lifecycle.

Try OpenResty XRay now to hunt down your memory leaks

It Looks Like a Leak, but It Isn’t

Not all memory growth is a leak. The following two common “false leak” patterns are rarely mentioned in generic memory debugging guides.

LuaJIT allocator’s free pool mechanism: LuaJIT’s memory allocator maintains a free pool, and it only returns memory to the operating system when contiguous free segments exist. This is by design, not a leak. In YUNDUN’s case, after traffic was drained away, in-use memory did drop, but RSS did not decrease noticeably—the trend graphs showed it clearly: as traffic drained, in-use memory shrank and free memory grew; when traffic returned, the free memory was reused. If you need those freed pages actually returned to the operating system (for example, under Kubernetes memory limits), see our article on fixing LuaJIT RSS bloat at the allocator level.

The interpreter’s own baseline overhead: Even if your business logic is lightweight, the modules loaded by the interpreter itself can occupy substantial memory. In a roughly 85MB Django process, .modules alone (the registry of loaded Python modules) accounted for 38.27MB—1,521 modules combined. A single module, openpyxl.utils.cell, took 2.6MB, and the standard library’s linecache took 648KB. This is not a leak; it is a fixed startup footprint.

The deciding criterion: To tell a real leak from a false one, check whether the growth comes from in-use memory or from free/cached memory. If in-use memory keeps growing and does not fall when load drops—that is a leak. If in-use memory is stable while free memory is simply not being returned—that is allocator behavior, not a leak.

How to Locate a Leak: Top-Down Layered Attribution

Locating a memory leak is not guesswork. The effective methodology is top-down, layer-by-layer convergence—from system-level totals, to the allocator layer, down to the specific object or line of code.

First, Attribute Memory by Allocator

The first step is not to rush after objects, but to figure out which allocator the memory is attributed to: the system allocator (Glibc), a language allocator (the LuaJIT/Python/Perl GC), or an application-level memory pool (Nginx pools)?

The judgment at this step is critical. In the LRU cache leak case above, concluding from the Glibc percentage that the leak was in the C layer and diving headfirst into a C code review would have led the investigation completely astray—the correct move is to keep checking whether language-level objects are holding that memory indirectly through C extensions. Likewise, in the Nginx memory pool case, the anomaly in the memory block size distribution narrowed the investigation directly from “Glibc totals are high” to “blocks of a specific size are accumulating abnormally”.

From GC Object Reference Paths to Source Lines

For garbage-collected languages, a GC object flame graph displays reference paths—the complete path from the GC roots to each live object, with width representing memory usage. Follow the widest path, and you find the objects holding the most memory.

Once you have the object name, the next step is to locate it in the source code. In the PHP case, the reference path pointed to the productPage property; a grep for that name in the source tree led straight to line 40 of ProdController.php. In the Python case, the reference path pointed to order_name_cache; copying the module name and letting its dots act as grep wildcards over the source files led just as quickly to the exact source file and line.

For Java, the same approach applies: you can diagnose Java memory leaks in production without heap dumps and without restarting the service—by analyzing the GC object reference chains of a live JVM to find objects still held by GC roots that should have been released.

Why Traditional Tools Fall Short in Production

Every traditional tool runs into a hard constraint in production—not “inconvenient”, but “unusable”:

OpenResty XRay’s Guided Analysis and Insights features offer a different path: they analyze running, unmodified processes directly—no restarts, no code changes, no attached debuggers. Using dynamic tracing, they generate both system-level (Glibc/memory pool) and language-level (Perl/Python/PHP/Lua/Java/C/C++) memory analyses simultaneously, and automatically derive the most significant reference paths along with root-cause suggestions.

Try OpenResty XRay now to hunt down your memory leaks

Frequently Asked Questions

How do I locate a memory leak in production without restarting the service?

Use OpenResty XRay’s Guided Analysis feature, select the “High memory usage” problem type, and analyze the running process directly. The system automatically generates GC object memory distribution flame graphs and memory allocation attribution reports, showing the largest memory objects along with their complete reference paths. The entire process requires no service restarts, no code changes, and no attached debuggers—OpenResty XRay performs non-invasive dynamic tracing analysis on live processes.

How do I tell a memory leak from high memory usage?

The decisive criterion is whether memory grows without bound. If a process’s memory is high but stable—say, a PHP process using 40MB because it read an entire web page into memory in one shot—that is a large memory footprint, not a leak. If memory keeps climbing, never stops, and is unrelated to load—that is a leak: some code path keeps allocating memory and never frees it.

Can garbage-collected languages leak memory too?

Yes. A garbage collector only reclaims objects that are no longer reachable through references. As long as a cache structure holds references to objects and never evicts them, the GC considers those objects live—their memory will never be reclaimed. In real cases, a Python module-level dictionary order_name_cache wrote a new record for every order but never deleted old ones, and a Lua cert_cache LRU cache was sized so large that it never evicted parsed SSL certificates—both are textbook leak patterns in garbage-collected languages.

Why does process memory keep growing when there is actually no leak?

There are two common reasons. One is the allocator’s free pool mechanism—for example, LuaJIT’s memory allocator does not necessarily return memory to the operating system right after objects are freed; RSS looks like it is growing, but in-use memory has already dropped, and the memory in the free pool gets reused as new requests arrive. The other is the interpreter’s own baseline overhead—for example, a Django process took 38.27MB just to load 1,521 Python modules; that is not a leak but a fixed startup footprint. The way to tell them apart is to check whether it is in-use memory or free/cached memory that is growing.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

Apply for a free trial of OpenResty XRay today

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.

Translations

We provide the English original text (this article) and a Chinese translation. We welcome interested readers to contribute translations in other languages as long as the full text is translated without any omissions. We thank them in advance.