Kong Plugin High CPU Usage: A Hidden Lua Exception Traced with Flame Graphs

A Kong custom plugin can drive high CPU usage even under light traffic when it repeatedly throws Lua exceptions. In one customer’s auth plugin, a nil value passed to string.lower repeatedly raised a Lua exception — and because it was swallowed by pcall, nothing showed up in Kong’s error log. OpenResty XRay’s CPU flame graph pinpointed the exception at handler.lua:35, and the simple fix cut CPU from 80% to 50%.

Below is how we traced it. Kong is a powerful, flexible API gateway built on top of our open-source OpenResty software, but custom plugins that extend it can introduce performance issues that are hard to notice and debug. OpenResty XRay is a noninvasive tool that monitors and analyzes any OpenResty application, including Kong, without changing its code or restarting it.

The Problem: Kong Plugin High CPU Usage Under Light Traffic

Our customer noticed that their Kong servers were consuming more CPU resources than expected, even though the incoming API traffic was not very high. They suspected that there might be some inefficiencies or errors in their custom plugins, but they had no clue where to look for them. They needed a tool that could help them pinpoint the root cause of the CPU bottleneck and provide actionable insights on how to fix it. (For a broader look at how much CPU and memory each Kong plugin consumes, see our companion article on per-plugin CPU and memory statistics.)

Finding the CPU Bottleneck with an OpenResty XRay Flame Graph

The customer installed OpenResty XRay on their Kong servers and configured it to automatically sample the online Kong processes either periodically or when the CPU usage spiked. OpenResty XRay also automatically updated the analysis report for the current day every hour, so that our customer and our team could see the latest performance data.

One of the first things we noticed in the report was the following hot code path in the CPU section:

This code path showed that Lua exceptions were being thrown by the string.lower standard function, which converts all the characters of an input string to lowercase. Exceptions are expensive operations in most programming languages and their implementations, because they usually require stack unwinding and error handling. We wanted to know why these exceptions were happening and where they came from.

By hovering the mouse over the Lua or C functions in the code path, we could see the Lua source locations, including the file name and the line number.

The report also showed us a CPU flame graph with this hot code path highlighted in red:

To confirm our findings, we checked out the Errors & Exceptions section of the same report, which showed us this error message:

The error message said “bad argument #1 to ‘lower’ (string expected, got nil)”, which meant that the Lua code was passing nil values to the lower function, which expected a string argument. By looking at its parent function frame, [builtin#string.lower], we knew that it was the Lua builtin function string.lower. From the report, we also learned that the exception was thrown from line 35 of the source file .../kong/plugins/auth/handler.lua.

It was interesting to note that this Lua exception was caught by pcall, a Lua function that calls another function in protected mode, meaning that it can catch any errors without interrupting the whole Lua handler. That’s why there was nothing useful in Kong’s error log files.

At this point, we had enough information to conclude that there was a bug in the customer’s own auth Kong plugin, which misused the standard Lua API function string.lower. The fix was also simple: just avoid passing nil values to string.lower at line 35 of .../kong/plugins/auth/handler.lua.

The Result: Kong CPU Usage Cut from 80% to 50%

After applying the fix to their custom auth plugin, our customer saw a dramatic improvement in their Kong server’s performance.

As we can see from this graph, for the same amount of incoming API traffic, the average CPU usage dropped from 80% to only 50%. That’s a 37.5% reduction in CPU consumption! Our customer was very happy with this result and thanked us for our help.

Frequently Asked Questions

Why is my Kong plugin using high CPU?

A custom Kong plugin can burn CPU even under light traffic when it repeatedly throws Lua exceptions. Exceptions are expensive because they require stack unwinding and error handling. In this case, an auth plugin repeatedly passed a nil value to string.lower, raising an exception each time and consuming CPU that the incoming traffic alone did not explain.

How do I find which Kong plugin is burning CPU?

OpenResty XRay samples the running Kong worker processes without changing their code or restarting them, and shows the hot code path in the CPU section, a CPU flame graph, and the exact Lua source file and line number. Here it traced the hot path to string.lower at handler.lua:35 in the customer’s own auth plugin.

Can a Lua exception caught by pcall still cause high CPU?

Yes. pcall runs a function in protected mode and catches the error, so nothing shows up in Kong’s error log — but the exception is still thrown and unwound each time it fires, which keeps burning CPU. That is why the problem was invisible in the logs until the flame graph exposed it.

Conclusion

This article demonstrated how OpenResty XRay helped our customer find and fix a CPU bottleneck caused by unexpected Lua exceptions in their custom Kong plugin. By using OpenResty XRay, they were able to quickly identify the hot code path and the source location of the exception, as well as the error message and the stack trace. They were also able to see the performance improvement after applying the fix.

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 2 decades of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.