Uneven Nginx Worker CPU Usage: Tracing a Missing reuseport Flag and a cjson Hotspot

Uneven Nginx worker CPU usage — one worker maxed out while others sit nearly idle — often comes down to a load-distribution problem such as the missing reuseport socket option. In this real customer case, OpenResty XRay traced a DNS service’s CPU imbalance to exactly that, plus a cjson_decode hotspot eating 60% of CPU time, and cut the core bottleneck by over 60% within minutes.

Here is how the flame graphs led us there, step by step.

The Symptom: Some Nginx Workers Maxed Out While Others Sat Idle

The DNS service system operated by our client is experiencing a severe imbalance in CPU usage. Some Nginx worker processes have excessively high CPU utilization, while others remain relatively idle. Additionally, the overall system response latency has increased, particularly under high load conditions. This imbalance not only affects the stability of the service but also leads to inefficient resource utilization, thereby increasing operational costs.

Traditional performance analysis methods have struggled to pinpoint the root cause precisely, as the issue involves complex interactions across multiple layers. In this situation, the client requested support from the OpenResty XRay team. We immediately conducted a comprehensive performance analysis of the system using OpenResty XRay.

How OpenResty XRay Pinpointed the Root Causes

Using the OpenResty XRay analyzer, we conducted an in-depth analysis of the target system and identified the following key issues:

Missing reuseport: Why Requests Were Unevenly Distributed

First, we checked the Nginx configuration status:

use_accept_mutex: 0
listening on: 0.0.0.0:8090, reuseport: 0
listening on: 0.0.0.0:3581, reuseport: 0
listening on: 0.0.0.0:8081, reuseport: 0
listening on: 0.0.0.0:8088, reuseport: 0
listening on: 0.0.0.0:11080, reuseport: 0
listening on: 0.0.0.0:8080, reuseport: 0
listening on: 0.0.0.0:9000, reuseport: 0
listening on: 0.0.0.0:9090, reuseport: 0
listening on: 0.0.0.0:1935, reuseport: 0
listening on: 0.0.0.0:80, reuseport: 0

We found that none of the listening ports had the reuseport option enabled, leading to uneven request distribution.

cjson_decode Consumed 60% of CPU Time in the Flame Graphs

Through C flame graph analysis, we discovered that approximately 60% of CPU time was consumed by the cjson module.

From the Lua flame graph, about 60% of the time was spent on cjson_decode operations, around 30% on shcache.lua:load, and only about 5% on the core business logic in dns_server.lua.

This indicates that JSON parsing has become the absolute performance bottleneck — the same class of problem we diagnosed in this JSON parsing memory case.

Cosocket Receive Overhead from an Outdated LuaJIT

Another significant CPU consumption point was the cosocket receive operation, which took up about 16% of CPU time:

ngx_stream_lua_socket_tcp_receive
  -> ngx_stream_lua_socket_tcp_receive_retval_handler
  -> ngx_stream_lua_socket_push_input_data
  -> luaL_addlstring [/etc/nginx/luajit/lib/libluajit-5.1.so.2.1.0]

Analysis showed that the customer was using an older version of LuaJIT, while the latest version has optimized this issue.

The Fix: Three Optimizations and Their Measured Gains

Based on the in-depth analysis results from OpenResty XRay, our technical team has developed a targeted optimization plan for our clients:

Evening Out Worker Load Distribution (20–30% Gain)

By fine-tuning the configuration, we resolved the issue of uneven load distribution among worker processes, significantly improving system resource utilization efficiency and enhancing overall performance by 20-30%.

Cutting the JSON Parsing Bottleneck by Over 60%

For the identified JSON processing performance bottleneck, we provided a multi-layered optimization strategy:

• Reconstructed data processing workflows to reduce unnecessary computational overhead
• Designed an intelligent caching mechanism to significantly lower the cost of repetitive operations
• Optimized configuration management to enhance system response efficiency

Through these optimizations, CPU consumption for the core bottleneck was reduced by over 60%, and system throughput saw a significant increase.

Upgrading the Runtime Stack (5–10% Gain)

Based on version compatibility analysis, we planned a technology stack upgrade path for our clients, further optimizing the performance of underlying components and achieving an additional 5-10% performance improvement.

This case demonstrates the powerful capabilities of OpenResty XRay in diagnosing complex performance issues, accurately pinpointing performance bottlenecks down to the code level, and providing clear directions for optimization.

Summary: What OpenResty XRay Found in Minutes

• Quickly and accurately identified all performance bottlenecks
• Discovered issues at the code level that traditional monitoring tools cannot detect
• Reduced core CPU usage by over 60%, improving overall system performance by 20% to 30%
• Identified that JSON parsing was consuming up to 60% of CPU resources, directly addressing the performance bottleneck
• Found severe imbalances in worker configuration, leading to highly inefficient load distribution
• Detected outdated components dragging down overall performance, allowing for timely upgrades and adjustments
• Significantly improved operational efficiency, saving clients considerable manpower and cost
• Utilized flame graph visualization throughout the process, making performance issues clear and optimization directions quantifiable

In today’s era of complex system architectures and high-concurrency business environments, performance issues are often not just surface phenomena. Traditional monitoring tools struggle to identify the true root causes. OpenResty XRay, as an industry-leading dynamic tracing platform, helps technical teams quickly delve into the core of problems and formulate precise, actionable optimization plans.

If you also want your system to run more stably, faster, and more cost-effectively, or if you wish to experience a deep performance optimization in advance, we invite you to apply for a product trial. Let OpenResty XRay become the most trusted tool in your team’s arsenal.

Frequently Asked Questions

Why is one Nginx worker process using much more CPU than the others?

In this case, none of the listening ports had the reuseport socket option enabled, so incoming requests were not evenly distributed across the Nginx worker processes — some workers ran hot while others stayed relatively idle. After the load-distribution configuration was fine-tuned, resource utilization evened out and overall performance improved by 20–30%.

How do I find out what is consuming CPU inside Nginx or OpenResty Lua code?

Flame graphs at two levels narrow it down quickly. In this case, the C-level flame graph from OpenResty XRay showed about 60% of CPU time inside the cjson module; the Lua-level flame graph then pinpointed cjson_decode at ~60%, shcache.lua:load at ~30%, and only ~5% in the actual business logic in dns_server.lua.

Why was cjson_decode consuming 60% of CPU time?

The service was repeatedly paying the cost of JSON parsing in its data-processing path, which made cjson_decode the single largest CPU consumer. Reconstructing the data-processing workflow and adding a caching mechanism to cut the cost of these repetitive operations reduced the core bottleneck’s CPU usage by over 60%.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.