Envoy proxy servers can burn a full CPU core on internal bookkeeping even for a simple “hello world” response. In this case study, a C++ CPU flame graph generated by OpenResty XRay reveals the three hottest code paths inside a live Envoy process — topped by the SchedulableCallbackImpl operator — without restarting or recompiling the binary. A throughput comparison then shows OpenResty handling over 200% more requests than Envoy on identical hardware.

Below: the full profiling walkthrough, flame graph readings, and automated reporting setup.

Try OpenResty XRay to analyze your running applications now

Scenario: Envoy Proxy Running at 90% CPU

Use the cat command to check this Envoy server configuration file.

Terminal showing Envoy server configuration file via the cat command

You can see that it listens on port 1088.

Envoy configuration listening on port 1088

It replies with the “Hello world” response body.

Envoy direct response configuration returning hello world

Test the response of this /hello interface.

curl request testing the Envoy /hello endpoint

The response is “hello world” indeed.

curl response confirming hello world from Envoy

Run the top command to check the CPU usage. Look at this process named envoy.

top command output showing Envoy process with high CPU usage

As shown, it consumes more than 90% of a CPU core.

Envoy process consuming over 90 percent of a CPU core

Run the ps command to see the full command line for this process. It is downloaded and installed from the Envoy official binary package repository.

ps command showing Envoy binary installed from the official repository

Profile Envoy CPU with C++ Flame Graphs

Profile the Live Envoy Process

Open the OpenResty XRay web console. The dashboard for this machine already shows near-100% CPU utilization and detects the running Envoy application.

OpenResty XRay dashboard showing envoy-app-server with high CPU utilization

Navigate to “Guided Analysis” and select “High CPU usage” as the problem type. Then pick the Envoy worker process — it shows 93% CPU, matching what top reported:

Selecting the Envoy worker process at 93 percent CPU for analysis

Leave the defaults (application type: Envoy, language level: C/C++, maximum running time: 300 seconds) and start the analysis. After two sampling rounds, OpenResty XRay automatically generates a report ranking the hottest C++ code paths:

OpenResty XRay analysis report showing top CPU code paths for Envoy

This is the type of problem we diagnose. It’s CPU.

Report identifying CPU as the diagnosed problem type

#1 Hot Path — SchedulableCallbackImpl

This is the #1 hottest C++ code path for the CPU time.

Number-one hottest C++ code path in the Envoy CPU report

This is an overloaded operator of the SchedulableCallbackImpl class.

SchedulableCallbackImpl overloaded operator identified as the top CPU consumer

Click “More” to see details.

Expanding details of the hottest code path

The hot code path was automatically derived from this C-land CPU flame graph.

C++ CPU flame graph of the Envoy process showing the SchedulableCallbackImpl hot path

Click the icon to enlarge the flame graph.

Enlarged Envoy CPU flame graph for detailed inspection

Continue to zoom in invoke_impl function.

Flame graph zoomed into the invoke_impl function

The write method of the Envoy network socket class performs socket writing. It sends out the HTTP response data.

Flame graph highlighting the Envoy socket write method

The drain method of the Envoy buffer class frees unused memory in the write buffer, and performs other cleanup work.

Flame graph showing the buffer drain method freeing write-buffer memory

The clearDeferredDeletedList method of the Envoy dispatcher class frees up all the resources associated with the current request. And does all the cleanup work.

Flame graph showing clearDeferredDeletedList cleanup in Envoy dispatcher

#2 Hot Path — emitLog Access-Log Formatting

Check the #2 hottest C++ code path for the CPU time.

Number-two hottest C++ code path in the Envoy CPU report

The function emitLog in the Envoy Proxy is used to write the access logs to a file.

emitLog function identified as the second CPU-intensive path in Envoy

Enlarge the flame graph.

Enlarged flame graph of the emitLog code path

Zoom in the emitLog function.

Flame graph zoomed into the emitLog function in Envoy

Most of the emitLog CPU time is spent formatting the log message string, not on writing file operations.

Flame graph revealing most emitLog CPU time spent on string formatting

#3 Hot Path — prepareLocalReplayViaFilterChain

Here is the third hottest code path that costs the most CPU time.

Number-three hottest C++ code path in the Envoy CPU report

This prepareLocalReplayViaFilterChain function is in the chain of Envoy response output filters. Each filter in the chain has the potential to modify the response.

prepareLocalReplayViaFilterChain in the Envoy response filter chain

Enlarge the flame graph.

Enlarged flame graph of the Envoy filter-chain code path

Zoom in the prepareLocalReplayViaFilterChain function.

Flame graph zoomed into prepareLocalReplayViaFilterChain

The createHeaderMap function is hot. It’s mainly for allocating new hash table for HTTP headers.

Flame graph showing createHeaderMap allocating hash tables for HTTP headers

The newUri function is mainly for allocating and formatting the URI string.

Flame graph showing newUri allocating and formatting the URI string

The setStatus function sets the response status code.

Flame graph showing setStatus setting the response status code

The format method of the BodyFormatter formats the responses body data.

Flame graph showing BodyFormatter formatting the response body

setContentLength method is used to set the response length header.

Flame graph showing setContentLength setting the response length header

setReferenceContentType method is for setting the Content-Type response header.

Flame graph showing setReferenceContentType setting the Content-Type header

Envoy vs OpenResty Throughput Comparison

This is the performance comparison chart between Envoy servers and OpenResty. You can see that the throughput of OpenResty is more than 200% higher than Envoy servers.

Throughput comparison chart showing OpenResty over 200 percent faster than Envoy

Boost your application's performance by OpenResty XRay for free

Continuous CPU Monitoring with Automated Reports

OpenResty XRay can also monitor online processes automatically and show analysis reports. Go to the “Insights” page.

OpenResty XRay Insights page for automated CPU reports

You can find the reports on the Insights page for daily and weekly periods. For this reason, you don’t have to use the “Guided Analysis” feature.

Daily and weekly CPU profiling reports on the Insights page

Though “Guided analysis” is useful for application development and demonstration purposes.

Guided Analysis option for development and demonstration purposes

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

Try out OpenResty XRay for free today

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

FAQ: Envoy CPU Profiling

Why is my Envoy proxy using so much CPU?

Common CPU consumers revealed by flame graphs in this case study: socket-write callbacks and buffer cleanup (SchedulableCallbackImpl), access-log string formatting (emitLog), and response filter-chain processing including header-map allocation (prepareLocalReplayViaFilterChain). Profiling the live process with CPU flame graphs — rather than guessing from metrics — pinpoints which C++ code path actually dominates.

How do I generate a CPU flame graph for Envoy?

Point a dynamic-tracing profiler at the Envoy worker process PID. OpenResty XRay’s guided analysis samples the running binary, generates a C-land CPU flame graph, and automatically ranks the hottest C++ code paths — no recompile, restart, or instrumentation required. Alternatively, rebuild Envoy with gperftools and use pprof, but that demands source access and a service restart.

Can I profile Envoy CPU in production without restarting?

Yes. Dynamic-tracing tools analyze the live process from the outside, without changing its code or requiring any collaboration from the target. Sampling overhead is negligible, and when the tool is not sampling the impact is strictly zero — making the approach safe for latency-sensitive production deployments.

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.