In this tutorial, we will walk through how to use OpenResty XRay to quantitatively analyze the CPU time consumption in the Envoy servers online. We will show the C++ code paths that consume the most CPU time. These hot code paths are obtained by OpenResty XRay automatically analyzing and interpreting the C++ language-level CPU flame graphs. It tries hard to minimize the performance impact, thus making it ideal for production environments sensitive to performance overhead and latency. At least for simple scenarios, we can see that Envoy is much less efficient than OpenResty and Nginx.

Problem: high CPU usage

Use the cat command to check this Envoy server configuration file.

You can see that it listens on port 1088.

It replies with the “Hello world” response body.

Test the response of this /hello interface.

The response is “hello world” indeed.

Run the top command to check the CPU usage. Look at this process named envoy.

As shown, it consumes more than 90% of a CPU core.

Run the ps command to see the full command line for this process. It is downloaded and installed from the Envoy official binary package repository.

Use the guidede analysis feature of OpenResty XRay to spot the most CPU-intensive C++ code paths inside Envoy Servers

Let’s use OpenResty XRay to check this unmodified process. You can analyze it in real-time and figure out what is happening. Open the OpenResty XRay web console in the web browser.

Make sure it is the right machine you are watching.

You can choose the right machine from the list below if the current one is incorrect.

Go to the “Guided Analysis” page.

Here, you can see different types of problems that you can diagnose.

Let’s select “High CPU usage”.

Click on “Next”.

Select the Envoy application.

Select the process that consumes over 90% of the CPU resources. This is what we saw previously in top.

Make sure that the application type is right. Usually, the default should be correct.

The language level here is just “C/C++”.

We can set the maximum analyzing time. We’ll leave it as 300 seconds, which is the default value.

Let’s start analyzing.

The system will keep performing different rounds of analysis. Now, it’s executing the first round.

The first round is done, and it’s already on to the second one. That’s enough for this case.

Let’s stop analyzing.

We can see it automatically created a report.

This is the type of problem we diagnose. It’s CPU.

This is the #1 hottest C++ code path for the CPU time.

This is an overloaded operator of the SchedulableCallbackImpl class.

Click “More” to see details.

The hot code path was automatically derived from this C-land CPU flame graph.

Click the icon to enlarge the flame graph.

Continue to zoom in invoke_impl function.

The write method of the Envoy network socket class performs socket writing. It sends out the HTTP response data.

The drain method of the Envoy buffer class frees unused memory in the write buffer, and performs other cleanup work.

The clearDeferredDeletedList method of the Envoy dispatcher class frees up all the resources associated with the current request. And does all the cleanup work.

Check the #2 hottest C++ code path for the CPU time.

The function emitLog in the Envoy Proxy is used to write the access logs to a file.

Enlarge the flame graph.

Zoom in the emitLog function.

Most of the emitLog CPU time is spent formatting the log message string, not on writing file operations.

Here is the third hottest code path that costs the most CPU time.

This prepareLocalReplayViaFilterChain function is in the chain of Envoy response output filters. Each filter in the chain has the potential to modify the response.

Enlarge the flame graph.

Zoom in the prepareLocalReplayViaFilterChain function.

The createHeaderMap function is hot. It’s mainly for allocating new hash table for HTTP headers.

The newUri function is mainly for allocating and formatting the URI string.

The setStatus function sets the response status code.

The format method of the BodyFormatter formats the responses body data.

setContentLength method is used to set the response length header.

setReferenceContentType method is for setting the Content-Type response header.

This is the performance comparison chart between Envoy servers and OpenResty. You can see that the throughput of OpenResty is more than 200% higher than Envoy servers.

Automatic analysis and reports

OpenResty XRay can also monitor online processes automatically and show analysis reports. Go to the “Insights” page.

You can find the reports on the Insights page for daily and weekly periods. For this reason, you don’t have to use the “Guided Analysis” feature.

Though “Guided analysis” is useful for application development and demonstration purposes.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.