OPENRESTY XRAY Updated Aug 24, 2022 6mins read

Resolving Bilibili’s major site incident with OpenResty XRay

* The incident * Process of resolution * Hardening after the incident * OpenResty XRay Services

Yichun Zhang

OPENRESTY XRAY Updated Jul 26, 2022 11mins read

Tracing the Slowest PCRE Regular Expressions in OpenResty or Nginx Processes

* System Environment * Narrowing Down the Culprit without Guessing * Limiting the execution overhead of PCRE * Non-Backtracking Regular Expression Engines * Lua's Builtin Patterns * Tracing Applications inside Containers * How The Tools are Implemented * The Overhead of the Tools

Yichun Zhang

OPENRESTY XRAY Updated Jul 26, 2022 9mins read

Listing Loaded Lua Modules in OpenResty or Nginx Processes

* System Environment * Names of Loaded Lua Modules * Running Directly in the Web Console * Tracing Applications inside Containers * How The Tools are Implemented * The Overhead of the Tools

Yichun Zhang

OPENRESTY XRAY Updated Jul 26, 2022 11mins read

Analyzing the Most CPU-Consuming Requests in OpenResty or Nginx

* System Environment * CPU-Hottest Request Hostnames * CPU-Hottest Request URIs * Digging Deeper * Running Directly in the Web Console * Tracing Applications inside Containers * How The Tools are Implemented * The Overhead of the Tools

Yichun Zhang

OPENRESTY XRAY Updated Jul 26, 2022 14mins read

Ylang: Universal Language for eBPF, Stap+, GDB, and More (Part 4)

* Transparent Container Tracing * Efficient Stack Unwinding * Analyzing Dead Processes (Core Dumps) * Very Low Tracing Overhead * Standard Ylang Libraries & Tools * Network Filtering & Control * The Ylang Compiler's Implementation * Operating System Support * Contributions to Open-Source Communities

Yichun Zhang

OPENRESTY XRAY Updated Jul 26, 2022 10mins read

Ylang: Universal Language for eBPF, Stap+, GDB, and More (Part 3)

* The Language Syntax (Continued) * String Literals * Built-in Regex Support * Full Control-Flow Support * Floating-Point Number Support * Compared to Open-Source Toolchains * Transparent Debug Symbol Handling * Debug Symbols: No Runtime Overhead * Centralized Package Database * Fuzzy Matching Debug Symbols

Yichun Zhang

OPENRESTY XRAY Updated Jul 26, 2022 14mins read

Ylang: Universal Language for eBPF, Stap+, GDB, and More (Part 2)

* The Language Syntax (Continued) * Macros with Extensions * Tracer and Tracee Spaces * Probes * Extended Variable Types

Yichun Zhang

OPENRESTY XRAY Updated Jul 26, 2022 14mins read

Ylang: Universal Language for eBPF, Stap+, GDB, and More (Part 1)

* What is Dynamic Tracing * Why the "Y" Name * Getting Started * Various Backends and Runtimes * Why a Unified Frontend Language * The Language Syntax

Yichun Zhang

OPENRESTY XRAY Updated Jul 26, 2022 19mins read

Introduction to Lua-Land CPU Flame Graphs

* What is a Flame Graph * Simple Lua samples * Complicated Lua applications * Sampling overhead * Safety * Compatibility * Other types of Lua-land Flame Graphs

Yichun Zhang

OPENRESTY XRAY Updated Jul 26, 2022 10mins read

Memory Fragmentation in OpenResty and Nginx's Shared Memory Zones

* An empty zone * Filling entries of similar sizes * Deleting odd-numbered keys * Deleting the keys in the first half * Mitigating Fragmentation

Yichun Zhang

OPENRESTY XRAY Updated Jul 26, 2022 14mins read

How OpenResty and Nginx Shared Memory Zones Consume RAM

* Slabs and pages * What is allocated is not what is paid for * Fake Memory Leaks * HUP reload

Yichun Zhang

OPENRESTY XRAY Updated Jul 25, 2022 8mins read

The Wonderland of Dynamic Tracing (Part 4 of 7)

Yichun Zhang

OPENRESTY XRAY Updated Jul 25, 2022 11mins read

The Wonderland of Dynamic Tracing (Part 3 of 7)

* Applications of SystemTap in Production * Case #1: Slow Debugging Code Left in Production * Case #2: Long request latency outliers * Case #3: From Network Issues to Disk Issues * Case #4: File Handle Cache Tradeoffs * Case #5: Compiled Regex Cache Tuning

Yichun Zhang

OPENRESTY XRAY Updated Jul 25, 2022 9mins read

The Wonderland of Dynamic Tracing (Part 2 of 7)

* DTrace * SystemTap * DTrace and SystemTap

Yichun Zhang

OPENRESTY XRAY Updated Jul 25, 2022 11mins read

The Wonderland of Dynamic Tracing (Part 1 of 7)

* Dynamic Tracing * What It Is * Too Good to be True? * How it Works * Still Having Doubts? * The Advantages of Dynamic Tracing

Yichun Zhang

OPENRESTY XRAY Updated Jul 26, 2022 11mins read

How OpenResty and Nginx Allocate and Manage Memory

* On The System Level * On The Application Level * For Traditional Nginx Servers

Yichun Zhang

OPENRESTY XRAY Updated Jul 25, 2022 16mins read

The LuaJIT GC64 Mode

* The Old Memory Limit * When Hitting The Memory Limit * The Memory Limit Is Per Process * GC-Managed Memory * Memory Allocated Outside GC * Extending The x64 Mode to The 4 GB Limit * The New GC64 Mode * How to Enable The GC64 Mode * Performance Impact * Debugging and Profiling Tool Chains

Yichun Zhang