Django Memory Profiler: Analyze Object-Level Memory Usage in a Live App

To profile a Django app’s memory usage, attach OpenResty XRay to the running, unmodified Python process — no code changes — and it renders a Python GC object memory distribution flame graph that shows exactly where the memory goes, down to individual objects like a module’s cache dictionary. This tutorial walks through that analysis on an unmodified 86 MB Django process, step by step. You can examine any Python object in detail, such as a string or a dictionary, via the detailed data reference paths OpenResty XRay infers automatically.

Spot a high-memory Django process with ps

Run ps aux | grep python3 to list the Python processes. One python3 manage.py runserver process stands out, using about 86 MB of resident memory (RSS) — and it is the standard /bin/python3 binary shipped with the Linux distribution, running unmodified.

Profile the Django app’s memory usage with Guided Analysis

Open the OpenResty XRay web console, make sure the right machine is selected, and go to the Guided Analysis page, where you can pick from the types of problems OpenResty XRay can diagnose.

Select High memory usage, choose the Python Django application and the child process that ps flagged (the one using 84 MB of RSS), keep both Python and C/C++ language levels, and start with the default 300-second limit. After a round or two, OpenResty XRay automatically produces a report. Under Python-Land, it shows the #1 hottest data reference path for GC object memory distribution: a dict holding 38.27 MB, reached through the interpreter’s .modules.

This data reference path shows that the the Python modules loaded in the interpreter use 38MB of memory.

Click “More” to see details.

The most significant data reference path was automatically inferred from this Python GC object memory distribution flame graph.

Below are more explanations and suggestions about the issue.

It mentions .modules.

And mentions that it contains everything related to Python modules.

Look at the flame graph again, zoom in and find out which modules are taking up the most memory.

We can see these 1521 Python modules are combined together. Each of these modules takes less than 1% of the total memory.

This module takes up a lot of memory. To see more details, click on it to zoom in.

You can see that the openpyxl.utils.cell module takes up more than 2.6MB of memory. openpyxl is a Python library for manipulating Excel files.

Copy the module name.

Now open the terminal and use the find command to find the corresponding source code file in this project directory. The project folder is the root directory of the Python module installation location.

Paste the module name we just copied.

Here we abuse the dots in the module name as a wildcard for the grep command.

Copy the full file path.

Use the vim editor to open the source file. And look at the underlying Python code in this file. You can use any editors you like.

Return to the flame graph.

The openpyxl module uses two dictionaries, _COL_STRING_CACHE and _STRING_COL_CACHE. They are used for converting between Excel cell coordinates and column names. These two variables use a lot of memory.

Copy the name of the first variable.

Search for _COL_STRING_CACHE and you can locate the object. You can find all places in this file where this variable is used.

The _STRING_COL_CACHE variable can also be found using the same way.

You can use the same method to analyze other modules on the flame graph.

Such as the linecache module. This is a Python standard module that caches file contents. It uses 648KB of memory.

It has a cache variable that uses a lot of memory inside.

Python uses the Libc allocator and the mmap syscall to allocate memory. The report shows that the Libc allocator uses 18.7MB of memory.

Monitor Django memory usage automatically with Insights

OpenResty XRay can also monitor online processes automatically and show analysis reports. On the Insights page you can find daily and weekly reports — the same memory reference paths and distribution, without running the Guided Analysis manually. Guided Analysis remains useful for application development and demonstration purposes.

Frequently Asked Questions

How do you profile memory usage in a Django app without changing the code?

Attach OpenResty XRay to the already-running, unmodified Python process and start a Guided Analysis. It reads the process in real time, so you don’t edit, instrument, or redeploy the Django app. The result is a report and flame graph showing where the memory goes, down to individual Python objects.

Which Python module uses the most memory in a Django app?

Zoom into the GC object memory distribution flame graph to rank modules by size. In this case the openpyxl.utils.cell module holds more than 2.6 MB in its _COL_STRING_CACHE and _STRING_COL_CACHE dictionaries, and the standard linecache module keeps 648 KB in its file-content cache.

What is a Python GC object memory distribution flame graph?

It is a flame graph, invented by OpenResty, that lays out every Python garbage-collected object by how much memory it holds and along which data reference path. OpenResty XRay generates and interprets it automatically, so you can trace a string or dictionary back to the module that retains it.

Why does a small Django process use so much memory?

Much of it is the interpreter itself: the .modules registry of loaded Python modules accounts for 38 MB (the 1521 module objects alone sum to 13.7 MB), and individual module-level caches — such as openpyxl’s cache dictionaries — add more on top. The flame graph attributes each megabyte to a specific object so you can see what is worth trimming.

What is OpenResty XRay

OpenResty XRay is a dynamic-tracing product that automatically analyzes your running applications to troubleshoot performance problems, behavioral issues, and security vulnerabilities with actionable suggestions. Under the hood, OpenResty XRay is powered by our Y language targeting various runtimes like Stap+, eBPF+, GDB, and ODB, depending on the contexts.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty^® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty^®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.