Hunting Performance in Python Code – Part 2. Measuring Memory Consumption

In this post I will talk about some tools that can help us solve a painful problem in Python, especially when using PyPy: memory consumption.

Why are we concerned with this in the first place? Why don’t we care only about performance? The answer to these questions is rather complex, but I’ll summarize it.

PyPy is an alternative Python interpreter, that features some great advantages over CPython: speed (through it’s Just in Time compiler), compatibility (it is almost a drop in replacement of CPython) and concurrency (using stackless and greenlets).

One downside of PyPy is that in general it uses more memory than CPython, due to it’s JIT  and garbage collector implementation. Nevertheless, in some cases, it is able to use less memory than CPython.

We will see next how you can measure the amount of memory used by your application.

Series index

The links below will go live once the posts are released:

  1. Setup
  2. Memory Profiling
  3. CPU Profiling – Python Scripts
  4. CPU Profiling – Python Interpreter

Diagnosing Memory Usage

memory_profiler

One library that you can use to measure the amount of memory used by the interpreter to run a workload is called memory_profiler. It is available through pip:

pip install memory_profiler

also install the psutil dependency:

pip install psutil

The advantage of this tool is that it shows the memory consumption line by line in a Python script. This helps us find the places from the script that can be we rewritten. But this analysis comes with a downside. Your code will run 10 to 20 times slower than the usual script.

How to use it? You just add a directive @profile() to the function you need to measure.

Let’s see it in action! We will use as a model the primes script as in our previous post, a bit modified to eliminate the statistics part. It is available on GitHub here.

To start measuring, use the following command for PyPy:

pypy -m memory_profiler 02.primes-v3.py

or shorter, by importing memory_profiler  directly in the script:

pypy -m memory_profiler 02.primes-v3.py

After the execution of this line, we will see something like this for PyPy

Line #    Mem usage    Increment   Line Contents
================================================
    54  35.312500 MiB   0.000000 MiB   @profile(precision=6)
    55                             def primes(n):
    56  35.351562 MiB   0.039062 MiB       if n == 2:
    57                                     return [2]
    58  35.355469 MiB   0.003906 MiB       elif n < 2:
    59                                     return []
    60  35.355469 MiB   0.000000 MiB       s = []
    61  59.515625 MiB  24.160156 MiB       for i in range(3, n+1):
    62  59.515625 MiB   0.000000 MiB           if i % 2 != 0:
    63  59.515625 MiB   0.000000 MiB               s.append(i)
    64  59.546875 MiB   0.031250 MiB       mroot = n ** 0.5
    65  59.550781 MiB   0.003906 MiB       half = (n + 1) / 2 - 1
    66  59.550781 MiB   0.000000 MiB       i = 0
    67  59.550781 MiB   0.000000 MiB       m = 3
    68  59.554688 MiB   0.003906 MiB       while m <= mroot:
    69  59.554688 MiB   0.000000 MiB           if s[i]:
    70  59.554688 MiB   0.000000 MiB               j = (m * m - 3) / 2
    71  59.554688 MiB   0.000000 MiB               s[j] = 0
    72  59.554688 MiB   0.000000 MiB               while j < half:
    73  59.554688 MiB   0.000000 MiB                   s[j] = 0
    74  59.554688 MiB   0.000000 MiB                   j += m
    75  59.554688 MiB   0.000000 MiB           i = i + 1
    76  59.554688 MiB   0.000000 MiB           m = 2 * i + 3
    77  59.554688 MiB   0.000000 MiB       l = [2]
    78  59.679688 MiB   0.125000 MiB       for x in s:
    79  59.679688 MiB   0.000000 MiB           if x:
    80  59.679688 MiB   0.000000 MiB               l.append(x)
    81  59.683594 MiB   0.003906 MiB       return l

 We can see that this script uses 24.371094 MiB of RAM. Let’s analyze this a bit. We see that most of it is used when building the number array. It excludes the even numbers and save all the other.

We can improve this a little bit by using the range call, with an increment parameter.  In this case, the script will look like this:

If we measure it again, we see the following:

Line #    Mem usage    Increment   Line Contents
================================================
    27  35.343750 MiB   0.000000 MiB   @profile(precision=6)
    28                             def primes(n):
    29  35.382812 MiB   0.039062 MiB       if n == 2:
    30                                     return [2]
    31  35.382812 MiB   0.000000 MiB       elif n < 2:
    32                                     return []
    33  35.386719 MiB   0.003906 MiB       s = range(3, n + 1, 2)
    34  35.417969 MiB   0.031250 MiB       mroot = n ** 0.5
    35  35.417969 MiB   0.000000 MiB       half = (n + 1) / 2 - 1
    36  35.417969 MiB   0.000000 MiB       i = 0
    37  35.421875 MiB   0.003906 MiB       m = 3
    38  58.019531 MiB  22.597656 MiB       while m <= mroot:
    39  58.019531 MiB   0.000000 MiB           if s[i]:
    40  58.019531 MiB   0.000000 MiB               j = (m * m - 3) / 2
    41  58.019531 MiB   0.000000 MiB               s[j] = 0
    42  58.019531 MiB   0.000000 MiB               while j < half:
    43  58.019531 MiB   0.000000 MiB                   s[j] = 0
    44  58.019531 MiB   0.000000 MiB                   j += m
    45  58.019531 MiB   0.000000 MiB           i = i + 1
    46  58.019531 MiB   0.000000 MiB           m = 2 * i + 3
    47  58.019531 MiB   0.000000 MiB       l = [2]
    48  58.089844 MiB   0.070312 MiB       for x in s:
    49  58.089844 MiB   0.000000 MiB           if x:
    50  58.089844 MiB   0.000000 MiB               l.append(x)
    51  58.093750 MiB   0.003906 MiB       return l

Great, now our memory consumption dropped to 22.75 MiB. It can also be improved a bit by using list comprehension.

Measured again:

Line #    Mem usage    Increment   Line Contents
================================================
     4  35.425781 MiB   0.000000 MiB   @profile(precision=6)
     5                             def primes(n):
     6  35.464844 MiB   0.039062 MiB       if n == 2:
     7                                     return [2]
     8  35.464844 MiB   0.000000 MiB       elif n < 2:
     9                                     return []
    10  35.464844 MiB   0.000000 MiB       s = range(3, n + 1, 2)
    11  35.500000 MiB   0.035156 MiB       mroot = n ** 0.5
    12  35.500000 MiB   0.000000 MiB       half = (n + 1) / 2 - 1
    13  35.500000 MiB   0.000000 MiB       i = 0
    14  35.500000 MiB   0.000000 MiB       m = 3
    15  57.683594 MiB  22.183594 MiB       while m <= mroot:
    16  57.683594 MiB   0.000000 MiB           if s[i]:
    17  57.683594 MiB   0.000000 MiB               j = (m * m - 3) / 2
    18  57.683594 MiB   0.000000 MiB               s[j] = 0
    19  57.683594 MiB   0.000000 MiB               while j < half:
    20  57.683594 MiB   0.000000 MiB                   s[j] = 0
    21  57.683594 MiB   0.000000 MiB                   j += m
    22  57.683594 MiB   0.000000 MiB           i = i + 1
    23  57.683594 MiB   0.000000 MiB           m = 2 * i + 3
    24  57.847656 MiB   0.164062 MiB       return [2] + [x for x in s if x]

Our final version of the script consumes only 22.421875 MiB. That’s almost a 10% reduction compared to the first version.


By Alecsandru Patrascu, alecsandru.patrascu [at] rinftech [dot] com

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s