# Hunting Performance in Python Code – Part 2. Measuring Memory Consumption

In this post I will talk about some tools that can help us solve a painful problem in Python, especially when using PyPy: memory consumption.

Why are we concerned with this in the first place? Why don’t we care only about performance? The answer to these questions is rather complex, but I’ll summarize it.

PyPy is an alternative Python interpreter, that features some great advantages over CPython: speed (through it’s Just in Time compiler), compatibility (it is almost a drop in replacement of CPython) and concurrency (using stackless and greenlets).

One downside of PyPy is that in general it uses more memory than CPython, due to it’s JIT  and garbage collector implementation. Nevertheless, in some cases, it is able to use less memory than CPython.

We will see next how you can measure the amount of memory used by your application.

# Series index

The links below will go live once the posts are released:

# Diagnosing Memory Usage

## memory_profiler

One library that you can use to measure the amount of memory used by the interpreter to run a workload is called `memory_profiler`. It is available through pip:

`pip install memory_profiler`

also install the `psutil` dependency:

`pip install psutil`

The advantage of this tool is that it shows the memory consumption line by line in a Python script. This helps us find the places from the script that can be we rewritten. But this analysis comes with a downside. Your code will run 10 to 20 times slower than the usual script.

How to use it? You just add a directive `@profile()` to the function you need to measure.

Let’s see it in action! We will use as a model the primes script as in our previous post, a bit modified to eliminate the statistics part. It is available on GitHub here.

 from memory_profiler import profile @profile(precision=6) def primes(n): if n == 2: return  elif n < 2: return [] s = range(3, n + 1, 2) mroot = n ** 0.5 half = (n + 1) / 2 – 1 i = 0 m = 3 while m <= mroot: if s[i]: j = (m * m – 3) / 2 s[j] = 0 while j < half: s[j] = 0 j += m i = i + 1 m = 2 * i + 3 return  + [x for x in s if x] len(primes(100000))

view raw
02.primes-v1.py
hosted with ❤ by GitHub

To start measuring, use the following command for PyPy:

`pypy -m memory_profiler 02.primes-v3.py`

or shorter, by importing `memory_profiler`  directly in the script:

`pypy -m memory_profiler 02.primes-v3.py`

After the execution of this line, we will see something like this for PyPy

```Line #    Mem usage    Increment   Line Contents
================================================
54  35.312500 MiB   0.000000 MiB   @profile(precision=6)
55                             def primes(n):
56  35.351562 MiB   0.039062 MiB       if n == 2:
57                                     return 
58  35.355469 MiB   0.003906 MiB       elif n < 2:
59                                     return []
60  35.355469 MiB   0.000000 MiB       s = []
61  59.515625 MiB  24.160156 MiB       for i in range(3, n+1):
62  59.515625 MiB   0.000000 MiB           if i % 2 != 0:
63  59.515625 MiB   0.000000 MiB               s.append(i)
64  59.546875 MiB   0.031250 MiB       mroot = n ** 0.5
65  59.550781 MiB   0.003906 MiB       half = (n + 1) / 2 - 1
66  59.550781 MiB   0.000000 MiB       i = 0
67  59.550781 MiB   0.000000 MiB       m = 3
68  59.554688 MiB   0.003906 MiB       while m <= mroot:
69  59.554688 MiB   0.000000 MiB           if s[i]:
70  59.554688 MiB   0.000000 MiB               j = (m * m - 3) / 2
71  59.554688 MiB   0.000000 MiB               s[j] = 0
72  59.554688 MiB   0.000000 MiB               while j < half:
73  59.554688 MiB   0.000000 MiB                   s[j] = 0
74  59.554688 MiB   0.000000 MiB                   j += m
75  59.554688 MiB   0.000000 MiB           i = i + 1
76  59.554688 MiB   0.000000 MiB           m = 2 * i + 3
77  59.554688 MiB   0.000000 MiB       l = 
78  59.679688 MiB   0.125000 MiB       for x in s:
79  59.679688 MiB   0.000000 MiB           if x:
80  59.679688 MiB   0.000000 MiB               l.append(x)
81  59.683594 MiB   0.003906 MiB       return l```

We can see that this script uses 24.371094 MiB of RAM. Let’s analyze this a bit. We see that most of it is used when building the number array. It excludes the even numbers and save all the other.

We can improve this a little bit by using the `range` call, with an increment parameter.  In this case, the script will look like this:

 from memory_profiler import profile @profile(precision=6) def primes(n): if n == 2: return  elif n < 2: return [] s = range(3, n + 1, 2) mroot = n ** 0.5 half = (n + 1) / 2 – 1 i = 0 m = 3 while m <= mroot: if s[i]: j = (m * m – 3) / 2 s[j] = 0 while j < half: s[j] = 0 j += m i = i + 1 m = 2 * i + 3 l =  for x in s: if x: l.append(x) return l len(primes(100000))

view raw
02.primes-v2.py
hosted with ❤ by GitHub

If we measure it again, we see the following:

```Line #    Mem usage    Increment   Line Contents
================================================
27  35.343750 MiB   0.000000 MiB   @profile(precision=6)
28                             def primes(n):
29  35.382812 MiB   0.039062 MiB       if n == 2:
30                                     return 
31  35.382812 MiB   0.000000 MiB       elif n < 2:
32                                     return []
33  35.386719 MiB   0.003906 MiB       s = range(3, n + 1, 2)
34  35.417969 MiB   0.031250 MiB       mroot = n ** 0.5
35  35.417969 MiB   0.000000 MiB       half = (n + 1) / 2 - 1
36  35.417969 MiB   0.000000 MiB       i = 0
37  35.421875 MiB   0.003906 MiB       m = 3
38  58.019531 MiB  22.597656 MiB       while m <= mroot:
39  58.019531 MiB   0.000000 MiB           if s[i]:
40  58.019531 MiB   0.000000 MiB               j = (m * m - 3) / 2
41  58.019531 MiB   0.000000 MiB               s[j] = 0
42  58.019531 MiB   0.000000 MiB               while j < half:
43  58.019531 MiB   0.000000 MiB                   s[j] = 0
44  58.019531 MiB   0.000000 MiB                   j += m
45  58.019531 MiB   0.000000 MiB           i = i + 1
46  58.019531 MiB   0.000000 MiB           m = 2 * i + 3
47  58.019531 MiB   0.000000 MiB       l = 
48  58.089844 MiB   0.070312 MiB       for x in s:
49  58.089844 MiB   0.000000 MiB           if x:
50  58.089844 MiB   0.000000 MiB               l.append(x)
51  58.093750 MiB   0.003906 MiB       return l```

Great, now our memory consumption dropped to 22.75 MiB. It can also be improved a bit by using list comprehension.

 from memory_profiler import profile @profile(precision=6) def primes(n): if n == 2: return  elif n < 2: return [] s = range(3, n + 1, 2) mroot = n ** 0.5 half = (n + 1) / 2 – 1 i = 0 m = 3 while m <= mroot: if s[i]: j = (m * m – 3) / 2 s[j] = 0 while j < half: s[j] = 0 j += m i = i + 1 m = 2 * i + 3 return  + [x for x in s if x] len(primes(100000))

view raw
02.primes-v1.py
hosted with ❤ by GitHub

Measured again:

```Line #    Mem usage    Increment   Line Contents
================================================
4  35.425781 MiB   0.000000 MiB   @profile(precision=6)
5                             def primes(n):
6  35.464844 MiB   0.039062 MiB       if n == 2:
7                                     return 
8  35.464844 MiB   0.000000 MiB       elif n < 2:
9                                     return []
10  35.464844 MiB   0.000000 MiB       s = range(3, n + 1, 2)
11  35.500000 MiB   0.035156 MiB       mroot = n ** 0.5
12  35.500000 MiB   0.000000 MiB       half = (n + 1) / 2 - 1
13  35.500000 MiB   0.000000 MiB       i = 0
14  35.500000 MiB   0.000000 MiB       m = 3
15  57.683594 MiB  22.183594 MiB       while m <= mroot:
16  57.683594 MiB   0.000000 MiB           if s[i]:
17  57.683594 MiB   0.000000 MiB               j = (m * m - 3) / 2
18  57.683594 MiB   0.000000 MiB               s[j] = 0
19  57.683594 MiB   0.000000 MiB               while j < half:
20  57.683594 MiB   0.000000 MiB                   s[j] = 0
21  57.683594 MiB   0.000000 MiB                   j += m
22  57.683594 MiB   0.000000 MiB           i = i + 1
23  57.683594 MiB   0.000000 MiB           m = 2 * i + 3
24  57.847656 MiB   0.164062 MiB       return  + [x for x in s if x]```

Our final version of the script consumes only 22.421875 MiB. That’s almost a 10% reduction compared to the first version.

By Alecsandru Patrascu, alecsandru.patrascu [at] rinftech [dot] com

## 2 thoughts on “Hunting Performance in Python Code – Part 2. Measuring Memory Consumption”

1. […] Part II: WordPress […]

Like