In this post I will talk about some tools that can help us solve a painful problem in Python, especially when using PyPy: memory consumption.
Why are we concerned with this in the first place? Why don’t we care only about performance? The answer to these questions is rather complex, but I’ll summarize it.
PyPy is an alternative Python interpreter, that features some great advantages over CPython: speed (through it’s Just in Time compiler), compatibility (it is almost a drop in replacement of CPython) and concurrency (using stackless and greenlets).
One downside of PyPy is that in general it uses more memory than CPython, due to it’s JIT and garbage collector implementation. Nevertheless, in some cases, it is able to use less memory than CPython.
We will see next how you can measure the amount of memory used by your application.
Series index
The links below will go live once the posts are released:
Diagnosing Memory Usage
memory_profiler
One library that you can use to measure the amount of memory used by the interpreter to run a workload is called memory_profiler. It is available through pip:
pip install memory_profiler
also install the psutil dependency:
pip install psutil
The advantage of this tool is that it shows the memory consumption line by line in a Python script. This helps us find the places from the script that can be we rewritten. But this analysis comes with a downside. Your code will run 10 to 20 times slower than the usual script.
How to use it? You just add a directive @profile() to the function you need to measure.
Let’s see it in action! We will use as a model the primes script as in our previous post, a bit modified to eliminate the statistics part. It is available on GitHub here.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from memory_profiler import profile | |
| @profile(precision=6) | |
| def primes(n): | |
| if n == 2: | |
| return [2] | |
| elif n < 2: | |
| return [] | |
| s = range(3, n + 1, 2) | |
| mroot = n ** 0.5 | |
| half = (n + 1) / 2 – 1 | |
| i = 0 | |
| m = 3 | |
| while m <= mroot: | |
| if s[i]: | |
| j = (m * m – 3) / 2 | |
| s[j] = 0 | |
| while j < half: | |
| s[j] = 0 | |
| j += m | |
| i = i + 1 | |
| m = 2 * i + 3 | |
| return [2] + [x for x in s if x] | |
| len(primes(100000)) |
To start measuring, use the following command for PyPy:
pypy -m memory_profiler 02.primes-v3.py
or shorter, by importing memory_profiler directly in the script:
pypy -m memory_profiler 02.primes-v3.py
After the execution of this line, we will see something like this for PyPy
Line # Mem usage Increment Line Contents
================================================
54 35.312500 MiB 0.000000 MiB @profile(precision=6)
55 def primes(n):
56 35.351562 MiB 0.039062 MiB if n == 2:
57 return [2]
58 35.355469 MiB 0.003906 MiB elif n < 2:
59 return []
60 35.355469 MiB 0.000000 MiB s = []
61 59.515625 MiB 24.160156 MiB for i in range(3, n+1):
62 59.515625 MiB 0.000000 MiB if i % 2 != 0:
63 59.515625 MiB 0.000000 MiB s.append(i)
64 59.546875 MiB 0.031250 MiB mroot = n ** 0.5
65 59.550781 MiB 0.003906 MiB half = (n + 1) / 2 - 1
66 59.550781 MiB 0.000000 MiB i = 0
67 59.550781 MiB 0.000000 MiB m = 3
68 59.554688 MiB 0.003906 MiB while m <= mroot:
69 59.554688 MiB 0.000000 MiB if s[i]:
70 59.554688 MiB 0.000000 MiB j = (m * m - 3) / 2
71 59.554688 MiB 0.000000 MiB s[j] = 0
72 59.554688 MiB 0.000000 MiB while j < half:
73 59.554688 MiB 0.000000 MiB s[j] = 0
74 59.554688 MiB 0.000000 MiB j += m
75 59.554688 MiB 0.000000 MiB i = i + 1
76 59.554688 MiB 0.000000 MiB m = 2 * i + 3
77 59.554688 MiB 0.000000 MiB l = [2]
78 59.679688 MiB 0.125000 MiB for x in s:
79 59.679688 MiB 0.000000 MiB if x:
80 59.679688 MiB 0.000000 MiB l.append(x)
81 59.683594 MiB 0.003906 MiB return l
We can see that this script uses 24.371094 MiB of RAM. Let’s analyze this a bit. We see that most of it is used when building the number array. It excludes the even numbers and save all the other.
We can improve this a little bit by using the range call, with an increment parameter. In this case, the script will look like this:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from memory_profiler import profile | |
| @profile(precision=6) | |
| def primes(n): | |
| if n == 2: | |
| return [2] | |
| elif n < 2: | |
| return [] | |
| s = range(3, n + 1, 2) | |
| mroot = n ** 0.5 | |
| half = (n + 1) / 2 – 1 | |
| i = 0 | |
| m = 3 | |
| while m <= mroot: | |
| if s[i]: | |
| j = (m * m – 3) / 2 | |
| s[j] = 0 | |
| while j < half: | |
| s[j] = 0 | |
| j += m | |
| i = i + 1 | |
| m = 2 * i + 3 | |
| l = [2] | |
| for x in s: | |
| if x: | |
| l.append(x) | |
| return l | |
| len(primes(100000)) |
If we measure it again, we see the following:
Line # Mem usage Increment Line Contents
================================================
27 35.343750 MiB 0.000000 MiB @profile(precision=6)
28 def primes(n):
29 35.382812 MiB 0.039062 MiB if n == 2:
30 return [2]
31 35.382812 MiB 0.000000 MiB elif n < 2:
32 return []
33 35.386719 MiB 0.003906 MiB s = range(3, n + 1, 2)
34 35.417969 MiB 0.031250 MiB mroot = n ** 0.5
35 35.417969 MiB 0.000000 MiB half = (n + 1) / 2 - 1
36 35.417969 MiB 0.000000 MiB i = 0
37 35.421875 MiB 0.003906 MiB m = 3
38 58.019531 MiB 22.597656 MiB while m <= mroot:
39 58.019531 MiB 0.000000 MiB if s[i]:
40 58.019531 MiB 0.000000 MiB j = (m * m - 3) / 2
41 58.019531 MiB 0.000000 MiB s[j] = 0
42 58.019531 MiB 0.000000 MiB while j < half:
43 58.019531 MiB 0.000000 MiB s[j] = 0
44 58.019531 MiB 0.000000 MiB j += m
45 58.019531 MiB 0.000000 MiB i = i + 1
46 58.019531 MiB 0.000000 MiB m = 2 * i + 3
47 58.019531 MiB 0.000000 MiB l = [2]
48 58.089844 MiB 0.070312 MiB for x in s:
49 58.089844 MiB 0.000000 MiB if x:
50 58.089844 MiB 0.000000 MiB l.append(x)
51 58.093750 MiB 0.003906 MiB return l
Great, now our memory consumption dropped to 22.75 MiB. It can also be improved a bit by using list comprehension.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from memory_profiler import profile | |
| @profile(precision=6) | |
| def primes(n): | |
| if n == 2: | |
| return [2] | |
| elif n < 2: | |
| return [] | |
| s = range(3, n + 1, 2) | |
| mroot = n ** 0.5 | |
| half = (n + 1) / 2 – 1 | |
| i = 0 | |
| m = 3 | |
| while m <= mroot: | |
| if s[i]: | |
| j = (m * m – 3) / 2 | |
| s[j] = 0 | |
| while j < half: | |
| s[j] = 0 | |
| j += m | |
| i = i + 1 | |
| m = 2 * i + 3 | |
| return [2] + [x for x in s if x] | |
| len(primes(100000)) |
Measured again:
Line # Mem usage Increment Line Contents
================================================
4 35.425781 MiB 0.000000 MiB @profile(precision=6)
5 def primes(n):
6 35.464844 MiB 0.039062 MiB if n == 2:
7 return [2]
8 35.464844 MiB 0.000000 MiB elif n < 2:
9 return []
10 35.464844 MiB 0.000000 MiB s = range(3, n + 1, 2)
11 35.500000 MiB 0.035156 MiB mroot = n ** 0.5
12 35.500000 MiB 0.000000 MiB half = (n + 1) / 2 - 1
13 35.500000 MiB 0.000000 MiB i = 0
14 35.500000 MiB 0.000000 MiB m = 3
15 57.683594 MiB 22.183594 MiB while m <= mroot:
16 57.683594 MiB 0.000000 MiB if s[i]:
17 57.683594 MiB 0.000000 MiB j = (m * m - 3) / 2
18 57.683594 MiB 0.000000 MiB s[j] = 0
19 57.683594 MiB 0.000000 MiB while j < half:
20 57.683594 MiB 0.000000 MiB s[j] = 0
21 57.683594 MiB 0.000000 MiB j += m
22 57.683594 MiB 0.000000 MiB i = i + 1
23 57.683594 MiB 0.000000 MiB m = 2 * i + 3
24 57.847656 MiB 0.164062 MiB return [2] + [x for x in s if x]
Our final version of the script consumes only 22.421875 MiB. That’s almost a 10% reduction compared to the first version.
By Alecsandru Patrascu, alecsandru.patrascu [at] rinftech [dot] com
Python Files 
[…] Part II: WordPress […]
LikeLike
[…] Memory Profiling […]
LikeLike