PyPy, compared to CPython relies more on achieving speed-up by “jitting” code as often as possible, rather than rely on its interpreter. However, jitting is not always an option, or at least not entirely. A good improvement for CPython, that we think might benefit PyPy as well, without impacting the JIT performance is Profile Guided Optimization (PGO or profopt).
I thank the PyPy developer community for their patience, kind advice and constant feedback they gave me in #pypy IRC or through email, which helped me to make this possible, especially to Carl Friedrich Bolz-Tereick and Armin Rigo.
Profile-guided optimizations can differ in implementation from compiler to compiler, but all of them, basically do the 3 steps:
- First, the source code is instrumented during compilation. It is worth noting that not all associated libraries need to be instrumented, but for best performance it is advisable to do so.
- Secondly, the binary that results from the first step will have some associated “profiles”, that will be updated every time you run it (this behavior for gcc, at least). This step is also known as the training phase. It is crucial here that you run workloads that are the most relevant for the code paths you usually expect / want to be taken. It is also important to remember that altering the code past this point would cause inconsistencies in your profiles, and therefore poorer performance.
- The third phase is cleaning everything but the profiles, and recompile everything based on the training they had. This will unvectorize small loops, ease inlining, improve branch prediction, improve hotspots etc.
So, how does all this benefit Python or PyPy? Well, since Python is an interpreted language, written in C, it is obvious that training its binary with common scenarios will benefit it greatly. It is worth mentioning though, that the training should not be everything that CPython can be used for. Simply put, if everything is a hot spot then nothing is. On the contrary, it would only make performance worse.
How about PyPy?
PyPy is a different, because it already has the benefit of a JIT, so it does not rely on the interpreter as much as Python does. The underlying issue here is that the Assembly code generated by the JIT has no way of benefiting from PGO, because it is was never instrumented. However, the interpreted code itself is roughly 3 times slower that CPython, mostly due to the instrumentation of the code it needs to do to start the JIT.
The target here was then to improve the interpreter of PyPy by compiling it with PGO, while also, avoid delaying the JIT, its main source of performance.
Ideally, we hoped for a speed-up similar to CPython, of about 10% on average, but we were aware that imperfect training would always introduce delays.
2. PGO for PyPy
Since PyPy is written in RPython, enabling the Profile-Guided optimizations for it is a bit more tricky than for CPython. This is mainly due to the fact that PyPy need to be translated and have it sources generated before they can be compiled.
However, to enable PGO we only needed the Makefile generated after the translation, so simply saved the sources and worked on them, in order to save time.
Therefore we altered the Makefile to create a new target for our profile optimized binary, by adding to the usual GCC command, the
--fprofile-generate flag for the first phase at both compilation and link phase, than train the resulting binary. Afterwards, we used the profiles to rebuild the project once again, by using:
--fprofile-use --fprofile-correction, again for both compilation and linking.
As it is quite hard, and also subjective, to determine a good training set for interpreters in general, we decided to start with the training tests that CPython uses and see whether this offers any performance improvement, and to have a baseline to compare any subsequent testing sets.
While we have more to explain / debate about how would profopt best work for PyPy, such as tips and tricks, different workloads (we also tried the actual benchmark as a training set 🙂 ), or what are the proper use cases for it, we will focus on the actual performance gains and how measured this.
An advantage of the implementation from PyPy, is that it’s not for the Python interpreter only, but it can be used for any existing or future implementation that are based on RPython!
Stay tuned for a more general implementation of PGO for both GCC and CLANG.
So here are the steps you should take to enable PGO for PyPy. As a sidenote, I need to mention that the tests have been performed on Ubuntu 16.04, with gcc 6.2.0. It is also important to mention that if you have a Darwin/Mac (I am working to enable this for CLANG on Mac) or Windows, you will most likely not be able to use PGO.
Clone the PyPy repo:
hg clone http://bitbucket.org/pypy/pypy pypy
apt-get install gcc make libffi-dev pkg-config libz-dev libbz2-dev \ libsqlite3-dev libncurses-dev libexpat1-dev libssl-dev libgdbm-dev \ tk-dev libgc-dev python-cffi \ liblzma-dev # For lzma on PyPy3.
Go to the clone and run:
cd pypy/goal # If you want to enable profopt for PyPy without too much hassle: python ../../rpython/bin/rpython --opt=jit --cc=/opt/gcc-6.2.0/gcc --profopt
Or, if you want to specify the training script for PGO yourself, you can specify the
profoptargs argument, which will take the absolute path to your script and the arguments it requires as well. For example, running the exact same script as in the previous case, but make the script use more cores:
cd pypy/goal python ../../rpython/bin/rpython --opt=jit --cc=/opt/gcc-6.2.0/gcc \ --profopt --profoptargs="/home/md/pypy/lib-python/2.7/test/regrtest.py --pgo -j 18 -x test_asyncore test_gdb test_multiprocessing test_subprocess || true" # By default the script above runs on 1 process, # while now the script will run on 18 processes. # I would advise to use the number of the cores you have for this. # Side Note: the "|| true" at the end ensures that the training finishes # successfully, as some of the regrtests fail for PyPy (12 out of 400)
Now, the translation process with PGO takes ~ 1h and 15 min, so grab a cup of coffee, enjoy the mandelbrot, etc. When it ends you should have your
pypy-c binary and the associated
libpypy-c.so in the goal directory, trained and ready for use.
On the other hand, you can apply the same concept to any binary that results from an RPythton translation. However, in their case there is, obviously, no default, so both the script and its arguments are required. Consider this example:
cd rpython/translator/goal ../../bin/rpython --profopt --profoptargs=1000 targetrpystonedalone.py
There is no actual training script for the resulting binary, as
rpystone is not an interpreter, but rather a binary that takes an integer value as a parameter. To get a clear picture of this, any training is conceptually run as follows:
Therefore, in the case of PyPy, since it is a Python interpreter:
./pypy-c /path/to/training/script.py arguments_of_the script
Measuring performance gains is not easy for several reasons (situations we have actually encountered):
- There is no standard benchmark. Or, it is inherently unfair to your setup.
- While there is a standard test suite for Python, called pyperformance, we have found that results can have differences that are deemed significant even from one run to another. Obviously, multiple runs of the same workload, with the same binary should not have relevant differences. It is also quite unfair for PyPy, because there is no warmup time, which usually takes a small amount of time, but makes a world of difference in the results. This problem brings us to the next point.
- Not all the tests in a benchmark are as reliable as you would like
- After pyperformance, we have decided to try a similar benchmark, implemented in the PyPy project (https://bitbucket.org/pypy/benchmarks/src) that is similar to pyperformance, except it actually does have a warmup phase so that the JIT can fully get in effect before the measurement starts. Even so, however, not all the tests have a low jitter in run-to-run variation. For example, in this benchmark, the translation test is a good example of a test that varies wildly because of its unusually long times, and due to the fact that it is not repeated. This is problem because in these certain cases, you might be unsure at what is your actual speedup or if you have any. The best way to make sure that you actually have a speedup, if you encounter such a scenario, is to repeat the tests, and calculate, statistically the coeficient of variance and the standard deviation.
- Most of your tests show a good speedup, but one of them is a lot slower.
- Such a scenario happened after we processed the results. We realized that most tests showed an average improvement of ~6% (which was quite good for a interpreter binary), but there was one of them (nbody simulations) which had almost 50% slowdown. At this point, it is nice if you have a strategy for such cases. In our case, we talked with the PyPy devs and they we satisfied with the results. However, their proposal was to leave profopt as an option to be enabled rather than it being the default setting. This made sense for both us and them, because whomever might use the interpreter should be able to be fully aware of the eventual shortcomings of enabling profile-guided optimizations: it works better in most cases, by an average of 6%, but if you do nbody simulations, you will be disappointed.
An excerpt of our results is in the table below. You can also see it in more detail, here (https://docs.google.com/spreadsheets/d/1aEUkgUcEXGSieBnn82_vVzORk9fRfdW2UKJlE2jFZCk/edit?usp=sharing).
|Benchmark||SPEEDUP vs 5.8.0 (%)||SPEEDUP vs 5.7.1(%)|
By Mihai Dodan. E-mail: mihai [dot] dodan [at] rinftech.com