Enabling PyPy as support for OpenStack Neutron

This is a guide to deploy OpenStack Neutron with PyPy support. The underlying purpose was to determine whether using PyPy instead of CPython works for the networking service, and whether we would notice any improvements and/or drawbacks. The performance results of this can be read in my post here.

However, the performance assessment will not be the scope of the current article, but rather a series of steps in order to reproduce our setup and have a fully functioning Neutron setup working over PyPy.

As a note, when we started this project, PyPy was at stable version 5.6.0 for Python 2.7 and OpenStack was at Ocata version. Aside from this, the setup was one physical machine with:

  • Processor: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
  • Memory: 32GB
  • 2 NICs: Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
  • OS: Ubuntu 16.04 LTS Xenial Xerus

Openstack comes in many flavors, including the option to put it together yourself by “assembling” it from the Git repos. However, even though this type of deployment is highly customizable, it has the major drawback that it is really hard to deploy / uninstall / redeploy, due to the manual steps you need to take every time. However, a “one-button-deploy” of Openstack was not desirable either, as you should also want some flexibility. Therefore, the solution settled on was Devstack, which is quite friendly documentation-wise and ideal for small, experimental setups.

Setup the Oslo libraries

Even though the Openstack services will be running with PyPy, the Apache server used by Keystone will not. By default, at the devstack deployment, the Apache server will use mod_wsgi, which is an Apache module use to host any Python web application. However, as of this date and, to the best of our knowledge, it is not yet compatible with PyPy. Moreover, by default, devstack will specifically look for the python2.7 when setting up mod_wsgi interpreter in /usr/bin and /usr/local/bin and will assume that the Oslo libraries have already been installed by the dependencies of the other modules.

Therefore, deploying with PyPy from the beginning will break the devstack process. The solutions is to pre-install these dependencies on the default Python 2.7 before changing to PyPy. The quickest way to do this is to clone the openstack the repositories, and install the requirements from each one of them.

 ~ $ sudo apt-get install python-dev python-pip
 ~ $ git clone https://github.com/openstack/nova.git
 ~ $ git clone https://github.com/openstack/neutron.git
 ~ $ git clone https://github.com/openstack/keystone.git
 ~ $ git clone https://github.com/openstack/glance.git
 ~ $ sudo pip install nova/requirements.txt
 ~ $ sudo pip install nova/test-requirements.txt
 ~ $ #repeat for every cloned module

As a side note, we are aware of the inconvenience of having 2 python interpreters using devstack related components in one system. There is a solution of replacing everything with PyPy, by using a another WSGI, called uWSGI, which has PyPy support for both Apache and Nginx. We will cover this is a later article, as for the moment, our end goal was to assess the performance of the Devstack services running with PyPy, while everything else was the same.

Setup PyPy as your default interpreter

Clone the desired PyPy version from the repository. In this case, it’s 5.6.0::

~ $ cd /opt
 ~ $ sudo su
 ~ # wget https://bitbucket.org/pypy/pypy/downloads/pypy2-v5.6.0-linux64.tar.bz2
 ~ # tar -xf pypy2-v5.6.0-linux64.tar.bz2
 ~ # cd pypy2-v5.6.0-linux64/bin
 ~ # ./pypy -m ensurepip
 ~ # ./pip install --upgrade pip
 ~ # ./pip install setuptools
 ~ # ./pip install --upgrade setuptools

At this point you need to specify pypy as the default python. It is preferable to do this in a script, because you do not want accidentally erase the /usr/bin/python2.7 (it takes a lot of time to fix):

~ # rm /usr/bin/python
 ~ # ln -sf /opt/pypy2-v5.6.0-linux64/bin/pypy /usr/bin/python

Prepend to PATH the path to the pypy bin folder. Example:

~ # vim /etc/environment
 #####FILE START /etc/environment######
 PATH="/opt/pypy/pypy2-v5.6.0-linux64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
 #####EOF
 ~ # source /etc/environment

Do a sanity check for your setup:

~ $ which python # this should /usr/bin/python, as the link was rewritten
 ~ $ python # this should yield some info about the Pypy version
 ~ $ which pip # this should be /opt/pypy......./pip, NOT /usr/bin/pip NOR /usr/local/bin/pip

Setup DevStack

Ideally, you would want to setup Devstack on a fresh install of Ubuntu, or, at least, be mindful of problems such webservers, used ports, etc.

The setup proposed is the minimum necessary for the Neutron service to work on a single machine. Therefore, you will also need Keystone (for authentication) and Nova (because some Neutron tasks require Nova to exist).

But first, you need to prepare your setup::

$ sudo apt-get update
 # Update the system
 ~ $ sudo apt-get upgrade
 # Install the required tools
 ~ $ sudo apt-get install -y git vim openssh-server openvswitch-switch ethtool
 # Edit the interfaces
 sudo vim /etc/network/interfaces

It is important here to understand the purpose of having 2 NICs with internet access, and how they actually work. You will use one of them for Neutron, specifically, to associate it with a bridge, in this case an OpenvSwitch Bridge. This bridge will be your external bridge, meaning the one that should link your cloud to the rest of the world. This external bridge is crucial to deploying Neutron, as all the other networks will connect to it and use it as a gateway. The reason you need a second NIC, is to keep it as your backup for internet access. When Neutron sets up, it will momentarily cut internet access to the NIC hosting the external bridge, while also performing some git updates, which causes Devstack to fail to deploy.

As a side note, it is possible to circumvent this issue by using another screen after the network outage. More here.

Therefore, your /etc/network/interfaces, you will become something like this:

# interfaces(5) file used by ifup(8) and ifdown(8)
 auto lo
 iface lo inet loopback

##This setup is network and PC specific, please use your own setup.
 ## However, keep the br-ex name as is because Neutron will specifically look for it
 ## External bridge
 auto br-ex
 iface br-ex inet static
 address 192.168.1.134 #my local ip, for the NIC I use for Neutron.
 netmask 255.255.255.0
 gateway 192.168.1.1
 dns-nameservers 8.8.8.8

## External network interface
 auto enp6s0 # the id of the NIC I used for Neutron; reconfigure for br-ex
 iface enp6s0 inet manual
 up ifconfig enp6s0 0.0.0.0 up
 up ip link set enp6s0 promisc on
 down ip link set enp6s0 promisc off
 down ifconfig enp6s0 down

Now create it from OpenvSwitch so that Neutron will find it at deployment.

~ $sudo ovs-vsctl add-br br-ex
~ $sudo ovs-vsctl add-port br-ex enp6s0

Now let’s do a sanity check, to see if all is ok.

 ~ $ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
 inet 127.0.0.1/8 scope host lo
 valid_lft forever preferred_lft forever
 inet6 ::1/128 scope host 
 valid_lft forever preferred_lft forever
 2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
 link/ether 0c:c4:7a:32:07:62 brd ff:ff:ff:ff:ff:ff
 inet 192.168.1.116/24 brd 192.168.1.255 scope global dynamic enp5s0
 valid_lft 50261sec preferred_lft 50261sec
 inet6 fe80::81bd:b3c0:3c73:74c8/64 scope link 
 valid_lft forever preferred_lft forever
 3: enp6s0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
 link/ether 0c:c4:7a:32:07:63 brd ff:ff:ff:ff:ff:ff
 inet6 fe80::ec4:7aff:fe32:763/64 scope link 
 valid_lft forever preferred_lft forever
 4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
 inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
 valid_lft forever preferred_lft forever
 5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
 link/ether 52:54:00:7c:c5:26 brd ff:ff:ff:ff:ff:ff
 6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1
 link/ether e6:f5:ef:f5:bf:60 brd ff:ff:ff:ff:ff:ff
 7: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1
 link/ether 0c:c4:7a:32:07:63 brd ff:ff:ff:ff:ff:ff
 inet 192.168.1.134/24 brd 192.168.1.255 scope global br-ex
 valid_lft forever preferred_lft forever
 inet6 2001:db8::2/64 scope global 
 valid_lft forever preferred_lft forever
 inet6 fe80::8459:23ff:fe87:484a/64 scope link 
 valid_lft forever preferred_lft forever
~ $sudo ovs-vsctl show
 fb49781d-903e-4889-ad72-cd506af65bcb 
 Bridge br-ex
 Port "enp6s0"
 Interface "enp6s0"
 Port br-ex
 Interface br-ex
 type: internal

Now, ideally, you would also want to add a dedicate user for DevStack, in order not to mix things up. You will only do this once, for your first deployment.

 ~ $ sudo adduser stack
 ~ $ sudo usermod -a -G sudo stack
 ~ $ sudo mkdir /opt/stack
 ~ $ sudo chown -R stack:stack /opt/stack
 ~ $ su stack

Great! Now you are logged in as stack and you are prepared to deploy.
Unfortunately, Devstack does not deploy with PyPy out of the box, because of tertiary dependencies that are not resolved yet, so we have to eliminate them manually (particularly, the postgres python libs are not compatible with PyPy). Therefore, you need to clone the Openstack repositories yourself and manually edit their requirements.txt and test-requirements.txt.

 ~ $ cd /opt/stack
 ~ $ git clone https://github.com/openstack/nova.git
 ~ $ git clone https://github.com/openstack/neutron.git
 ~ $ git clone https://github.com/openstack/keystone.git
 ~ $ git clone https://github.com/openstack/glance.git
 ~ $ cd /opt/glance
 ~ $ vim requirements.txt #comment out every psycopg and postgres dependency
 ~ $ vim test-requirements.txt #comment out every psycopg and postgres dependency

Go back to your home folder (user stack’s home folder) and clone devstack. Afterwards, you will need to prepare the local configuration file.

 ~ $ git clone https://github.com/openstack-dev/devstack.git
 ~ $ cd devstack
 ~ $ vim local.conf

The settings in the local.conf need to mirror the setup we prepared up to now. Particularly, we will need to disable some services, specify which bridge to be used for external communication and the restrictions on the network the bridge is attached to.

[[local|localrc]]

HOST_IP=192.168.1.134
 SERVICE_HOST=192.168.1.134
 MYSQL_HOST=192.168.1.134
 RABBIT_HOST=192.168.1.134

DEVSTACK_PASSWORD=password
 # Change the following passwords
 DATABASE_PASSWORD=$DEVSTACK_PASSWORD
 RABBIT_PASSWORD=$DEVSTACK_PASSWORD
 SERVICE_TOKEN=$DEVSTACK_PASSWORD
 SERVICE_PASSWORD=$DEVSTACK_PASSWORD
 ADMIN_PASSWORD=$DEVSTACK_PASSWORD


 Q_USE_SECGROUP=True
 FLOATING_RANGE="192.168.1.0/24"
 IPV4_ADDRS_SAFE_TO_USE="10.0.0.0/22"
 Q_FLOATING_ALLOCATION_POOL=start=192.168.1.200,end=192.168.1.254 #The sample setup had other machines in the network and we did not want interference 
 PUBLIC_NETWORK_GATEWAY="192.168.1.1"
 PUBLIC_INTERFACE=enp6s0

Q_USE_PROVIDERNET_FOR_PUBLIC=True
 OVS_PHYSICAL_BRIDGE=br-ex
 PUBLIC_BRIDGE=br-ex
 OVS_BRIDGE_MAPPINGS=public:br-ex

LIBVIRT_TYPE=kvm
 API_RATE_LIMIT=False

SCREEN_LOGDIR=/opt/stack/logs/screen
 VERBOSE=True
 LOG_COLOR=False


 #Services to be started

# Requirements
 enable_service rabbit
 enable_service mysql
 disable_service postgresql

# Keystone
 enable_service key

# Neutron
 enable_service neutron
 enable_service q-svc
 enable_service q-agt
 enable_service q-dhcp
 enable_service q-l3
 enable_service q-meta
 enable_service q-lbaas
 enable_service q-fwaas
 enable_service q-metering
 enable_service q-vpn

# Horizon
 disable_service horizon

# Glance
 disable_service glance
 disable_service g-api 
 disable_service g-reg

# Cinder
 disable_service cinder
 disable_service c-api
 disable_service c-vol
 disable_service c-sch
 disable_service c-bak

#Nova
 enable_service n-api
 enable_service n-crt
 enable_service n-obj
 enable_service n-cond
 enable_service n-sch
 enable_service n-cauth
 enable_service n-novnc
 enable_service n-cpu

#Tempest
 disable_service tempest


 disable_service n-net
 disable_service s-proxy
 disable_service s-object
 disable_service s-container
 disable_service s-account
 disable_service heat
 disable_service h-api
 disable_service h-api-cfn
 disable_service h-api-cw
 disable_service h-eng
 disable_service ceilometer-acompute
 disable_service ceilometer-acentral
 disable_service ceilometer-collector
 disable_service ceilometer-api

Now call the deployment script and wait. It takes roughly 10 minutes.

./stack.sh

As a side note, in the case you encounter any problems, you should check the error that was thrown, and also check the openstack logs (in our case /opt/stack/logs; the key.log is usually telling) before you redeploy. After that, you should:

./unstack.sh
 ./clean.sh

Also, be mindful here, before you run ./stack.sh again, that the ovs-bridge created for the previous deployment was deleted by ./unstack.sh process, therefore you need to create it again.

~ $sudo ifdown enp6s0
 ~ $sudo ifup enp6s0
 ~ $sudo ovs-vsctl add-br br-ex
 ~ $sudo ovs-vsctl add-port br-ex enp6s0

By Mihai Dodan, mihai.dodan [at] rinftech [dot] com

Advertisements

5 thoughts on “Enabling PyPy as support for OpenStack Neutron

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s