This is a guide to deploy OpenStack Neutron with PyPy support. The underlying purpose was to determine whether using PyPy instead of CPython works for the networking service, and whether we would notice any improvements and/or drawbacks. The performance results of this can be read in my post here.
However, the performance assessment will not be the scope of the current article, but rather a series of steps in order to reproduce our setup and have a fully functioning Neutron setup working over PyPy.
As a note, when we started this project, PyPy was at stable version 5.6.0 for Python 2.7 and OpenStack was at Ocata version. Aside from this, the setup was one physical machine with:
- Processor: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
- Memory: 32GB
- 2 NICs: Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
- OS: Ubuntu 16.04 LTS Xenial Xerus
Openstack comes in many flavors, including the option to put it together yourself by “assembling” it from the Git repos. However, even though this type of deployment is highly customizable, it has the major drawback that it is really hard to deploy / uninstall / redeploy, due to the manual steps you need to take every time. However, a “one-button-deploy” of Openstack was not desirable either, as you should also want some flexibility. Therefore, the solution settled on was Devstack, which is quite friendly documentation-wise and ideal for small, experimental setups.
Setup the Oslo libraries
Even though the Openstack services will be running with PyPy, the Apache server used by Keystone will not. By default, at the devstack deployment, the Apache server will use mod_wsgi, which is an Apache module use to host any Python web application. However, as of this date and, to the best of our knowledge, it is not yet compatible with PyPy. Moreover, by default, devstack will specifically look for the python2.7 when setting up mod_wsgi interpreter in /usr/bin and /usr/local/bin and will assume that the Oslo libraries have already been installed by the dependencies of the other modules.
Therefore, deploying with PyPy from the beginning will break the devstack process. The solutions is to pre-install these dependencies on the default Python 2.7 before changing to PyPy. The quickest way to do this is to clone the openstack the repositories, and install the requirements from each one of them.
~ $ sudo apt-get install python-dev python-pip ~ $ git clone https://github.com/openstack/nova.git ~ $ git clone https://github.com/openstack/neutron.git ~ $ git clone https://github.com/openstack/keystone.git ~ $ git clone https://github.com/openstack/glance.git ~ $ sudo pip install nova/requirements.txt ~ $ sudo pip install nova/test-requirements.txt ~ $ #repeat for every cloned module
As a side note, we are aware of the inconvenience of having 2 python interpreters using devstack related components in one system. There is a solution of replacing everything with PyPy, by using a another WSGI, called uWSGI, which has PyPy support for both Apache and Nginx. We will cover this is a later article, as for the moment, our end goal was to assess the performance of the Devstack services running with PyPy, while everything else was the same.
Setup PyPy as your default interpreter
Clone the desired PyPy version from the repository. In this case, it’s 5.6.0::
~ $ cd /opt ~ $ sudo su ~ # wget https://bitbucket.org/pypy/pypy/downloads/pypy2-v5.6.0-linux64.tar.bz2 ~ # tar -xf pypy2-v5.6.0-linux64.tar.bz2 ~ # cd pypy2-v5.6.0-linux64/bin ~ # ./pypy -m ensurepip ~ # ./pip install --upgrade pip ~ # ./pip install setuptools ~ # ./pip install --upgrade setuptools
At this point you need to specify pypy as the default python. It is preferable to do this in a script, because you do not want accidentally erase the /usr/bin/python2.7 (it takes a lot of time to fix):
~ # rm /usr/bin/python ~ # ln -sf /opt/pypy2-v5.6.0-linux64/bin/pypy /usr/bin/python
Prepend to PATH the path to the pypy bin folder. Example:
~ # vim /etc/environment #####FILE START /etc/environment###### PATH="/opt/pypy/pypy2-v5.6.0-linux64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games" #####EOF ~ # source /etc/environment
Do a sanity check for your setup:
~ $ which python # this should /usr/bin/python, as the link was rewritten ~ $ python # this should yield some info about the Pypy version ~ $ which pip # this should be /opt/pypy......./pip, NOT /usr/bin/pip NOR /usr/local/bin/pip
Ideally, you would want to setup Devstack on a fresh install of Ubuntu, or, at least, be mindful of problems such webservers, used ports, etc.
The setup proposed is the minimum necessary for the Neutron service to work on a single machine. Therefore, you will also need Keystone (for authentication) and Nova (because some Neutron tasks require Nova to exist).
But first, you need to prepare your setup::
$ sudo apt-get update # Update the system ~ $ sudo apt-get upgrade # Install the required tools ~ $ sudo apt-get install -y git vim openssh-server openvswitch-switch ethtool # Edit the interfaces sudo vim /etc/network/interfaces
It is important here to understand the purpose of having 2 NICs with internet access, and how they actually work. You will use one of them for Neutron, specifically, to associate it with a bridge, in this case an OpenvSwitch Bridge. This bridge will be your external bridge, meaning the one that should link your cloud to the rest of the world. This external bridge is crucial to deploying Neutron, as all the other networks will connect to it and use it as a gateway. The reason you need a second NIC, is to keep it as your backup for internet access. When Neutron sets up, it will momentarily cut internet access to the NIC hosting the external bridge, while also performing some git updates, which causes Devstack to fail to deploy.
As a side note, it is possible to circumvent this issue by using another screen after the network outage. More here.
Therefore, your /etc/network/interfaces, you will become something like this:
# interfaces(5) file used by ifup(8) and ifdown(8) auto lo iface lo inet loopback ##This setup is network and PC specific, please use your own setup. ## However, keep the br-ex name as is because Neutron will specifically look for it ## External bridge auto br-ex iface br-ex inet static address 192.168.1.134 #my local ip, for the NIC I use for Neutron. netmask 255.255.255.0 gateway 192.168.1.1 dns-nameservers 22.214.171.124 ## External network interface auto enp6s0 # the id of the NIC I used for Neutron; reconfigure for br-ex iface enp6s0 inet manual up ifconfig enp6s0 0.0.0.0 up up ip link set enp6s0 promisc on down ip link set enp6s0 promisc off down ifconfig enp6s0 down
Now create it from OpenvSwitch so that Neutron will find it at deployment.
~ $sudo ovs-vsctl add-br br-ex ~ $sudo ovs-vsctl add-port br-ex enp6s0
Now let’s do a sanity check, to see if all is ok.
~ $ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 0c:c4:7a:32:07:62 brd ff:ff:ff:ff:ff:ff inet 192.168.1.116/24 brd 192.168.1.255 scope global dynamic enp5s0 valid_lft 50261sec preferred_lft 50261sec inet6 fe80::81bd:b3c0:3c73:74c8/64 scope link valid_lft forever preferred_lft forever 3: enp6s0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000 link/ether 0c:c4:7a:32:07:63 brd ff:ff:ff:ff:ff:ff inet6 fe80::ec4:7aff:fe32:763/64 scope link valid_lft forever preferred_lft forever 4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever 5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000 link/ether 52:54:00:7c:c5:26 brd ff:ff:ff:ff:ff:ff 6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1 link/ether e6:f5:ef:f5:bf:60 brd ff:ff:ff:ff:ff:ff 7: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1 link/ether 0c:c4:7a:32:07:63 brd ff:ff:ff:ff:ff:ff inet 192.168.1.134/24 brd 192.168.1.255 scope global br-ex valid_lft forever preferred_lft forever inet6 2001:db8::2/64 scope global valid_lft forever preferred_lft forever inet6 fe80::8459:23ff:fe87:484a/64 scope link valid_lft forever preferred_lft forever
~ $sudo ovs-vsctl show fb49781d-903e-4889-ad72-cd506af65bcb Bridge br-ex Port "enp6s0" Interface "enp6s0" Port br-ex Interface br-ex type: internal
Now, ideally, you would also want to add a dedicate user for DevStack, in order not to mix things up. You will only do this once, for your first deployment.
~ $ sudo adduser stack ~ $ sudo usermod -a -G sudo stack ~ $ sudo mkdir /opt/stack ~ $ sudo chown -R stack:stack /opt/stack ~ $ su stack
Great! Now you are logged in as stack and you are prepared to deploy.
Unfortunately, Devstack does not deploy with PyPy out of the box, because of tertiary dependencies that are not resolved yet, so we have to eliminate them manually (particularly, the postgres python libs are not compatible with PyPy). Therefore, you need to clone the Openstack repositories yourself and manually edit their requirements.txt and test-requirements.txt.
~ $ cd /opt/stack ~ $ git clone https://github.com/openstack/nova.git ~ $ git clone https://github.com/openstack/neutron.git ~ $ git clone https://github.com/openstack/keystone.git ~ $ git clone https://github.com/openstack/glance.git ~ $ cd /opt/glance ~ $ vim requirements.txt #comment out every psycopg and postgres dependency ~ $ vim test-requirements.txt #comment out every psycopg and postgres dependency
Go back to your home folder (user stack’s home folder) and clone devstack. Afterwards, you will need to prepare the local configuration file.
~ $ git clone https://github.com/openstack-dev/devstack.git ~ $ cd devstack ~ $ vim local.conf
The settings in the local.conf need to mirror the setup we prepared up to now. Particularly, we will need to disable some services, specify which bridge to be used for external communication and the restrictions on the network the bridge is attached to.
[[local|localrc]] HOST_IP=192.168.1.134 SERVICE_HOST=192.168.1.134 MYSQL_HOST=192.168.1.134 RABBIT_HOST=192.168.1.134 DEVSTACK_PASSWORD=password # Change the following passwords DATABASE_PASSWORD=$DEVSTACK_PASSWORD RABBIT_PASSWORD=$DEVSTACK_PASSWORD SERVICE_TOKEN=$DEVSTACK_PASSWORD SERVICE_PASSWORD=$DEVSTACK_PASSWORD ADMIN_PASSWORD=$DEVSTACK_PASSWORD Q_USE_SECGROUP=True FLOATING_RANGE="192.168.1.0/24" IPV4_ADDRS_SAFE_TO_USE="10.0.0.0/22" Q_FLOATING_ALLOCATION_POOL=start=192.168.1.200,end=192.168.1.254 #The sample setup had other machines in the network and we did not want interference PUBLIC_NETWORK_GATEWAY="192.168.1.1" PUBLIC_INTERFACE=enp6s0 Q_USE_PROVIDERNET_FOR_PUBLIC=True OVS_PHYSICAL_BRIDGE=br-ex PUBLIC_BRIDGE=br-ex OVS_BRIDGE_MAPPINGS=public:br-ex LIBVIRT_TYPE=kvm API_RATE_LIMIT=False SCREEN_LOGDIR=/opt/stack/logs/screen VERBOSE=True LOG_COLOR=False #Services to be started # Requirements enable_service rabbit enable_service mysql disable_service postgresql # Keystone enable_service key # Neutron enable_service neutron enable_service q-svc enable_service q-agt enable_service q-dhcp enable_service q-l3 enable_service q-meta enable_service q-lbaas enable_service q-fwaas enable_service q-metering enable_service q-vpn # Horizon disable_service horizon # Glance disable_service glance disable_service g-api disable_service g-reg # Cinder disable_service cinder disable_service c-api disable_service c-vol disable_service c-sch disable_service c-bak #Nova enable_service n-api enable_service n-crt enable_service n-obj enable_service n-cond enable_service n-sch enable_service n-cauth enable_service n-novnc enable_service n-cpu #Tempest disable_service tempest disable_service n-net disable_service s-proxy disable_service s-object disable_service s-container disable_service s-account disable_service heat disable_service h-api disable_service h-api-cfn disable_service h-api-cw disable_service h-eng disable_service ceilometer-acompute disable_service ceilometer-acentral disable_service ceilometer-collector disable_service ceilometer-api
Now call the deployment script and wait. It takes roughly 10 minutes.
As a side note, in the case you encounter any problems, you should check the error that was thrown, and also check the openstack logs (in our case /opt/stack/logs; the key.log is usually telling) before you redeploy. After that, you should:
Also, be mindful here, before you run ./stack.sh again, that the ovs-bridge created for the previous deployment was deleted by ./unstack.sh process, therefore you need to create it again.
~ $sudo ifdown enp6s0 ~ $sudo ifup enp6s0 ~ $sudo ovs-vsctl add-br br-ex ~ $sudo ovs-vsctl add-port br-ex enp6s0
By Mihai Dodan, mihai.dodan [at] rinftech [dot] com