PyPy vs CPython Performance on OpenStack Neutron

This post contains details about a performance comparison between PyPy and CPython, when applied to OpenStack Neutron.  The steps did to switch from CPython to PyPy are described in my post here.

The server setup for CPython is quite similar to PyPy. Basically, simply cloning devstack, setting up the Network Interface Adapter and having the same local.conf in the devstack folder should do the trick (as described in my previous post).

The client

Having setup the devstack server, we now need a client and a benchmark to evaluate the setup.  For benchmarking purposes we used Rally, an Openstack suite with the purpose of creating scenarios and workloads.

We advise you to setup Rally in a virtual environment, on a separate machine, for the following reasons:

  •  We do not want the client workload generation to affect the server performance. Rally can use a sizable amount of resources, especially for large workloads, and we want measurements that are as accurate as possible.
  • Rally does not work with PyPy as the default interpreter, only with CPython. Proof of concept runs on the same machine show that Rally has issues when interacting with its database with PyPy.

Now, on the client machine::

 ~ $ git clone git clone https://github.com/openstack/rally
 ~ $ cd rally
 ~ $ ./install.sh -p /usr/bin/python2.7 -d /your/virtualenv/rally/folder

Rally will create a virtual env folder. You need to activate it and prepare the deployment of Rally to communicate with the Openstack machine::

 ~ $ source /your/virtualenv/rally/folder/bin/activate
 ~ $ cd samples/deployments
 ~ $ #Replace existing-keystone-v3.json with this; these are SAMPLE IPs, please use your own.
 ~ $ #auth_url : http://server_machine_ip:keystone_admin_port/v3
 ##########################FILE STARTS HERE#################################
 {
 "type": "ExistingCloud",
 "auth_url": "http://192.168.1.134:35357/v3",
 "region_name": "RegionOne",
 "endpoint_type": "public",
 "admin": {
 "username": "admin",
 "password": "password",
 "user_domain_name": "default",
 "project_name": "admin",
 "project_domain_name": "default"
 },
 "https_insecure": false,
 "https_cacert": ""
 }
 #########################FILE ENDS HERE##################################
 ~ $ rally deployment create --file=existing-keystone-v3.json --name=FooBar
 ~ $ ######Sanity check#####
 ~ $ rally deployment check

This should out put something like::

keystone endpoints are valid and following services are available:
 +-------------+----------------+-----------+
 | services    | type           | status    |
 +-------------+----------------+-----------+
 | __unknown__ | compute_legacy | Available |
 | __unknown__ | placement      | Available |
 | keystone    | identity       | Available |
 | neutron     | network        | Available |
 | nova        | compute        | Available |
 +-------------+----------------+-----------+

This means that Rally has successfully authenticated to Openstack on your server, via Keystone, and is correctly able to identify working services.

Creating the workload

In order to better understand what we refer to as a benchmark, and its subdivision, we define in our case this legend::

1 benchmark = 10 workloads or tasks
1 workload or task = 21 scenarios
1 scenario = 100 iterations

For some Openstack modules, including Neutron, Rally comes with a default series of tasks, with predefined scenarios and with their own SLAs. We need to define now what will we be aiming for in our benchmark.

Since this is a comparison between the performance of the Neutron service under CPython and, respectively, PyPy, what we will be measuring will be the deployment time of networking services, at large workload scenarios. We will not be measuring the traffic statistics of the said networking services (as we would be testing OpenvSwitch more than Neutron, on certain aspects), but rather, the efficiency of deployment.

Also it is important to note here that the default Rally scenarios are useful conceptually, because of the diversity of the workload and the SLA, but are problematic, due to the fact that they are generally to short. The problem is that, PyPy obtains most of its speedup from the fact that it has a JIT. The JIT, however, requires some warm up, which in the case of simple Rally scenarios, lasted too long to see any meaningful difference. (we have also seen this during profiling, which will be discussed in another article)

Therefore, what we proposed, are for each individual Rally – Neutron scenario, to increase the number of iterations per tests, so that PyPy has a fair chance to catch up. Also we have eliminated any scenarios that called Nova or any other service besides Neutron. See our proposed Neutron workload at `4. Rally workload`_.

To ensure the stability of our results, the tasks have been run 10 times for both CPython and PyPy, by executing from the client:

rally task start big-neutron-scenario.yaml

The results have then been aggregated into 1 html file per task run, and afterwards, the data has been transferred in an Excel.

Results and methodology

Below we show the results of individual scenarios of the workload with PyPy vs CPython. The results show the average time calculated without the minimum and maximum values.

en-detail-pypyvscpython

Next, we show the results of all the 10 task runs, for both CPython and PyPy. The time for one task is the summation of all its individual scenarios.

pypyvscpythontimes1

Now, we show the results of the speedup of PyPy when compared to CPython. It is calculated by the average time value of all 10 workloads (except min-max) for CPython, divided by the average time value of all 10 tasks for both PyPy and CPython.

pypyvscpythonspeedup

Rally workload

Workload

       {% set smoke = 0 %}

	---
	  NeutronNetworks.create_and_list_networks:
	    -
	      args:
		network_create_args: {}
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		     network: -1
	      sla:
		failure_rate:
		  max: 20
	    -
	      args:
		network_create_args:
		  provider:network_type: "vxlan"
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		     network: -1
		roles:
		  - "admin"
	      sla:
		failure_rate:
		  max: 20

	  NeutronNetworks.create_and_show_network:
	    -
	      args:
		network_create_args: {}
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    network: -1
	      sla:
		failure_rate:
		  max: 0

	  NeutronNetworks.create_and_list_subnets:
	    -
	      args:
		network_create_args:
		subnet_create_args:
		subnet_cidr_start: "1.1.0.0/30"
		subnets_per_network: 2
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		network: {}
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		     network: -1
		     subnet: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronSecurityGroup.create_and_list_security_groups:
	    -
	      args:
		security_group_create_args: {}
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    security_group: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronSecurityGroup.create_and_delete_security_groups:
	    -
	      args:
		security_group_create_args: {}
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    security_group: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronSecurityGroup.create_and_update_security_groups:
	    -
	      args:
		security_group_create_args: {}
		security_group_update_args: {}
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    security_group: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronNetworks.create_and_list_floating_ips:
	    -
	      args:
		floating_network: "public"
		floating_ip_args: {}
	      runner:
		type: "constant"
		times: 40
		concurrency: 20
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		     floatingip: -1
	      sla:
		failure_rate:
		  max: 0

	  NeutronNetworks.create_and_list_routers:
	    -
	      args:
		network_create_args:
		subnet_create_args:
		subnet_cidr_start: "1.1.0.0/30"
		subnets_per_network: 2
		router_create_args:
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		network: {}
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		     network: -1
		     subnet: -1
		     router: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronNetworks.create_and_list_ports:
	    -
	      args:
		network_create_args:
		port_create_args:
		ports_per_network: 4
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		network: {}
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		     network: -1
		     subnet: -1
		     router: -1
		     port: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronNetworks.list_agents:
	    -
	      args:
		agent_args: {}
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
	      sla:
		failure_rate:
		  max: 0

	  NeutronNetworks.create_and_update_networks:
	    -
	      args:
		network_create_args: {}
		network_update_args:
		    admin_state_up: False
		    name: "_updated"
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    network: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronNetworks.create_and_update_subnets:
	    -
	      args:
		network_create_args: {}
		subnet_create_args: {}
		subnet_cidr_start: "1.4.0.0/16"
		subnets_per_network: 2
		subnet_update_args:
		    enable_dhcp: False
		    name: "_subnet_updated"
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		network: {}
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    network: -1
		    subnet: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronNetworks.create_and_update_routers:
	    -
	      args:
		network_create_args: {}
		subnet_create_args: {}
		subnet_cidr_start: "1.1.0.0/30"
		subnets_per_network: 2
		router_create_args: {}
		router_update_args:
		    admin_state_up: False
		    name: "_router_updated"
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		network: {}
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    network: -1
		    subnet: -1
		    router: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronNetworks.create_and_delete_networks:
	    -
	      args:
		network_create_args: {}
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    network: -1
		    subnet: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronNetworks.create_and_delete_subnets:
	    -
	      args:
		network_create_args: {}
		subnet_create_args: {}
		subnet_cidr_start: "1.1.0.0/30"
		subnets_per_network: 2
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		network: {}
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    network: -1
		    subnet: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronNetworks.create_and_delete_floating_ips:
	    -
	      args:
		floating_network: "public"
		floating_ip_args: {}
	      runner:
		type: "constant"
		times: {{smoke or 40}}
		concurrency: {{smoke or 20}}
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		     floatingip: -1
	      sla:
		failure_rate:
		  max: 0

	  NeutronNetworks.create_and_delete_routers:
	    -
	      args:
		network_create_args: {}
		subnet_create_args: {}
		subnet_cidr_start: "1.1.0.0/30"
		subnets_per_network: 2
		router_create_args: {}
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		network: {}
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    network: -1
		    subnet: -1
		    router: -1
	      sla:
		  failure_rate:
		    max: 20

	  NeutronNetworks.create_and_delete_ports:
	    -
	      args:
		network_create_args: {}
		port_create_args: {}
		ports_per_network: 10
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		network: {}
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    network: -1
		    port: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronNetworks.create_and_update_ports:
	    -
	      args:
		network_create_args: {}
		port_create_args: {}
		ports_per_network: 5
		port_update_args:
		    admin_state_up: False
		    device_id: "dummy_id"
		    device_owner: "dummy_owner"
		    name: "_port_updated"
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		network: {}
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    network: -1
		    port: -1
	      sla:
		failure_rate:
		  max: 20

	  Quotas.neutron_update:
	    -
	      args:
		max_quota: 1024
	      runner:
		type: "constant"
		times: 100
		concurrency: 5
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
	      sla:
		failure_rate:
		  max: 0
       {% set smoke = 0 %}

	---
	  NeutronNetworks.create_and_list_networks:
	    -
	      args:
		network_create_args: {}
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		     network: -1
	      sla:
		failure_rate:
		  max: 20
	    -
	      args:
		network_create_args:
		  provider:network_type: "vxlan"
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		     network: -1
		roles:
		  - "admin"
	      sla:
		failure_rate:
		  max: 20

	  NeutronNetworks.create_and_show_network:
	    -
	      args:
		network_create_args: {}
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    network: -1
	      sla:
		failure_rate:
		  max: 0

	  NeutronNetworks.create_and_list_subnets:
	    -
	      args:
		network_create_args:
		subnet_create_args:
		subnet_cidr_start: "1.1.0.0/30"
		subnets_per_network: 2
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		network: {}
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		     network: -1
		     subnet: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronSecurityGroup.create_and_list_security_groups:
	    -
	      args:
		security_group_create_args: {}
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    security_group: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronSecurityGroup.create_and_delete_security_groups:
	    -
	      args:
		security_group_create_args: {}
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    security_group: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronSecurityGroup.create_and_update_security_groups:
	    -
	      args:
		security_group_create_args: {}
		security_group_update_args: {}
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    security_group: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronNetworks.create_and_list_floating_ips:
	    -
	      args:
		floating_network: "public"
		floating_ip_args: {}
	      runner:
		type: "constant"
		times: 40
		concurrency: 20
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		     floatingip: -1
	      sla:
		failure_rate:
		  max: 0

	  NeutronNetworks.create_and_list_routers:
	    -
	      args:
		network_create_args:
		subnet_create_args:
		subnet_cidr_start: "1.1.0.0/30"
		subnets_per_network: 2
		router_create_args:
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		network: {}
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		     network: -1
		     subnet: -1
		     router: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronNetworks.create_and_list_ports:
	    -
	      args:
		network_create_args:
		port_create_args:
		ports_per_network: 4
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		network: {}
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		     network: -1
		     subnet: -1
		     router: -1
		     port: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronNetworks.list_agents:
	    -
	      args:
		agent_args: {}
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
	      sla:
		failure_rate:
		  max: 0

	  NeutronNetworks.create_and_update_networks:
	    -
	      args:
		network_create_args: {}
		network_update_args:
		    admin_state_up: False
		    name: "_updated"
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    network: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronNetworks.create_and_update_subnets:
	    -
	      args:
		network_create_args: {}
		subnet_create_args: {}
		subnet_cidr_start: "1.4.0.0/16"
		subnets_per_network: 2
		subnet_update_args:
		    enable_dhcp: False
		    name: "_subnet_updated"
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		network: {}
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    network: -1
		    subnet: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronNetworks.create_and_update_routers:
	    -
	      args:
		network_create_args: {}
		subnet_create_args: {}
		subnet_cidr_start: "1.1.0.0/30"
		subnets_per_network: 2
		router_create_args: {}
		router_update_args:
		    admin_state_up: False
		    name: "_router_updated"
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		network: {}
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    network: -1
		    subnet: -1
		    router: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronNetworks.create_and_delete_networks:
	    -
	      args:
		network_create_args: {}
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    network: -1
		    subnet: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronNetworks.create_and_delete_subnets:
	    -
	      args:
		network_create_args: {}
		subnet_create_args: {}
		subnet_cidr_start: "1.1.0.0/30"
		subnets_per_network: 2
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		network: {}
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    network: -1
		    subnet: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronNetworks.create_and_delete_floating_ips:
	    -
	      args:
		floating_network: "public"
		floating_ip_args: {}
	      runner:
		type: "constant"
		times: {{smoke or 40}}
		concurrency: {{smoke or 20}}
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		     floatingip: -1
	      sla:
		failure_rate:
		  max: 0

	  NeutronNetworks.create_and_delete_routers:
	    -
	      args:
		network_create_args: {}
		subnet_create_args: {}
		subnet_cidr_start: "1.1.0.0/30"
		subnets_per_network: 2
		router_create_args: {}
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		network: {}
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    network: -1
		    subnet: -1
		    router: -1
	      sla:
		  failure_rate:
		    max: 20

	  NeutronNetworks.create_and_delete_ports:
	    -
	      args:
		network_create_args: {}
		port_create_args: {}
		ports_per_network: 10
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		network: {}
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    network: -1
		    port: -1
	      sla:
		failure_rate:
		  max: 20

	  NeutronNetworks.create_and_update_ports:
	    -
	      args:
		network_create_args: {}
		port_create_args: {}
		ports_per_network: 5
		port_update_args:
		    admin_state_up: False
		    device_id: "dummy_id"
		    device_owner: "dummy_owner"
		    name: "_port_updated"
	      runner:
		type: "constant"
		times: 100
		concurrency: 20
	      context:
		network: {}
		users:
		  tenants: 4
		  users_per_tenant: 2
		quotas:
		  neutron:
		    network: -1
		    port: -1
	      sla:
		failure_rate:
		  max: 20

	  Quotas.neutron_update:
	    -
	      args:
		max_quota: 1024
	      runner:
		type: "constant"
		times: 100
		concurrency: 5
	      context:
		users:
		  tenants: 4
		  users_per_tenant: 2
	      sla:
		failure_rate:
		  max: 0

By Mihai Dodan, mihai.dodan [at] rinftech [dot] com

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s