openstack/performance-docs
Code Issues Proposed changes

Merge "OpenStack load test plan and reports"

Browse Source
This commit is contained in:
Jenkins 2016-07-08 12:06:35 +00:00 committed by Gerrit Code Review
commit 284f238ec9
25 changed files with 10956 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
doc/source/test_plans/index.rst
View File

@ -19,5 +19,6 @@ Test Plans
container_cluster_systems/plan
neutron_features/l3_ha/test_plan
hardware_features/index
openstack_load/plan
1000_nodes/plan
reliability/plan

188
doc/source/test_plans/openstack_load/plan.rst Normal file
View File

@ -0,0 +1,188 @@
.. _openstack_load_test_plan:
======================
OpenStack load testing
======================
:status: **ready**
:version: 1.0
:Abstract:
This test plan describes a set of scenarios to measure maximum number
of requests per second for a particular OpenStack API service.
:Conventions:
- **RPS** Requests-per-second - number of requests send to an API endpoint
per second
Test Plan
=========
**Rally** is a benchmarking tool that was designed specifically for OpenStack
API testing. To make this possible, **Rally** automates and unifies multi-node
OpenStack deployment, cloud verification, benchmarking & profiling. This is a
simple way to check cloud workability and performance of control plane
operations running on it. This test plan describes several Rally scenarios
that can cover almost all most important in perms of performance basic cloud
operations e.g. VMs creation, work with the security groups, authentication
and other operations.
Test Environment
----------------
Preparation
^^^^^^^^^^^
This test plan can be executed with help of Rally tool.
Environment description
^^^^^^^^^^^^^^^^^^^^^^^
The environment description includes hardware specification of servers,
network parameters, operation system and OpenStack deployment characteristics.
Hardware
~~~~~~~~
This section contains list of all types of hardware nodes.
+-----------+-------+----------------------------------------------------+
| Parameter | Value | Comments |
+-----------+-------+----------------------------------------------------+
| model | | e.g. Supermicro X9SRD-F |
+-----------+-------+----------------------------------------------------+
| CPU | | e.g. 6 x Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz |
+-----------+-------+----------------------------------------------------+
| role | | e.g. compute or network |
+-----------+-------+----------------------------------------------------+
Network
~~~~~~~
This section contains list of interfaces and network parameters.
For complicated cases this section may include topology diagram and switch
parameters.
+------------------+-------+-------------------------+
| Parameter | Value | Comments |
+------------------+-------+-------------------------+
| network role | | e.g. provider or public |
+------------------+-------+-------------------------+
| card model | | e.g. Intel |
+------------------+-------+-------------------------+
| driver | | e.g. ixgbe |
+------------------+-------+-------------------------+
| speed | | e.g. 10G or 1G |
+------------------+-------+-------------------------+
| MTU | | e.g. 9000 |
+------------------+-------+-------------------------+
| offloading modes | | e.g. default |
+------------------+-------+-------------------------+
Software
~~~~~~~~
This section describes installed software.
+-----------------+-------+---------------------------+
| Parameter | Value | Comments |
+-----------------+-------+---------------------------+
| OS | | e.g. Ubuntu 14.04.3 |
+-----------------+-------+---------------------------+
| OpenStack | | e.g. Mitaka |
+-----------------+-------+---------------------------+
| Hypervisor | | e.g. KVM |
+-----------------+-------+---------------------------+
| Neutron plugin | | e.g. ML2 + OVS |
+-----------------+-------+---------------------------+
| L2 segmentation | | e.g. VLAN / VxLAN / GRE |
+-----------------+-------+---------------------------+
| virtual routers | | e.g. HA / DVR |
+-----------------+-------+---------------------------+
Test Case 1: Neutron Load Testing
---------------------------------
Description
^^^^^^^^^^^
In this scenario Neutron API is loaded with constant flow of requests. The
number of requests per second is tuned to keep success rate at 100%.
Duration of operations is collected and stats are calculated. The result
is presented in table format.
List of performance metrics
^^^^^^^^^^^^^^^^^^^^^^^^^^^
+----------+-------+-------------------+-----------------------------------+
| Priority | Value | Measurement Units | Description |
+==========+=======+===================+===================================+
| 1 | | RPS | Number of API requests per second |
+----------+-------+-------------------+-----------------------------------+
Tools
=====
This test plan can be executed with help of Rally. Following is the list
of Rally scenarios.
Create Neutron networks
-----------------------
In this scenario Rally creates networks at constant rate. The single
iteration includes the following operations:
#. create network
.. literalinclude:: rally_scenarios/neutron_create_networks.json
Create Neutron network with 1 port
----------------------------------
In this scenario Rally creates Neutron network with one port at constant rate.
The single iteration includes the following operations:
#. create network
#. create one port
.. literalinclude:: rally_scenarios/neutron_create_network_with_port.json
Create Neutron ports
--------------------
In this scenario Rally creates Neutron network and ports at constant rate.
The single iteration includes the following operations:
#. create network
#. create 20 ports in each of network
#. list all ports
.. literalinclude:: rally_scenarios/neutron_create_and_list_ports.json
Create Cinder volumes
---------------------
In this scenario Rally creates volumes at constant rate. The single
iteration includes the following operations:
#. create volume
.. literalinclude:: rally_scenarios/cinder_create_volumes.json
Boot Nova servers
-----------------
In this scenario Rally boots Nova servers at constant rate:
#. boot server
.. literalinclude:: rally_scenarios/nova_boot_servers.json

36
doc/source/test_plans/openstack_load/rally_scenarios/cinder_create_volumes.json Normal file
View File

@ -0,0 +1,36 @@
{
"CinderVolumes.create_volume": [
{
"runner": {
"rps": 8,
"type": "rps",
"times": 1000
},
"args": {
"size": 10
},
"sla": {
"failure_rate": {
"max": 0
}
},
"context": {
"users": {
"users_per_tenant": 1,
"project_domain": "default",
"user_choice_method": "random",
"user_domain": "default",
"tenants": 1,
"resource_management_workers": 5
},
"quotas": {
"cinder": {
"gigabytes": -1,
"volumes": -1,
"snapshots": -1
}
}
}
}
]
}

37
doc/source/test_plans/openstack_load/rally_scenarios/neutron_create_and_list_ports.json Normal file
View File

@ -0,0 +1,37 @@
{
"NeutronNetworks.create_and_list_ports": [
{
"runner": {
"rps": 0.1,
"type": "rps",
"times": 50
},
"args": {
"network_create_args": {},
"ports_per_network": 20,
"port_create_args": {}
},
"sla": {
"failure_rate": {
"max": 0
}
},
"context": {
"users": {
"users_per_tenant": 2,
"project_domain": "default",
"user_choice_method": "random",
"user_domain": "default",
"tenants": 20,
"resource_management_workers": 30
},
"quotas": {
"neutron": {
"network": -1,
"port": -1
}
}
}
}
]
}

37
doc/source/test_plans/openstack_load/rally_scenarios/neutron_create_network_with_port.json Normal file
View File

@ -0,0 +1,37 @@
{
"NeutronNetworks.create_port": [
{
"runner": {
"rps": 5,
"type": "rps",
"times": 500
},
"args": {
"network_create_args": {},
"ports_per_network": 1,
"port_create_args": {}
},
"sla": {
"failure_rate": {
"max": 0
}
},
"context": {
"users": {
"users_per_tenant": 2,
"project_domain": "default",
"user_choice_method": "random",
"user_domain": "default",
"tenants": 20,
"resource_management_workers": 30
},
"quotas": {
"neutron": {
"network": -1,
"port": -1
}
}
}
}
]
}

34
doc/source/test_plans/openstack_load/rally_scenarios/neutron_create_networks.json Normal file
View File

@ -0,0 +1,34 @@
{
"NeutronNetworks.create_network": [
{
"runner": {
"rps": 5,
"type": "rps",
"times": 1000
},
"args": {
"network_create_args": {}
},
"sla": {
"failure_rate": {
"max": 0
}
},
"context": {
"users": {
"users_per_tenant": 4,
"project_domain": "default",
"user_choice_method": "random",
"user_domain": "default",
"tenants": 30,
"resource_management_workers": 30
},
"quotas": {
"neutron": {
"network": -1
}
}
}
}
]
}

54
doc/source/test_plans/openstack_load/rally_scenarios/nova_boot_servers.json Normal file
View File

@ -0,0 +1,54 @@
{
"NovaServers.boot_server": [
{
"runner": {
"rps": 1.7,
"type": "rps",
"times": 500
},
"args": {
"flavor": {
"name": "m1.tiny"
},
"image": {
"name": "TestVM"
}
},
"sla": {
"failure_rate": {
"max": 0
}
},
"context": {
"network": {
"subnets_per_network": 1,
"start_cidr": "100.1.0.0/21",
"networks_per_tenant": 1
},
"users": {
"users_per_tenant": 2,
"project_domain": "default",
"user_choice_method": "random",
"user_domain": "default",
"tenants": 2,
"resource_management_workers": 30
},
"quotas": {
"neutron": {
"subnet": -1,
"network": -1,
"port": -1
},
"nova": {
"ram": -1,
"floating_ips": -1,
"security_group_rules": -1,
"instances": -1,
"cores": -1,
"security_groups": -1
}
}
}
}
]
}

1
doc/source/test_results/index.rst
View File

@ -19,5 +19,6 @@ Test Results
neutron_features/index
hardware_features/index
provisioning/index
openstack_load/index
1000_nodes/index
reliability/index

BIN
doc/source/test_results/openstack_load/cinder_create_volumes_rps.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 10 KiB

296
doc/source/test_results/openstack_load/index.rst Normal file
View File

@ -0,0 +1,296 @@
.. _openstack_load_report:
*****************************
OpenStack load testing report
*****************************
:Abstract:
This document contains OpenStack load testing results collected for
:ref:`openstack_load_test_plan`
Environment description
=======================
Environment contains servers of the following roles:
- rally node
- controller node
- compute-osd node
- compute node
.. table:: Amount of servers each role
+------------+--------------+
|Role |Servers count |
+============+==============+
|rally |1 |
+------------+--------------+
|controller |3 |
+------------+--------------+
|compute |176 |
+------------+--------------+
|compute-osd |20 |
+------------+--------------+
Hardware configuration of each server
-------------------------------------
All servers have same configuration describing in table below
.. table:: Description of servers hardware
+-------+----------------+-------------------------------+
|server |vendor,model |HP,DL380 Gen9 |
+-------+----------------+-------------------------------+
|CPU |vendor,model |Intel,E5-2680 v3 |
| +----------------+-------------------------------+
| |processor_count |2 |
| +----------------+-------------------------------+
| |core_count |12 |
| +----------------+-------------------------------+
| |frequency_MHz |2500 |
+-------+----------------+-------------------------------+
|RAM |vendor,model |HP,752369-081 |
| +----------------+-------------------------------+
| |amount_MB |262144 |
+-------+----------------+-------------------------------+
|NETWORK|interface_name |p1p1 |
| +----------------+-------------------------------+
| |vendor,model |Intel,X710 Dual Port |
| +----------------+-------------------------------+
| |bandwidth |10G |
+-------+----------------+-------------------------------+
|STORAGE|dev_name |/dev/sda |
| +----------------+-------------------------------+
| |vendor,model | | raid10 - HP P840 |
| | | | 12 disks EH0600JEDHE |
| +----------------+-------------------------------+
| |SSD/HDD |HDD |
| +----------------+-------------------------------+
| |size | 3,6TB |
+-------+----------------+-------------------------------+
Network configuration of each server
------------------------------------
All servers have same network configuration:
.. image:: network_scheme.png
:alt: Network Scheme of the environment
Here is the part of switch configuration for each switch port which connected to
ens1f0 interface of a server:
.. code:: bash
switchport mode trunk
switchport trunk native vlan 600
switchport trunk allowed vlan 600-602,630-649
spanning-tree port type edge trunk
spanning-tree bpduguard enable
no snmp trap link-status
Software configuration on environment servers
---------------------------------------------
.. table:: Services on servers by role
+------------+----------------------------+
|Role |Service name |
+============+============================+
|controller || horizon |
| || keystone |
| || nova-api |
| || nava-scheduler |
| || nova-cert |
| || nova-conductor |
| || nova-consoleauth |
| || nova-consoleproxy |
| || cinder-api |
| || cinder-backup |
| || cinder-scheduler |
| || cinder-volume |
| || glance-api |
| || glance-glare |
| || glance-registry |
| || neutron-dhcp-agent |
| || neutron-l3-agent |
| || neutron-metadata-agent |
| || neutron-openvswitch-agent |
| || neutron-server |
| || heat-api |
| || heat-api-cfn |
| || heat-api-cloudwatch |
| || ceph-mon |
| || rados-gw |
| || heat-engine |
| || memcached |
| || rabbitmq-server |
| || mysqld |
| || galera |
| || corosync |
| || pacemaker |
| || haproxy |
+------------+----------------------------+
|compute || nova-compute |
| || neutron-l3-agent |
| || neutron-metadata-agent |
| || neutron-openvswitch-agent |
+------------+----------------------------+
|compute-osd || nova-compute |
| || neutron-l3-agent |
| || neutron-metadata-agent |
| || neutron-openvswitch-agent |
| || ceph-osd |
+------------+----------------------------+
|osd |ceph-osd |
+------------+----------------------------+
.. table:: Software version of OpenStack nodes
+------------+-------------------+
|Software |Version |
+============+===================+
|OpenStack |Mitaka |
+------------+-------------------+
|Ceph |Hammer |
+------------+-------------------+
|Ubuntu |Ubuntu 14.04.3 LTS |
+------------+-------------------+
Software configuration on servers with rally role
-------------------------------------------------
On this server should be installed Rally. How to do it you can find in
`Rally installation documentation`_
.. table:: Software version of Rally server
+------------+-------------------+
|Software |Version |
+============+===================+
|Rally |0.4.0 |
+------------+-------------------+
|Ubuntu |Ubuntu 14.04.3 LTS |
+------------+-------------------+
Test results
============
Create Neutron networks
-----------------------
In this test case Rally creates Neutron networks at constant rate. Achieved
maximum rate: **5 RPS**.
+---------------------------------------------------------------------------------------------------------------------------+
| Response Times (sec) |
+------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| neutron.create_network | 0.165 | 0.197 | 0.277 | 0.295 | 0.462 | 0.211 | 100.0% | 1000 |
| total | 0.166 | 0.197 | 0.277 | 0.295 | 0.462 | 0.211 | 100.0% | 1000 |
+------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
Rally report: :download:`rally_neutron_create_port.html <rally_reports/neutron_create_networks.html>`
Create Neutron network with port
--------------------------------
In this test case Rally creates Neutron network with one Neutron port at
constant rate. The summary duration time is measured and dependency on number
of requests per second is shown.
.. image:: neutron_create_network_with_port.png
===== =========== ============== =========== ===========
RPS Min (sec) Median (sec) Max (sec) Avg (sec)
===== =========== ============== =========== ===========
5 0.505 0.658 4.712 0.795
10 0.468 0.635 2.789 0.712
===== =========== ============== =========== ===========
Rally reports:
* :download:`Create Neutron network with port 5 rps <rally_reports/neutron_create_network_with_port_5rps.html>`
* :download:`Create Neutron network with port 10 rps <rally_reports/neutron_create_network_with_port_10rps.html>`
Create and list Neutron ports
-----------------------------
In this test case Rally creates Neutron network with 20 ports and lists them.
Operations are requested at constant rate. Achieved maximum rate: **0.1 RPS**.
+-----------------------------------------------------------------------------------------------------------------------------+
| Response Times (sec) |
+--------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+--------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| neutron.create_port | 0.257 | 0.370 | 0.994 | 1.405 | 1.81 | 0.452 | 100.0% | 50 |
| neutron.list_ports | 0.045 | 0.108 | 0.222 | 0.236 | 0.256 | 0.129 | 100.0% | 50 |
| neutron.create_network | 0.221 | 0.311 | 0.418 | 0.44 | 0.525 | 0.312 | 100.0% | 50 |
| total | 8.243 | 9.487 | 10.213 | 10.686 | 12.259 | 9.485 | 100.0% | 50 |
+--------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
Rally report: :download:`rally_neutron_create_port.html <rally_reports/neutron_create_and_list_ports.html>`
Create Cinder volumes
---------------------
The following chart and table show how operation duration depends on number
of requests per second. The median grows, at 8 RPS one of request falls out
of 1 minute.
.. image:: cinder_create_volumes_rps.png
===== =========== ============== =========== ===========
RPS Min (sec) Median (sec) Max (sec) Avg (sec)
===== =========== ============== =========== ===========
1 2.462 2.578 3.158 2.607
2 2.474 2.616 3.029 2.644
5 2.52 2.757 7.701 2.788
6 2.565 2.792 3.975 2.83
8 2.654 3.97 73.869 16.608
===== =========== ============== =========== ===========
Rally reports:
* :download:`Create Cinder volumes 1 rps <rally_reports/cinder_create_volumes_1rps.html>`
* :download:`Create Cinder volumes 2 rps <rally_reports/cinder_create_volumes_2rps.html>`
* :download:`Create Cinder volumes 5 rps <rally_reports/cinder_create_volumes_5rps.html>`
* :download:`Create Cinder volumes 6 rps <rally_reports/cinder_create_volumes_6rps.html>`
* :download:`Create Cinder volumes 8 rps <rally_reports/cinder_create_volumes_8rps.html>`
Boot Nova Server
----------------
The following chart and table show how operation duration depends on number
of requests per second. Success rate falls beyond 100% as some of operations
take longer than the timeout limit.
.. image:: nova_boot_servers_rps.png
===== =========== ============== =========== =========== =============
RPS Min (sec) Median (sec) Max (sec) Avg (sec) Success (%)
===== =========== ============== =========== =========== =============
1.5 12.234 29.197 118.353 34.074 100
1.7 12.626 34.993 182.083 48.736 98.4
2 13.426 37.698 298.564 50.021 70
===== =========== ============== =========== =========== =============
Rally reports:
* :download:`Nova Boot server 1.5 rps <rally_reports/nova_boot_server_1_5rps.html>`
* :download:`Nova Boot server 1.7 rps <rally_reports/nova_boot_server_1_7rps.html>`
* :download:`Nova Boot server 2 rps <rally_reports/nova_boot_server_2rps.html>`
.. references:
.. _Rally installation documentation: https://rally.readthedocs.io/en/latest/install.html

BIN
doc/source/test_results/openstack_load/network_scheme.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 20 KiB

BIN
doc/source/test_results/openstack_load/neutron_create_network_with_port.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 9.9 KiB

BIN
doc/source/test_results/openstack_load/nova_boot_servers_rps.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

856
doc/source/test_results/openstack_load/rally_reports/cinder_create_volumes_1rps.html Normal file
View File

File diff suppressed because one or more lines are too long

856
doc/source/test_results/openstack_load/rally_reports/cinder_create_volumes_2rps.html Normal file
View File

File diff suppressed because one or more lines are too long

856
doc/source/test_results/openstack_load/rally_reports/cinder_create_volumes_5rps.html Normal file
View File

File diff suppressed because one or more lines are too long

856
doc/source/test_results/openstack_load/rally_reports/cinder_create_volumes_6rps.html Normal file
View File

File diff suppressed because one or more lines are too long

856
doc/source/test_results/openstack_load/rally_reports/cinder_create_volumes_8rps.html Normal file
View File

File diff suppressed because one or more lines are too long

856
doc/source/test_results/openstack_load/rally_reports/neutron_create_and_list_ports.html Normal file
View File

File diff suppressed because one or more lines are too long

856
doc/source/test_results/openstack_load/rally_reports/neutron_create_network_with_port_10rps.html Normal file
View File

File diff suppressed because one or more lines are too long

856
doc/source/test_results/openstack_load/rally_reports/neutron_create_network_with_port_5rps.html Normal file
View File

File diff suppressed because one or more lines are too long

856
doc/source/test_results/openstack_load/rally_reports/neutron_create_networks.html Normal file
View File

File diff suppressed because one or more lines are too long

856
doc/source/test_results/openstack_load/rally_reports/nova_boot_server_1_5rps.html Normal file
View File

File diff suppressed because one or more lines are too long

856
doc/source/test_results/openstack_load/rally_reports/nova_boot_server_1_7rps.html Normal file
View File

File diff suppressed because one or more lines are too long

856
doc/source/test_results/openstack_load/rally_reports/nova_boot_server_2rps.html Normal file
View File

File diff suppressed because one or more lines are too long