191 lines
6.7 KiB
ReStructuredText
191 lines
6.7 KiB
ReStructuredText
.. _openstack_api_workers:
|
|
|
|
======================================================================
|
|
OpenStack API worker performance and scalability testing with Browbeat
|
|
======================================================================
|
|
|
|
:status: **draft**
|
|
:version: 1.0
|
|
|
|
:Abstract:
|
|
|
|
The work described in this test plan is to evaluate changes
|
|
that can occure upstream like [1][2], and review the impact of
|
|
API response times due to these changes. This test plan will also seek
|
|
to determine a possible os_worker default that performs close to the
|
|
previous os_worker count.
|
|
|
|
[1] http://lists.openstack.org/pipermail/openstack-dev/2016-September/104819.html
|
|
[2] https://github.com/openstack/puppet-openstacklib/blob/master/lib/facter/os_workers.rb
|
|
|
|
Test Plan
|
|
=========
|
|
|
|
Test Environment
|
|
----------------
|
|
|
|
Preparation
|
|
^^^^^^^^^^^
|
|
Ideally this is run on a newly deployed cloud each time for repeatability
|
|
purposes. Cloud deployment should be documented for each test case / run as
|
|
deployment will set many configuration values which will impact performance.
|
|
|
|
Environment description
|
|
^^^^^^^^^^^^^^^^^^^^^^^
|
|
The environment description includes hardware specs, software versions, tunings
|
|
and configuration of the OpenStack Cloud under test.
|
|
|
|
Hardware
|
|
~~~~~~~~
|
|
List details of hardware for each node type here.
|
|
|
|
Deployment node (Undercloud)
|
|
|
|
+-----------+------------------------------------------------------------+
|
|
| Parameter | Value |
|
|
+-----------+------------------------------------------------------------+
|
|
| model | e.g. Dell PowerEdge r620 |
|
|
+-----------+------------------------------------------------------------+
|
|
| CPU | e.g. 2xIntel(R) Xeon(R) E2620 @ 2GHz (12Cores/24Threads)|
|
|
+-----------+------------------------------------------------------------+
|
|
| Memory | e.g. 64GiB (@1333MHz) |
|
|
+-----------+------------------------------------------------------------+
|
|
| Network | e.g. 2x10Gb/s Intel X520 |
|
|
+-----------+------------------------------------------------------------+
|
|
|
|
Controller
|
|
|
|
+-----------+------------------------------------------------------------+
|
|
| Parameter | Value |
|
|
+-----------+------------------------------------------------------------+
|
|
| model | e.g. Dell PowerEdge r620 |
|
|
+-----------+------------------------------------------------------------+
|
|
| CPU | e.g. 2xIntel(R) Xeon(R) E2620 @ 2GHz (12Cores/24Threads)|
|
|
+-----------+------------------------------------------------------------+
|
|
| Memory | e.g. 64GiB (@1333MHz) |
|
|
+-----------+------------------------------------------------------------+
|
|
| Network | e.g. 2x10Gb/s Intel X520 |
|
|
+-----------+------------------------------------------------------------+
|
|
|
|
Compute (7)
|
|
|
|
+-----------+------------------------------------------------------------+
|
|
| Parameter | Value |
|
|
+-----------+------------------------------------------------------------+
|
|
| model | e.g. Dell PowerEdge r620 |
|
|
+-----------+------------------------------------------------------------+
|
|
| CPU | e.g. 2xIntel(R) Xeon(R) E2620 @ 2GHz (12Cores/24Threads)|
|
|
+-----------+------------------------------------------------------------+
|
|
| Memory | e.g. 64GiB (@1333MHz) |
|
|
+-----------+------------------------------------------------------------+
|
|
| Network | e.g. 2x10Gb/s Intel X520 |
|
|
+-----------+------------------------------------------------------------+
|
|
|
|
Software
|
|
~~~~~~~~
|
|
Record versions of Linux kernel, Base Operating System (RHEL 7.3),
|
|
OpenStack version 10 (Newton), OpenStack Packages, testing harness/framework
|
|
and any other pertinent software.
|
|
|
|
Tuning/Configuration
|
|
~~~~~~~~~~~~~~~~~~~~
|
|
With TripleO we can use an extra configuration file that explicitly sets the
|
|
worker count for us. However with these templates we cannot do simple
|
|
calculations based on the $::processorcount. However we can use it to statically
|
|
set our deployment to a known number of workers. This tuning file was used for
|
|
this work to set the number of workers per-service.
|
|
|
|
**Example of tunings.yaml**
|
|
|
|
.. code-block:: none
|
|
|
|
parameter_defaults:
|
|
controllerExtraConfig:
|
|
keystone::wsgi::apache::workers: 24
|
|
keystone::wsgi::apache::threads: 1
|
|
|
|
Test Diagram
|
|
~~~~~~~~~~~~
|
|
Using OpenStack Browbeat execute the same test scenarios[1] across different
|
|
os_worker counts within the overcloud. Our SUT is equipped with Intel Westmere
|
|
CPUs (dual socket), so we have a total of 24 logical cores to work with. Based
|
|
on the current upstream calculation, os_worker count will be 6.
|
|
|
|
.. code-block:: none
|
|
|
|
os_worker_count = (((processor_count/4),2).max,8).min
|
|
os_worker_count = (((24/4),2).max,8).min
|
|
|
|
This testing will run with the following number of os_workers: 6, 8, 12 and 24
|
|
to determine if the current default performs favorably, or if we should suggest
|
|
changing it.
|
|
|
|
We will use TripleO to deploy the SUT OpenStack cloud. We will vary the os_worker
|
|
count using the tunings.yaml script described above.
|
|
|
|
Each OpenStack Rally scenario defined in [1] will execute 3 times per os_worker
|
|
environment. Our results will show the Min, Max, Average and 95%tile per atomic action.
|
|
|
|
[1] https://gist.github.com/jtaleric/fcc8d0d0d989d7a593a6e8c595252150
|
|
|
|
Test Case 1: Keystone
|
|
----------------------
|
|
|
|
Description
|
|
^^^^^^^^^^^
|
|
Each Rally scenario below was executed until they reach 1500 times, at 32 and 64 concurrencies.
|
|
|
|
**Scenarios**
|
|
* Authenticate-keystone
|
|
* Authenticate-neutron
|
|
* Authenticate-nova
|
|
|
|
Setup
|
|
^^^^^^
|
|
#. Deploy OpenStack Cloud
|
|
#. Install testing and monitoring tooling
|
|
#. Gather metadata on Cloud
|
|
#. Run test
|
|
|
|
Analysis
|
|
^^^^^^^^^
|
|
Review Rally scenario results, noting the API response times at different concurrencies.
|
|
|
|
List of performance metrics
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
Performance
|
|
|
|
- Service API response times
|
|
|
|
Test Case 2: Neutron
|
|
----------------------
|
|
|
|
Description
|
|
^^^^^^^^^^^
|
|
Each Rally scenario below will until they reach 500 times, at 32 and 64 concurrencies.
|
|
|
|
**Scenarios**
|
|
* Create-list-network
|
|
* Create-list-port
|
|
* Create-list-router
|
|
* Create-list-security-group
|
|
* Create-list-subnet
|
|
|
|
Setup
|
|
^^^^^^
|
|
#. Deploy OpenStack Cloud
|
|
#. Install testing and monitoring tooling
|
|
#. Gather metadata on Cloud
|
|
#. Run test
|
|
|
|
Analysis
|
|
^^^^^^^^^
|
|
Review Rally scenario results, noting the API response times at different concurrencies.
|
|
|
|
List of performance metrics
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
Performance
|
|
|
|
- Service API response times
|
|
|