diff --git a/doc/source/test_plans/fuel_ccp/plan.rst b/doc/source/test_plans/fuel_ccp/plan.rst new file mode 100644 index 0000000..601a6ac --- /dev/null +++ b/doc/source/test_plans/fuel_ccp/plan.rst @@ -0,0 +1,573 @@ +.. _fuel_ccp_test_plan: + +==================================================== +Fuel Containerized Control Plane performance testing +==================================================== + +:status: **ready** +:version: 1.0 + +:Abstract: + + This test plan aims to provide set of tests to identify OpenStack + performance against given containerized OpenStack cloud (installed + on the top of pre-deployed Kubernetes cluster) using simple minimalistic set + of Rally tests. + +Test Plan +========= + +This document is inspired by +:ref:`openstack_control_plane_performance_test_plan`, and aims to cover +baseline cloud operations and extend this test suite to verify containerized +deployment approach. As :ref:`openstack_control_plane_performance_test_plan` +this test plan covers basic cloud operations e.g. VMs creation, work with the +security groups, authentication and more, as well as long-running test suites +to verify cloud stability. + +Test Environment +---------------- + +Preparation +^^^^^^^^^^^ + +This test plan is performed against existing OpenStack cloud installed on top +of pre-deployed Kubernetes cluster with `fuel-ccp`_ tool with pre-installed Rally +framework. + +.. _fuel-ccp: http://fuel-ccp.readthedocs.io/en/latest/ + +Environment description +^^^^^^^^^^^^^^^^^^^^^^^ + +The environment description includes hardware specification of servers, +network parameters, operation system and OpenStack deployment characteristics. + +Hardware +~~~~~~~~ + +This section contains list of all types of hardware nodes (table below is +an example). + ++-----------+-------+----------------------------------------------------+ +| Parameter | Value | Comments | ++-----------+-------+----------------------------------------------------+ +| model | | e.g. Supermicro X9SRD-F | ++-----------+-------+----------------------------------------------------+ +| CPU | | e.g. 6 x Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz | ++-----------+-------+----------------------------------------------------+ +| role | | e.g. compute or network | ++-----------+-------+----------------------------------------------------+ + +Network +~~~~~~~ + +This section contains list of interfaces and network parameters. For +complicated cases this section may include topology diagram and switch +parameters (table below is an example). + ++------------------+-------+-------------------------+ +| Parameter | Value | Comments | ++------------------+-------+-------------------------+ +| network role | | e.g. provider or public | ++------------------+-------+-------------------------+ +| card model | | e.g. Intel | ++------------------+-------+-------------------------+ +| driver | | e.g. ixgbe | ++------------------+-------+-------------------------+ +| speed | | e.g. 10G or 1G | ++------------------+-------+-------------------------+ +| MTU | | e.g. 9000 | ++------------------+-------+-------------------------+ +| offloading modes | | e.g. default | ++------------------+-------+-------------------------+ + +Software +~~~~~~~~ + +This section describes installed software (table below is an example). + ++-----------------+-------+---------------------------+ +| Parameter | Value | Comments | ++-----------------+-------+---------------------------+ +| OS | | e.g. Ubuntu 14.04.3 | ++-----------------+-------+---------------------------+ +| OpenStack | | e.g. Mitaka | ++-----------------+-------+---------------------------+ +| Hypervisor | | e.g. KVM | ++-----------------+-------+---------------------------+ +| Neutron plugin | | e.g. ML2 + OVS | ++-----------------+-------+---------------------------+ +| L2 segmentation | | e.g. VLAN / VxLAN / GRE | ++-----------------+-------+---------------------------+ +| virtual routers | | e.g. HA / DVR | ++-----------------+-------+---------------------------+ + +Test tool +--------- + +**Rally** is a benchmarking tool that was designed specifically for OpenStack +API testing. To make this possible, **Rally** automates and unifies multi-node +OpenStack deployment, cloud verification, benchmarking & profiling. This is a +simple way to check cloud workability and performance of control plane +operations running on it. + +Test Case 1: Boot and delete server +----------------------------------- + +Description +^^^^^^^^^^^ + +The most user-facing control plane operation is new virtual machine creation. +This scenario covers the most basic OpenStack server creation to present the +baseline numbers for Nova (OpenStack Compute) control plane. + +Parameters +^^^^^^^^^^ + ++-------------------------+-----------------------------------------+ +|Name | Description | ++=========================+=========================================+ +|IMAGE | Image from which boot server | ++-------------------------+-----------------------------------------+ +|FLAVOR | Flavor type from which boot server | ++-------------------------+-----------------------------------------+ +|ASSIGN_NIC | Bool, whether or not to auto assign | +| | NICs in Rally scenario | ++-------------------------+-----------------------------------------+ +|CONCURRENCY | Amount of parallel executors | ++-------------------------+-----------------------------------------+ +|ITERATIONS | Total amount of iterations processed by | +| | all executors | ++-------------------------+-----------------------------------------+ + + +List of performance metrics +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + ++----------+-------+-------------------+---------------------------+ +| Priority | Value | Measurement Units | Description | ++==========+=======+===================+===========================+ +| 1 | | sec | Time of atomic operations | ++----------+-------+-------------------+---------------------------+ + +Measuring performance values +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +1. Create server with FLAVOR flavor from IMAGE image through Nova API +2. Delete server through Nova API. + +These 2 steps executed successively in CONCURRENCY parallel executors. +ASSIGN_NIC parameter reflects Rally scenario configuration whether to assign +NIC to the booted server in automatic fashion. + +One cycle of these 2 steps is called an iteration. +ITERATIONS is a total amount of iterations which was processed by executors. + +At the end of this test case you should calculate average, 90% percentile, +50% percentile, minimum and maximum for each step. You need to fill the +following tables with calculated values: + +Nova +---- + ++---------------+------+--------+--------+-------+-------+ +| Operation | Mean | 90%ile | 50%ile | Max | Min | +| | (sec)| (sec) | (sec) | (sec) | (sec) | ++===============+======+========+========+=======+=======+ +| create_server | | | | | | ++---------------+------+--------+--------+-------+-------+ +| delete_server | | | | | | ++---------------+------+--------+--------+-------+-------+ + +Example of Rally scenario configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. literalinclude:: test_plans/021-nova-boot-and-delete-baseline.yaml + :language: bash + +Test Case 2: Boot and delete server with security groups +-------------------------------------------------------- + +Description +^^^^^^^^^^^ + +The most user-facing control plane operation is new virtual machine creation. +At the same time security groups management is very time consuming operation +in case of lots VMs attached to the same security group, therefore it's vital +to understand these operations performance. Standard Rally scanario is used for +this purpose (comparing with +:ref:`openstack_control_plane_performance_test_plan` fuel-ccp does not support +yet Cinder installation or appropriate Nova configuration to support live +migrations, so there is no need to write separated plugin to cover these +operations). + +Parameters +^^^^^^^^^^ + ++-------------------------+-----------------------------------------+ +|Name | Description | ++=========================+=========================================+ +|IMAGE | Image from which boot server | ++-------------------------+-----------------------------------------+ +|FLAVOR | Flavor type from which boot server | ++-------------------------+-----------------------------------------+ +|SEC_GROUP_COUNT | Count of security groups | +| | to be created in one iteration | ++-------------------------+-----------------------------------------+ +|RULES_PER_SECURITY_GROUP | Count of rules to be added to | +| | each security group | ++-------------------------+-----------------------------------------+ +|CONCURRENCY | Amount of parallel executors | ++-------------------------+-----------------------------------------+ +|ITERATIONS | Total amount of iterations processed by | +| | all executors | ++-------------------------+-----------------------------------------+ + + +List of performance metrics +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + ++----------+-------+-------------------+---------------------------+ +| Priority | Value | Measurement Units | Description | ++==========+=======+===================+===========================+ +| 1 | | sec | Time of atomic operations | ++----------+-------+-------------------+---------------------------+ + +Measuring performance values +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +1. Successively create SEC_GROUP_COUNT security groups through Nova API. + Duration of this step represent time that control plane process + create_SEC_GROUP_COUNT_security_groups atomic operation. +2. Successively create RULES_PER_SECURITY_GROUP rules for security + groups through Nova API. +3. Create server with FLAVOR flavor from IMAGE image through Nova API +4. List all security groups through Nova API. +5. Delete server through Nova API. +6. Successively delete SEC_GROUP_COUNT security group through Nova API. + +These 6 steps executed successively in CONCURRENCY parallel executors. +One cycle of these 6 steps is called an iteration. + +ITERATIONS is a total amount of iterations which was processed by executors. + +At the end of this test case you should calculate average, 90% percentile, +50% percentile, minimum and maximum for each step. You need to fill the +following tables with calculated values: + +Neutron +------- + ++--------------------------+------+--------+--------+-------+-------+ +| Operation | Mean | 90%ile | 50%ile | Max | Min | +| | (sec)| (sec) | (sec) | (sec) | (sec) | ++==========================+======+========+========+=======+=======+ +| create_N_security_groups | | | | | | ++--------------------------+------+--------+--------+-------+-------+ +| delete_N_security_groups | | | | | | ++--------------------------+------+--------+--------+-------+-------+ +| create_M_rules | | | | | | ++--------------------------+------+--------+--------+-------+-------+ +| delete_M_rules | | | | | | ++--------------------------+------+--------+--------+-------+-------+ + +.. note:: + Change operation name to appropriate regarding SEC_GROUP_COUNT and + RULES_PER_SECURITY_GROUP values. + +Nova +---- + ++---------------+------+--------+--------+-------+-------+ +| Operation | Mean | 90%ile | 50%ile | Max | Min | +| | (sec)| (sec) | (sec) | (sec) | (sec) | ++===============+======+========+========+=======+=======+ +| create_server | | | | | | ++---------------+------+--------+--------+-------+-------+ +| delete_server | | | | | | ++---------------+------+--------+--------+-------+-------+ + +Example of Rally scenario configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. literalinclude:: test_plans/041-nova-boot-and-delete-with-secgroups-baseline.yaml + :language: bash + + +Test Case 3: Boot and list servers +---------------------------------- + +Description +^^^^^^^^^^^ + +This scenario covers density aspect of server creation control plane +operation and checks how many virtual machines can be booted on top +of containerized OpenStack. + +Parameters +^^^^^^^^^^ + ++-------------------------+-----------------------------------------+ +|Name | Description | ++=========================+=========================================+ +|IMAGE | Image from which boot server | ++-------------------------+-----------------------------------------+ +|FLAVOR | Flavor type from which boot server | ++-------------------------+-----------------------------------------+ +|CONCURRENCY | Amount of parallel executors | ++-------------------------+-----------------------------------------+ +|ITERATIONS | Total amount of iterations processed by | +| | all executors | ++-------------------------+-----------------------------------------+ + +List of performance metrics +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + ++----------+-------+-------------------+---------------------------+ +| Priority | Value | Measurement Units | Description | ++==========+=======+===================+===========================+ +| 1 | | sec | Time of atomic operations | ++----------+-------+-------------------+---------------------------+ + +Measuring performance values +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +1. Create server with FLAVOR flavor from IMAGE image through Nova API +2. List all existing servers through Nova API. + +These 2 steps executed successively in CONCURRENCY parallel executors. + +One cycle of these 2 steps is called an iteration. +ITERATIONS is a total amount of iterations which was processed by executors. + +At the end of this test case you should calculate average, 90% percentile, +50% percentile, minimum and maximum for each step. You need to fill the +following tables with calculated values: + +Nova +---- + ++---------------+------+--------+--------+-------+-------+ +| Operation | Mean | 90%ile | 50%ile | Max | Min | +| | (sec)| (sec) | (sec) | (sec) | (sec) | ++===============+======+========+========+=======+=======+ +| create_server | | | | | | ++---------------+------+--------+--------+-------+-------+ +| list_servers | | | | | | ++---------------+------+--------+--------+-------+-------+ + +Example of Rally scenario configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. literalinclude:: test_plans/051-nova-boot-and-list-baseline.yaml + :language: bash + +Test Case 4: Create and delete image +------------------------------------ + +Description +^^^^^^^^^^^ + +To cover Glance control plane operations simple create and delete image +scenario can be used. + +Parameters +^^^^^^^^^^ + ++-----------------+-----------------------------------------+ +|Name | Description | ++=================+=========================================+ +|IMAGE | Image to upload to glance | ++-----------------+-----------------------------------------+ +|CONTAINER_FORMAT | Container format to create | ++-----------------+-----------------------------------------+ +|DISK_FORMAT | Disk format to create | ++-----------------+-----------------------------------------+ +|CONCURRENCY | Amount of parallel executors | ++-----------------+-----------------------------------------+ +|ITERATIONS | Total amount of iterations processed by | +| | all executors | ++-----------------+-----------------------------------------+ + +List of performance metrics +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + ++----------+-------+-------------------+---------------------------+ +| Priority | Value | Measurement Units | Description | ++==========+=======+===================+===========================+ +| 1 | | sec | Time of atomic operations | ++----------+-------+-------------------+---------------------------+ + +Measuring performance values +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +1. Create image from IMAGE with CONTAINER_FORMAT container format and + DISK_FORMAT disk format through Glance API. +2. Delete image from Glance through Glance API. + +These 2 steps executed successively in CONCURRENCY parallel executors. +One cycle of thee 2 steps is called an iteration. + +ITERATIONS is a total amount of iterations which was processed by executors. + +At the end of this test case you should calculate average, 90% percentile, +50% percentile, minimum and maximum for each step. You need to fill the +following tables with calculated values: + +Glance +------ + ++--------------+------+--------+--------+-------+-------+ +| Operation | Mean | 90%ile | 50%ile | Max | Min | +| | (sec)| (sec) | (sec) | (sec) | (sec) | ++==============+======+========+========+=======+=======+ +| create_image | | | | | | ++--------------+------+--------+--------+-------+-------+ +| delete_image | | | | | | ++--------------+------+--------+--------+-------+-------+ + +Example of Rally scenario configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. literalinclude:: test_plans/011-glance-create-and-delete-1g-image.yaml + :language: bash + +Test Case 5: Create and list images +----------------------------------- + +Description +^^^^^^^^^^^ + +To cover Glance control plane operations simple create and list images scenario +can be used. + +Parameters +^^^^^^^^^^ + ++-----------------+-----------------------------------------+ +|Name | Description | ++=================+=========================================+ +|IMAGE | Image to upload to glance | ++-----------------+-----------------------------------------+ +|CONTAINER_FORMAT | Container format to create | ++-----------------+-----------------------------------------+ +|DISK_FORMAT | Disk format to create | ++-----------------+-----------------------------------------+ +|CONCURRENCY | Amount of parallel executors | ++-----------------+-----------------------------------------+ +|ITERATIONS | Total amount of iterations processed by | +| | all executors | ++-----------------+-----------------------------------------+ + +List of performance metrics +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + ++----------+-------+-------------------+---------------------------+ +| Priority | Value | Measurement Units | Description | ++==========+=======+===================+===========================+ +| 1 | | sec | Time of atomic operations | ++----------+-------+-------------------+---------------------------+ + +Measuring performance values +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +1. Create image from IMAGE with CONTAINER_FORMAT container format and + DISK_FORMAT disk format through Glance API. +2. List existing images through Glance API. + +These 2 steps executed successively in CONCURRENCY parallel executors. +One cycle of thee 2 steps is called an iteration. + +ITERATIONS is a total amount of iterations which was processed by executors. + +At the end of this test case you should calculate average, 90% percentile, +50% percentile, minimum and maximum for each step. You need to fill the +following tables with calculated values: + +Glance +------ + ++--------------+------+--------+--------+-------+-------+ +| Operation | Mean | 90%ile | 50%ile | Max | Min | +| | (sec)| (sec) | (sec) | (sec) | (sec) | ++==============+======+========+========+=======+=======+ +| create_image | | | | | | ++--------------+------+--------+--------+-------+-------+ +| list_images | | | | | | ++--------------+------+--------+--------+-------+-------+ + +Example of Rally scenario configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. literalinclude:: test_plans/012-glance-create-and-list-image.yaml + :language: bash + +Test case 6: Keystone authentication +------------------------------------ + +Description +^^^^^^^^^^^ + +To cover Keystone control plane operations simple authenticate +scenario can be used. + +Parameters +^^^^^^^^^^ + ++------------------+-----------------------------------------+ +| Name | Description | ++==================+=========================================+ +| RPS | Generated load | ++------------------+-----------------------------------------+ +| ITERATIONS | Total amount of iterations processed by | +| | all executors | ++------------------+-----------------------------------------+ + +Measuring performance values +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +1. Authenticate in Keystone through Keystone API. + +This step is executed in parallel on multiple executors to generate +RPS load. + +Execution of this step is called an iteration. + +ITERATIONS is a total amount of iterations which was processed by executors. + +At the end of this test case you should calculate average, 90% percentile, +50% percentile, minimum and maximum for each step. You need to fill the +following tables with calculated values: + +Keystone +-------- + ++--------------+------+--------+--------+-------+-------+ +| Operation | Mean | 90%ile | 50%ile | Max | Min | +| | (sec)| (sec) | (sec) | (sec) | (sec) | ++==============+======+========+========+=======+=======+ +| authenticate | | | | | | ++--------------+------+--------+--------+-------+-------+ + +List of performance metrics +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + ++----------+-------+-------------------+---------------------------+ +| Priority | Value | Measurement Units | Description | ++==========+=======+===================+===========================+ +| 1 | | sec | Time of atomic operations | ++----------+-------+-------------------+---------------------------+ + +Example of Rally scenario configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. literalinclude:: test_plans/001-keystone-authenticate-90-rps.yaml + :language: bash + +Reports +======= + +Test plan execution reports: + +* :ref:`fuel_ccp_test_report` diff --git a/doc/source/test_plans/fuel_ccp/test_plans/000-keystone-authenticate-60-rps.yaml b/doc/source/test_plans/fuel_ccp/test_plans/000-keystone-authenticate-60-rps.yaml new file mode 100644 index 0000000..39dcf63 --- /dev/null +++ b/doc/source/test_plans/fuel_ccp/test_plans/000-keystone-authenticate-60-rps.yaml @@ -0,0 +1,14 @@ +--- + Authenticate.keystone: + - + runner: + type: "rps" + times: 120000 + rps: 60 + context: + users: + tenants: 10 + users_per_tenant: 10 + sla: + failure_rate: + max: 0 diff --git a/doc/source/test_plans/fuel_ccp/test_plans/001-keystone-authenticate-90-rps.yaml b/doc/source/test_plans/fuel_ccp/test_plans/001-keystone-authenticate-90-rps.yaml new file mode 100644 index 0000000..bd1054a --- /dev/null +++ b/doc/source/test_plans/fuel_ccp/test_plans/001-keystone-authenticate-90-rps.yaml @@ -0,0 +1,14 @@ +--- + Authenticate.keystone: + - + runner: + type: "rps" + times: 120000 + rps: 90 + context: + users: + tenants: 10 + users_per_tenant: 10 + sla: + failure_rate: + max: 0 diff --git a/doc/source/test_plans/fuel_ccp/test_plans/002-keystone-authenticate-120-rps.yaml b/doc/source/test_plans/fuel_ccp/test_plans/002-keystone-authenticate-120-rps.yaml new file mode 100644 index 0000000..53dfa1d --- /dev/null +++ b/doc/source/test_plans/fuel_ccp/test_plans/002-keystone-authenticate-120-rps.yaml @@ -0,0 +1,14 @@ +--- + Authenticate.keystone: + - + runner: + type: "rps" + times: 120000 + rps: 120 + context: + users: + tenants: 10 + users_per_tenant: 10 + sla: + failure_rate: + max: 0 diff --git a/doc/source/test_plans/fuel_ccp/test_plans/003-keystone-authenticate-baseline-rps.yaml b/doc/source/test_plans/fuel_ccp/test_plans/003-keystone-authenticate-baseline-rps.yaml new file mode 100644 index 0000000..85823aa --- /dev/null +++ b/doc/source/test_plans/fuel_ccp/test_plans/003-keystone-authenticate-baseline-rps.yaml @@ -0,0 +1,14 @@ +--- + Authenticate.keystone: + - + runner: + type: "rps" + times: 120000 + rps: 180 + context: + users: + tenants: 10 + users_per_tenant: 10 + sla: + failure_rate: + max: 0 diff --git a/doc/source/test_plans/fuel_ccp/test_plans/010-glance-create-and-delete-cirros-image.yaml b/doc/source/test_plans/fuel_ccp/test_plans/010-glance-create-and-delete-cirros-image.yaml new file mode 100644 index 0000000..2a1d214 --- /dev/null +++ b/doc/source/test_plans/fuel_ccp/test_plans/010-glance-create-and-delete-cirros-image.yaml @@ -0,0 +1,18 @@ +--- + GlanceImages.create_and_delete_image: + - + args: + image_location: "{{ image_server_url }}/cirros.img" + container_format: "bare" + disk_format: "qcow2" + runner: + type: "constant" + times: 400 + concurrency: 50 + context: + users: + tenants: 2 + users_per_tenant: 3 + sla: + failure_rate: + max: 0 diff --git a/doc/source/test_plans/fuel_ccp/test_plans/011-glance-create-and-delete-1g-image.yaml b/doc/source/test_plans/fuel_ccp/test_plans/011-glance-create-and-delete-1g-image.yaml new file mode 100644 index 0000000..143478f --- /dev/null +++ b/doc/source/test_plans/fuel_ccp/test_plans/011-glance-create-and-delete-1g-image.yaml @@ -0,0 +1,18 @@ +--- + GlanceImages.create_and_delete_image: + - + args: + image_location: "{{ image_server_url }}/1g.img" + container_format: "bare" + disk_format: "qcow2" + runner: + type: "constant" + times: 400 + concurrency: 50 + context: + users: + tenants: 2 + users_per_tenant: 3 + sla: + failure_rate: + max: 0 diff --git a/doc/source/test_plans/fuel_ccp/test_plans/012-glance-create-and-list-image.yaml b/doc/source/test_plans/fuel_ccp/test_plans/012-glance-create-and-list-image.yaml new file mode 100644 index 0000000..d3c846c --- /dev/null +++ b/doc/source/test_plans/fuel_ccp/test_plans/012-glance-create-and-list-image.yaml @@ -0,0 +1,18 @@ +--- + GlanceImages.create_and_list_image: + - + args: + image_location: "{{ image_server_url }}/cirros.img" + container_format: "bare" + disk_format: "qcow2" + runner: + type: "constant" + times: 200 + concurrency: 50 + context: + users: + tenants: 2 + users_per_tenant: 3 + sla: + failure_rate: + max: 0 diff --git a/doc/source/test_plans/fuel_ccp/test_plans/020-nova-boot-and-delete-check.yaml b/doc/source/test_plans/fuel_ccp/test_plans/020-nova-boot-and-delete-check.yaml new file mode 100644 index 0000000..fd68a14 --- /dev/null +++ b/doc/source/test_plans/fuel_ccp/test_plans/020-nova-boot-and-delete-check.yaml @@ -0,0 +1,35 @@ +--- + NovaServers.boot_and_delete_server: + - + args: + flavor: + name: "^scaletest$" + image: + name: "^cirros$" + force_delete: false + runner: + type: "constant" + times: {{ 1 * computes_count }} + concurrency: 5 + context: + users: + tenants: 10 + users_per_tenant: 10 + quotas: + nova: + instances: -1 + cores: -1 + ram: -1 + floating_ips: -1 + security_groups: -1 + security_group_rules: -1 + neutron: + network: -1 + security_group: -1 + security_group_rule: -1 + port: -1 + subnet: -1 + router: -1 + sla: + failure_rate: + max: 0 diff --git a/doc/source/test_plans/fuel_ccp/test_plans/021-nova-boot-and-delete-baseline.yaml b/doc/source/test_plans/fuel_ccp/test_plans/021-nova-boot-and-delete-baseline.yaml new file mode 100644 index 0000000..10fab19 --- /dev/null +++ b/doc/source/test_plans/fuel_ccp/test_plans/021-nova-boot-and-delete-baseline.yaml @@ -0,0 +1,35 @@ +--- + NovaServers.boot_and_delete_server: + - + args: + flavor: + name: "^scaletest$" + image: + name: "^cirros$" + force_delete: false + runner: + type: "constant" + times: {{ 5 * computes_count }} + concurrency: 5 + context: + users: + tenants: 10 + users_per_tenant: 10 + quotas: + nova: + instances: -1 + cores: -1 + ram: -1 + floating_ips: -1 + security_groups: -1 + security_group_rules: -1 + neutron: + network: -1 + security_group: -1 + security_group_rule: -1 + port: -1 + subnet: -1 + router: -1 + sla: + failure_rate: + max: 0 diff --git a/doc/source/test_plans/fuel_ccp/test_plans/022-nova-boot-and-delete-burn.yaml b/doc/source/test_plans/fuel_ccp/test_plans/022-nova-boot-and-delete-burn.yaml new file mode 100644 index 0000000..aa6a1a1 --- /dev/null +++ b/doc/source/test_plans/fuel_ccp/test_plans/022-nova-boot-and-delete-burn.yaml @@ -0,0 +1,35 @@ +--- + NovaServers.boot_and_delete_server: + - + args: + flavor: + name: "^scaletest$" + image: + name: "^cirros$" + force_delete: false + runner: + type: "constant" + times: {{ 40 * computes_count }} + concurrency: 10 + context: + users: + tenants: 10 + users_per_tenant: 10 + quotas: + nova: + instances: -1 + cores: -1 + ram: -1 + floating_ips: -1 + security_groups: -1 + security_group_rules: -1 + neutron: + network: -1 + security_group: -1 + security_group_rule: -1 + port: -1 + subnet: -1 + router: -1 + sla: + failure_rate: + max: 0 diff --git a/doc/source/test_plans/fuel_ccp/test_plans/030-nova-boot-and-delete-auto-nic-check.yaml b/doc/source/test_plans/fuel_ccp/test_plans/030-nova-boot-and-delete-auto-nic-check.yaml new file mode 100644 index 0000000..8f5b9c4 --- /dev/null +++ b/doc/source/test_plans/fuel_ccp/test_plans/030-nova-boot-and-delete-auto-nic-check.yaml @@ -0,0 +1,38 @@ +--- + NovaServers.boot_and_delete_server: + - + args: + flavor: + name: "^scaletest$" + image: + name: "^cirros$" + auto_assign_nic: true + runner: + type: "constant" + times: {{ 1 * computes_count }} + concurrency: 5 + context: + users: + tenants: 10 + users_per_tenant: 10 + quotas: + nova: + instances: -1 + cores: -1 + ram: -1 + floating_ips: -1 + security_groups: -1 + security_group_rules: -1 + neutron: + network: -1 + security_group: -1 + security_group_rule: -1 + port: -1 + subnet: -1 + router: -1 + network: + start_cidr: "{{ network_cidr }}" + networks_per_tenant: 2 + sla: + failure_rate: + max: 0 diff --git a/doc/source/test_plans/fuel_ccp/test_plans/031-nova-boot-and-delete-auto-nic-baseline.yaml b/doc/source/test_plans/fuel_ccp/test_plans/031-nova-boot-and-delete-auto-nic-baseline.yaml new file mode 100644 index 0000000..300f4dd --- /dev/null +++ b/doc/source/test_plans/fuel_ccp/test_plans/031-nova-boot-and-delete-auto-nic-baseline.yaml @@ -0,0 +1,38 @@ +--- + NovaServers.boot_and_delete_server: + - + args: + flavor: + name: "^scaletest$" + image: + name: "^cirros$" + auto_assign_nic: true + runner: + type: "constant" + times: {{ 5 * computes_count }} + concurrency: 5 + context: + users: + tenants: 10 + users_per_tenant: 10 + quotas: + nova: + instances: -1 + cores: -1 + ram: -1 + floating_ips: -1 + security_groups: -1 + security_group_rules: -1 + neutron: + network: -1 + security_group: -1 + security_group_rule: -1 + port: -1 + subnet: -1 + router: -1 + network: + start_cidr: "{{ network_cidr }}" + networks_per_tenant: 2 + sla: + failure_rate: + max: 0 diff --git a/doc/source/test_plans/fuel_ccp/test_plans/032-nova-boot-and-delete-auto-nic-burn.yaml b/doc/source/test_plans/fuel_ccp/test_plans/032-nova-boot-and-delete-auto-nic-burn.yaml new file mode 100644 index 0000000..0ed0673 --- /dev/null +++ b/doc/source/test_plans/fuel_ccp/test_plans/032-nova-boot-and-delete-auto-nic-burn.yaml @@ -0,0 +1,38 @@ +--- + NovaServers.boot_and_delete_server: + - + args: + flavor: + name: "^scaletest$" + image: + name: "^cirros$" + auto_assign_nic: true + runner: + type: "constant" + times: {{ 40 * computes_count }} + concurrency: 10 + context: + users: + tenants: 10 + users_per_tenant: 10 + quotas: + nova: + instances: -1 + cores: -1 + ram: -1 + floating_ips: -1 + security_groups: -1 + security_group_rules: -1 + neutron: + network: -1 + security_group: -1 + security_group_rule: -1 + port: -1 + subnet: -1 + router: -1 + network: + start_cidr: "{{ network_cidr }}" + networks_per_tenant: 2 + sla: + failure_rate: + max: 0 diff --git a/doc/source/test_plans/fuel_ccp/test_plans/040-nova-boot-and-delete-with-secgroups-check.yaml b/doc/source/test_plans/fuel_ccp/test_plans/040-nova-boot-and-delete-with-secgroups-check.yaml new file mode 100644 index 0000000..e12e9fc --- /dev/null +++ b/doc/source/test_plans/fuel_ccp/test_plans/040-nova-boot-and-delete-with-secgroups-check.yaml @@ -0,0 +1,40 @@ +--- + NovaSecGroup.boot_and_delete_server_with_secgroups: + - + args: + flavor: + name: "^scaletest$" + image: + name: "^cirros$" + security_group_count: 2 + rules_per_security_group: 10 + runner: + type: "constant" + times: {{ 1 * computes_count }} + concurrency: 5 + context: + users: + # it's important to have only 1 tenant to have single network to load + tenants: 1 + users_per_tenant: 10 + network: + start_cidr: "{{ network_cidr }}" + networks_per_tenant: 1 + quotas: + nova: + instances: -1 + cores: -1 + ram: -1 + floating_ips: -1 + security_groups: -1 + security_group_rules: -1 + neutron: + network: -1 + security_group: -1 + security_group_rule: -1 + port: -1 + subnet: -1 + router: -1 + sla: + failure_rate: + max: 0 diff --git a/doc/source/test_plans/fuel_ccp/test_plans/041-nova-boot-and-delete-with-secgroups-baseline.yaml b/doc/source/test_plans/fuel_ccp/test_plans/041-nova-boot-and-delete-with-secgroups-baseline.yaml new file mode 100644 index 0000000..965b45a --- /dev/null +++ b/doc/source/test_plans/fuel_ccp/test_plans/041-nova-boot-and-delete-with-secgroups-baseline.yaml @@ -0,0 +1,40 @@ +--- + NovaSecGroup.boot_and_delete_server_with_secgroups: + - + args: + flavor: + name: "^scaletest$" + image: + name: "^cirros$" + security_group_count: 2 + rules_per_security_group: 10 + runner: + type: "constant" + times: {{ 5 * computes_count }} + concurrency: 5 + context: + users: + # it's important to have only 1 tenant to have single network to load + tenants: 1 + users_per_tenant: 10 + network: + start_cidr: "{{ network_cidr }}" + networks_per_tenant: 1 + quotas: + nova: + instances: -1 + cores: -1 + ram: -1 + floating_ips: -1 + security_groups: -1 + security_group_rules: -1 + neutron: + network: -1 + security_group: -1 + security_group_rule: -1 + port: -1 + subnet: -1 + router: -1 + sla: + failure_rate: + max: 0 diff --git a/doc/source/test_plans/fuel_ccp/test_plans/042-nova-boot-and-delete-with-secgroups-burn.yaml b/doc/source/test_plans/fuel_ccp/test_plans/042-nova-boot-and-delete-with-secgroups-burn.yaml new file mode 100644 index 0000000..cdd194e --- /dev/null +++ b/doc/source/test_plans/fuel_ccp/test_plans/042-nova-boot-and-delete-with-secgroups-burn.yaml @@ -0,0 +1,39 @@ +--- + NovaSecGroup.boot_and_delete_server_with_secgroups: + - + args: + flavor: + name: "^scaletest$" + image: + name: "^cirros$" + security_group_count: 2 + rules_per_security_group: 10 + runner: + type: "constant" + times: {{ 40 * computes_count }} + concurrency: 10 + context: + users: + # it's important to have only 1 tenant to have single network to load + tenants: 1 + users_per_tenant: 10 + network: + start_cidr: "{{ network_cidr }}" + quotas: + nova: + instances: -1 + cores: -1 + ram: -1 + floating_ips: -1 + security_groups: -1 + security_group_rules: -1 + neutron: + network: -1 + security_group: -1 + security_group_rule: -1 + port: -1 + subnet: -1 + router: -1 + sla: + failure_rate: + max: 0 diff --git a/doc/source/test_plans/fuel_ccp/test_plans/050-nova-boot-and-list-check.yaml b/doc/source/test_plans/fuel_ccp/test_plans/050-nova-boot-and-list-check.yaml new file mode 100644 index 0000000..72e7e19 --- /dev/null +++ b/doc/source/test_plans/fuel_ccp/test_plans/050-nova-boot-and-list-check.yaml @@ -0,0 +1,35 @@ +--- + NovaServers.boot_and_list_server: + - + args: + flavor: + name: "^scaletest$" + image: + name: "^cirros$" + detailed: True + runner: + type: "constant" + times: {{ 1 * computes_count }} + concurrency: 5 + context: + users: + tenants: 10 + users_per_tenant: 10 + quotas: + nova: + instances: -1 + cores: -1 + ram: -1 + floating_ips: -1 + security_groups: -1 + security_group_rules: -1 + neutron: + network: -1 + security_group: -1 + security_group_rule: -1 + port: -1 + subnet: -1 + router: -1 + sla: + failure_rate: + max: 0 diff --git a/doc/source/test_plans/fuel_ccp/test_plans/051-nova-boot-and-list-baseline.yaml b/doc/source/test_plans/fuel_ccp/test_plans/051-nova-boot-and-list-baseline.yaml new file mode 100644 index 0000000..b478a7e --- /dev/null +++ b/doc/source/test_plans/fuel_ccp/test_plans/051-nova-boot-and-list-baseline.yaml @@ -0,0 +1,35 @@ +--- + NovaServers.boot_and_list_server: + - + args: + flavor: + name: "^scaletest$" + image: + name: "^cirros$" + detailed: True + runner: + type: "constant" + times: {{ 5 * computes_count }} + concurrency: 5 + context: + users: + tenants: 10 + users_per_tenant: 10 + quotas: + nova: + instances: -1 + cores: -1 + ram: -1 + floating_ips: -1 + security_groups: -1 + security_group_rules: -1 + neutron: + network: -1 + security_group: -1 + security_group_rule: -1 + port: -1 + subnet: -1 + router: -1 + sla: + failure_rate: + max: 0 diff --git a/doc/source/test_plans/fuel_ccp/test_plans/052-nova-boot-and-list-burn.yaml b/doc/source/test_plans/fuel_ccp/test_plans/052-nova-boot-and-list-burn.yaml new file mode 100644 index 0000000..5ab3b98 --- /dev/null +++ b/doc/source/test_plans/fuel_ccp/test_plans/052-nova-boot-and-list-burn.yaml @@ -0,0 +1,35 @@ +--- + NovaServers.boot_and_list_server: + - + args: + flavor: + name: "^scaletest$" + image: + name: "^cirros$" + detailed: True + runner: + type: "constant" + times: {{ 40 * computes_count }} + concurrency: 10 + context: + users: + tenants: 10 + users_per_tenant: 10 + quotas: + nova: + instances: -1 + cores: -1 + ram: -1 + floating_ips: -1 + security_groups: -1 + security_group_rules: -1 + neutron: + network: -1 + security_group: -1 + security_group_rule: -1 + port: -1 + subnet: -1 + router: -1 + sla: + failure_rate: + max: 0 diff --git a/doc/source/test_plans/fuel_ccp/test_plans/053-nova-boot-and-list-superburn.yaml b/doc/source/test_plans/fuel_ccp/test_plans/053-nova-boot-and-list-superburn.yaml new file mode 100644 index 0000000..2edc045 --- /dev/null +++ b/doc/source/test_plans/fuel_ccp/test_plans/053-nova-boot-and-list-superburn.yaml @@ -0,0 +1,35 @@ +--- + NovaServers.boot_and_list_server: + - + args: + flavor: + name: "^scaletest$" + image: + name: "^cirros$" + detailed: True + runner: + type: "constant" + times: {{ 40 * computes_count }} + concurrency: 40 + context: + users: + tenants: 10 + users_per_tenant: 10 + quotas: + nova: + instances: -1 + cores: -1 + ram: -1 + floating_ips: -1 + security_groups: -1 + security_group_rules: -1 + neutron: + network: -1 + security_group: -1 + security_group_rule: -1 + port: -1 + subnet: -1 + router: -1 + sla: + failure_rate: + max: 0 diff --git a/doc/source/test_plans/fuel_ccp/test_plans/054-nova-boot-and-list-hellburn.yaml b/doc/source/test_plans/fuel_ccp/test_plans/054-nova-boot-and-list-hellburn.yaml new file mode 100644 index 0000000..fe28777 --- /dev/null +++ b/doc/source/test_plans/fuel_ccp/test_plans/054-nova-boot-and-list-hellburn.yaml @@ -0,0 +1,35 @@ +--- + NovaServers.boot_and_list_server: + - + args: + flavor: + name: "^scaletest$" + image: + name: "^cirros$" + detailed: True + runner: + type: "constant" + times: {{ 80 * computes_count }} + concurrency: 80 + context: + users: + tenants: 10 + users_per_tenant: 10 + quotas: + nova: + instances: -1 + cores: -1 + ram: -1 + floating_ips: -1 + security_groups: -1 + security_group_rules: -1 + neutron: + network: -1 + security_group: -1 + security_group_rule: -1 + port: -1 + subnet: -1 + router: -1 + sla: + failure_rate: + max: 0 diff --git a/doc/source/test_results/fuel_ccp/configs/Network_Scheme.png b/doc/source/test_results/fuel_ccp/configs/Network_Scheme.png new file mode 100644 index 0000000..5c800df Binary files /dev/null and b/doc/source/test_results/fuel_ccp/configs/Network_Scheme.png differ diff --git a/doc/source/test_results/fuel_ccp/index.rst b/doc/source/test_results/fuel_ccp/index.rst new file mode 100644 index 0000000..08b153d --- /dev/null +++ b/doc/source/test_results/fuel_ccp/index.rst @@ -0,0 +1,373 @@ +.. _`fuel_ccp_test_report`: + +=================================================== +Fuel Containerized Control Plane performance report +=================================================== + +:Abstract: + + This document includes Fuel CCP control plane performance test results for + various environments (from 50 to 350 nodes). All tests have been performed + regarding :ref:`fuel_ccp_test_plan` + +Environment description +======================= + +For Kubernetes pre-deployment `Kargo tool`_ was used. More information about +*fuel-ccp* and how it can be installed can be found in +`official documentation`_. + +.. _Kargo tool: https://github.com/kubespray/kargo +.. _official documentation: http://fuel-ccp.readthedocs.io/en/latest/ + +Hardware configuration of each server +------------------------------------- + +All servers have 3 types of configuration describing in table below + +.. table:: Description of servers hardware type 1 + + +-------+----------------+---------------------------------+ + |server |vendor,model |Dell PowerEdge R630 | + +-------+----------------+---------------------------------+ + |CPU |vendor,model |Intel,E5-2680 v3 | + | +----------------+---------------------------------+ + | |processor_count |2 | + | +----------------+---------------------------------+ + | |core_count |12 | + | +----------------+---------------------------------+ + | |frequency_MHz |2500 | + +-------+----------------+---------------------------------+ + |RAM |vendor,model |Samsung, M393A2G40DB0-CPB | + | +----------------+---------------------------------+ + | |amount_MB |262144 | + +-------+----------------+---------------------------------+ + |NETWORK|interface_name s|eno1, eno2 | + | +----------------+---------------------------------+ + | |vendor,model |Intel,X710 Dual Port | + | +----------------+---------------------------------+ + | |bandwidth |10G | + | +----------------+---------------------------------+ + | |interface_names |enp3s0f0, enp3s0f1 | + | +----------------+---------------------------------+ + | |vendor,model |Intel,X710 Dual Port | + | +----------------+---------------------------------+ + | |bandwidth |10G | + +-------+----------------+---------------------------------+ + |STORAGE|dev_name |/dev/sda | + | +----------------+---------------------------------+ + | |vendor,model | | raid1 - Dell, PERC H730P Mini | + | | | | 2 disks Intel S3610 | + | +----------------+---------------------------------+ + | |SSD/HDD |SSD | + | +----------------+---------------------------------+ + | |size | 3,6TB | + +-------+----------------+---------------------------------+ + +.. table:: Description of servers hardware type 2 + + +-------+----------------+-------------------------------+ + |server |vendor,model |Lenovo ThinkServer RD550 | + +-------+----------------+-------------------------------+ + |CPU |vendor,model |Intel,E5-2680 v3 | + | +----------------+-------------------------------+ + | |processor_count |2 | + | +----------------+-------------------------------+ + | |core_count |12 | + | +----------------+-------------------------------+ + | |frequency_MHz |2500 | + +-------+----------------+-------------------------------+ + |RAM |vendor,model |Samsung, M393A2G40DB0-CPB | + | +----------------+-------------------------------+ + | |amount_MB |262144 | + +-------+----------------+-------------------------------+ + |NETWORK|interface_name s|enp3s0f0, enp3s0f1 | + | +----------------+-------------------------------+ + | |vendor,model |Intel,X710 Dual Port | + | +----------------+-------------------------------+ + | |bandwidth |10G | + | +----------------+-------------------------------+ + | |interface_names |ens2f0, ens2f1 | + | +----------------+-------------------------------+ + | |vendor,model |Intel,X710 Dual Port | + | +----------------+-------------------------------+ + | |bandwidth |10G | + +-------+----------------+-------------------------------+ + |STORAGE|dev_name |/dev/sda | + | +----------------+-------------------------------+ + | |vendor,model | | raid1 - Lenovo 720ix | + | | | | 2 disks Intel S3610 | + | +----------------+-------------------------------+ + | |SSD/HDD |SSD | + | +----------------+-------------------------------+ + | |size |799GB | + +-------+----------------+-------------------------------+ + + .. table:: Description of servers hardware type 3 + + +-------+----------------+-------------------------------+ + |server |vendor,model |Lenovo ThinkServer RD650 | + +-------+----------------+-------------------------------+ + |CPU |vendor,model |Intel,E5-2670 v3 | + | +----------------+-------------------------------+ + | |processor_count |2 | + | +----------------+-------------------------------+ + | |core_count |12 | + | +----------------+-------------------------------+ + | |frequency_MHz |2500 | + +-------+----------------+-------------------------------+ + |RAM |vendor,model |Samsung, M393A2G40DB0-CPB | + | +----------------+-------------------------------+ + | |amount_MB |131916 | + +-------+----------------+-------------------------------+ + |NETWORK|interface_name s|enp3s0f0, enp3s0f1 | + | +----------------+-------------------------------+ + | |vendor,model |Intel,X710 Dual Port | + | +----------------+-------------------------------+ + | |bandwidth |10G | + | +----------------+-------------------------------+ + | |interface_names |ens2f0, ens2f1 | + | +----------------+-------------------------------+ + | |vendor,model |Intel,X710 Dual Port | + | +----------------+-------------------------------+ + | |bandwidth |10G | + +-------+----------------+-------------------------------+ + |STORAGE|dev_name |/dev/sda | + | +----------------+-------------------------------+ + | |vendor,model | | raid1 - Lenovo 720ix | + | | | | 2 disks Intel S3610 | + | +----------------+-------------------------------+ + | |SSD/HDD |SSD | + | +----------------+-------------------------------+ + | |size |799GB | + | +----------------+-------------------------------+ + | |dev_name |/dev/sdb | + | +----------------+-------------------------------+ + | |vendor,model | | raid10 - Lenovo 720ix | + | | | | 10 disks 2T | + | +----------------+-------------------------------+ + | |SSD/HDD |HDD | + | +----------------+-------------------------------+ + | |size |9999GB | + +-------+----------------+-------------------------------+ + +Network configuration of each server +------------------------------------ + +All servers have same network configuration: + +.. image:: configs/Network_Scheme.png + :alt: Network Scheme of the environment + :width: 650px + +Test results +============ + +Test Case 1: Boot and delete server +----------------------------------- + +The following set of results is dedicated to investigate how Nova installed +against Kubernetes cluster via fuel-ccp tool is behaving on various scale. + +200 nodes OpenStack cluster, concurrency 5, 910 iterations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +NovaServers.boot_and_delete_server scenario in +:download:`nova_200_nodes.html ` + +.. image:: reports/200_nodes_boot_and_delete.png + :alt: Boot and delete servers Rally scenario (200 nodes) + :width: 650px + ++----------------------+-----------+-----------+-----------+-----------+-----------+ +| Operation | Median| 90%ile | 95%ile | Max | Min | +| | (sec) | (sec) | (sec) | (sec) | (sec) | ++======================+===========+===========+===========+===========+===========+ +| boot_server | 4.001 | 5.001 | 5.135 | 7.686 | 3.785 | ++----------------------+-----------+-----------+-----------+-----------+-----------+ +| delete_server | 2.346 | 2.487 | 2.517 | 3.769 | 2.297 | ++----------------------+-----------+-----------+-----------+-----------+-----------+ + +350 nodes OpenStack cluster, concurrency 5, 1750 iterations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +NovaServers.boot_and_delete_server scenario in +:download:`nova_350_nodes.html ` + +.. image:: reports/350_nodes_boot_and_delete.png + :alt: Boot and delete servers Rally scenario (350 nodes) + :width: 650px + ++----------------------+-----------+-----------+-----------+-----------+-----------+ +| Operation | Median| 90%ile | 95%ile | Max | Min | +| | (sec) | (sec) | (sec) | (sec) | (sec) | ++======================+===========+===========+===========+===========+===========+ +| boot_server | 12.679 | 16.853 | 18.539 | 71.071 | 3.899 | ++----------------------+-----------+-----------+-----------+-----------+-----------+ +| delete_server | 6.261 | 8.365 | 8.613 | 14.747 | 0.842 | ++----------------------+-----------+-----------+-----------+-----------+-----------+ + +Test Case 2: Boot and list servers +---------------------------------- + +The following set of results is dedicated to investigate how Nova installed +against Kubernetes cluster via fuel-ccp tool is behaving on various scale. + +150 nodes OpenStack cluster, concurrency 20, 5520 iterations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Tested density: 40 VMs per compute node + +NovaServers.boot_and_list_server scenario in +:download:`nova_150_nodes_20.html ` + +.. image:: reports/150_nodes_boot_and_list_20.png + :alt: Boot and list servers Rally scenario (150 nodes) + :width: 650px + ++----------------------+-----------+-----------+-----------+-----------+-----------+ +| Operation | Median| 90%ile | 95%ile | Max | Min | +| | (sec) | (sec) | (sec) | (sec) | (sec) | ++======================+===========+===========+===========+===========+===========+ +| list_servers | 4.381 | 7.135 | 7.69 | 172.788 | 0.105 | ++----------------------+-----------+-----------+-----------+-----------+-----------+ +| boot_server | 16.931 | 21.05 | 22.203 | 102.507 | 4.717 | ++----------------------+-----------+-----------+-----------+-----------+-----------+ + +150 nodes OpenStack cluster, concurrency 40, 5520 iterations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Tested density: 40 VMs per compute node + +NovaServers.boot_and_list_server scenario in +:download:`nova_150_nodes_40.html ` + +.. image:: reports/150_nodes_boot_and_list_40.png + :alt: Boot and list servers Rally scenario (150 nodes) + :width: 650px + ++----------------------+-----------+-----------+-----------+-----------+-----------+ +| Operation | Median| 90%ile | 95%ile | Max | Min | +| | (sec) | (sec) | (sec) | (sec) | (sec) | ++======================+===========+===========+===========+===========+===========+ +| list_servers | 4.656 | 7.607 | 7.963 | 166.783 | 0.188 | ++----------------------+-----------+-----------+-----------+-----------+-----------+ +| boot_server | 19.773 | 25.958 | 29.396 | 197.92 | 9.574 | ++----------------------+-----------+-----------+-----------+-----------+-----------+ + +200 nodes OpenStack cluster, concurrency 5, 910 iterations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Tested density: 5 VMs per compute node + +NovaServers.boot_and_list_server scenario in +:download:`nova_200_nodes.html ` + +.. image:: reports/200_nodes_boot_and_list_5.png + :alt: Boot and list servers Rally scenario (200 nodes) + :width: 650px + ++----------------------+-----------+-----------+-----------+-----------+-----------+ +| Operation | Median| 90%ile | 95%ile | Max | Min | +| | (sec) | (sec) | (sec) | (sec) | (sec) | ++======================+===========+===========+===========+===========+===========+ +| list_servers | 0.272 | 0.424 | 0.476 | 0.644 | 0.106 | ++----------------------+-----------+-----------+-----------+-----------+-----------+ +| boot_server | 4.017 | 5.037 | 5.13 | 5.437 | 3.794 | ++----------------------+-----------+-----------+-----------+-----------+-----------+ + +200 nodes OpenStack cluster, concurrency 40, 7280 iterations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Tested density: 40 VMs per compute node + +NovaServers.boot_and_list_server [2] scenario in +:download:`nova_200_nodes.html ` + +.. image:: reports/200_nodes_boot_and_list_40.png + :alt: Boot and list servers Rally scenario (200 nodes) + :width: 650px + ++----------------------+-----------+-----------+-----------+-----------+-----------+ +| Operation | Median| 90%ile | 95%ile | Max | Min | +| | (sec) | (sec) | (sec) | (sec) | (sec) | ++======================+===========+===========+===========+===========+===========+ +| list_servers | 4.237 | 5.944 | 7.994 | 130.39 | 0.111 | ++----------------------+-----------+-----------+-----------+-----------+-----------+ +| boot_server | 11.978 | 15.767 | 17.764 | 55.431 | 3.911 | ++----------------------+-----------+-----------+-----------+-----------+-----------+ + +Test case 3: Keystone authentication +------------------------------------ + +The following set of results is dedicated to investigate how Keystone installed +against Kubernetes cluster via fuel-ccp tool is behaving under various requests +per second load. For more information the full Rally report can be used: +:download:`keystone.html `. Failed scenarios are +related either to the Keystone configuration tuning under Containerized Control +Plane repository or to the huge enough RPS being set for all in one Keystone. + +50 nodes OpenStack cluster, 30 RPS, 12000 iterations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Authenticate.keystone [4] scenario in +:download:`keystone.html ` + ++--------------+-----------+----------+-----------+----------+-----------+ +| Operation | Median| 90%ile | 95%ile | Max | Min | +| | (sec) | (sec) | (sec) | (sec) | (sec) | ++==============+===========+==========+===========+==========+===========+ +| authenticate | 0.08 | 0.102 | 0.113 | 1.015 | 0.06 | ++--------------+-----------+----------+-----------+----------+-----------+ + +50 nodes OpenStack cluster, 60 RPS, 12000 iterations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Authenticate.keystone [6] scenario in +:download:`keystone.html ` + ++--------------+-----------+----------+-----------+----------+-----------+ +| Operation | Median| 90%ile | 95%ile | Max | Min | +| | (sec) | (sec) | (sec) | (sec) | (sec) | ++==============+===========+==========+===========+==========+===========+ +| authenticate | 0.097 | 0.132 | 0.147 | 1.113 | 0.073 | ++--------------+-----------+----------+-----------+----------+-----------+ + +50 nodes OpenStack cluster, 90 RPS, 12000 iterations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Authenticate.keystone [5] scenario in +:download:`keystone.html ` + ++--------------+-----------+----------+-----------+----------+-----------+ +| Operation | Median| 90%ile | 95%ile | Max | Min | +| | (sec) | (sec) | (sec) | (sec) | (sec) | ++==============+===========+==========+===========+==========+===========+ +| authenticate | 0.128 | 0.179 | 0.199 | 1.111 | 0.082 | ++--------------+-----------+----------+-----------+----------+-----------+ + +50 nodes OpenStack cluster, 120 RPS, 12000 iterations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Authenticate.keystone [7] scenario in +:download:`keystone.html ` + ++--------------+-----------+----------+-----------+----------+-----------+ +| Operation | Median| 90%ile | 95%ile | Max | Min | +| | (sec) | (sec) | (sec) | (sec) | (sec) | ++==============+===========+==========+===========+==========+===========+ +| authenticate | 0.195 | 0.269 | 0.298 | 1.455 | 0.096 | ++--------------+-----------+----------+-----------+----------+-----------+ + +50 nodes OpenStack cluster, 150 RPS, 12000 iterations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Authenticate.keystone scenario in +:download:`keystone.html ` + ++--------------+-----------+----------+-----------+----------+-----------+ +| Operation | Median| 90%ile | 95%ile | Max | Min | +| | (sec) | (sec) | (sec) | (sec) | (sec) | ++==============+===========+==========+===========+==========+===========+ +| authenticate | 0.478 | 0.738 | 0.817 | 2.024 | 0.109 | ++--------------+-----------+----------+-----------+----------+-----------+ diff --git a/doc/source/test_results/fuel_ccp/reports/150_nodes_boot_and_list_20.png b/doc/source/test_results/fuel_ccp/reports/150_nodes_boot_and_list_20.png new file mode 100644 index 0000000..360814a Binary files /dev/null and b/doc/source/test_results/fuel_ccp/reports/150_nodes_boot_and_list_20.png differ diff --git a/doc/source/test_results/fuel_ccp/reports/150_nodes_boot_and_list_40.png b/doc/source/test_results/fuel_ccp/reports/150_nodes_boot_and_list_40.png new file mode 100644 index 0000000..8c5f2af Binary files /dev/null and b/doc/source/test_results/fuel_ccp/reports/150_nodes_boot_and_list_40.png differ diff --git a/doc/source/test_results/fuel_ccp/reports/200_nodes_boot_and_delete.png b/doc/source/test_results/fuel_ccp/reports/200_nodes_boot_and_delete.png new file mode 100644 index 0000000..38e3f6a Binary files /dev/null and b/doc/source/test_results/fuel_ccp/reports/200_nodes_boot_and_delete.png differ diff --git a/doc/source/test_results/fuel_ccp/reports/200_nodes_boot_and_list_40.png b/doc/source/test_results/fuel_ccp/reports/200_nodes_boot_and_list_40.png new file mode 100644 index 0000000..3c36b23 Binary files /dev/null and b/doc/source/test_results/fuel_ccp/reports/200_nodes_boot_and_list_40.png differ diff --git a/doc/source/test_results/fuel_ccp/reports/200_nodes_boot_and_list_5.png b/doc/source/test_results/fuel_ccp/reports/200_nodes_boot_and_list_5.png new file mode 100644 index 0000000..bb6f130 Binary files /dev/null and b/doc/source/test_results/fuel_ccp/reports/200_nodes_boot_and_list_5.png differ diff --git a/doc/source/test_results/fuel_ccp/reports/350_nodes_boot_and_delete.png b/doc/source/test_results/fuel_ccp/reports/350_nodes_boot_and_delete.png new file mode 100644 index 0000000..07df016 Binary files /dev/null and b/doc/source/test_results/fuel_ccp/reports/350_nodes_boot_and_delete.png differ diff --git a/doc/source/test_results/fuel_ccp/reports/keystone_tuning.html b/doc/source/test_results/fuel_ccp/reports/keystone_tuning.html new file mode 100644 index 0000000..03bc1c0 --- /dev/null +++ b/doc/source/test_results/fuel_ccp/reports/keystone_tuning.html @@ -0,0 +1,873 @@ + + + + + + + Rally | Rally Task Report + + + + + + + + + + + + + +
+
+ Rally  + task results +
+
+ +
+ + + + +
+
+
Task overview
+
Input file
+
+
+
+ ► + ▼ + {{n.cls}}
+
{{m.name}}
+
+
+ +
+ +
+

Task overview

+ + + + + + + + + +
+ Scenario + + ▴ + ▾ + + + Load duration (s) + + ▴ + ▾ + + + Full duration (s) + + ▴ + ▾ + + + Iterations + + ▴ + ▾ + + + Runner + + ▴ + ▾ + + + Errors + + ▴ + ▾ + + + Success (SLA) + + ▴ + ▾ + +
{{sc.ref}} + {{sc.load_duration | number:3}} + {{sc.full_duration | number:3}} + {{sc.iterations_count}} + {{sc.runner}} + {{sc.errors.length}} + + ✔ + ✖ +
+
+ +
+

Input file

+ 
{{source}}
+
+ +
+

{{scenario.cls}}.{{scenario.name}} ({{scenario.full_duration | number:3}}s)

+
    +
  • +
    {{t.name}}
    +
    • +
    +
+
+ + + + + + + + + + +
+ +
+
+ + +
+ + + + \ No newline at end of file diff --git a/doc/source/test_results/fuel_ccp/reports/nova_150_nodes_20.html b/doc/source/test_results/fuel_ccp/reports/nova_150_nodes_20.html new file mode 100644 index 0000000..5bc0613 --- /dev/null +++ b/doc/source/test_results/fuel_ccp/reports/nova_150_nodes_20.html @@ -0,0 +1,873 @@ + + + + + + + Rally | Rally Task Report + + + + + + + + + + + + + +
+
+ Rally  + task results +
+
+ +
+ + + + +
+
+
Task overview
+
Input file
+
+
+
+ ► + ▼ + {{n.cls}}
+
{{m.name}}
+
+
+ +
+ +
+

Task overview

+ + + + + + + + + +
+ Scenario + + ▴ + ▾ + + + Load duration (s) + + ▴ + ▾ + + + Full duration (s) + + ▴ + ▾ + + + Iterations + + ▴ + ▾ + + + Runner + + ▴ + ▾ + + + Errors + + ▴ + ▾ + + + Success (SLA) + + ▴ + ▾ + +
{{sc.ref}} + {{sc.load_duration | number:3}} + {{sc.full_duration | number:3}} + {{sc.iterations_count}} + {{sc.runner}} + {{sc.errors.length}} + + ✔ + ✖ +
+
+ +
+

Input file

+ 
{{source}}
+
+ +
+

{{scenario.cls}}.{{scenario.name}} ({{scenario.full_duration | number:3}}s)

+
    +
  • +
    {{t.name}}
    +
    • +
    +
+
+ + + + + + + + + + +
+ +
+
+ + +
+ + + + \ No newline at end of file diff --git a/doc/source/test_results/fuel_ccp/reports/nova_150_nodes_40.html b/doc/source/test_results/fuel_ccp/reports/nova_150_nodes_40.html new file mode 100644 index 0000000..a20c9d5 --- /dev/null +++ b/doc/source/test_results/fuel_ccp/reports/nova_150_nodes_40.html @@ -0,0 +1,873 @@ + + + + + + + Rally | Rally Task Report + + + + + + + + + + + + + +
+
+ Rally  + task results +
+
+ +
+ + + + +
+
+
Task overview
+
Input file
+
+
+
+ ► + ▼ + {{n.cls}}
+
{{m.name}}
+
+
+ +
+ +
+

Task overview

+ + + + + + + + + +
+ Scenario + + ▴ + ▾ + + + Load duration (s) + + ▴ + ▾ + + + Full duration (s) + + ▴ + ▾ + + + Iterations + + ▴ + ▾ + + + Runner + + ▴ + ▾ + + + Errors + + ▴ + ▾ + + + Success (SLA) + + ▴ + ▾ + +
{{sc.ref}} + {{sc.load_duration | number:3}} + {{sc.full_duration | number:3}} + {{sc.iterations_count}} + {{sc.runner}} + {{sc.errors.length}} + + ✔ + ✖ +
+
+ +
+

Input file

+ 
{{source}}
+
+ +
+

{{scenario.cls}}.{{scenario.name}} ({{scenario.full_duration | number:3}}s)

+
    +
  • +
    {{t.name}}
    +
    • +
    +
+
+ + + + + + + + + + +
+ +
+
+ + +
+ + + + \ No newline at end of file diff --git a/doc/source/test_results/fuel_ccp/reports/nova_200_nodes.html b/doc/source/test_results/fuel_ccp/reports/nova_200_nodes.html new file mode 100644 index 0000000..bb9f0a5 --- /dev/null +++ b/doc/source/test_results/fuel_ccp/reports/nova_200_nodes.html @@ -0,0 +1,873 @@ + + + + + + + Rally | Rally Task Report + + + + + + + + + + + + + +
+
+ Rally  + task results +
+
+ +
+ + + + +
+
+
Task overview
+
Input file
+
+
+
+ ► + ▼ + {{n.cls}}
+
{{m.name}}
+
+
+ +
+ +
+

Task overview

+ + + + + + + + + +
+ Scenario + + ▴ + ▾ + + + Load duration (s) + + ▴ + ▾ + + + Full duration (s) + + ▴ + ▾ + + + Iterations + + ▴ + ▾ + + + Runner + + ▴ + ▾ + + + Errors + + ▴ + ▾ + + + Success (SLA) + + ▴ + ▾ + +
{{sc.ref}} + {{sc.load_duration | number:3}} + {{sc.full_duration | number:3}} + {{sc.iterations_count}} + {{sc.runner}} + {{sc.errors.length}} + + ✔ + ✖ +
+
+ +
+

Input file

+ 
{{source}}
+
+ +
+

{{scenario.cls}}.{{scenario.name}} ({{scenario.full_duration | number:3}}s)

+
    +
  • +
    {{t.name}}
    +
    • +
    +
+
+ + + + + + + + + + +
+ +
+
+ + +
+ + + + \ No newline at end of file diff --git a/doc/source/test_results/fuel_ccp/reports/nova_350_nodes.html b/doc/source/test_results/fuel_ccp/reports/nova_350_nodes.html new file mode 100644 index 0000000..1fdf4b5 --- /dev/null +++ b/doc/source/test_results/fuel_ccp/reports/nova_350_nodes.html @@ -0,0 +1,873 @@ + + + + + + + Rally | Rally Task Report + + + + + + + + + + + + + +
+
+ Rally  + task results +
+
+ +
+ + + + +
+
+
Task overview
+
Input file
+
+
+
+ ► + ▼ + {{n.cls}}
+
{{m.name}}
+
+
+ +
+ +
+

Task overview

+ + + + + + + + + +
+ Scenario + + ▴ + ▾ + + + Load duration (s) + + ▴ + ▾ + + + Full duration (s) + + ▴ + ▾ + + + Iterations + + ▴ + ▾ + + + Runner + + ▴ + ▾ + + + Errors + + ▴ + ▾ + + + Success (SLA) + + ▴ + ▾ + +
{{sc.ref}} + {{sc.load_duration | number:3}} + {{sc.full_duration | number:3}} + {{sc.iterations_count}} + {{sc.runner}} + {{sc.errors.length}} + + ✔ + ✖ +
+
+ +
+

Input file

+ 
{{source}}
+
+ +
+

{{scenario.cls}}.{{scenario.name}} ({{scenario.full_duration | number:3}}s)

+
    +
  • +
    {{t.name}}
    +
    • +
    +
+
+ + + + + + + + + + +
+ +
+
+ + +
+ + + + \ No newline at end of file