Merge "Spec with test cases to extend HA testing"
This commit is contained in:
Jenkins
Gerrit Code Review
commit
21349d36dd
258
specs/6.0/ha_tests.rst
Normal file
258
specs/6.0/ha_tests.rst
Normal file
@ -0,0 +1,258 @@
|
|||||||
|
|
||||||
|
==========================================
|
||||||
|
HA tests improvements
|
||||||
|
==========================================
|
||||||
|
|
||||||
|
Include the URL of your launchpad blueprint:
|
||||||
|
|
||||||
|
https://blueprints.launchpad.net/fuel/+spec/ha-test-improvements
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
Need to add new HA tests and modify the existing one
|
||||||
|
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
We need to clarify the list of new tests and new checks
|
||||||
|
and then implement it in system tests
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
No alternatives
|
||||||
|
|
||||||
|
Data model impact
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
No impact
|
||||||
|
|
||||||
|
REST API impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
No impact
|
||||||
|
|
||||||
|
Upgrade impact
|
||||||
|
--------------
|
||||||
|
|
||||||
|
No impact
|
||||||
|
|
||||||
|
Security impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
No impact
|
||||||
|
|
||||||
|
Notifications impact
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
No impact
|
||||||
|
|
||||||
|
Other end user impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
No impact
|
||||||
|
|
||||||
|
Performance Impact
|
||||||
|
------------------
|
||||||
|
|
||||||
|
No impact
|
||||||
|
|
||||||
|
Other deployer impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
No impact
|
||||||
|
|
||||||
|
Developer impact
|
||||||
|
----------------
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Can be implemented by fuel-qa team in parallel
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
1. Shut down public vip two times
|
||||||
|
(link to bug https://bugs.launchpad.net/fuel/+bug/1311749)
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Deploy HA cluster with Nova-network, 3 controllers, 2 compute
|
||||||
|
2. Find node with public vip
|
||||||
|
3. Shut down eth with public vip
|
||||||
|
4. Check vip is recovered
|
||||||
|
5. Find node on which vip is recovered
|
||||||
|
6. Shut down eth with public vip one more time
|
||||||
|
7. Check vip is recovered
|
||||||
|
8. Run OSTF
|
||||||
|
9. Do the same for management vip
|
||||||
|
|
||||||
|
2. Galera does not reassemble on galera quorum loss
|
||||||
|
(link to bug https://bugs.launchpad.net/fuel/+bug/1350545)
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Deploy HA cluster with Nova-network, 3 controllers, 2 compute
|
||||||
|
2. Shut down one controller
|
||||||
|
3. Wait for galera cluster to reassemble (HA health check has passed)
|
||||||
|
4. Kill mysqld on second controller
|
||||||
|
5. Start first controller
|
||||||
|
6. Wait for 5 minutes that galera reassembles and check it reassembles
|
||||||
|
7. Run OSTF
|
||||||
|
8. Check rabbit status with MOS script
|
||||||
|
|
||||||
|
3. Corrupt root file system on primary controller
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Deploy HA cluster with Nova-network, 3 controllers, 2 compute
|
||||||
|
2. Corrupt root file system on primary controller
|
||||||
|
3. Run OSTF
|
||||||
|
|
||||||
|
4. Block corosync traffic
|
||||||
|
(link to bug https://bugs.launchpad.net/fuel/+bug/1354520)
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Deploy HA cluster with Nova-network, 3 controllers, 2 compute
|
||||||
|
2. Login to rabbit master node
|
||||||
|
3. Block corosync traffic by extracting interface from management bridge
|
||||||
|
4. Unblock corosync traffic back
|
||||||
|
5. Check rabbitmqctl cluster_status at rabbit master node
|
||||||
|
6. Run OSTF HA tests
|
||||||
|
|
||||||
|
5. HA scalability for mongo
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Deploy HA cluster with Nova-network, 1 controller and 3 mongo nodes
|
||||||
|
2. Add 2 controller nodes and re-deploy cluster
|
||||||
|
3. Run OSTF
|
||||||
|
4. Add 2 mongo nodes and re-deploy cluster
|
||||||
|
5. Run OSTF
|
||||||
|
|
||||||
|
6. Lock DB access on primary controller
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Deploy HA cluster with Nova-network, 3 controllers, 2 compute
|
||||||
|
2. Lock DB access on primary controller
|
||||||
|
3. Run OSTF
|
||||||
|
|
||||||
|
7. Need to test HA failover on clusters with bonding
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Deploy HA cluster with Neutron Vlan, 3 controllers, 2 compute,
|
||||||
|
eth1-eth4 interfaces are bonded in active backup mode
|
||||||
|
2. Destroy primary controller
|
||||||
|
3. Check pacemaker status
|
||||||
|
4. Run OSTF
|
||||||
|
5. Check rabbit status with MOS script
|
||||||
|
(retry it during 5 min till successful result)
|
||||||
|
|
||||||
|
8. HA load testing with rally
|
||||||
|
(May be not a part of this blueprint)
|
||||||
|
|
||||||
|
9. Need to test HA Neutron cluster under high load and simultaneous
|
||||||
|
removing of virtual router ports
|
||||||
|
(related link http://lists.openstack.org/pipermail/openstack-operators/
|
||||||
|
2014-September/005165.html)
|
||||||
|
|
||||||
|
10. Cinder Neutron Plugin
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Deploy HA cluster with Neutron GRE, 3 controllers, 2 compute,
|
||||||
|
cinder-neutron plugin enabled
|
||||||
|
2. Run network verification
|
||||||
|
3. Run OSTF
|
||||||
|
|
||||||
|
11. Rmq failover test for compute service
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Deploy HA cluster with Nova-network, 3 controllers,
|
||||||
|
2 compute with cinder roles
|
||||||
|
2. Disable one compute node with
|
||||||
|
nova-manage service disable --host=<compute_node_name> --service=nova-compute
|
||||||
|
3. On controller node under test (which compute node under test is connected
|
||||||
|
to via rmq port 5673) repeat spawn / destroy instance requests continuosly
|
||||||
|
(sleep 60) while the test is running
|
||||||
|
4. Add iptables block rule from compute IP to controller IP:5673
|
||||||
|
(take care for conntrack as well)
|
||||||
|
iptables -I INPUT 1 -s compute_IP -p tcp --dport 5673 -m state
|
||||||
|
--state NEW,ESTABLISHED,RELATED -j DROP
|
||||||
|
5. Wait 3 min for compute node under test should be marked as down
|
||||||
|
in the nova service-list
|
||||||
|
6. Wait for another 3 min for it to be brought up back
|
||||||
|
7. Check for the compute node under test queue - it should be zero messages
|
||||||
|
in it
|
||||||
|
8. Check if the instance could be spawned at the node
|
||||||
|
|
||||||
|
12. Check monit on compute nodes
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Deploy HA cluster with Nova-network, 3 controllers, 2 compute
|
||||||
|
2. Ssh to every compute node
|
||||||
|
3. Kill nova-compute service
|
||||||
|
4. Check that service was restarted by monit
|
||||||
|
|
||||||
|
13. Check pacemaker restarts heat-engine in case of losing amqp connection
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Deploy HA cluster with Nova-network, 3 controllers, 2 compute
|
||||||
|
2. SSH to controller with running heat-engine
|
||||||
|
3. Check heat-engine status
|
||||||
|
4. Block heat-engine amqp connections
|
||||||
|
5. Check if heat-engine was moved to another controller or stopped
|
||||||
|
on current controller
|
||||||
|
6. If moved - ssh to node with running heat-engine
|
||||||
|
6.1 Check heat-engine is running
|
||||||
|
6.2 Check heat-engine has some amqp connections
|
||||||
|
7. If stopped - check heat-engine process is running with new pid
|
||||||
|
7.1 Unblock heat-engine amqp connections
|
||||||
|
7.2 Check amqp connection re-appears for heat-engine
|
||||||
|
|
||||||
|
|
||||||
|
14. Neutron agent rescheduling
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Deploy HA cluster with Neutron GRE, 3 controllers, 2 compute
|
||||||
|
2. Check the neutron-agents list consitency (no duplicates,
|
||||||
|
alive statuses, etc)
|
||||||
|
3. On host with l3 agent create one more router
|
||||||
|
4. Check there are 2 namespaces
|
||||||
|
5. Destroy controller with l3 agent
|
||||||
|
6. Check it was moved to another controller, check all routers
|
||||||
|
and namespaces were moved
|
||||||
|
7. Check metadata agent was also moved, there is process in router
|
||||||
|
namespace that listen to 8775 port
|
||||||
|
|
||||||
|
15. DHCP agent rescheduling
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Deploy HA cluster with Neutron GRE, 3 controllers, 2 compute
|
||||||
|
2. Destroy controller with dhcp agent
|
||||||
|
3. Check it was moved to another controller
|
||||||
|
4. Check metadata agent was also moved, there is process in router
|
||||||
|
namespace that listen to 8775 port
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Testing
|
||||||
|
=======
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Documentation Impact
|
||||||
|
====================
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
|
|
||||||
Loading…
Reference in New Issue
Block a user