vitrage/doc/source/contributor/manual-tests.rst

Vitrage manual tests

General

The purpose of these tests is to manually check functionality that is not tested using tempest tests, and to double-check the correctness and validity of a devstack in general.

Vitrage Health

Services

Services Status

Run

sudo systemctl status devstack@vitrage-graph.service
sudo systemctl status devstack@vitrage-persistor.service
sudo systemctl status devstack@vitrage-notifier.service
sudo systemctl status devstack@vitrage-api.service

Expected result

Make sure that the status is active.

Check the logs for WARNING/ERROR/Exception/traceback

Services Restart

Run

sudo service devstack@vitrage-graph restart
sudo service devstack@vitrage-notifier restart
sudo service devstack@vitrage-persistor restart

Expected result

Make sure the restart is quick and does not reach a timeout

Services Information

Run

vitrage service list

Expected result

+----------------------------------+------------+-------------+---------------------------+
| Name                             | Process Id | Hostname    | Created At                |
+----------------------------------+------------+-------------+---------------------------+
| ApiWorker_worker_0               |       1084 | my-devstack | 2019-03-13T14:31:46+00:00 |
| EvaluatorWorker_worker_0         |       1082 | my-devstack | 2019-03-13T14:31:46+00:00 |
| MachineLearningService_worker_0  |       5956 | my-devstack | 2019-03-13T10:30:54+00:00 |
| PersistorService_worker_0        |      22536 | my-devstack | 2019-03-13T14:14:15+00:00 |
| SnmpParsingService_worker_0      |       6170 | my-devstack | 2019-03-13T10:30:56+00:00 |
| VitrageNotifierService_worker_0  |      22746 | my-devstack | 2019-03-13T14:14:27+00:00 |
| vitrageuWSGI_worker_1            |       2847 | my-devstack | 2019-03-13T10:30:47+00:00 |
| vitrageuWSGI_worker_2            |       2848 | my-devstack | 2019-03-13T10:30:47+00:00 |
+----------------------------------+------------+-------------+---------------------------+

Processes

Run

ps -aux | grep vitrage-graph

Expected result

vitrage-graph: master process

vitrage-graph: EvaluatorWorker

vitrage-graph: ApiWorker

There may be more than one EvaluatorWorker processes, according to the definition in vitrage.conf (the default is one worker).

Healthcheck API

Run

vitrage healthcheck

Expected result

+----------+---------+
| Field    | Value   |
+----------+---------+
| detailed | False   |
| reasons  | ['OK'] |
+----------+---------+

Vitrage Help

Run

vitrage -h

Expected result

Should display all Vitrage commands.

CLI commands

Most of the functionality is covered in tempest tests, but we have no automatic tests for the CLI itself.

Topology

Run

vitrage topology show

Expected result

Should display a graph with Vitrage entities and relationships.

Templates

Template Validate

What to execute	Expected results
vitrage template validate	Error, --path is required
vitrage template validate -path ./v1_template.yaml	Validation failed - Unknown template type
vitrage template validate --path ./v1_template.yaml --type standard	Template validation is OK
vitrage template validate --path ./v1_template.yaml --type definition	Validation failed, definition template can not contain scenarios block
vitrage template validate --path ./v2_high_cpu_load.yaml	Template validation is OK
vitrage template validate --path .	Validates all templates in the path. Some succeed and some fail.
vitrage template validate --path ./v2_definition_template.yaml	Template validation is OK
vitrage template validate --path v2_equivalence.yaml	No validation
vitrage template validate --path v3_high_mem_consumption.yaml	Template validation is OK

Template Add

What to execute	Expected results
vitrage template list	An empty list
vitrage template add	Error, --path is required
vitrage template add --path ./v1_template.yaml	Template added with status ERROR: Unknown template type
vitrage template add --path ./v1_template.yaml --type kuku	--type: invalid choice: kuku
vitrage template add --path ./v1_template.yaml --type standard	Template added with status LOADING
vitrage template add --path ./v1_template.yaml --type standard	ERROR: Duplicate template name
vitrage template list	One template with status ACTIVE
vitrage template delete	No output
vitrage template list	An empty list
vitrage template add --path ./v2_high_cpu_load.yaml	Template added with status LOADING
vitrage template add --path ./v2_definition_template.yaml	Template added with status LOADING
vitrage template add --path ./v2_with_include.yaml	Template added with status LOADING
vitrage template add --path ./v2_with_invalid_include.yaml	ERROR: Trying to include a template that does not exist
vitrage template add --path v2_equivalence.yaml	Template added with status LOADING
vitrage template add --path v3_high_mem_consumption.yaml	Template validation is OK
vitrage template list	Five templates with status ACTIVE and one in ERROR
vitrage template show <uuid>	Shows the template json representation

Templates with parameters

What to execute	Expected results
vitrage template validate --path v3_with_params.yaml	Failed to resolve parameter - template_name
vitrage template validate --path v3_with_params.yaml --params template_name=template1	Template validation is OK
vitrage template validate --path v3_with_params.yaml --params alarm_name=alarm1	Failed to resolve parameter - template_name
vitrage template validate --path v3_with_params.yaml --params template_name=template1 alarm_name=alarm1	Template validation is OK
vitrage template add --path v3_with_params.yaml --params template_name=template1 alarm_name=alarm1	Template added with status LOADING
vitrage template add --path v2_with_params.yaml --params template_name=t1 alarm_name=a1 alarm_type=zabbix	Template added with status LOADING
vitrage template show <uuid>	Shows the template json representation. All parameters are resolved and the parameters section is removed.

Deduced Alarms and RCA

What to execute	Expected results
create an instance in Nova	An instance is created
vitrage alarm list	An empty list
vitrage event post --type="High CPU load" --details=' {"hostname": "my-devstack","source":"sample_monitor", "cause":"link-down","severity":"critical","status":"down", "monitor_id":"monitor-1","monitor_event_id":"123"}'	Make sure to use hostname that contains an instance No output
vitrage alarm list	Shows 'High CPU load' on the host and also an alarm on the instance.
vitrage alarm show <uuid>	Shows the alarm details. Verify for both alarms.
vitrage rca show <uuid>	Shows the alarm RCA graph. Verify for both alarms.
vitrage alarm count	Shows one WARNING and one CRITICAL alarm

Resources

What to execute	Expected results
vitrage resource list	Shows a list of resources from different datasources. Does not show alarms
vitrage resource list --filter ..... TBD	Shows a list of resources that match the given filter.
vitrage resource show <uuid>	Shows a the defails of the selected resource.
vitrage resource count	Shows how many resources there are of every type.

Multi Tenancy

TBD

Datasources

Note: The resources and alarms are visible only to the tenant that created them.

• For a resource that was created in the UI, check the Vitrage entity graph of the same tenant.
• For a resource that was created in the CLI, check the Vitrage entity graph of admin tenant.

Basic OpenStack datasources

What to execute	Expected results
Create an instance/volume/network/stack	A new entity immediately appears in Vitrage entity graph and is connected to the right neighbors.
Delete these resources	The entities are immediately removed from the graph

Static Topology

What to execute	Expected results
Copy switch_and_nic.yaml under /etc/vitrage/static_datasources	The switche and nic should appear in the graph within 30 seconds

Aodh

What to execute	Expected results
aodh alarm create --name 'test_1' --state alarm --event-type 'my.event' --type event --query 'traits.resource_id=string::my-devstack'	An alarm is created in Aodh with state alarm. Make sure to use as traits.resource_id the name of your devstack.
vitrage alarm list	The Aodh alarm appears and is connected to the devstack
aodh alarm create --name 'Gnocchi alarm 1' --type gnocchi_resources_threshold --resource-type instance --resource-id f9335fc1-f3bf-4915-bed2-2c7350628ac9 --metric cpu_util --threshold 0.001 --granularity 300 --state alarm --aggregation-method mean --comparison-operator gt	An alarm is created in Aodh with state alarm. Make sure to assign --resource-id with a valid instance uuid.
vitrage alarm list	Two Aodh alarms, connected to different resources
aodh alarm delete <alarm UUID>
vitrage alarm list	One Aodh alarm remains

Notifiers

Nova Notifier

What to execute	Expected results
vitrage template add --path ./host_down.yaml	Template added with status LOADING
nova service-list	The state of nova-compute is up
vitrage event post --type="compute.host.down" --details= '{"hostname": "my-devstack","source":"sample_monitor", "cause":"link-down","severity":"critical","status":"down", "monitor_id":"monitor-1","monitor_event_id":"123"}'	Make sure to use hostname of your devstack. No output.
vitrage alarm list	A compute.host.down alarm, connected to the right host
nova service-list	The state of nova-compute is down
vitrage event post --type="compute.host.down" --details= '{"hostname": "my-devstack","source":"sample_monitor", "cause":"link-down","severity":"critical","status":"up", "monitor_id":"monitor-1","monitor_event_id":"123"}'	Make sure to use hostname of your devstack. No output.
nova service-list	The state of nova-compute is up

Webhook Notifier

Configure a web client

Copy test_web_server.py under /opt/stack Run: sudo pip install web.py

Test the Webhook Notifier

What to execute	Expected results
vitrage webhook list	Empty list
python /opt/stack/test_web_server.py 8001	server started (in a different console window)
python /opt/stack/test_web_server.py 8002	server started (in a different console window)
vitrage webhook add --url http://localhost:8001/alarm	Outputs the webhook details
vitrage webhook add --url http://localhost:8002/alarm --regex '{"vitrage_type": "doctor"}'	Outputs the webhook details
vitrage webhook list	A list with the details of both webhooks
vitrage webhook show <webhook uuid>	Shows the details of the webhook
vitrage event post --type="compute.host.down" --details= '{"hostname": "compute-0-0","source":"sample_monitor", "cause":"link-down","severity":"critical","status":"down", "monitor_id":"monitor-1","monitor_event_id":"123"}'	Both webhooks print the details of compute.host.down alarm to the console. The webhook on port 8001 prints also the details of the deduced alarms to the console.
vitrage webhook delete <webhook uuid>	Deletes a webhook
vitrage webhook list	Shows only one webhook
vitrage event post --type="compute.host.down" --details= '{"hostname": "compute-0-0","source":"sample_monitor", "cause":"link-down","severity":"critical","status":"up", "monitor_id":"monitor-1","monitor_event_id":"123"}'	The deleted webhook does not print anything. The other webhook prints the cleared alarm(s).

Mistral Notifier

What to execute	Expected results
mistral workflow-create ./workflow1.yaml	The workflow is created
vitrage template add --path ./v3_execute_mistral.yaml	Template added with status LOADING
vitrage event post --type="alarm_for_mistral" --details= '{"hostname": "my-devstack","source":"sample_monitor", "cause":"link-down","severity":"critical","status":"down", "monitor_id":"monitor-1","monitor_event_id":"123"}'	Make sure to use hostname of your devstack. No output.
mistral execution-list	workflow1 is in the list.
mistral execution-get <uuid of the latest execution>	Shows details about workflow1 execution.
mistral execution-get-input <uuid of the latest execution>	"farewell": "my-devstack"

Zaqar Notifier

TBD

SNMP Notifier

TBD

UI Tests

The UI tests are included in this document, so we'll have a complete set of manual sanity tests in one place.

Alarm Banner

What to execute	Expected results
Go to compute->instances menu	The alarm banner should be on the top right corner with the correct number of alarms
Click on the alarm banner	You should be redirected to Vitrage alarms view

Alarm View

What to execute	Expected results
Raise an alarm of type doctor (examples above)	The alarm appears in the Active Alarms list
Filter By alarm type doctor	Only doctor alarms remain
Clear the doctor filter	All alarms appear
Sort by name	Alarms are sorted
Click the RCA button for the High CPU load alarm	An RCA graph displays the alarms on the host and on the instance(s). Make sure all properties are ok.
Switch to Alarm History tab	Several alarms should appear with different statuses. The alarms that are currently active should not have an Ended value.
Click the RCA button for one of the alarms	An RCA graph displays the alarm(s) as inactive.
Change the Ended filter and click Fetch Alarms	The list of alarms is different
Filter By resource type nova.host	Only alarms on the host are displayed
Go back to Active Alarms	The list of active alarms is displayed

Topology View

What to execute	Expected results
Go to Topology View	The sunburst shows the compute hierarchy in different colors.
Switch tenants	The number of instances changes accordingly
Drill down to the host and instances	The sunburst changes. On the left there may be alarms
Click the RCA button on one of the alarms	The RCA view is opened

Entity Graph

What to execute	Expected results
Open the Entity Graph, zoom in and out	The graph changes accordingly
Click the Pin button, drag 1-2 entities, and refresh	The graph is not changed
Click the Unpin button	The graph is changed
Double-click two entites to pin them and drag them to different places. Then refresh the graph.	The pinned entities stay in the same location. The rest of the graph is changed.
Click several entities, one at the time	The properties of the selected entity appear on the left panel
Write a text in the search box	All entities with the selected text are highlighted
Switch to a different tenant and see how the graph changes	There are less entities (all instances are gone)
Filter by a specific Heat stack, modify the details level	The graph changes accordingly

Template View

Template View for tenant

What to execute	Expected results
Open the Template view under Project->Vitrage, check the list of templates	Few templates appear with Template validation is OK Details. There are no Add and Delete buttons.
Write a name in the filter	Only templates with this name appear
Clear the filter	All templates appear
Click the Show icon	The template content is displayed
Expand some of the entities, relationships and scenarios	The details are displayed
Switch to Yaml View	A json representation is displayed
Switch back to Simple View	The simple view is displayed

+------------------------------------------------------------+--------------------------------------------------------+

Template View for admin

What to execute	Expected results
Open the Template view under Admin->Vitrage and do the same checks as for tenant
Delete all existing templates	Templates are deleted, the list is empty
Click Add template and add v1_template.yaml. In the Type drop-down, select definition	ERROR: A template definition can not contain includes or scenarios blocks
Click Add template and add v1_template.yaml. In the Type drop-down, select standard	The template is added with status ACTIVE
Click Add template and add v2_definition_template.yaml	The template is added with status ACTIVE
Click Add template and add v2_equivalence.yaml	The template is added with status ACTIVE and and details No Validation
Click Show icon for templates of types standard (v1, v2 and v3), equivalence and definition	All templates are displayed correctly
Click Add template and add v2_wrong.yaml	ERROR: Action definition must contain action_target field. The template is not added.

Templates with parameters

What to execute	Expected results
Click Add template and add v2_with_params.yaml	Error: Failed to resolve parameter - alarm_type The template is not added.
Click Add template and add v2_with_params.yaml. Add parameter alarm_type.	Error: Failed to resolve parameter - alarm_name The template is not added.
Click Add template and add v2_with_params.yaml. Add parameters alarm_type, alarm_name, template_name and new_state. Add and remove another parameter before clicking Done.	The template is added with status ACTIVE
Click the Show icon	There is no parameters section. All parameters are resolved with the given values.
Click Add template and add v2_with_params.yaml. Add parameters alarm_type and alarm_name only	The template is added with status ACTIVE
Click the Show icon	There is no parameters section. alarm_type and alarm_name parameters are resolved with the given values. new_state has default value of ERROR.
Perform similar tests for v3_with_params.yaml
Click Add template and add v3_with_default_params.yaml	The template is added with status ACTIVE