openstack/tripleo-quickstart-extras
Code Issues Proposed changes

Updating skip file mechanism

Browse Source 
This way to load the skip file will allow a more concise format
to be consumed not only by the skip file itself but for other
scripts

Change-Id: I09130b0e46711e1e01a6b5b5077ae9249f01c597
This commit is contained in:
Arx Cruz 2017-02-27 12:41:58 +01:00
parent 535544be37
commit 1de4b1f627
14 changed files with 252 additions and 136 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

33
roles/validate-tempest/README.md
View File

@ -27,6 +27,37 @@ Role Variables
* `tempest_failing`: false/true - default is false, run only tests known to be failing
* `tempest_exit_on_failure`: true/false - whether to exit from role with tempest exit code (default: true)
Skip tests file
---------------
Tempest test can be skipped if they are in a skip file, using the whole test
name or a regular expression.
Since the intention is to use the list of skip tests in other parts of
quickstart-extras, it's easier to have this list of skip tests in yaml format
where you can set the skip test, reason and launchpad or bugzilla. This make
easier later to check whether the test still valid or not to be skipped.
Skip files are under roles/validate-tempest/vars/tempest_skip_VERSION.yaml.
In these files you can add a regex, together with a reason and a bugzilla
or launchpad bug related to that particular test.
Launchpad and/or bugzilla re optional, however you must specify a reason why
you are skipping the test.
Example of skip file:
---
known_failures:
- test: 'tempest.scenario.test_volume_boot_pattern'
reason: 'rdo-manager tempest smoke test failing on "floating ip pool not found"'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1272289'
- test: 'tempest.api.volume.test_volumes_get'
reason: 'rdo-manager tempest smoke test failing on "floating ip pool not found"'
- test: 'tempest.api.identity.*v3'
reason: 'Tempest Identify tests failed on V3 api on OSP-D puddle'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1266947'
- test: 'tempest.api.image.v2.test_images_metadefs_namespace_properties.MetadataNamespacePropertiesTest.test_basic_meta_def_namespace_property'
reason: 'glance is not calling glance-manage db_load_metadefs'
lp: 'https://bugs.launchpad.net/tripleo/+bug/1664995'
Dependencies
------------
@ -35,8 +66,6 @@ No dependencies
Example Playbook
----------------
Including an example of how to use your role (for instance, with variables passed in as parameters) is always nice for users too:
---
- name: Run tempest
hosts: undercloud

2
roles/validate-tempest/defaults/main.yml
View File

@ -23,7 +23,7 @@ tempest_overcloud: true
run_tempest: true
post_tempest: true
tempest_format: packages # venv or packages
skip_file_src: "{{ release }}_skip_file.j2"
skip_file_src: "skip_file.j2"
skip_file: skip_file
tempest_isolated: false
tempest_until_failure: false

4
roles/validate-tempest/tasks/pre-tempest.yml
View File

@ -12,6 +12,10 @@
dest: "{{ working_dir }}/tempest-setup.sh"
mode: 0744
- name: Load skip list variables
include_vars:
file: "tempest_skip_{{ release }}.yml"
- name: Create tempest skipfile
template:
src: "{{ skip_file_src }}"

8
roles/validate-tempest/templates/liberty_skip_file.j2
View File

@ -1,8 +0,0 @@
# old bug rhbz1272289
-tempest.scenario.test_volume_boot_pattern
-tempest.api.volume.test_volumes_get
# rhbz1266947
-tempest.api.identity.*v3
# TripleO does not deploy Barbican by default, which is needed for this test to pass
-tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_cryptsetup
-tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_luks

44
roles/validate-tempest/templates/master_skip_file.j2
View File

@ -1,44 +0,0 @@
# rhbz1304933
tempest.api.telemetry.test_telemetry_notification_api
# rhbz1266947
tempest.api.identity.*v3
# rhbz1357667
tempest.api.data_processing
# lpbug 1605654
tempest.scenario.test_network_basic_ops.TestNetworkBasicOps
# rhbz1378989
tempest.api.network.test_ports.PortsIpV6TestJSON
tempest.api.network.test_ports.PortsTestJSON
# rhbz1379829
tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON
#rhbz1380146
tempest.api.orchestration.stacks.test_resource_types.ResourceTypesTest
#rhbz1380145
tempest.api.identity.admin.v2.test_services.ServicesTestJSON
#rhbz1380187
tempest.api.network.test_routers.RoutersIpV6Test.test_add_multiple_router_interfaces
# TripleO does not deploy Barbican by default, which is needed for this test to pass
tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_cryptsetup
tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_luks
#rhbz1411690
tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic
tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_multiple_security_groups
{% if not enable_cinder_backup|default(false)|bool %}
# Cinder backup is not enabled by default in tripleo only in particular
# scenarios
tempest.api.volume.admin.test_volumes_backup.VolumesBackupsAdminV1Test
tempest.api.volume.admin.test_volumes_backup.VolumesBackupsAdminV2Test
tempest.api.volume.test_volumes_backup.VolumesBackupsV1Test
tempest.api.volume.test_volumes_backup.VolumesBackupsV2Test
{% endif %}
# Glance isn't populating metadata definition database LP1664995
tempest.api.image.v2.test_images_metadefs_namespaces.MetadataNamespacesTest.test_basic_metadata_definition_namespaces
tempest.api.image.v2.test_images_metadefs_resource_types.MetadataResourceTypesTest.test_basic_meta_def_resource_type_association
tempest.api.image.v2.test_images_metadefs_namespace_properties.MetadataNamespacePropertiesTest.test_basic_meta_def_namespace_property
# Skip Mistral tests as it needs to use mistral v2 endpoints
# FIXME(chkumar246): It needs to be fixed in a new patch
mistral_tempest_tests.tests
# Skip Ceilometer tests as it required gnocchi to be configured
# FIXME(chkumar246): It needs to be fixed in a new patch
ceilometer.tests.tempest.api.test_telemetry_notification_api.TelemetryNotificationAPITest.test_check_glance_v1_notifications
ceilometer.tests.tempest.api.test_telemetry_notification_api.TelemetryNotificationAPITest.test_check_glance_v2_notifications

12
roles/validate-tempest/templates/mitaka_skip_file.j2
View File

@ -1,12 +0,0 @@
# old bug rhbz1272289
tempest.scenario.test_volume_boot_pattern
tempest.api.volume.test_volumes_get
# rhbz1266947
tempest.api.identity.*v3
#NEED BUGS or RETEST or RECONFIG
tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON.test_add_remove_fixed_ip
tempest.api.network.test_routers.RoutersIpV6Test.test_add_remove_router_interface_with_subnet_id
tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops
# TripleO does not deploy Barbican by default, which is needed for this test to pass
tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_cryptsetup
tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_luks

25
roles/validate-tempest/templates/newton_skip_file.j2
View File

@ -1,25 +0,0 @@
# rhbz1304933
tempest.api.telemetry.test_telemetry_notification_api
# old bug rhbz1272289
tempest.scenario.test_volume_boot_pattern
tempest.api.volume.test_volumes_get
# rhbz1266947
tempest.api.identity.*v3
# rhbz1357667
tempest.api.data_processing
# lpbug 1605654
tempest.scenario.test_network_basic_ops.TestNetworkBasicOps
# rhbz1386421
tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON
# TripleO does not deploy Barbican by default, which is needed for this test to pass
tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_cryptsetup
tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_luks
# rhbz1411690
tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic
tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_multiple_security_groups
# Skip Mistral tests as Mistral is not configured on Newton
mistral_tempest_tests.tests
# Skip Ceilometer tests as it required gnocchi to be configured
# FIXME(chkumar246): It needs to be fixed in a new patch
ceilometer.tests.tempest.api.test_telemetry_notification_api.TelemetryNotificationAPITest.test_check_glance_v1_notifications
ceilometer.tests.tempest.api.test_telemetry_notification_api.TelemetryNotificationAPITest.test_check_glance_v2_notifications

44
roles/validate-tempest/templates/ocata_skip_file.j2
View File

@ -1,44 +0,0 @@
# rhbz1304933
tempest.api.telemetry.test_telemetry_notification_api
# rhbz1266947
tempest.api.identity.*v3
# rhbz1357667
tempest.api.data_processing
# lpbug 1605654
tempest.scenario.test_network_basic_ops.TestNetworkBasicOps
# rhbz1378989
tempest.api.network.test_ports.PortsIpV6TestJSON
tempest.api.network.test_ports.PortsTestJSON
# rhbz1379829
tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON
#rhbz1380146
tempest.api.orchestration.stacks.test_resource_types.ResourceTypesTest
#rhbz1380145
tempest.api.identity.admin.v2.test_services.ServicesTestJSON
#rhbz1380187
tempest.api.network.test_routers.RoutersIpV6Test.test_add_multiple_router_interfaces
# TripleO does not deploy Barbican by default, which is needed for this test to pass
tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_cryptsetup
tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_luks
#rhbz1411690
tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic
tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_multiple_security_groups
{% if not enable_cinder_backup|default(false)|bool %}
# Cinder backup is not enabled by default in tripleo only in particular
# scenarios
tempest.api.volume.admin.test_volumes_backup.VolumesBackupsAdminV1Test
tempest.api.volume.admin.test_volumes_backup.VolumesBackupsAdminV2Test
tempest.api.volume.test_volumes_backup.VolumesBackupsV1Test
tempest.api.volume.test_volumes_backup.VolumesBackupsV2Test
{% endif %}
# Glance isn't populating metadata definition database LP1664995
tempest.api.image.v2.test_images_metadefs_namespaces.MetadataNamespacesTest.test_basic_metadata_definition_namespaces
tempest.api.image.v2.test_images_metadefs_resource_types.MetadataResourceTypesTest.test_basic_meta_def_resource_type_association
tempest.api.image.v2.test_images_metadefs_namespace_properties.MetadataNamespacePropertiesTest.test_basic_meta_def_namespace_property
# Skip Mistral tests as it needs to use mistral v2 endpoints
# FIXME(chkumar246): It needs to be fixed in a new patch
mistral_tempest_tests.tests
# Skip Ceilometer tests as it required gnocchi to be configured
# FIXME(chkumar246): It needs to be fixed in a new patch
ceilometer.tests.tempest.api.test_telemetry_notification_api.TelemetryNotificationAPITest.test_check_glance_v1_notifications
ceilometer.tests.tempest.api.test_telemetry_notification_api.TelemetryNotificationAPITest.test_check_glance_v2_notifications

10
roles/validate-tempest/templates/skip_file.j2 Normal file
View File

@ -0,0 +1,10 @@
{% for skip_test in known_failures %}
# {{ skip_test.reason }}
{% if skip_test.bz is defined %}
# {{ skip_test.bz }}
{% endif %}
{% if skip_test.lp is defined %}
# {{ skip_test.lp }}
{% endif %}
{{ skip_test.test }}
{% endfor %}

14
roles/validate-tempest/vars/tempest_skip_liberty.yml Normal file
View File

@ -0,0 +1,14 @@
known_failures:
- test: 'tempest.scenario.test_volume_boot_pattern'
reason: 'rdo-manager tempest smoke test failing on "floating ip pool not found"'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1272289'
- test: 'tempest.api.volume.test_volumes_get'
reason: 'rdo-manager tempest smoke test failing on "floating ip pool not found"'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1272289'
- test: 'tempest.api.identity.*v3'
reason: 'Tempest Identify tests failed on V3 api on OSP-D puddle'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1266947'
- test: 'tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_cryptsetup'
reason: 'TripleO does not deploy Barbican by default, which is needed for this test to pass'
- test: 'tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_luks'
reason: 'TripleO does not deploy Barbican by default, which is needed for this test to pass'

64
roles/validate-tempest/vars/tempest_skip_master.yml Normal file
View File

@ -0,0 +1,64 @@
known_failures:
- test: '.*test_external_network_visibility'
reason: 'Tempest test "external network visibility" fails with tripleo'
lp: 'https://bugs.launchpad.net/tripleo/+bug/1577769'
- test: 'tempest.api.data_processing'
reason: 'tempest.api.data_processing tests failing on newton'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1357667'
- test: 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
reason: 'TestNetworkBasicOps often times out running against tripleo'
lp: 'https://bugs.launchpad.net/tripleo/+bug/1605654'
- test: 'tempest.api.network.test_ports.*'
reason: 'tempest.api.network.test_ports failures'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1378989'
- test: 'tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON'
reason: 'tempest failure: tempest.api.compute.servers.test_server_actions'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1379829'
- test: 'tempest.api.orchestration.stacks.test_resource_types.ResourceTypesTest'
reason: 'tempest fail: tempest.api.orchestration.stacks.test_resource_types'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1380146'
- test: 'tempest.api.identity.admin.v2.test_services.ServicesTestJSON'
reason: 'tempest fail: tempest.api.identity.admin.v2.test_services.ServicesTestJSON'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1380145'
- test: 'tempest.api.network.test_routers.RoutersIpV6Test.test_add_multiple_router_interfaces'
reason: 'tempest fail: tempest.api.network.test_routers.RoutersIpV6Test.test_add_multiple_router_interfaces'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1380187'
- test: 'tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes'
reason: 'TripleO does not deploy Barbican by default'
- test: 'tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic'
reason: 'Security groups firewall rules do not get applied on the compute nodes'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1411690'
- test: 'tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_multiple_security_groups'
reason: 'Security groups firewall rules do not get applied on the compute nodes'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1411690'
- test: 'tempest.api.volume.admin.test_volumes_backup.VolumesBackupsAdminV1Test'
reason: 'Cinder backup is not enabled by default'
- test: 'tempest.api.volume.admin.test_volumes_backup.VolumesBackupsAdminV2Test'
reason: 'Cinder backup is not enabled by default'
- test: 'tempest.api.volume.test_volumes_backup.VolumesBackupsV1Test'
reason: 'Cinder backup is not enabled by default'
- test: 'cinder.tests.tempest.api.volume.test_volume_backup.VolumesBackupsTest'
reason: 'Cinder backup is not enabled by default'
- test: 'tempest.api.volume.test_volumes_backup.VolumesBackupsV2Test'
reason: 'Cinder backup is not enabled by default'
- test: 'tempest.api.image.v2.test_images_metadefs_namespaces.MetadataNamespacesTest.test_basic_metadata_definition_namespaces'
reason: 'glance is not calling glance-manage db_load_metadefs'
lp: 'https://bugs.launchpad.net/tripleo/+bug/1664995'
- test: 'tempest.api.image.v2.test_images_metadefs_resource_types.MetadataResourceTypesTest.test_basic_meta_def_resource_type_association'
reason: 'glance is not calling glance-manage db_load_metadefs'
lp: 'https://bugs.launchpad.net/tripleo/+bug/1664995'
- test: 'tempest.api.image.v2.test_images_metadefs_namespace_properties.MetadataNamespacePropertiesTest.test_basic_meta_def_namespace_property'
reason: 'glance is not calling glance-manage db_load_metadefs'
lp: 'https://bugs.launchpad.net/tripleo/+bug/1664995'
- test: 'mistral_tempest_tests.tests'
reason: 'Skip Mistral tests as it needs to use mistral v2 endpoints'
- test: 'ceilometer.tests.tempest.api.test_telemetry_notification_api.TelemetryNotificationAPITest'
reason: 'This telemetry installation is configured to use Gnocchi'
- test: 'ceilometer.tests.tempest.api.test_telemetry_notification_api.TelemetryNotificationAdminAPITest'
reason: 'This telemetry installation is configured to use Gnocchi'
- test: 'ceilometer.tests.tempest.scenario.test_object_storage_telemetry_middleware.TestObjectStorageTelemetry'
reason: 'This telemetry installation is configured to use Gnocchi'
- test: 'neutron.tests.tempest.scenario.test_dvr.NetworkDvrTest'
reason: 'TODO(arxcruz) New test, need investigation'
- test: 'neutron.tests.tempest.scenario.test_trunk.TrunkTest'
reason: 'TODO(arxcruz) New test, need investigation'

20
roles/validate-tempest/vars/tempest_skip_mitaka.yml Normal file
View File

@ -0,0 +1,20 @@
known_failures:
- test: 'tempest.scenario.test_volume_boot_pattern'
reason: 'rdo-manager tempest smoke test failing on "floating ip pool not found"'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1272289'
- test: 'tempest.api.volume.test_volumes_get'
reason: 'rdo-manager tempest smoke test failing on "floating ip pool not found"'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1272289'
- test: 'tempest.api.identity.*v3'
reason: 'Tempest Identify tests failed on V3 api on OSP-D puddle'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1266947'
- test: 'tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON.test_add_remove_fixed_ip'
reason: 'Need bug, retest or reconfig'
- test: 'tempest.api.network.test_routers.RoutersIpV6Test.test_add_remove_router_interface_with_subnet_id'
reason: 'Need bug, retest or reconfig'
- test: 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops'
reason: 'Need bug, retest or reconfig'
- test: 'tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_cryptsetup'
reason: 'TripleO does not deploy Barbican by default, which is needed for this test to pass'
- test: 'tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_luks'
reason: 'TripleO does not deploy Barbican by default, which is needed for this test to pass'

44
roles/validate-tempest/vars/tempest_skip_newton.yml Normal file
View File

@ -0,0 +1,44 @@
known_failures:
- test: 'tempest.api.telemetry.test_telemetry_notification_api'
reason: 'ospd-8 poodle 2016-02-04.1 tempest.api.telemetry.test_telemetry_notification_api failing'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1304933'
- test: 'tempest.scenario.test_volume_boot_pattern'
reason: 'rdo-manager tempest smoke test failing on "floating ip pool not found"'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1272289'
- test: 'tempest.api.volume.test_volumes_get'
reason: 'rdo-manager tempest smoke test failing on "floating ip pool not found"'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1272289'
- test: 'tempest.api.identity.*v3'
reason: 'Tempest Identify tests failed on V3 api on OSP-D puddle'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1266947'
- test: 'tempest.api.data_processing'
reason: 'tempest.api.data_processing tests failing on newton'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1357667'
- test: 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
reason: 'TestNetworkBasicOps often times out running against tripleo'
lp: 'https://bugs.launchpad.net/tripleo/+bug/1605654'
- test: 'tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON'
reason: 'tempest fail: tempest.api.identity.admin.v2.test_services.ServicesTestJSON'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1386421'
- test: 'tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_cryptsetup'
reason: 'TripleO does not deploy Barbican by default, which is needed for this test to pass'
- test: 'tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_luks'
reason: 'TripleO does not deploy Barbican by default, which is needed for this test to pass'
- test: 'tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic'
reason: 'Security groups firewall rules do not get applied on the compute nodes'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1411690'
- test: 'tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_multiple_security_groups'
reason: 'Security groups firewall rules do not get applied on the compute nodes'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1411690'
- test: 'mistral_tempest_tests.tests'
reason: 'Skip Mistral tests as it needs to use mistral v2 endpoints'
- test: 'ceilometer.tests.tempest.api.test_telemetry_notification_api.TelemetryNotificationAPITest'
reason: 'This telemetry installation is configured to use Gnocchi'
- test: 'ceilometer.tests.tempest.api.test_telemetry_notification_api.TelemetryNotificationAdminAPITest'
reason: 'This telemetry installation is configured to use Gnocchi'
- test: 'ceilometer.tests.tempest.scenario.test_object_storage_telemetry_middleware.TestObjectStorageTelemetry'
reason: 'This telemetry installation is configured to use Gnocchi'
- test: 'neutron.tests.tempest.scenario.test_dvr.NetworkDvrTest'
reason: 'TODO(arxcruz) New test, need investigation'
- test: 'neutron.tests.tempest.scenario.test_trunk.TrunkTest'
reason: 'TODO(arxcruz) New test, need investigation'

64
roles/validate-tempest/vars/tempest_skip_ocata.yml Normal file
View File

@ -0,0 +1,64 @@
known_failures:
- test: '.*test_external_network_visibility'
reason: 'Tempest test "external network visibility" fails with tripleo'
lp: 'https://bugs.launchpad.net/tripleo/+bug/1577769'
- test: 'tempest.api.data_processing'
reason: 'tempest.api.data_processing tests failing on newton'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1357667'
- test: 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
reason: 'TestNetworkBasicOps often times out running against tripleo'
lp: 'https://bugs.launchpad.net/tripleo/+bug/1605654'
- test: 'tempest.api.network.test_ports.*'
reason: 'tempest.api.network.test_ports failures'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1378989'
- test: 'tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON'
reason: 'tempest failure: tempest.api.compute.servers.test_server_actions'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1379829'
- test: 'tempest.api.orchestration.stacks.test_resource_types.ResourceTypesTest'
reason: 'tempest fail: tempest.api.orchestration.stacks.test_resource_types'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1380146'
- test: 'tempest.api.identity.admin.v2.test_services.ServicesTestJSON'
reason: 'tempest fail: tempest.api.identity.admin.v2.test_services.ServicesTestJSON'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1380145'
- test: 'tempest.api.network.test_routers.RoutersIpV6Test.test_add_multiple_router_interfaces'
reason: 'tempest fail: tempest.api.network.test_routers.RoutersIpV6Test.test_add_multiple_router_interfaces'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1380187'
- test: 'tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes'
reason: 'TripleO does not deploy Barbican by default'
- test: 'tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic'
reason: 'Security groups firewall rules do not get applied on the compute nodes'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1411690'
- test: 'tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_multiple_security_groups'
reason: 'Security groups firewall rules do not get applied on the compute nodes'
bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1411690'
- test: 'tempest.api.volume.admin.test_volumes_backup.VolumesBackupsAdminV1Test'
reason: 'Cinder backup is not enabled by default'
- test: 'tempest.api.volume.admin.test_volumes_backup.VolumesBackupsAdminV2Test'
reason: 'Cinder backup is not enabled by default'
- test: 'tempest.api.volume.test_volumes_backup.VolumesBackupsV1Test'
reason: 'Cinder backup is not enabled by default'
- test: 'cinder.tests.tempest.api.volume.test_volume_backup.VolumesBackupsTest'
reason: 'Cinder backup is not enabled by default'
- test: 'tempest.api.volume.test_volumes_backup.VolumesBackupsV2Test'
reason: 'Cinder backup is not enabled by default'
- test: 'tempest.api.image.v2.test_images_metadefs_namespaces.MetadataNamespacesTest.test_basic_metadata_definition_namespaces'
reason: 'glance is not calling glance-manage db_load_metadefs'
lp: 'https://bugs.launchpad.net/tripleo/+bug/1664995'
- test: 'tempest.api.image.v2.test_images_metadefs_resource_types.MetadataResourceTypesTest.test_basic_meta_def_resource_type_association'
reason: 'glance is not calling glance-manage db_load_metadefs'
lp: 'https://bugs.launchpad.net/tripleo/+bug/1664995'
- test: 'tempest.api.image.v2.test_images_metadefs_namespace_properties.MetadataNamespacePropertiesTest.test_basic_meta_def_namespace_property'
reason: 'glance is not calling glance-manage db_load_metadefs'
lp: 'https://bugs.launchpad.net/tripleo/+bug/1664995'
- test: 'mistral_tempest_tests.tests'
reason: 'Skip Mistral tests as it needs to use mistral v2 endpoints'
- test: 'ceilometer.tests.tempest.api.test_telemetry_notification_api.TelemetryNotificationAPITest'
reason: 'This telemetry installation is configured to use Gnocchi'
- test: 'ceilometer.tests.tempest.api.test_telemetry_notification_api.TelemetryNotificationAdminAPITest'
reason: 'This telemetry installation is configured to use Gnocchi'
- test: 'ceilometer.tests.tempest.scenario.test_object_storage_telemetry_middleware.TestObjectStorageTelemetry'
reason: 'This telemetry installation is configured to use Gnocchi'
- test: 'neutron.tests.tempest.scenario.test_dvr.NetworkDvrTest'
reason: 'TODO(arxcruz) New test, need investigation'
- test: 'neutron.tests.tempest.scenario.test_trunk.TrunkTest'
reason: 'TODO(arxcruz) New test, need investigation'