diff --git a/TESTING.rst b/TESTING.rst index 412d12e1d9d..4a928dd05f5 100644 --- a/TESTING.rst +++ b/TESTING.rst @@ -58,8 +58,8 @@ that broad categorization, here are a few more characteristic: (tools/configure_for_func_testing.sh). Typically test a component such as an agent using no mocks. * Integration tests - Run against a running cloud, often target the API level, - but also 'scenarios' or 'user stories'. You may find such tests under - tests/fullstack, and in the Tempest, Rally and + but also 'scenarios', 'user stories' or 'grenade'. You may find such tests + under tests/fullstack, and in the Tempest, Rally, Grenade and neutron-tempest-plugin(neutron_tempest_plugin/api|scenario) projects. Tests in the Neutron tree are typically organized by the testing infrastructure @@ -326,6 +326,24 @@ This is any devstack configuration required to make sure your environment can successfully process your scenario requests; 3) edit neutron-neutron.yaml. This is your scenario 'contract' or SLA. +Grenade Tests +~~~~~~~~~~~~~ + +Grenade is a tool to test upgrade process between OpenStack releases. It +actually not introduces any new tests but it is a tool which uses Tempest tests +to verify upgrade process between releases. +Neutron runs couple of Grenade jobs in check and gate queue - see +:ref:`CI Testing ` summary. + +You can run Grenade tests locally on the virtual machine(s). It is pretty +similar to deploying OpenStack using Devstack. All is described in the +`Project's wiki `_ and `documentation +`_. + +More info about how to troubleshoot Grenade failures in the CI jobs can be found +in the :ref:`Troubleshooting Grenade jobs ` +document. + Development Process ------------------- diff --git a/doc/source/contributor/policies/gate-failure-triage.rst b/doc/source/contributor/policies/gate-failure-triage.rst index 40c46c39a6f..f9b2cf5ff38 100644 --- a/doc/source/contributor/policies/gate-failure-triage.rst +++ b/doc/source/contributor/policies/gate-failure-triage.rst @@ -72,6 +72,47 @@ Troubleshooting functional/fullstack job from testr_results.html.gz. 5. Create an :ref:`Elastic Recheck Query ` + +.. _troubleshooting-grenade-jobs: + +Troubleshooting Grenade jobs +---------------------------- +Grenade is used in the Neutron gate to test every patch proposed to Neutron to +ensure it will not break the upgrade process. +Upgrading from the N-1 to the N branch is constantly being tested. So if you +send patch to the Neutron ``master`` branch Grenade jobs will first deploy +Neutron from the last stable release and then upgrade it to the master branch +with your patch. +Details about how Grenade works are available in +`the documentation `_. + +In Neutron CI jobs that use Grenade are run in the multinode jobs configuration +which means that we have deployed OpenStack on 2 VMs: + +- one called ``controller`` which is in an "all in one" + node so it runs neutron-server, as well as the neutron-ovs-agent and + nova-compute services, + +- one called ``compute1`` which runs only services like nova-compute and + neutron-ovs-agent. + +Neutron supports that neutron-server in N version will always work with the +agents which runs in N-1 version. To test such scenario all our Grenade jobs +upgrade OpenStack services only on the ``controller`` node. Services which run +on the ``compute1`` node are always run with the "old" release during that job. + +Debugging of failures in the Grenade job is very similar to debugging any +other Tempest based job. +The difference is that in the logs of the Grenade job, there is always +"logs/old" and "logs/new" directories which contain Devstack logs from each run +of the Devstack's stack.sh script. +In the "logs/grenade.sh_log.txt" file there is a full log of the grenade.sh run +and you should always start checking failures from that file. +Logs of the Neutron services for "old" and "new" versions are in the same files, +like, for example, "logs/screen-q-svc.txt" for neutron-server logs. You will +find in that log when the service was restarted - that is the moment when it +was upgraded by Grenade and it is now running the new version. + Advanced Troubleshooting of Gate Jobs ------------------------------------- As a first step of troubleshooting a failing gate job, you should always check diff --git a/doc/source/contributor/testing/ci_scenario_jobs.rst b/doc/source/contributor/testing/ci_scenario_jobs.rst index fc2670333f3..b1fdf304838 100644 --- a/doc/source/contributor/testing/ci_scenario_jobs.rst +++ b/doc/source/contributor/testing/ci_scenario_jobs.rst @@ -20,6 +20,7 @@ ''''''' Heading 4 (Avoid deeper levels because they do not render well.) +.. _ci_jobs: Neutron jobs running in Zuul CI ===============================