[Docs] Add doc about debugging of the Grenade jobs
Change-Id: I218a7a2584a8669a0da0a7ff328985e12f8ec92f
This commit is contained in:
parent
774908c50e
commit
59827fa840
22
TESTING.rst
22
TESTING.rst
|
@ -58,8 +58,8 @@ that broad categorization, here are a few more characteristic:
|
||||||
(tools/configure_for_func_testing.sh). Typically test a component
|
(tools/configure_for_func_testing.sh). Typically test a component
|
||||||
such as an agent using no mocks.
|
such as an agent using no mocks.
|
||||||
* Integration tests - Run against a running cloud, often target the API level,
|
* Integration tests - Run against a running cloud, often target the API level,
|
||||||
but also 'scenarios' or 'user stories'. You may find such tests under
|
but also 'scenarios', 'user stories' or 'grenade'. You may find such tests
|
||||||
tests/fullstack, and in the Tempest, Rally and
|
under tests/fullstack, and in the Tempest, Rally, Grenade and
|
||||||
neutron-tempest-plugin(neutron_tempest_plugin/api|scenario) projects.
|
neutron-tempest-plugin(neutron_tempest_plugin/api|scenario) projects.
|
||||||
|
|
||||||
Tests in the Neutron tree are typically organized by the testing infrastructure
|
Tests in the Neutron tree are typically organized by the testing infrastructure
|
||||||
|
@ -326,6 +326,24 @@ This is any devstack configuration required to make sure your environment can
|
||||||
successfully process your scenario requests; 3) edit neutron-neutron.yaml. This
|
successfully process your scenario requests; 3) edit neutron-neutron.yaml. This
|
||||||
is your scenario 'contract' or SLA.
|
is your scenario 'contract' or SLA.
|
||||||
|
|
||||||
|
Grenade Tests
|
||||||
|
~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Grenade is a tool to test upgrade process between OpenStack releases. It
|
||||||
|
actually not introduces any new tests but it is a tool which uses Tempest tests
|
||||||
|
to verify upgrade process between releases.
|
||||||
|
Neutron runs couple of Grenade jobs in check and gate queue - see
|
||||||
|
:ref:`CI Testing <ci_jobs>` summary.
|
||||||
|
|
||||||
|
You can run Grenade tests locally on the virtual machine(s). It is pretty
|
||||||
|
similar to deploying OpenStack using Devstack. All is described in the
|
||||||
|
`Project's wiki <https://wiki.openstack.org/wiki/Grenade>`_ and `documentation
|
||||||
|
<https://opendev.org/openstack/grenade/src/branch/master/README.rst>`_.
|
||||||
|
|
||||||
|
More info about how to troubleshoot Grenade failures in the CI jobs can be found
|
||||||
|
in the :ref:`Troubleshooting Grenade jobs <troubleshooting-grenade-jobs>`
|
||||||
|
document.
|
||||||
|
|
||||||
Development Process
|
Development Process
|
||||||
-------------------
|
-------------------
|
||||||
|
|
||||||
|
|
|
@ -72,6 +72,47 @@ Troubleshooting functional/fullstack job
|
||||||
from testr_results.html.gz.
|
from testr_results.html.gz.
|
||||||
5. Create an :ref:`Elastic Recheck Query <elastic-recheck-query>`
|
5. Create an :ref:`Elastic Recheck Query <elastic-recheck-query>`
|
||||||
|
|
||||||
|
|
||||||
|
.. _troubleshooting-grenade-jobs:
|
||||||
|
|
||||||
|
Troubleshooting Grenade jobs
|
||||||
|
----------------------------
|
||||||
|
Grenade is used in the Neutron gate to test every patch proposed to Neutron to
|
||||||
|
ensure it will not break the upgrade process.
|
||||||
|
Upgrading from the N-1 to the N branch is constantly being tested. So if you
|
||||||
|
send patch to the Neutron ``master`` branch Grenade jobs will first deploy
|
||||||
|
Neutron from the last stable release and then upgrade it to the master branch
|
||||||
|
with your patch.
|
||||||
|
Details about how Grenade works are available in
|
||||||
|
`the documentation <https://docs.openstack.org/grenade/latest/readme.html>`_.
|
||||||
|
|
||||||
|
In Neutron CI jobs that use Grenade are run in the multinode jobs configuration
|
||||||
|
which means that we have deployed OpenStack on 2 VMs:
|
||||||
|
|
||||||
|
- one called ``controller`` which is in an "all in one"
|
||||||
|
node so it runs neutron-server, as well as the neutron-ovs-agent and
|
||||||
|
nova-compute services,
|
||||||
|
|
||||||
|
- one called ``compute1`` which runs only services like nova-compute and
|
||||||
|
neutron-ovs-agent.
|
||||||
|
|
||||||
|
Neutron supports that neutron-server in N version will always work with the
|
||||||
|
agents which runs in N-1 version. To test such scenario all our Grenade jobs
|
||||||
|
upgrade OpenStack services only on the ``controller`` node. Services which run
|
||||||
|
on the ``compute1`` node are always run with the "old" release during that job.
|
||||||
|
|
||||||
|
Debugging of failures in the Grenade job is very similar to debugging any
|
||||||
|
other Tempest based job.
|
||||||
|
The difference is that in the logs of the Grenade job, there is always
|
||||||
|
"logs/old" and "logs/new" directories which contain Devstack logs from each run
|
||||||
|
of the Devstack's stack.sh script.
|
||||||
|
In the "logs/grenade.sh_log.txt" file there is a full log of the grenade.sh run
|
||||||
|
and you should always start checking failures from that file.
|
||||||
|
Logs of the Neutron services for "old" and "new" versions are in the same files,
|
||||||
|
like, for example, "logs/screen-q-svc.txt" for neutron-server logs. You will
|
||||||
|
find in that log when the service was restarted - that is the moment when it
|
||||||
|
was upgraded by Grenade and it is now running the new version.
|
||||||
|
|
||||||
Advanced Troubleshooting of Gate Jobs
|
Advanced Troubleshooting of Gate Jobs
|
||||||
-------------------------------------
|
-------------------------------------
|
||||||
As a first step of troubleshooting a failing gate job, you should always check
|
As a first step of troubleshooting a failing gate job, you should always check
|
||||||
|
|
|
@ -20,6 +20,7 @@
|
||||||
''''''' Heading 4
|
''''''' Heading 4
|
||||||
(Avoid deeper levels because they do not render well.)
|
(Avoid deeper levels because they do not render well.)
|
||||||
|
|
||||||
|
.. _ci_jobs:
|
||||||
|
|
||||||
Neutron jobs running in Zuul CI
|
Neutron jobs running in Zuul CI
|
||||||
===============================
|
===============================
|
||||||
|
|
Loading…
Reference in New Issue