[Docs] Add doc about debugging of the Grenade jobs

Change-Id: I218a7a2584a8669a0da0a7ff328985e12f8ec92f

TESTING.rst

@@ -58,8 +58,8 @@ that broad categorization, here are a few more characteristic:
   (tools/configure_for_func_testing.sh). Typically test a component
   such as an agent using no mocks.
 * Integration tests - Run against a running cloud, often target the API level,
-  but also 'scenarios' or 'user stories'. You may find such tests under
-  tests/fullstack, and in the Tempest, Rally and
+  but also 'scenarios', 'user stories' or 'grenade'. You may find such tests
+  under tests/fullstack, and in the Tempest, Rally, Grenade and
   neutron-tempest-plugin(neutron_tempest_plugin/api|scenario) projects.
 
 Tests in the Neutron tree are typically organized by the testing infrastructure
@@ -326,6 +326,24 @@ This is any devstack configuration required to make sure your environment can
 successfully process your scenario requests; 3) edit neutron-neutron.yaml. This
 is your scenario 'contract' or SLA.
 
+Grenade Tests
+~~~~~~~~~~~~~
+
+Grenade is a tool to test upgrade process between OpenStack releases. It
+actually not introduces any new tests but it is a tool which uses Tempest tests
+to verify upgrade process between releases.
+Neutron runs couple of Grenade jobs in check and gate queue - see
+:ref:`CI Testing <ci_jobs>` summary.
+
+You can run Grenade tests locally on the virtual machine(s). It is pretty
+similar to deploying OpenStack using Devstack. All is described in the
+`Project's wiki <https://wiki.openstack.org/wiki/Grenade>`_ and `documentation
+<https://opendev.org/openstack/grenade/src/branch/master/README.rst>`_.
+
+More info about how to troubleshoot Grenade failures in the CI jobs can be found
+in the :ref:`Troubleshooting Grenade jobs <troubleshooting-grenade-jobs>`
+document.
+
 Development Process
 -------------------
 

doc/source/contributor/policies/gate-failure-triage.rst

@@ -72,6 +72,47 @@ Troubleshooting functional/fullstack job
    from testr_results.html.gz.
 5. Create an :ref:`Elastic Recheck Query <elastic-recheck-query>`
 
+
+.. _troubleshooting-grenade-jobs:
+
+Troubleshooting Grenade jobs
+----------------------------
+Grenade is used in the Neutron gate to test every patch proposed to Neutron to
+ensure it will not break the upgrade process.
+Upgrading from the N-1 to the N branch is constantly being tested. So if you
+send patch to the Neutron ``master`` branch Grenade jobs will first deploy
+Neutron from the last stable release and then upgrade it to the master branch
+with your patch.
+Details about how Grenade works are available in
+`the documentation <https://docs.openstack.org/grenade/latest/readme.html>`_.
+
+In Neutron CI jobs that use Grenade are run in the multinode jobs configuration
+which means that we have deployed OpenStack on 2 VMs:
+
+- one called ``controller`` which is in an "all in one"
+  node so it runs neutron-server, as well as the neutron-ovs-agent and
+  nova-compute services,
+
+- one called ``compute1`` which runs only services like nova-compute and
+  neutron-ovs-agent.
+
+Neutron supports that neutron-server in N version will always work with the
+agents which runs in N-1 version. To test such scenario all our Grenade jobs
+upgrade OpenStack services only on the ``controller`` node. Services which run
+on the ``compute1`` node are always run with the "old" release during that job.
+
+Debugging of failures in the Grenade job is very similar to debugging any
+other Tempest based job.
+The difference is that in the logs of the Grenade job, there is always
+"logs/old" and "logs/new" directories which contain Devstack logs from each run
+of the Devstack's stack.sh script.
+In the "logs/grenade.sh_log.txt" file there is a full log of the grenade.sh run
+and you should always start checking failures from that file.
+Logs of the Neutron services for "old" and "new" versions are in the same files,
+like, for example, "logs/screen-q-svc.txt" for neutron-server logs. You will
+find in that log when the service was restarted - that is the moment when it
+was upgraded by Grenade and it is now running the new version.
+
 Advanced Troubleshooting of Gate Jobs
 -------------------------------------
 As a first step of troubleshooting a failing gate job, you should always check

doc/source/contributor/testing/ci_scenario_jobs.rst

@@ -20,6 +20,7 @@
       '''''''  Heading 4
       (Avoid deeper levels because they do not render well.)
 
+.. _ci_jobs:
 
 Neutron jobs running in Zuul CI
 ===============================