Run DevStack in the gate
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

README.rst 11KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260
  1. Devstack Gate
  2. =============
  3. Devstack-gate is a collection of scripts used by the OpenStack CI team
  4. to test every change to core OpenStack projects by deploying OpenStack
  5. via devstack on a cloud server.
  6. What It Is
  7. ==========
  8. All changes to core OpenStack projects are "gated" on a set of tests
  9. so that it will not be merged into the main repository unless it
  10. passes all of the configured tests. Most projects require unit tests
  11. with pep8 and several versions of Python. Those tests are all run only
  12. on the project in question. The devstack gate test, however, is an
  13. integration test and ensures that a proposed change still enables
  14. several of the projects to work together.
  15. Obviously we test integrated OpenStack components and their clients
  16. because they all work closely together to form an OpenStack
  17. system. Changes to devstack itself are also required to pass this test
  18. so that we can be assured that devstack is always able to produce a
  19. system capable of testing the next change to nova. The devstack gate
  20. scripts themselves are included for the same reason.
  21. How It Works
  22. ============
  23. The devstack test starts with an essentially bare virtual machine,
  24. installs devstack on it, and runs tests of the resulting OpenStack
  25. installation. In order to ensure that each test run is independent,
  26. the virtual machine is discarded at the end of the run, and a new
  27. machine is used for the next run. In order to keep the actual test run
  28. as short and reliable as possible, the virtual machines are prepared
  29. ahead of time and kept in a pool ready for immediate use. The process
  30. of preparing the machines ahead of time reduces network traffic and
  31. external dependencies during the run.
  32. The `Nodepool`_ project is used to maintain this pool of machines.
  33. .. _Nodepool: https://opendev.org/zuul/nodepool
  34. How to Debug a Devstack Gate Failure
  35. ====================================
  36. When Jenkins runs gate tests for a change, it leaves comments on the
  37. change in Gerrit with a link to the resulting logs, including the
  38. console log. If a change fails in a devstack-gate test, you can follow
  39. these links to find out what went wrong. Start at the bottom of the log
  40. file with the failure, scroll up to look for errors related to failed
  41. tests.
  42. You might need some information about the specific run of the test. In
  43. the devstack-gate-setup-workspace log, you can see all the git commands
  44. used to set up the repositories, and they will output the (short) sha1
  45. and commit subjects of the head of each repository.
  46. It's possible that a failure could be a false negative related to a
  47. specific provider, especially if there is a pattern of failures from
  48. tests that run on nodes from that provider. In order to find out which
  49. provider supplied the node the test ran on, look at the name of the
  50. jenkins slave in the devstack-gate-setup-host log, the name of the
  51. provider is included.
  52. Below that, you'll find the output from devstack as it installs all of
  53. the debian and pip packages required for the test, and then configures
  54. and runs the services. Most of what it needs should already be cached
  55. on the test host, but if the change to be tested includes a dependency
  56. change, or there has been such a change since the snapshot image was
  57. created, the updated dependency will be downloaded from the Internet,
  58. which could cause a false negative if that fails.
  59. Assuming that there are no visible failures in the console log, you
  60. may need to examine the log output from the OpenStack services, located
  61. in the logs/ directory. All of the OpenStack services are configured to
  62. syslog, so you may find helpful log messages by clicking on the
  63. "syslog.txt[.gz]" file. Some error messages are so basic they don't
  64. make it to syslog, such as if a service fails to start. Devstack
  65. starts all of the services in screen, and you can see the output
  66. captured by screen in files named "screen-\*.txt". You may find a
  67. traceback there that isn't in syslog.
  68. After examining the output from the test, if you believe the result
  69. was a false negative, you can retrigger the test by running a recheck,
  70. this is done by leaving a review comment with simply the text: recheck
  71. If a test failure is a result of a race condition in the OpenStack code,
  72. you also have the opportunity to try to identify it, and file a bug report,
  73. help fix the problem or leverage `elastic-recheck
  74. <http://docs.openstack.org/infra/elastic-recheck/readme.html>`_ to help
  75. track the problem. If it seems to be related to a specific devstack gate
  76. node provider, we'd love it if you could help identify what the variable
  77. might be (whether in the devstack-gate scripts, devstack itself, Nodepool,
  78. OpenStack, or even the provider's service).
  79. Simulating Devstack Gate Tests
  80. ==============================
  81. Developers often have a need to recreate gating integration tests
  82. manually, and this provides a walkthrough of making a DG-slave-like
  83. throwaway server without the overhead of building other CI
  84. infrastructure to manage a pool of them. This can be useful to reproduce
  85. and troubleshoot failures or tease out nondeterministic bugs.
  86. First, you can build an image identical to the images running in the gate using
  87. `diskimage-builder <https://docs.openstack.org/developer/diskimage-builder>`_.
  88. The specific operating systems built and DIB elements for each image type are
  89. defined in `nodepool.yaml <https://opendev.org/openstack/project-config/
  90. src/branch/master/nodepool/nodepool.yaml>`_. There is a handy script
  91. available in the project-config repo to build this for you::
  92. git clone https://opendev.org/openstack/project-config
  93. cd project-config
  94. ./tools/build-image.sh
  95. Take a look at the documentation within the `build-image.sh` script for specific
  96. build options.
  97. These days Tempest testing is requiring in excess of 2GiB RAM (4 should
  98. be enough but we typically use 8) and completes within an hour on a
  99. 4-CPU virtual machine.
  100. If you're using an OpenStack provider, it's usually helpful to set up a
  101. `clouds.yaml` file. More information on `clouds.yaml` files can be found in the
  102. `os-client-config documentation <https://docs.openstack.org/developer/os-client-config/#config-files`_.
  103. A `clouds.yaml` file for Rackspace would look something like::
  104. clouds:
  105. rackspace:
  106. auth:
  107. profile: rackspace
  108. username: '<provider_username>'
  109. password: '<provider_password>'
  110. project_name: '<provider_project_name>'
  111. Where provider_username and provider_password are the user / password
  112. for a valid user in your account, and provider_project_name is the project_name
  113. you want to use (sometimes called 'tenant name' on older clouds)
  114. You can then use the `openstack` command line client (found in the python
  115. package
  116. `python-openstackclient <http://pypi.python.org/pypi/python-openstackclient>`_)
  117. to create a VM on the cloud.
  118. You can tell `openstack` to use the `DFW` region
  119. of the `rackspace` cloud you defined either by setting environment variables::
  120. export OS_CLOUD=rackspace
  121. export OS_REGION_NAME=DFW
  122. openstack servers list
  123. or command line options:
  124. openstack --os-cloud=rackspace --os-region-name=DFW servers list
  125. It will be assumed in remaining examples that environment varialbes have been
  126. set.
  127. If you haven't already, create an SSH keypair "my-keypair" (name it whatever
  128. you like)::
  129. openstack keypair create --public-key=$HOME/.ssh/id_rsa.pub my-keypair
  130. Upload your image, boot a server named "testserver" (chosen arbitrarily for
  131. this example) with your SSH key allowed, and log into it::
  132. FLAVOR='8GB Standard Instance'
  133. openstack image create --file devstack-gate.qcow2 devstack-gate
  134. openstack server create --wait --flavor "$FLAVOR" --image "devstack-gate" \
  135. --key-name=my-keypair testserver
  136. openstack server ssh testserver
  137. If you get a cryptic error like ``ERROR: 'public'`` then you may need to
  138. manually look up the IP address with ``openstack server show testserver`` and
  139. connect by running ``ssh root@<ip_address>`` instead. Once logged in, switch to
  140. the jenkins user and set up parts of the environment expected by devstack-gate
  141. testing::
  142. su - jenkins
  143. export REPO_URL=https://git.openstack.org
  144. export ZUUL_URL=/home/jenkins/workspace-cache
  145. export ZUUL_REF=HEAD
  146. export WORKSPACE=/home/jenkins/workspace/testing
  147. mkdir -p $WORKSPACE
  148. Specify the project and branch you want to test for integration::
  149. export ZUUL_PROJECT=openstack/nova
  150. export ZUUL_BRANCH=master
  151. Get a copy of the tested project. After these steps, apply relevant
  152. patches on the target branch (via cherry-pick, rebase, et cetera) and
  153. make sure ``HEAD`` is at the ref you want tested::
  154. git clone $REPO_URL/$ZUUL_PROJECT $ZUUL_URL/$ZUUL_PROJECT \
  155. && cd $ZUUL_URL/$ZUUL_PROJECT \
  156. && git checkout remotes/origin/$ZUUL_BRANCH
  157. Switch to the workspace and get a copy of devstack-gate::
  158. cd $WORKSPACE \
  159. && git clone --depth 1 $REPO_URL/openstack/devstack-gate
  160. At this point you're ready to set the same environment variables and run
  161. the same commands/scripts as used in the desired job. The definitions
  162. for these are found in the openstack/project-config project under
  163. the jenkins/jobs directory in a file named devstack-gate.yaml. It will
  164. probably look something like::
  165. export PYTHONUNBUFFERED=true
  166. export DEVSTACK_GATE_TEMPEST=1
  167. export DEVSTACK_GATE_TEMPEST_FULL=1
  168. cp devstack-gate/devstack-vm-gate-wrap.sh ./safe-devstack-vm-gate-wrap.sh
  169. ./safe-devstack-vm-gate-wrap.sh
  170. If you're trying to figure out which devstack gate jobs run for a given
  171. project+branch combination, this is encoded in the
  172. openstack/project-config project under the zuul/ directory in a file
  173. named layout.yaml. You'll want to look in the "projects" section for a list
  174. of jobs run on a given project in the "gate" pipeline, and then consult the
  175. "jobs" section of the file to see if there are any overrides indicating
  176. which branches qualify for the job and whether or not its voting is
  177. disabled.
  178. After the script completes, investigate any failures. Then log out and
  179. ``openstack server delete testserver`` or similar to get rid of it once no
  180. longer needed. It's possible to re-run certain jobs or specific tests on a used
  181. VM (sometimes with a bit of manual clean-up in between runs), but for
  182. proper testing you'll want to validate your fixes on a completely fresh
  183. one.
  184. Refer to the `Jenkins Job Builder`_ and Zuul_ documentation for more
  185. information on their configuration file formats.
  186. .. _`Jenkins Job Builder`: http://docs.openstack.org/infra/system-config/jjb.html
  187. .. _Zuul: http://docs.openstack.org/infra/system-config/zuul.html
  188. Contributions Welcome
  189. =====================
  190. All of the OpenStack developer infrastructure is freely available and
  191. managed in source code repositories just like the code of OpenStack
  192. itself. If you'd like to contribute, just clone and propose a patch to
  193. the relevant repository::
  194. https://opendev.org/openstack/devstack-gate
  195. https://opendev.org/zuul/nodepool
  196. https://opendev.org/opendev/system-config
  197. https://opendev.org/openstack/project-config
  198. You can file bugs on the storyboard devstack-gate project::
  199. https://storyboard.openstack.org/#!/project/712
  200. And you can chat with us on Freenode in #openstack-qa or #openstack-infra.
  201. It's worth noting that, while devstack-gate is generally licensed under the
  202. Apache license, `playbooks/plugins/callback/devstack.py` is GPLv3 due to having
  203. derived from the Ansible source code.