Ramy Asselin 14fd503aaa Migrate to puppet-openstackci openstackci::nodepool

Manage nodepool configurations using the common-ci solution
in puppet-openstackci

Remove nodepool.yaml.erb from this repo as openstackci::nodepool
will pull it in from project-config/nodepool/nodepool.yaml

Remove the tox nodepool environment and test dependency as it
has been migrated to project-config

The nodepool logging template file and associated tool that generates the file
will remain in this repo. In the short term, updates to nodepool.yaml in
project-config repo may require a related change in this repo to update the logging
configuration. In the longer term, nodepool will be updated to automatically
log image creations without needing a customized logging configuration.

Depends-on: I89207d100eb4b6bbb502a6ed38831f49e4b29096
Depends-on: I473a1b78acdb035eb379394a7ed5f771434dc942
Depends-on: I6b01ab7260a41927fff34b9b81b631ea2c933f22
Change-Id: I2b45a7145805368b1598d3a3e8a94f0e4eb8cf2d

2015-11-10 15:13:50 -08:00

4.8 KiB

Raw Blame History

title: Nodepool

Nodepool

Nodepool is a service used by the OpenStack CI team to deploy and manage a pool of devstack images on a cloud server for use in OpenStack project testing.

At a Glance

Hosts

nodepool.openstack.org

Puppet

https://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/manifests/nodepool.pp
modules/openstack_project/manifests/single_use_slave.pp

Configuration

nodepool/nodepool.yaml
nodepool/scripts/
nodepool/elements/

Projects

https://git.openstack.org/cgit/openstack-infra/nodepool

Bugs

https://storyboard.openstack.org/#!/project/668

Resources

Nodepool Reference Manual

Overview

Once per day, for every image type (and provider) configured by nodepool, a new image with cached data for use by devstack. Nodepool spins up new instances and tears down old as tests are queued up and completed, always maintaining a consistent number of available instances for tests up to the set limits of the CI infrastructure.

Bad Images

Since nodepool takes a while to build images, and generally only does it once per day, occasionally the images it produces may have significant behavior changes from the previous versions. For instance, a provider's base image or operating system package may update, or some of the scripts or system configuration that we apply to the images may change. If this occurs, it is easy to revert to the last good image.

Nodepool periodically deletes old images, however, it never deletes the current or next most recent image in the ready state for any image-provider combination. So if you find that the devstack-precise images for a single or all providers are problematic, you can run:

$ sudo nodepool image-list

+--------+--------------------+------------------------+----------------------------------------------------------+------------+--------------------------------------+--------------------------------------+----------+-------------+
| ID     | Provider           | Image                  | Hostname                                                 | Version    | Image ID                             | Server ID                            | State    | Age (hours) |
+--------+--------------------+------------------------+----------------------------------------------------------+------------+--------------------------------------+--------------------------------------+----------+-------------+
| 168655 | hpcloud-az2        | devstack-precise       | devstack-precise-1394417686.template.openstack.org       | 1394417686 | 387612                               | 4909797                              | ready    | 26.83       |
| 168696 | hpcloud-az2        | devstack-precise       | devstack-precise-1394514268.template.openstack.org       | 1394514268 | 388782                               | 4930213                              | ready    | 0.75        |
+--------+--------------------+------------------------+----------------------------------------------------------+------------+--------------------------------------+--------------------------------------+----------+-------------+

Image 168655 is the previous image and 168696 is the current image (they are both marked as ready and the current image is simply the image with the shortest age. Delete the problematic image with:

$ sudo nodepool image-delete 168696

Then the previous image, 168655, will become the current image and nodepool will use it when creating new nodes. When nodepool next creates an image, it will still retain 168655 since it will still be considered the next-most-recent image.

vhd-util

Creating images for Rackspace requires a patched version of vhd-util to convert the images into the appropriate VHD format. A package is manaually managed at ppa:openstack-ci-core/vhd-util and is based on a git repo at https://github.com/emonty/vhd-util

Updating vhd-util

Should it become required to update vhd-util before Infra has a proper packaging repo or solution in place, one should clone from the git repo:

$ git clone git://github.com/emonty/vhd-util
$ cd vhd-util

Then perform whatever updates and packaging work are needed. The repo is formatted as a git-buildpackage repo with --pristine-tar. When you're ready to upload a new verion, commit, create a source package and a tag:

$ git-buildpackage --git-tag --git-sign-tags -S

This will make a source package in the parent directory. Upload it to launchpad:

$ cd ..
$ dput ppa:openstack-ci-core/vhd-util vhd-util_$version_source.changes

Then probably pushing the repo to github and submitting a pull request so that we can keep up with the change is not a terrible idea.

4.8 KiB Raw Blame History