infra-specs/specs/puppet-modules.rst

::

Copyright 2014 Hewlett-Packard Development Company, L.P.

This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode

Separate puppet-module projects

Rather than having all puppet modules in a single project, each should be in its own project.

Problem Description

The openstack-infra/config project is a monolithic ecosystem of puppet modules. This makes it hard to pick-and-choose specific modules to use downstream such as asterisk or graphite when the larger ecosystem is not wanted, and also makes it easier to accidentally write changes in the wrong puppet module. Some modules, such as jenkins or gerrit, should be generic and reusable across deployments while others (the openstack_project module and possibly others in the future) are very site-specific. Making the site-specific modules a downstream consumer of the generic modules will more clearly delineate where site-specific changes should be made and make the generic modules more easily re-consumable.

Proposed Change

Put each (or at least most) puppet modules in their own project.

Alternatives

• Leave openstack-infra/config as is (monolithic)
• Create several projects, each with a logical group of modules (e.g. logstash + log_processor + elasticseacrh + kibana)

Implementation

The openstack_project module and manifests/site.pp will remain in config. The devstack_host, remove_nginx, and salt modules will be deleted. All other modules will be put in their own project.

Assignee(s)

Primary assignee(s):

• jesusaurus
• nibalizer

Additional assignee(s):

• bodepd

Work Items

We will need to create integration tests to ensure that api compatibility is maintained between puppet modules. This will be difficult to create without having multiple modules, but is important to have before actively developing the new modules. After breaking out the first module we should create an integration test to make sure that changes to it do not break openstack-infra/config and then we can continue with the rest of the modules.

The following process must be done for each module separately:

1. Freeze changes to a specific module within config

2. Isolate the module's history with git-subtree:

   git subtree split --prefix=modules/$MODULE --branch=$MODULE

   The resulting tree can be pushed to a remote.

   • git-subtree will do the right thing by re-rooting the resulting code at modules/$MODULE but will require git >= 1.8.3.2 (and even then may be in a separate contrib package)

3. Push the new branch to a temporary project (e.g. personal github repo)

4. Add a new gerrit project for the module (using the temporary project as upstream)

5. Modify install_modules.sh to install the module from the new gerrit project and add the new project to the puppet integration tests. Remove the old module from openstack_infra/config with rm.

   • We should continuously deploy the master branch

6. Propose a review to add some of the files that are needed by the module:

   • Rakefile :

     require 'rubygems'
     require 'puppetlabs_spec_helper/rake_tasks'
     require 'puppet-lint/tasks/puppet-lint'
     PuppetLint.configuration.fail_on_warnings = true
     PuppetLint.configuration.send('disable_80chars')
     PuppetLint.configuration.send('disable_autoloader_layout')
     PuppetLint.configuration.send('disable_class_inherits_from_params_class')
     PuppetLint.configuration.send('disable_class_parameter_defaults')

   • Modulefile :

     name 'openstack-pip'
     version '0.0.1'
     source 'git://git.openstack.org/openstack-infra/puppet-pip.git'
     author 'openstackci'
     license 'Apache 2.0'
     summary 'Puppet module for the pip package manager'
     description 'This module provides providers to help pip install well. Only tested against 2.7.'
     project_page 'https://github.com/openstack-ci/puppet-pip'

   • README.md :

     # OpenStack Pip Module
     
     Adds a pip3 provider to handle pip installs in the Python3 ecosystem
     
     
     ## Example
     
     ```puppet
     
     package {'flask':
       ensure   => latest,
       provider => 'pip3'
     }
     ```

   • Metadata.json :

     {
       "name": "openstackci-pip",
       "version": "0.0.1",
       "author": "Openstack CI",
       "summary": "Pip provider",
       "license": "ASL 2.0",
       "source": "git://git.openstack.org/openstack-infra/puppet-pip.git",
       "project_page": "https://github.com/openstack-ci/puppet-pip",
       "issues_url": "https://github.com/openstack-ci/puppet-pip",
       "dependencies": []
     }

# Note that determining dependencies may not be immediately obvious,

we must count on the code review process to ensure that we've done this right.

7. Lather, rinse, and repeat

Repositories

• openstack-infra/puppet-accessbot
• openstack-infra/puppet-asterisk
• openstack-infra/puppet-bugdaystats
• openstack-infra/puppet-bup
• openstack-infra/puppet-cgit
• openstack-infra/puppet-drupal
• openstack-infra/puppet-elastic_recheck
• openstack-infra/puppet-elasticsearch
• openstack-infra/puppet-exim
• openstack-infra/puppet-gerrit
• openstack-infra/puppet-gerritbot
• openstack-infra/puppet-github
• openstack-infra/puppet-graphite
• openstack-infra/puppet-iptables
• openstack-infra/puppet-jeepyb
• openstack-infra/puppet-jenkins
• openstack-infra/puppet-kibana
• openstack-infra/puppet-lodgeit
• openstack-infra/puppet-log_processor
• openstack-infra/puppet-logrotate
• openstack-infra/puppet-logstash
• openstack-infra/puppet-mailman
• openstack-infra/puppet-mediawiki
• openstack-infra/puppet-meetbot
• openstack-infra/puppet-mysql_backup
• openstack-infra/puppet-nodepool
• openstack-infra/puppet-openssl
• openstack-infra/puppet-openstackid
• openstack-infra/puppet-packagekit
• openstack-infra/puppet-pip
• openstack-infra/puppet-planet
• openstack-infra/puppet-puppetboot
• openstack-infra/puppet-recheckwatch
• openstack-infra/puppet-redis
• openstack-infra/puppet-releasestatus
• openstack-infra/puppet-remove_nginx
• openstack-infra/puppet-reviewday
• openstack-infra/puppet-salt
• openstack-infra/puppet-snmpd
• openstack-infra/puppet-ssh
• openstack-infra/puppet-ssl_cert_check
• openstack-infra/puppet-statusbot
• openstack-infra/puppet-storyboard
• openstack-infra/puppet-subversion
• openstack-infra/puppet-sudoers
• openstack-infra/puppet-tmpreaper
• openstack-infra/puppet-ulimit
• openstack-infra/puppet-unattended_upgrades
• openstack-infra/puppet-unbound
• openstack-infra/puppet-user
• openstack-infra/puppet-zuul

Servers

None

DNS Entries

None

Documentation

Each new module will have its own documentation.

Security

None

Testing

• Unit tests: We currently only lint and syntax-check the modules in config. They should also have rspec-beaker and server-spec tests written for them (even if we don't move them to their own project).
• Integration tests: We need to test that changes to the new projects do not break config (such as with changes to a class's parameter list).

Developer Impact

By migrating from a single project to many projects, developers will no longer be able to atomically change multiple modules at the same time. This means that changes that touch multiple modules will have to be made in a backwards-compatible way with soft dependencies between changes (such as two changes mentioning each other in their commit messages). Requiring backwards-compatible changes will also make it easier for downstream consumers to use the modules.

Dependencies

None