Support Ansible collections

This change adds support for installing Ansible collections via
requirements.yml in Kayobe or Kayobe config.

Story: 2008391
Task: 41315

Change-Id: I764ff019a18266b593add7ab80ee095d7d07a869
This commit is contained in:
Mark Goddard 2021-06-22 15:10:50 +00:00
parent bb05adbdcf
commit 5535832c10
10 changed files with 353 additions and 94 deletions

View File

@ -38,25 +38,27 @@ in `etc/kayobe/*.yml
<https://opendev.org/openstack/kayobe/src/branch/master/etc/kayobe/>`__.
A number of custom Jinja filters exist in `ansible/filter_plugins/*.py
<https://opendev.org/openstack/kayobe/src/branch/master/ansible/filter_plugins>`__.
Kayobe depends on roles hosted on Ansible Galaxy, and these and their version
requirements are defined in `requirements.yml
Kayobe depends on roles and collections hosted on Ansible Galaxy, and these and
their version requirements are defined in `requirements.yml
<https://opendev.org/openstack/kayobe/src/branch/master/requirements.yml>`__.
Ansible Galaxy
==============
Kayobe uses a number of Ansible roles hosted on Ansible Galaxy. The role
dependencies are tracked in ``requirements.yml``, and specify required
versions. The process for changing a Galaxy role is as follows:
Kayobe uses a number of Ansible roles and collections hosted on Ansible Galaxy.
The role dependencies are tracked in ``requirements.yml``, and specify required
versions. The process for changing a Galaxy role or collection is as follows:
#. If required, develop changes for the role. This may be done outside of
Kayobe, or by modifying the role in place during development. If upstream
changes to the role have already been made, this step can be skipped.
#. Commit changes to the role, typically via a Github pull request.
#. Request that a tagged release of the role be made, or make one if you have
the necessary privileges.
#. Ensure that automatic imports are configured for the role using e.g. a
TravisCI webhook notification, or perform a manual import of the role on
Ansible Galaxy.
#. If required, develop changes for the role or collection. This may be done
outside of Kayobe, or by modifying the code in place during development. If
upstream changes to the code have already been made, this step can be
skipped.
#. Commit changes to the role or collection, typically via a Github pull
request.
#. Request that a tagged release of the role or collection be made, or make one
if you have the necessary privileges.
#. Ensure that automatic imports are configured for the repository using e.g. a
webhook notification, or perform a manual import of the role on Ansible
Galaxy.
#. Modify the version in ``requirements.yml`` to match the new release of the
role.
role or collection.

View File

@ -75,14 +75,16 @@ These symlinks can even be committed to the kayobe-config Git repository.
Ansible Galaxy
--------------
Ansible Galaxy provides a means for sharing Ansible roles. Kayobe
configuration may provide a Galaxy requirements file that defines roles to be
installed from Galaxy. These roles may then be used by custom playbooks.
Ansible Galaxy provides a means for sharing Ansible roles and collections.
Kayobe configuration may provide a Galaxy requirements file that defines roles
and collections to be installed from Galaxy. These roles and collections may
then be used by custom playbooks.
Galaxy role dependencies may be defined in
``$KAYOBE_CONFIG_PATH/ansible/requirements.yml``. These roles will be
installed in ``$KAYOBE_CONFIG_PATH/ansible/roles/`` when bootstrapping the
Ansible control host::
Galaxy dependencies may be defined in
``$KAYOBE_CONFIG_PATH/ansible/requirements.yml``. These roles and collections
will be installed in ``$KAYOBE_CONFIG_PATH/ansible/roles/`` and
``$KAYOBE_CONFIG_PATH/ansible/collections`` when bootstrapping the Ansible
control host::
(kayobe) $ kayobe control host bootstrap
@ -90,8 +92,8 @@ And updated when upgrading the Ansible control host::
(kayobe) $ kayobe control host upgrade
Example
=======
Example: roles
==============
The following example adds a ``foo.yml`` playbook to a set of kayobe
configuration. The playbook uses a Galaxy role, ``bar.baz``.
@ -116,6 +118,7 @@ Here is the playbook, ``ansible/foo.yml``::
Here is the Galaxy requirements file, ``ansible/requirements.yml``::
---
roles:
- bar.baz
We should first install the Galaxy role dependencies, to download the
@ -127,6 +130,45 @@ Then, to run the ``foo.yml`` playbook::
(kayobe) $ kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/foo.yml
Example: collections
====================
The following example adds a ``foo.yml`` playbook to a set of kayobe
configuration. The playbook uses a role from a Galaxy collection,
``bar.baz.qux``.
Here is the kayobe configuration repository structure::
etc/kayobe/
ansible/
collections/
foo.yml
requirements.yml
bifrost.yml
...
Here is the playbook, ``ansible/foo.yml``::
---
- hosts: controllers
roles:
- name: bar.baz.qux
Here is the Galaxy requirements file, ``ansible/requirements.yml``::
---
collections:
- bar.baz
We should first install the Galaxy dependencies, to download the ``bar.baz``
collection::
(kayobe) $ kayobe control host bootstrap
Then, to run the ``foo.yml`` playbook::
(kayobe) $ kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/foo.yml
Hooks
=====

View File

@ -291,7 +291,7 @@ def config_dump(parsed_args, host=None, hosts=None, var_name=None,
def install_galaxy_roles(parsed_args, force=False):
"""Install Ansible Galaxy role dependencies.
Installs dependencies specified in kayobe, and if present, in kayobe
Installs role dependencies specified in kayobe, and if present, in kayobe
configuration.
:param parsed_args: Parsed command line arguments.
@ -300,7 +300,7 @@ def install_galaxy_roles(parsed_args, force=False):
LOG.info("Installing galaxy role dependencies from kayobe")
requirements = utils.get_data_files_path("requirements.yml")
roles_destination = utils.get_data_files_path('ansible', 'roles')
utils.galaxy_install(requirements, roles_destination, force=force)
utils.galaxy_role_install(requirements, roles_destination, force=force)
# Check for requirements in kayobe configuration.
kc_reqs_path = os.path.join(parsed_args.config_path,
@ -323,7 +323,49 @@ def install_galaxy_roles(parsed_args, force=False):
(parsed_args.config_path, str(e)))
# Install roles from kayobe-config.
utils.galaxy_install(kc_reqs_path, kc_roles_path, force=force)
utils.galaxy_role_install(kc_reqs_path, kc_roles_path, force=force)
def install_galaxy_collections(parsed_args, force=False):
"""Install Ansible Galaxy collection dependencies.
Installs collection dependencies specified in kayobe, and if present, in
kayobe configuration.
:param parsed_args: Parsed command line arguments.
:param force: Whether to force reinstallation of roles.
"""
LOG.info("Installing galaxy collection dependencies from kayobe")
requirements = utils.get_data_files_path("requirements.yml")
collections_destination = utils.get_data_files_path('ansible',
'collections')
utils.galaxy_collection_install(requirements, collections_destination,
force=force)
# Check for requirements in kayobe configuration.
kc_reqs_path = os.path.join(parsed_args.config_path,
"ansible", "requirements.yml")
if not utils.is_readable_file(kc_reqs_path)["result"]:
LOG.info("Not installing galaxy collection dependencies from kayobe "
"config - requirements.yml not present")
return
LOG.info("Installing galaxy collection dependencies from kayobe config")
# Ensure a collections directory exists in kayobe-config.
kc_collections_path = os.path.join(parsed_args.config_path,
"ansible", "collections")
try:
os.makedirs(kc_collections_path)
except OSError as e:
if e.errno != errno.EEXIST:
raise exception.Error("Failed to create directory "
"ansible/collections/ "
"in kayobe configuration at %s: %s" %
(parsed_args.config_path, str(e)))
# Install collections from kayobe-config.
utils.galaxy_collection_install(kc_reqs_path, kc_collections_path,
force=force)
def prune_galaxy_roles(parsed_args):

View File

@ -232,6 +232,7 @@ class ControlHostBootstrap(KayobeAnsibleMixin, KollaAnsibleMixin, VaultMixin,
def take_action(self, parsed_args):
self.app.LOG.debug("Bootstrapping Kayobe Ansible control host")
ansible.install_galaxy_roles(parsed_args)
ansible.install_galaxy_collections(parsed_args)
playbooks = _build_playbook_list("bootstrap")
self.run_kayobe_playbooks(parsed_args, playbooks, ignore_limit=True)
@ -271,8 +272,9 @@ class ControlHostUpgrade(KayobeAnsibleMixin, VaultMixin, Command):
# Remove roles that are no longer used. Do this before installing new
# ones, just in case a custom role dependency includes any.
ansible.prune_galaxy_roles(parsed_args)
# Use force to upgrade roles.
# Use force to upgrade roles and collections.
ansible.install_galaxy_roles(parsed_args, force=True)
ansible.install_galaxy_collections(parsed_args, force=True)
playbooks = _build_playbook_list("bootstrap")
self.run_kayobe_playbooks(parsed_args, playbooks, ignore_limit=True)
playbooks = _build_playbook_list("kolla-ansible")

View File

@ -35,18 +35,21 @@ class TestApp(cliff.app.App):
class TestCase(unittest.TestCase):
@mock.patch.object(ansible, "install_galaxy_roles", autospec=True)
@mock.patch.object(ansible, "install_galaxy_collections", autospec=True)
@mock.patch.object(ansible, "passwords_yml_exists", autospec=True)
@mock.patch.object(commands.KayobeAnsibleMixin,
"run_kayobe_playbooks")
def test_control_host_bootstrap(self, mock_run, mock_passwords,
mock_install):
mock_install_collections,
mock_install_roles):
mock_passwords.return_value = False
command = commands.ControlHostBootstrap(TestApp(), [])
parser = command.get_parser("test")
parsed_args = parser.parse_args([])
result = command.run(parsed_args)
self.assertEqual(0, result)
mock_install.assert_called_once_with(parsed_args)
mock_install_roles.assert_called_once_with(parsed_args)
mock_install_collections.assert_called_once_with(parsed_args)
expected_calls = [
mock.call(
mock.ANY,
@ -63,20 +66,23 @@ class TestCase(unittest.TestCase):
self.assertEqual(expected_calls, mock_run.call_args_list)
@mock.patch.object(ansible, "install_galaxy_roles", autospec=True)
@mock.patch.object(ansible, "install_galaxy_collections", autospec=True)
@mock.patch.object(ansible, "passwords_yml_exists", autospec=True)
@mock.patch.object(commands.KayobeAnsibleMixin,
"run_kayobe_playbooks")
@mock.patch.object(commands.KollaAnsibleMixin,
"run_kolla_ansible_overcloud")
def test_control_host_bootstrap_with_passwords(
self, mock_kolla_run, mock_run, mock_passwords, mock_install):
self, mock_kolla_run, mock_run, mock_passwords,
mock_install_collections, mock_install_roles):
mock_passwords.return_value = True
command = commands.ControlHostBootstrap(TestApp(), [])
parser = command.get_parser("test")
parsed_args = parser.parse_args([])
result = command.run(parsed_args)
self.assertEqual(0, result)
mock_install.assert_called_once_with(parsed_args)
mock_install_roles.assert_called_once_with(parsed_args)
mock_install_collections.assert_called_once_with(parsed_args)
expected_calls = [
mock.call(
mock.ANY,
@ -106,16 +112,21 @@ class TestCase(unittest.TestCase):
self.assertEqual(expected_calls, mock_kolla_run.call_args_list)
@mock.patch.object(ansible, "install_galaxy_roles", autospec=True)
@mock.patch.object(ansible, "install_galaxy_collections", autospec=True)
@mock.patch.object(ansible, "prune_galaxy_roles", autospec=True)
@mock.patch.object(commands.KayobeAnsibleMixin,
"run_kayobe_playbooks")
def test_control_host_upgrade(self, mock_run, mock_prune, mock_install):
def test_control_host_upgrade(self, mock_run, mock_prune,
mock_install_roles,
mock_install_collections):
command = commands.ControlHostUpgrade(TestApp(), [])
parser = command.get_parser("test")
parsed_args = parser.parse_args([])
result = command.run(parsed_args)
self.assertEqual(0, result)
mock_install.assert_called_once_with(parsed_args, force=True)
mock_install_roles.assert_called_once_with(parsed_args, force=True)
mock_install_collections.assert_called_once_with(parsed_args,
force=True)
mock_prune.assert_called_once_with(parsed_args)
expected_calls = [
mock.call(

View File

@ -434,7 +434,7 @@ class TestCase(unittest.TestCase):
mock.call(os.path.join(dump_dir, "host2.yml")),
])
@mock.patch.object(utils, 'galaxy_install', autospec=True)
@mock.patch.object(utils, 'galaxy_role_install', autospec=True)
@mock.patch.object(utils, 'is_readable_file', autospec=True)
@mock.patch.object(os, 'makedirs', autospec=True)
def test_install_galaxy_roles(self, mock_mkdirs, mock_is_readable,
@ -453,7 +453,7 @@ class TestCase(unittest.TestCase):
"/etc/kayobe/ansible/requirements.yml")
self.assertFalse(mock_mkdirs.called)
@mock.patch.object(utils, 'galaxy_install', autospec=True)
@mock.patch.object(utils, 'galaxy_role_install', autospec=True)
@mock.patch.object(utils, 'is_readable_file', autospec=True)
@mock.patch.object(os, 'makedirs', autospec=True)
def test_install_galaxy_roles_with_kayobe_config(
@ -476,7 +476,7 @@ class TestCase(unittest.TestCase):
"/etc/kayobe/ansible/requirements.yml")
mock_mkdirs.assert_called_once_with("/etc/kayobe/ansible/roles")
@mock.patch.object(utils, 'galaxy_install', autospec=True)
@mock.patch.object(utils, 'galaxy_role_install', autospec=True)
@mock.patch.object(utils, 'is_readable_file', autospec=True)
@mock.patch.object(os, 'makedirs', autospec=True)
def test_install_galaxy_roles_with_kayobe_config_forced(
@ -499,7 +499,7 @@ class TestCase(unittest.TestCase):
"/etc/kayobe/ansible/requirements.yml")
mock_mkdirs.assert_called_once_with("/etc/kayobe/ansible/roles")
@mock.patch.object(utils, 'galaxy_install', autospec=True)
@mock.patch.object(utils, 'galaxy_role_install', autospec=True)
@mock.patch.object(utils, 'is_readable_file', autospec=True)
@mock.patch.object(os, 'makedirs', autospec=True)
def test_install_galaxy_roles_with_kayobe_config_mkdirs_failure(
@ -520,6 +520,92 @@ class TestCase(unittest.TestCase):
"/etc/kayobe/ansible/requirements.yml")
mock_mkdirs.assert_called_once_with("/etc/kayobe/ansible/roles")
@mock.patch.object(utils, 'galaxy_collection_install', autospec=True)
@mock.patch.object(utils, 'is_readable_file', autospec=True)
@mock.patch.object(os, 'makedirs', autospec=True)
def test_install_galaxy_collections(self, mock_mkdirs, mock_is_readable,
mock_install):
parser = argparse.ArgumentParser()
ansible.add_args(parser)
parsed_args = parser.parse_args([])
mock_is_readable.return_value = {"result": False}
ansible.install_galaxy_collections(parsed_args)
mock_install.assert_called_once_with(utils.get_data_files_path(
"requirements.yml"), utils.get_data_files_path(
"ansible", "collections"), force=False)
mock_is_readable.assert_called_once_with(
"/etc/kayobe/ansible/requirements.yml")
self.assertFalse(mock_mkdirs.called)
@mock.patch.object(utils, 'galaxy_collection_install', autospec=True)
@mock.patch.object(utils, 'is_readable_file', autospec=True)
@mock.patch.object(os, 'makedirs', autospec=True)
def test_install_galaxy_collections_with_kayobe_config(
self, mock_mkdirs, mock_is_readable, mock_install):
parser = argparse.ArgumentParser()
ansible.add_args(parser)
parsed_args = parser.parse_args([])
mock_is_readable.return_value = {"result": True}
ansible.install_galaxy_collections(parsed_args)
expected_calls = [
mock.call(utils.get_data_files_path("requirements.yml"),
utils.get_data_files_path("ansible", "collections"),
force=False),
mock.call("/etc/kayobe/ansible/requirements.yml",
"/etc/kayobe/ansible/collections", force=False)]
self.assertEqual(expected_calls, mock_install.call_args_list)
mock_is_readable.assert_called_once_with(
"/etc/kayobe/ansible/requirements.yml")
mock_mkdirs.assert_called_once_with("/etc/kayobe/ansible/collections")
@mock.patch.object(utils, 'galaxy_collection_install', autospec=True)
@mock.patch.object(utils, 'is_readable_file', autospec=True)
@mock.patch.object(os, 'makedirs', autospec=True)
def test_install_galaxy_collections_with_kayobe_config_forced(
self, mock_mkdirs, mock_is_readable, mock_install):
parser = argparse.ArgumentParser()
ansible.add_args(parser)
parsed_args = parser.parse_args([])
mock_is_readable.return_value = {"result": True}
ansible.install_galaxy_collections(parsed_args, force=True)
expected_calls = [
mock.call(utils.get_data_files_path("requirements.yml"),
utils.get_data_files_path("ansible", "collections"),
force=True),
mock.call("/etc/kayobe/ansible/requirements.yml",
"/etc/kayobe/ansible/collections", force=True)]
self.assertEqual(expected_calls, mock_install.call_args_list)
mock_is_readable.assert_called_once_with(
"/etc/kayobe/ansible/requirements.yml")
mock_mkdirs.assert_called_once_with("/etc/kayobe/ansible/collections")
@mock.patch.object(utils, 'galaxy_collection_install', autospec=True)
@mock.patch.object(utils, 'is_readable_file', autospec=True)
@mock.patch.object(os, 'makedirs', autospec=True)
def test_install_galaxy_collections_with_kayobe_config_mkdirs_failure(
self, mock_mkdirs, mock_is_readable, mock_install):
parser = argparse.ArgumentParser()
ansible.add_args(parser)
parsed_args = parser.parse_args([])
mock_is_readable.return_value = {"result": True}
mock_mkdirs.side_effect = OSError(errno.EPERM)
self.assertRaises(exception.Error,
ansible.install_galaxy_collections, parsed_args)
mock_install.assert_called_once_with(
utils.get_data_files_path("requirements.yml"),
utils.get_data_files_path("ansible", "collections"), force=False)
mock_is_readable.assert_called_once_with(
"/etc/kayobe/ansible/requirements.yml")
mock_mkdirs.assert_called_once_with("/etc/kayobe/ansible/collections")
@mock.patch.object(utils, 'galaxy_remove', autospec=True)
def test_prune_galaxy_roles(self, mock_remove):
parser = argparse.ArgumentParser()

View File

@ -26,23 +26,72 @@ from kayobe import utils
class TestCase(unittest.TestCase):
@mock.patch.object(utils, "run_command")
def test_galaxy_install(self, mock_run):
utils.galaxy_install("/path/to/role/file", "/path/to/roles")
mock_run.assert_called_once_with(["ansible-galaxy", "install",
def test_galaxy_role_install(self, mock_run):
utils.galaxy_role_install("/path/to/role/file", "/path/to/roles")
mock_run.assert_called_once_with(["ansible-galaxy", "role", "install",
"--roles-path", "/path/to/roles",
"--role-file", "/path/to/role/file"])
@mock.patch.object(utils, "run_command")
def test_galaxy_install_failure(self, mock_run):
def test_galaxy_role_install_failure(self, mock_run):
mock_run.side_effect = subprocess.CalledProcessError(1, "command")
self.assertRaises(SystemExit,
utils.galaxy_install, "/path/to/role/file",
utils.galaxy_role_install, "/path/to/role/file",
"/path/to/roles")
@mock.patch.object(utils, "run_command")
@mock.patch.object(utils, "read_yaml_file")
def test_galaxy_collection_install(self, mock_read, mock_run):
mock_read.return_value = {"collections": []}
utils.galaxy_collection_install("/path/to/collection/file",
"/path/to/collections")
mock_run.assert_called_once_with(["ansible-galaxy", "collection",
"install", "--collections-path",
"/path/to/collections",
"--requirements-file",
"/path/to/collection/file"])
@mock.patch.object(utils, "run_command")
@mock.patch.object(utils, "read_yaml_file")
def test_galaxy_collection_install_failure(self, mock_read, mock_run):
mock_read.return_value = {"collections": []}
mock_run.side_effect = subprocess.CalledProcessError(1, "command")
self.assertRaises(SystemExit,
utils.galaxy_collection_install,
"/path/to/collection/file", "/path/to/collections")
@mock.patch.object(utils, "run_command")
@mock.patch.object(utils, "read_file")
def test_galaxy_collection_read_failure(self, mock_read, mock_run):
mock_read.side_effect = IOError
self.assertRaises(SystemExit,
utils.galaxy_collection_install,
"/path/to/collection/file", "/path/to/collections")
@mock.patch.object(utils, "run_command")
@mock.patch.object(utils, "read_yaml_file")
def test_galaxy_collection_no_collections(self, mock_read, mock_run):
mock_read.return_value = {"roles": []}
utils.galaxy_collection_install("/path/to/collection/file",
"/path/to/collections")
mock_run.assert_called_once_with(["ansible-galaxy", "collection",
"install", "--collections-path",
"/path/to/collections",
"--requirements-file",
"/path/to/collection/file"])
@mock.patch.object(utils, "run_command")
@mock.patch.object(utils, "read_yaml_file")
def test_galaxy_collection_legacy_format(self, mock_read, mock_run):
mock_read.return_value = []
utils.galaxy_collection_install("/path/to/collection/file",
"/path/to/collections")
self.assertFalse(mock_run.called)
@mock.patch.object(utils, "run_command")
def test_galaxy_remove(self, mock_run):
utils.galaxy_remove(["role1", "role2"], "/path/to/roles")
mock_run.assert_called_once_with(["ansible-galaxy", "remove",
mock_run.assert_called_once_with(["ansible-galaxy", "role", "remove",
"--roles-path", "/path/to/roles",
"role1", "role2"])
@ -50,7 +99,7 @@ class TestCase(unittest.TestCase):
def test_galaxy_remove_failure(self, mock_run):
mock_run.side_effect = subprocess.CalledProcessError(1, "command")
self.assertRaises(SystemExit,
utils.galaxy_install, ["role1", "role2"],
utils.galaxy_remove, ["role1", "role2"],
"/path/to/roles")
@mock.patch.object(utils, "read_file")

View File

@ -72,9 +72,9 @@ def _get_base_path():
return os.path.join(os.path.realpath(__file__), "..")
def galaxy_install(role_file, roles_path, force=False):
def galaxy_role_install(role_file, roles_path, force=False):
"""Install Ansible roles via Ansible Galaxy."""
cmd = ["ansible-galaxy", "install"]
cmd = ["ansible-galaxy", "role", "install"]
cmd += ["--roles-path", roles_path]
cmd += ["--role-file", role_file]
if force:
@ -87,10 +87,29 @@ def galaxy_install(role_file, roles_path, force=False):
sys.exit(e.returncode)
def galaxy_collection_install(requirements_file, collections_path,
force=False):
requirements = read_yaml_file(requirements_file)
if not isinstance(requirements, dict):
# Handle legacy role list format, which causes the command to fail.
return
cmd = ["ansible-galaxy", "collection", "install"]
cmd += ["--collections-path", collections_path]
cmd += ["--requirements-file", requirements_file]
if force:
cmd += ["--force"]
try:
run_command(cmd)
except subprocess.CalledProcessError as e:
LOG.error("Failed to install Ansible collections from %s via Ansible "
"Galaxy: returncode %d", requirements_file, e.returncode)
sys.exit(e.returncode)
def galaxy_remove(roles_to_remove, roles_path):
"""Remove Ansible roles via Ansible Galaxy."""
cmd = ["ansible-galaxy", "remove"]
cmd = ["ansible-galaxy", "role", "remove"]
cmd += ["--roles-path", roles_path]
cmd += roles_to_remove
try:

View File

@ -0,0 +1,5 @@
---
features:
- |
Adds support for installing Ansible collections. See `story 2008391
<https://storyboard.openstack.org/#!/story/2008391>`__ for details.

View File

@ -1,4 +1,5 @@
---
roles:
- src: ahuffman.resolv
version: 1.3.1
- src: stackhpc.systemd_networkd