The variables swift_developer_mode and swift_venv_download
no longer carry any meaning. This review changes swift to
do the equivalent of what developer_mode was all the time,
meaning that it always builds the venv and never requires
the repo server, but it will use a repo server when available.
As part of this, we move the source build out of its own file
because it's now a single task to include the venv build role.
This is just to make it easier to follow the code.
Change-Id: I9233e4daff0034339750477fd21d5cfa181afd83
Currently the devel packages are installed everywhere,
but they only need to be where the wheels are built.
Also, there is already a task to install the packages
needed on the target hosts when installing - so we do
not need to give the same list to the venv install role
because they will already have been installed.
We remove the unnecessary installation of the compiling
packages because the python venv build role already does
it. We also remove the curl and which packages as they
appear to be unused. The git package is moved to the
devel package list as it's only used there.
We also re-order the package lists alphabetically to
make them easier to follow.
Depends-On: https://review.openstack.org/613585
Change-Id: I6d037c45aee8b54502a2e3f3add56b23be34109d
In order to enable the service setup host python interpreter to
be changed easily, we make it a variable. This will be useful
when someone sets the service setup host to be the utility
container, because we'll be able to set this var by default.
Change-Id: Icb3ea9ad782218c357ec0e3c577ecbe6fbf60461
The ssh service on ubuntu based systems is "ssh" which is established by
the service unit path `/lib/systemd/system/ssh.service`. When running
the service will respond to the name "sshd" however this is just an
alias. This change adds a variable to set the service unit name
based on the distro family which will allow the service to start should
it be masked.
The change will now delegate to all nodes within the swift cluster
ensuring ssh is enabled and started. If SSH is not running everywhere at
the same time swift ring distribution will not be possible later on in
the role.
Change-Id: Ifbc748019403f0c0712d5f241f118e998d2061d4
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
When running an included role with_items + loop control is slow. This
change feeds the role the entire list of curated services which will
improve execution speed, especially when dealing with hundreds of hosts.
Change-Id: I6258d23300c8b26469f8480aa042d53b91d90ad1
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
In order to radically simplify how we prepare the service
venvs, we use a common role to do the wheel builds and the
venv preparation. This makes the process far simpler to
understand, because the role does its own building and
installing. It also reduces the code maintenance burden,
because instead of duplicating the build processes in the
repo_build role and the service role - we only have it all
done in a single place.
We also change the role venv tag var to use the integrated
build's common venv tag so that we can remove the role's
venv tag in group_vars in the integrated build. This reduces
memory consumption and also reduces the duplication.
This is by no means the final stop in the simplification
process, but it is a step forward. The will be work to follow
which:
1. Replaces 'developer mode' with an equivalent mechanism
that uses the common role and is simpler to understand.
We will also simplify the provisioning of pip install
arguments when doing this.
2. Simplifies the installation of optional pip packages.
Right now it's more complicated than it needs to be due
to us needing to keep the py_pkgs plugin working in the
integrated build.
3. Deduplicates the distro package installs. Right now the
role installs the distro packages twice - just before
building the venv, and during the python_venv_build role
execution.
Depends-On: https://review.openstack.org/598957
Change-Id: Iecb64d28afe3acfbae7060af55c1a891310e5ef4
Implements: blueprint python-build-install-simplification
Signed-off-by: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
The use of 'include_tasks' and a loop of variables creates
a situation where a user is unable to use tags to scope the
inclusion of only the MQ tasks when running the playbooks.
The use-case this is important for is when the rabbitmq
containers are destroyed and rebuilt in order to resolve
an issue with them, and the user wishes to quickly recreate
all the vhosts/users.
Ansible's 'include_tasks' is a dynamic inclusion, and dynamic
inclusions are not included when using tags. The nice thing
about dynamic inclusions is that they completely skip all
tasks when the condition does not apply, cutting down deploy
time. However, given the use-case, we should rather take on
the extra deployment time.
This patch changes the dynamic inclusion to a static one,
adds a 'common-mq' tag to cover all MQ implementations,
and re-implements the 'common-rabbitmq' tag for the tasks
that relate to RabbitMQ specifically.
It also implements conditionals for each task set so that
the rpc/notify tasks can be skipped if a vhost/user is not
required for that purpose (eg: swift does not use RPC, and
most roles will not use notifications by default).
Depends-On: https://review.openstack.org/588191
Change-Id: I994ccc0d20e4ec72955e625119a35856a7171ba3
In order to reduce the packages required to pip install on to the hosts,
we allow the service setup to be delegated to a specific host, defaulting
to the deploy host. We also switch as many tasks as possible to using the
built-in Ansible modules which make use of the shade library.
The 'virtualenv' package is now installed appropriately by the openstack_hosts
role, so there's no need to install it any more. The 'httplib2' package is a
legacy Ansible requirement for the get_url/get_uri module which is no longer
needed. The keystone client library is not required any more now that we're
using the upstream modules. As there are no required packages left, the task
to install them is also removed.
Depends-On: https://review.openstack.org/582359
Depends-On: https://review.openstack.org/587376
Change-Id: I68f3a0bf2b7a3a12cbf40d7d6a853d5b4c6dd0f3
With the more recent versions of ansible, we should now use
"is" instead of the "|" sign for the tests.
This should fix it.
Change-Id: Ia1c412489554ab629a8ffa8e385a9b634f471861
There is no record for why we implement the MQ vhost/user creation
outside of the role in the playbook, when we could do it inside the
role.
Implementing it inside the role allows us to reduce the quantity of
group_vars duplicated from the role, and allows us to better document
the required variables in the role. The delegation can still be done
as it is done in the playbook too.
In this patch we implement the new variable:
- swift_oslomsg_notify_setup_host
This is used in the role to allow delegation of the MQ vhost/user
setup for each type to any host, but they default to using the first
member of the applicable oslomsg host group.
We also adjust some of the defaults to automatically inherit existing
vars set in group_vars form the integrated build so that we do not
need to do the wiring in the integrated build's group vars. We still
default them in the role too for independent role usage.
Depends-On: https://review.openstack.org/584630
Change-Id: I3d5e00e090d37ea7aa95460965749ef066b63b23
The rules are not currently maintained, nor do we have the resources
to maintain them. In addition, they most likely don't work in our
integrated repos.
For future, it would be best to depend on upstream packages for
SELinux support such as `openstack-selinux`.
Change-Id: I6203b98a96a341ce52930ceeed609d9c118ae8b8
There are times when a deployer will need to reconfigure parts of
an environment and having a general purpose tag to run said operation
will be important especicially should the deployer be needing to
reconfigure systemd unit files in a downtime event. This change adds
a general purpose systemd tag where include_role and systemd is found
which will assit operators with day2 operational tasks.
Change-Id: Id65084aff939a240922e46f31c7edb2fae013b38
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Distributions provide packages for the OpenStack services so we add
support for using these instead of the pip ones. However, functional
testing is not complete yet since it requires tempest to be in the swift
virtual environment which doesn't exist for distro installs. As such,
for functional testing to cover this method of installation it needs
to be changed quite a bit.
Change-Id: Id75e56440685df407a9991018dc07b5a75429082
Implements: blueprint openstack-distribution-packages
This removes the systemd service templates and tasks from this role and
leverages a common systemd service role instead. This change removes a
lot of code duplication across all roles all without sacrificing features
or functionality. The intention of this change is to ensure uniformity and
reduce the maintenance burden on the community when sweeping changes are
needed. The exterior role is built to be OSA compatible and may be pulled
into tree should we deem it necessary.
Change-Id: Icb7ca523cb19c560de5c84b0d60a06305029192c
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This prevents data to be leaked into the callback plugin.
Some style changes are included to make it easier to read,
the copyright notice is added as it should have been from
the start, and a duplicate retry argument is removed.
Change-Id: I11c3da84fa6469764a19b6a387ef312124dfba2a
We reinitialize the venv to ensure that the right version of
python is in the venv, but we do not want virtualenv to also
replace pip, setuptools and wheel so we tell it not to. If we
don't do this then virtualenv will install the latest available
version, which is not what we want.
Change-Id: I05d9ee40e596afcc35559b53f6a773521b1d1534
Partial-Bug: #1764470
The ansible_selinux variable is always populated with a 'status'
property, even if SELinux is not installed or configured. This
patch simplifies the check.
Change-Id: I9a7150dec09b486010d533a25692962532708035
Signed-off-by: Major Hayden <major@mhtx.net>
This removes warnings in Ansible 2.4+.
The patch also removes the "static:" arguments since they are no
longer used by Ansible.
Change-Id: I1448501507daae92022a803af80839286f3542f2
In the previous patch removing the virtualenv-tools, we've updated
paths to the proper binaries, but we don't reset the virtualenv
like what we used to do.
Change-Id: I33906abc0e145dc671c2781924c4c20e9df3ba48
Partial-Bug: #1741634
virtualenv-tools has a bug which gets triggered in gates: it can't
change the shebang of a virtualenv python bin/ files if they
were generated with a virtualenv script whose shebang ends with
python2 instead of python.
Because we can't modify virtualenv-tools, we use shell scripts
instead.
Change-Id: I2dc9a507162fcd2323e19a3a2daa97c6db9cdcdb
Partial-Bug: #1741634
This patch removes Jinja2 delimiters from when clauses and fixes the
capitalization of task names to match the other tasks in the file.
Closes-Bug: 1735785
Change-Id: I16334a11c551026c593b1872a44ed68f786719a5
Swift fails to deploy when using >1 storage node. The fact `need_sync`
was being set using information from ansible_local in an effort to
support rolling uprades sometime in the future. Sadly this variable
causes storage nodes to be removed from the key sync which results in a
failed build. If we decide serialized rolling uprades are required for
swift we can revisit this however until then we need to remove this
constraint and ensure we're allowing successfule deployments and
upgrades of swift.
This reverts commit 176452bebf7e1ceabd2cf72a89c558c0fc7f1420.
Change-Id: I21318d7c737c2415ef760b46176c3f9b372bb634
Ansible throws warnings when we use rsync with the command
module. This patch suppresses those warnings.
Change-Id: I42652318f128120b1ecec4781066c89ee9023636
This patch contains five combined improvements.
1) Remove RabbitMQ from the gate
RabbitMQ is only needed in the swift gate if the ceilometer
middleware is being deloyed. This isn't enabled in the gate
job, so let's save some time and deploy memcached/galera
without RabbitMQ.
2) Bindep pyOpenSSL change
This commit also includes the pyOpenSSL bindep change in the
hopes that we can get the swift gate moving along again. ;)
3) Disable erasure code test for CentOS
Due to #1707220 we are running into issues with CentOS 7 running Erasure
Code storage policies on RAX hardware.
This patch temoprarily disables EC testing for CentOS 7, but we can
re-enable that once the package is backported to stable/ocata branch.
4) Pass packages as a list
This patch passes a list of packages to the package module
rather than using "with_items" and it should speed up the
package installation process.
5) Set package state to present
Change-Id: Ic7b13853ce00b0f8ac2181e46e2bd8dea5ce377e
Co-Authored-By: Andy McCrae <andy.mccrae@gmail.com>
Swift should mirror other roles and use a list instead of a dictionary
for it's filtered_services.
This patch makes that change.
Change-Id: Ie6bf282a36ed63d73996447a88c3c5f6056465a7
Add support for the openSUSE Leap distributions. Nothing special is
required for this except for adding the appropriate distro variables
file and also update the zypper cache before package installation.
Moreover, the syslog user belongs to the 'users' group instead of a
dedicated 'syslog' group so we adjust the defaults for openSUSE.
Depends-On: I96c02fb2ee26691f1d7dd449d7205baa231795fe
Change-Id: I86beac2b3e038a0a4a3bf9618218bc1e393bdf08
This patch implements the use of dynamic includes,
the filtered service list and the elimination of
redundant tasks in order to optimise the role
execution.
Change-Id: Ia957bd80ec6a97a29b4b3a1c28bf37dfc9226ab1
When running a rolling upgrade, each swift node is addressed
individually. This can result in undefined variables when a node doesn't
have the necessary SSH and ring information.
For minor upgrades, these changes are not necessary. They are only
necessary in fresh installs, or modifying storage capacity. Thus, this
patch checks to see if it's a new install (no swift venv_tag previously
defined) or it's the same version (current venv tag matches the previous
one) and only does syncs in those cases.
Change-Id: Ice3b4526609f5a5fd4472c8408de9ed08aea29e6
In order to do a developer mode that allows installation of packages
from local file, which take precedence over the developer_mode
constraints, we need to allow the order of the constraints to be
changed.
This patch adds a "pip_install_developer_constraints" var which is used
to set the developer mode constraints. By default this will leave the
same behaviour but will allow additional constraints to be added, or the
developermode constraints file to be overriden altogether.
Change-Id: I67f880def4b16e519865cc18937807e80b0b82c8
In order to make it easier to detect the currently deployed
venv for a service, and therefore allow smarter decisions
for things like upgrading, we implement the venv tag as a
local fact.
The file used to store facts will be the same for all
OpenStack services, with each service using its own section.
Example:
"ansible_local": {
"openstack_ansible": {
"swift": {
"venv_tag": "14.2.1"
}
}
}
Change-Id: I54f6317ba1b82c29a2b64c59e64cc7a34a82e880
We use an SSH bastion host which we do our deployment through. The
deployment host doesn't have direct access to the same network as the
host. As a result the venv local checksum lookup fails.
I have described this here:
https://bugs.launchpad.net/openstack-ansible/+bug/1689283
This is a simple fix for this problem, assuming everything is good it
will need repeating in multiple places in the code base.
Change-Id: Ib0693f6ff28e2f9e9b2befbd1abc05945a7b6d61
Use the defaults file to set the 'swift_vars' and 'swift_proxy_vars'
variables instead of set_fact tasks.
Change-Id: I52636950652180c4192ad7596dc96c397a7c78f9