194 Commits

Author SHA1 Message Date
Mohammed Naser
52aacc5f0b Update role for new source build process
The variables swift_developer_mode and swift_venv_download
no longer carry any meaning. This review changes swift to
do the equivalent of what developer_mode was all the time,
meaning that it always builds the venv and never requires
the repo server, but it will use a repo server when available.

As part of this, we move the source build out of its own file
because it's now a single task to include the venv build role.
This is just to make it easier to follow the code.

Change-Id: I9233e4daff0034339750477fd21d5cfa181afd83
2019-03-29 03:10:31 +00:00
Jesse Pretorius
430932f274 Remove unnecessary package install duplication
Currently the devel packages are installed everywhere,
but they only need to be where the wheels are built.

Also, there is already a task to install the packages
needed on the target hosts when installing - so we do
not need to give the same list to the venv install role
because they will already have been installed.

We remove the unnecessary installation of the compiling
packages because the python venv build role already does
it. We also remove the curl and which packages as they
appear to be unused. The git package is moved to the
devel package list as it's only used there.

We also re-order the package lists alphabetically to
make them easier to follow.

Depends-On: https://review.openstack.org/613585
Change-Id: I6d037c45aee8b54502a2e3f3add56b23be34109d
2019-01-03 09:27:55 +00:00
Jesse Pretorius
c7e11dfcbd Enable overriding the service setup host python interpreter
In order to enable the service setup host python interpreter to
be changed easily, we make it a variable. This will be useful
when someone sets the service setup host to be the utility
container, because we'll be able to set this var by default.

Change-Id: Icb3ea9ad782218c357ec0e3c577ecbe6fbf60461
2018-11-30 16:42:46 +00:00
Zuul
55f27fdab3 Merge "Add variable for the ssh service and ensure its enabled" 2018-11-30 05:53:13 +00:00
Kevin Carter
d40a020678 Add variable for the ssh service and ensure its enabled
The ssh service on ubuntu based systems is "ssh" which is established by
the service unit path `/lib/systemd/system/ssh.service`. When running
the service will respond to the name "sshd" however this is just an
alias. This change adds a variable to set the service unit name
based on the distro family which will allow the service to start should
it be masked.

The change will now delegate to all nodes within the swift cluster
ensuring ssh is enabled and started. If SSH is not running everywhere at
the same time swift ring distribution will not be possible later on in
the role.

Change-Id: Ifbc748019403f0c0712d5f241f118e998d2061d4
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-11-28 17:52:41 +00:00
Kevin Carter
ca6b0d729a
remove loop control when running include_role
When running an included role with_items + loop control is slow. This
change feeds the role the entire list of curated services which will
improve execution speed, especially when dealing with hundreds of hosts.

Change-Id: I6258d23300c8b26469f8480aa042d53b91d90ad1
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-10-04 00:44:56 -05:00
Jesse Pretorius
7126647d7d Use a common python build/install role
In order to radically simplify how we prepare the service
venvs, we use a common role to do the wheel builds and the
venv preparation. This makes the process far simpler to
understand, because the role does its own building and
installing. It also reduces the code maintenance burden,
because instead of duplicating the build processes in the
repo_build role and the service role - we only have it all
done in a single place.

We also change the role venv tag var to use the integrated
build's common venv tag so that we can remove the role's
venv tag in group_vars in the integrated build. This reduces
memory consumption and also reduces the duplication.

This is by no means the final stop in the simplification
process, but it is a step forward. The will be work to follow
which:

1. Replaces 'developer mode' with an equivalent mechanism
   that uses the common role and is simpler to understand.
   We will also simplify the provisioning of pip install
   arguments when doing this.
2. Simplifies the installation of optional pip packages.
   Right now it's more complicated than it needs to be due
   to us needing to keep the py_pkgs plugin working in the
   integrated build.
3. Deduplicates the distro package installs. Right now the
   role installs the distro packages twice - just before
   building the venv, and during the python_venv_build role
   execution.

Depends-On: https://review.openstack.org/598957
Change-Id: Iecb64d28afe3acfbae7060af55c1a891310e5ef4
Implements: blueprint python-build-install-simplification
Signed-off-by: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
2018-09-03 11:07:02 +00:00
OpenStack Proposal Bot
bfd0db6fb8 Updated from OpenStack Ansible Tests
Change-Id: Iff0bcc1216f9c7b20d6eb32dd8cba50e04808821
2018-08-20 10:50:21 +00:00
Zuul
0960ad9352 Merge "Updated from OpenStack Ansible Tests" 2018-08-18 00:18:04 +00:00
Byron McCollum
5e37f5bc40 Move swift ring tasks before systemd tasks
Ensure swift rings are present before setting up systemd units
and starting the swift services.

Change-Id: I2d3d1355d9624a5f5caa30e49e086326bed13976
2018-08-16 19:40:09 -05:00
OpenStack Proposal Bot
0d4f905964 Updated from OpenStack Ansible Tests
Change-Id: Ie752549598f1793d9b0d9159b1ce2225aa883b7a
2018-08-15 13:42:07 +00:00
Jesse Pretorius
ccc2f8cc49 Allow tags to be used for MQ tasks
The use of 'include_tasks' and a loop of variables creates
a situation where a user is unable to use tags to scope the
inclusion of only the MQ tasks when running the playbooks.

The use-case this is important for is when the rabbitmq
containers are destroyed and rebuilt in order to resolve
an issue with them, and the user wishes to quickly recreate
all the vhosts/users.

Ansible's 'include_tasks' is a dynamic inclusion, and dynamic
inclusions are not included when using tags. The nice thing
about dynamic inclusions is that they completely skip all
tasks when the condition does not apply, cutting down deploy
time. However, given the use-case, we should rather take on
the extra deployment time.

This patch changes the dynamic inclusion to a static one,
adds a 'common-mq' tag to cover all MQ implementations,
and re-implements the 'common-rabbitmq' tag for the tasks
that relate to RabbitMQ specifically.

It also implements conditionals for each task set so that
the rpc/notify tasks can be skipped if a vhost/user is not
required for that purpose (eg: swift does not use RPC, and
most roles will not use notifications by default).

Depends-On: https://review.openstack.org/588191
Change-Id: I994ccc0d20e4ec72955e625119a35856a7171ba3
2018-08-03 09:59:38 +00:00
Jesse Pretorius
fc6c1935d7 Execute service setup against a delegated host using Ansible built-in modules
In order to reduce the packages required to pip install on to the hosts,
we allow the service setup to be delegated to a specific host, defaulting
to the deploy host. We also switch as many tasks as possible to using the
built-in Ansible modules which make use of the shade library.

The 'virtualenv' package is now installed appropriately by the openstack_hosts
role, so there's no need to install it any more. The 'httplib2' package is a
legacy Ansible requirement for the get_url/get_uri module which is no longer
needed. The keystone client library is not required any more now that we're
using the upstream modules. As there are no required packages left, the task
to install them is also removed.

Depends-On: https://review.openstack.org/582359
Depends-On: https://review.openstack.org/587376
Change-Id: I68f3a0bf2b7a3a12cbf40d7d6a853d5b4c6dd0f3
2018-07-31 17:37:02 +00:00
Jean-Philippe Evrard
92766000ed Fix usage of "|" for tests
With the more recent versions of ansible, we should now use
"is" instead of the "|" sign for the tests.

This should fix it.

Change-Id: Ia1c412489554ab629a8ffa8e385a9b634f471861
2018-07-30 08:39:35 +00:00
Jesse Pretorius
1ddd7590cf Move MQ vhost/user creation into role
There is no record for why we implement the MQ vhost/user creation
outside of the role in the playbook, when we could do it inside the
role.

Implementing it inside the role allows us to reduce the quantity of
group_vars duplicated from the role, and allows us to better document
the required variables in the role. The delegation can still be done
as it is done in the playbook too.

In this patch we implement the new variable:
- swift_oslomsg_notify_setup_host

This is used in the role to allow delegation of the MQ vhost/user
setup for each type to any host, but they default to using the first
member of the applicable oslomsg host group.

We also adjust some of the defaults to automatically inherit existing
vars set in group_vars form the integrated build so that we do not
need to do the wiring in the integrated build's group vars. We still
default them in the role too for independent role usage.

Depends-On: https://review.openstack.org/584630
Change-Id: I3d5e00e090d37ea7aa95460965749ef066b63b23
2018-07-26 09:43:10 +00:00
Zuul
11e2b94554 Merge "Drop SELinux support from os_swift" 2018-07-02 22:40:41 +00:00
Mohammed Naser
bad3e9f393 Drop SELinux support from os_swift
The rules are not currently maintained, nor do we have the resources
to maintain them.  In addition, they most likely don't work in our
integrated repos.

For future, it would be best to depend on upstream packages for
SELinux support such as `openstack-selinux`.

Change-Id: I6203b98a96a341ce52930ceeed609d9c118ae8b8
2018-06-30 13:30:47 -04:00
Kevin Carter
2ba6659578
Add systemd tags to include role
There are times when a deployer will need to reconfigure parts of
an environment and having a general purpose tag to run said operation
will be important especicially should the deployer be needing to
reconfigure systemd unit files in a downtime event. This change adds
a general purpose systemd tag where include_role and systemd is found
which will assit operators with day2 operational tasks.

Change-Id: Id65084aff939a240922e46f31c7edb2fae013b38
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-06-27 19:00:19 -05:00
Kevin Carter
882c051bcd
correct typo in service setup
Change-Id: I9100efe6a594c65e17876b682475f0cd8093d21f
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-06-27 18:04:59 -05:00
Markos Chandras
51ea328b7c Add support for using distribution packages for OpenStack services
Distributions provide packages for the OpenStack services so we add
support for using these instead of the pip ones. However, functional
testing is not complete yet since it requires tempest to be in the swift
virtual environment which doesn't exist for distro installs. As such,
for functional testing to cover this method of installation it needs
to be changed quite a bit.

Change-Id: Id75e56440685df407a9991018dc07b5a75429082
Implements: blueprint openstack-distribution-packages
2018-06-04 08:41:19 +01:00
Kevin Carter
8f1cb4dde5 Convert role to use a common systemd service role
This removes the systemd service templates and tasks from this role and
leverages a common systemd service role instead. This change removes a
lot of code duplication across all roles all without sacrificing features
or functionality. The intention of this change is to ensure uniformity and
reduce the maintenance burden on the community when sweeping changes are
needed. The exterior role is built to be OSA compatible and may be pulled
into tree should we deem it necessary.

Change-Id: Icb7ca523cb19c560de5c84b0d60a06305029192c
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-04-22 14:00:27 +00:00
Zuul
d0fe470d50 Merge "Only replace python when re-initializing the venv" 2018-04-19 18:23:50 +00:00
Jesse Pretorius
130e8da005 Do not log passwords
This prevents data to be leaked into the callback plugin.

Some style changes are included to make it easier to read,
the copyright notice is added as it should have been from
the start, and a duplicate retry argument is removed.

Change-Id: I11c3da84fa6469764a19b6a387ef312124dfba2a
2018-04-18 12:57:31 +01:00
Jesse Pretorius
1a789263bd Only replace python when re-initializing the venv
We reinitialize the venv to ensure that the right version of
python is in the venv, but we do not want virtualenv to also
replace pip, setuptools and wheel so we tell it not to. If we
don't do this then virtualenv will install the latest available
version, which is not what we want.

Change-Id: I05d9ee40e596afcc35559b53f6a773521b1d1534
Partial-Bug: #1764470
2018-04-18 11:02:08 +01:00
Major Hayden
5af62fd254
Simplify SELinux check
The ansible_selinux variable is always populated with a 'status'
property, even if SELinux is not installed or configured. This
patch simplifies the check.

Change-Id: I9a7150dec09b486010d533a25692962532708035
Signed-off-by: Major Hayden <major@mhtx.net>
2018-03-06 13:04:25 -06:00
Major Hayden
e115fe9d6a Remove systemd conditionals
Change-Id: I9a81e660d57525be08d9cce44c00fa22c0a53226
2018-02-16 19:32:57 +00:00
Major Hayden
ca76d4b910 Change include: to include_tasks:
This removes warnings in Ansible 2.4+.

The patch also removes the "static:" arguments since they are no
longer used by Ansible.

Change-Id: I1448501507daae92022a803af80839286f3542f2
2018-02-16 19:32:48 +00:00
Major Hayden
1a9c6fb23d Remove Pike task that is labeled for removal
Change-Id: I87691b934fab8cf67196b79c580c8e1fe359ff34
2018-02-16 16:20:12 +00:00
Jean-Philippe Evrard
cbf01f694a Reinitialize the virtualenv
In the previous patch removing the virtualenv-tools, we've updated
paths to the proper binaries, but we don't reset the virtualenv
like what we used to do.

Change-Id: I33906abc0e145dc671c2781924c4c20e9df3ba48
Partial-Bug: #1741634
2018-01-22 17:36:04 +00:00
Zuul
0f26b461f8 Merge "Replace virtualenv-tools by a script" 2018-01-16 13:24:18 +00:00
Jean-Philippe Evrard
d5ad55cf4c Replace virtualenv-tools by a script
virtualenv-tools has a bug which gets triggered in gates: it can't
change the shebang of a virtualenv python bin/ files if they
were generated with a virtualenv script whose shebang ends with
python2 instead of python.

Because we can't modify virtualenv-tools, we use shell scripts
instead.

Change-Id: I2dc9a507162fcd2323e19a3a2daa97c6db9cdcdb
Partial-Bug: #1741634
2018-01-15 14:17:07 +00:00
Major Hayden
6cd15acb26
Remove jinja2 delimiters from when
This patch removes Jinja2 delimiters from when clauses and fixes the
capitalization of task names to match the other tasks in the file.

Closes-Bug: 1735785
Change-Id: I16334a11c551026c593b1872a44ed68f786719a5
2017-12-01 10:24:05 -06:00
Kevin Carter (cloudnull)
988fea87b5 Revert "Don't run sync in upgrades"
Swift fails to deploy when using >1 storage node. The fact `need_sync`
was being set using information from ansible_local in an effort to
support rolling uprades sometime in the future. Sadly this variable
causes storage nodes to be removed from the key sync which results in a
failed build. If we decide serialized rolling uprades are required for
swift we can revisit this however until then we need to remove this
constraint and ensure we're allowing successfule deployments and
upgrades of swift.

This reverts commit 176452bebf7e1ceabd2cf72a89c558c0fc7f1420.

Change-Id: I21318d7c737c2415ef760b46176c3f9b372bb634
2017-10-30 14:52:02 +00:00
Jenkins
729277d436 Merge "Revert "Flag service for restart if the venv changes"" 2017-09-03 06:18:37 +00:00
Major Hayden
20801aa010
Suppress rsync warnings w/command
Ansible throws warnings when we use rsync with the command
module. This patch suppresses those warnings.

Change-Id: I42652318f128120b1ecec4781066c89ee9023636
2017-09-01 09:39:40 -05:00
Jesse Pretorius (odyssey4me)
5b8e343821 Revert "Flag service for restart if the venv changes"
This fact is set, but never used.

This reverts commit 49b82a3573be1e8e56986b73d53da62c3953d08e.

Change-Id: I63e115b37dc89800a68d3c71e8b08cef0e971df6
2017-08-29 11:26:13 +00:00
Nolan Brubaker
49b82a3573 Flag service for restart if the venv changes
Depends-On: I73ff5c97de03cab0cc040f7869379f18c1448b0a
Change-Id: I837e31e5f17b31d4cd6191df196acd3c91bfc91e
2017-08-22 14:50:07 +00:00
Markos Chandras
546378a730 tasks: swift_install: Fix virtualenv-tools issue on openSUSE
openSUSE is also suffering from the same virtualenv-tools issue so
we need to remove the existing python-2.7 symlink.

Link: https://github.com/fireteam/virtualenv-tools/issues/5
Related-Bug: #1637509
Partial-Bug: #1644629
Change-Id: I9011007d949ba4848c4e767b6cfab42e10eb1128
2017-08-17 11:09:03 +00:00
Major Hayden
76bdb799bf Add dnf support
This patch adds dnf support for CentOS.

Implements: blueprint centos-and-dnf
Change-Id: I8114ce7105ba252539c71c0b7fe1ddf73a672d29
2017-08-15 03:11:49 +00:00
Jenkins
4bd2bea7ec Merge "Use a list instead of a dict for filtered_services" 2017-08-14 23:42:44 +00:00
Major Hayden
fa9f41f823
Optimize swift gate
This patch contains five combined improvements.

1) Remove RabbitMQ from the gate

RabbitMQ is only needed in the swift gate if the ceilometer
middleware is being deloyed. This isn't enabled in the gate
job, so let's save some time and deploy memcached/galera
without RabbitMQ.

2) Bindep pyOpenSSL change

This commit also includes the pyOpenSSL bindep change in the
hopes that we can get the swift gate moving along again. ;)

3) Disable erasure code test for CentOS

Due to #1707220 we are running into issues with CentOS 7 running Erasure
Code storage policies on RAX hardware.

This patch temoprarily disables EC testing for CentOS 7, but we can
re-enable that once the package is backported to stable/ocata branch.

4) Pass packages as a list

This patch passes a list of packages to the package module
rather than using "with_items" and it should speed up the
package installation process.

5) Set package state to present

Change-Id: Ic7b13853ce00b0f8ac2181e46e2bd8dea5ce377e
Co-Authored-By: Andy McCrae <andy.mccrae@gmail.com>
2017-08-09 08:43:25 -05:00
Andy McCrae
d860153400 Use a list instead of a dict for filtered_services
Swift should mirror other roles and use a list instead of a dictionary
for it's filtered_services.

This patch makes that change.

Change-Id: Ie6bf282a36ed63d73996447a88c3c5f6056465a7
2017-07-31 17:24:34 +01:00
Markos Chandras
5a3ee5489b Add support for the openSUSE Leap distributions
Add support for the openSUSE Leap distributions. Nothing special is
required for this except for adding the appropriate distro variables
file and also update the zypper cache before package installation.
Moreover, the syslog user belongs to the 'users' group instead of a
dedicated 'syslog' group so we adjust the defaults for openSUSE.

Depends-On: I96c02fb2ee26691f1d7dd449d7205baa231795fe
Change-Id: I86beac2b3e038a0a4a3bf9618218bc1e393bdf08
2017-07-14 10:37:58 +01:00
Jesse Pretorius
01c3728efc Optimise the execution of the role
This patch implements the use of dynamic includes,
the filtered service list and the elimination of
redundant tasks in order to optimise the role
execution.

Change-Id: Ia957bd80ec6a97a29b4b3a1c28bf37dfc9226ab1
2017-07-07 17:52:12 +01:00
Nolan Brubaker
176452bebf Don't run sync in upgrades
When running a rolling upgrade, each swift node is addressed
individually. This can result in undefined variables when a node doesn't
have the necessary SSH and ring information.

For minor upgrades, these changes are not necessary. They are only
necessary in fresh installs, or modifying storage capacity. Thus, this
patch checks to see if it's a new install (no swift venv_tag previously
defined) or it's the same version (current venv tag matches the previous
one) and only does syncs in those cases.

Change-Id: Ice3b4526609f5a5fd4472c8408de9ed08aea29e6
2017-07-05 15:10:01 +00:00
Andy McCrae
33fc6b7673 Allow the developer constraints to be adjusted
In order to do a developer mode that allows installation of packages
from local file, which take precedence over the developer_mode
constraints, we need to allow the order of the constraints to be
changed.

This patch adds a "pip_install_developer_constraints" var which is used
to set the developer mode constraints. By default this will leave the
same behaviour but will allow additional constraints to be added, or the
developermode constraints file to be overriden altogether.

Change-Id: I67f880def4b16e519865cc18937807e80b0b82c8
2017-05-26 11:59:33 +01:00
Jesse Pretorius
39c16e1148 Add venv_tag local fact
In order to make it easier to detect the currently deployed
venv for a service, and therefore allow smarter decisions
for things like upgrading, we implement the venv tag as a
local fact.

The file used to store facts will be the same for all
OpenStack services, with each service using its own section.

Example:

"ansible_local": {
    "openstack_ansible": {
        "swift": {
            "venv_tag": "14.2.1"
        }
    }
}

Change-Id: I54f6317ba1b82c29a2b64c59e64cc7a34a82e880
2017-05-22 14:51:58 +00:00
Rick Box
14d1a2feb8 Fix for lookup and get_url occuring in different places
We use an SSH bastion host which we do our deployment through. The
deployment host doesn't have direct access to the same network as the
host. As a result the venv local checksum lookup fails.

I have described this here:

https://bugs.launchpad.net/openstack-ansible/+bug/1689283

This is a simple fix for this problem, assuming everything is good it
will need repeating in multiple places in the code base.

Change-Id: Ib0693f6ff28e2f9e9b2befbd1abc05945a7b6d61
2017-05-10 14:17:35 +01:00
Jenkins
5043a6d43f Merge "Set defaults for swift_vars and swift_proxy_vars" 2017-04-26 21:02:28 +00:00
Jimmy McCrory
dbd37c4337 Set defaults for swift_vars and swift_proxy_vars
Use the defaults file to set the 'swift_vars' and 'swift_proxy_vars'
variables instead of set_fact tasks.

Change-Id: I52636950652180c4192ad7596dc96c397a7c78f9
2017-04-26 17:11:55 +00:00