110 Commits

Author SHA1 Message Date
Mohammed Naser
4de1219730 swift3: remove support
The swift3 middleware has been deprecated in replacement with the
s3api middleware instead.  This removes all the swift3 references
to hopefully enable someone to cleanly add s3api eventually.

Change-Id: I3a8a1ab861ec81b1f4f8dbc02a6a332d4ce495b8
2019-03-29 09:41:50 -04:00
Mohammed Naser
52aacc5f0b Update role for new source build process
The variables swift_developer_mode and swift_venv_download
no longer carry any meaning. This review changes swift to
do the equivalent of what developer_mode was all the time,
meaning that it always builds the venv and never requires
the repo server, but it will use a repo server when available.

As part of this, we move the source build out of its own file
because it's now a single task to include the venv build role.
This is just to make it easier to follow the code.

Change-Id: I9233e4daff0034339750477fd21d5cfa181afd83
2019-03-29 03:10:31 +00:00
Jesse Pretorius
c7e11dfcbd Enable overriding the service setup host python interpreter
In order to enable the service setup host python interpreter to
be changed easily, we make it a variable. This will be useful
when someone sets the service setup host to be the utility
container, because we'll be able to set this var by default.

Change-Id: Icb3ea9ad782218c357ec0e3c577ecbe6fbf60461
2018-11-30 16:42:46 +00:00
Jonathan Rosser
87fb727472 Correct swift replication service units
These pointed to the wrong binaries, and in the case of the object
replicator, a non-existant binary. The names of the binaries have
been checked here:

https://packages.ubuntu.com/bionic/all/swift-account/filelist
https://packages.ubuntu.com/bionic/all/swift-container/filelist
https://packages.ubuntu.com/bionic/all/swift-object/filelist

Change-Id: Ib8d4875d44130048c5a5d8b2973d6da3a4321abe
2018-11-02 20:47:15 +00:00
Jesse Pretorius
7126647d7d Use a common python build/install role
In order to radically simplify how we prepare the service
venvs, we use a common role to do the wheel builds and the
venv preparation. This makes the process far simpler to
understand, because the role does its own building and
installing. It also reduces the code maintenance burden,
because instead of duplicating the build processes in the
repo_build role and the service role - we only have it all
done in a single place.

We also change the role venv tag var to use the integrated
build's common venv tag so that we can remove the role's
venv tag in group_vars in the integrated build. This reduces
memory consumption and also reduces the duplication.

This is by no means the final stop in the simplification
process, but it is a step forward. The will be work to follow
which:

1. Replaces 'developer mode' with an equivalent mechanism
   that uses the common role and is simpler to understand.
   We will also simplify the provisioning of pip install
   arguments when doing this.
2. Simplifies the installation of optional pip packages.
   Right now it's more complicated than it needs to be due
   to us needing to keep the py_pkgs plugin working in the
   integrated build.
3. Deduplicates the distro package installs. Right now the
   role installs the distro packages twice - just before
   building the venv, and during the python_venv_build role
   execution.

Depends-On: https://review.openstack.org/598957
Change-Id: Iecb64d28afe3acfbae7060af55c1a891310e5ef4
Implements: blueprint python-build-install-simplification
Signed-off-by: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
2018-09-03 11:07:02 +00:00
Zuul
c3c1bffde9 Merge "Revert "Unblock gate: Provide default rabbitmq password"" 2018-08-07 19:46:35 +00:00
Jesse Pretorius (odyssey4me)
941febe046 Revert "Unblock gate: Provide default rabbitmq password"
This reverts commit 6ee96f224c795e8f4d24b394dc95ccb3bb6d8920.

This variable is no longer referenced anywhere, so we can remove it.

Change-Id: I70a262020918d51b20486cf819dc9131abfce7f1
2018-08-07 12:31:20 +00:00
Zuul
9df587fa4d Merge "Execute service setup against a delegated host using Ansible built-in modules" 2018-08-01 15:00:03 +00:00
Jesse Pretorius
fc6c1935d7 Execute service setup against a delegated host using Ansible built-in modules
In order to reduce the packages required to pip install on to the hosts,
we allow the service setup to be delegated to a specific host, defaulting
to the deploy host. We also switch as many tasks as possible to using the
built-in Ansible modules which make use of the shade library.

The 'virtualenv' package is now installed appropriately by the openstack_hosts
role, so there's no need to install it any more. The 'httplib2' package is a
legacy Ansible requirement for the get_url/get_uri module which is no longer
needed. The keystone client library is not required any more now that we're
using the upstream modules. As there are no required packages left, the task
to install them is also removed.

Depends-On: https://review.openstack.org/582359
Depends-On: https://review.openstack.org/587376
Change-Id: I68f3a0bf2b7a3a12cbf40d7d6a853d5b4c6dd0f3
2018-07-31 17:37:02 +00:00
Jesse Pretorius
6ee96f224c Unblock gate: Provide default rabbitmq password
The integrated build has no value for swift_rabbitmq_password
and yet the oslomsg vars reference it. Given that the integrated
gate is quite hard to merge anything into right now - set a
default value here temporarily until we can merge a fix there.

Change-Id: I8c13fe063cd5ec454196bc630cb6362f97c4f146
2018-07-31 13:23:55 +01:00
Zuul
d97fa0fd53 Merge "Move MQ vhost/user creation into role" 2018-07-28 08:20:19 +00:00
Jesse Pretorius
1ddd7590cf Move MQ vhost/user creation into role
There is no record for why we implement the MQ vhost/user creation
outside of the role in the playbook, when we could do it inside the
role.

Implementing it inside the role allows us to reduce the quantity of
group_vars duplicated from the role, and allows us to better document
the required variables in the role. The delegation can still be done
as it is done in the playbook too.

In this patch we implement the new variable:
- swift_oslomsg_notify_setup_host

This is used in the role to allow delegation of the MQ vhost/user
setup for each type to any host, but they default to using the first
member of the applicable oslomsg host group.

We also adjust some of the defaults to automatically inherit existing
vars set in group_vars form the integrated build so that we do not
need to do the wiring in the integrated build's group vars. We still
default them in the role too for independent role usage.

Depends-On: https://review.openstack.org/584630
Change-Id: I3d5e00e090d37ea7aa95460965749ef066b63b23
2018-07-26 09:43:10 +00:00
Jean-Philippe Evrard
9379dc8aee Remove httplib
The 'httplib2' package is a legacy Ansible requirement for the
get_url/get_uri module which is no longer needed.

Change-Id: I420dfd74e78963486ed9d5a66b8c3b72372b5ae6
2018-07-25 07:39:18 +00:00
Zuul
cf7d3077a9 Merge "Update to use oslo.messaging service for Notify" 2018-07-02 10:19:57 +00:00
Kevin Carter
6175ff1e8d
Add packages required for osprofiler
The following packages are required in-order to run osprofiler.
these packages will provide deployers the ability to profile
a service on demand should they choose to enable the profile
functionality.

Change-Id: If60f98fc069c040680f58658aacc63a156c4317f
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-06-11 22:59:09 -05:00
Andrew Smith
91af246bdf Update to use oslo.messaging service for Notify
This introduces oslo.messaging variables that define the Notify transport
in place of the rabbitmq values.

This patch:
* Add oslo.messaging variable for Notify to defaults
* Update url generation
* Add oslo.messaging to inventory
* Add release note

Change-Id: I1c2e844c4c7a2256087bcc4521f970ca8e8c6b16
2018-06-11 16:45:16 -04:00
Markos Chandras
4bf5de8d30 Do not pip install keystoneclient on the host
The keystoneclient package is being installed on the host by PIP but
that means that a whole bunch of required dependencies are being pulled
in as well.

This brings the host to a rather messed up state when installing
keystone from distro packages, since distribution and
PIP packages are being mixed together. We only need the client to
register the service with keystone so we can simply use the distro
package for that to avoid installing lots of PIP packages on the
host.

Change-Id: Id5d79db00e1a4aa4983aafd92c088ef8f13a7da0
Implements: blueprint openstack-distribution-packages
2018-06-04 08:41:20 +01:00
Markos Chandras
51ea328b7c Add support for using distribution packages for OpenStack services
Distributions provide packages for the OpenStack services so we add
support for using these instead of the pip ones. However, functional
testing is not complete yet since it requires tempest to be in the swift
virtual environment which doesn't exist for distro installs. As such,
for functional testing to cover this method of installation it needs
to be changed quite a bit.

Change-Id: Id75e56440685df407a9991018dc07b5a75429082
Implements: blueprint openstack-distribution-packages
2018-06-04 08:41:19 +01:00
Markos Chandras
ae8e01aada defaults: Do not install the cinderclient package
Nothing in the swift role needs the cinderclient package so there is
no need to install it.

Change-Id: I01b2a71fa185745b1b73546222779ef71889258a
2018-05-31 09:26:57 +01:00
Kevin Carter
8f1cb4dde5 Convert role to use a common systemd service role
This removes the systemd service templates and tasks from this role and
leverages a common systemd service role instead. This change removes a
lot of code duplication across all roles all without sacrificing features
or functionality. The intention of this change is to ensure uniformity and
reduce the maintenance burden on the community when sweeping changes are
needed. The exterior role is built to be OSA compatible and may be pulled
into tree should we deem it necessary.

Change-Id: Icb7ca523cb19c560de5c84b0d60a06305029192c
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-04-22 14:00:27 +00:00
Zuul
0f26b461f8 Merge "Replace virtualenv-tools by a script" 2018-01-16 13:24:18 +00:00
Jean-Philippe Evrard
d5ad55cf4c Replace virtualenv-tools by a script
virtualenv-tools has a bug which gets triggered in gates: it can't
change the shebang of a virtualenv python bin/ files if they
were generated with a virtualenv script whose shebang ends with
python2 instead of python.

Because we can't modify virtualenv-tools, we use shell scripts
instead.

Change-Id: I2dc9a507162fcd2323e19a3a2daa97c6db9cdcdb
Partial-Bug: #1741634
2018-01-15 14:17:07 +00:00
Jean-Philippe Evrard
33505b9482 Update PyPy version
This is the time of the cycle where we update all the static
elements in the roles. PyPy has a static file version, so we
bump it.

Change-Id: I15c21c177c32dcae0a7f2710b50f91829be09586
2017-11-27 09:41:08 +00:00
Charles Farquhar
8b2fc7afc7 Fix ordering of swift3 in middleware pipeline
Beginning with commit 6ffcc29 of swift3, the swift3 and s3token middlewares
must come between authtoken and keystoneauth in the swift proxy pipeline.

When 6ffcc29 was committed, reordering was optional at the expense of an
unneceessary call to keystone.  However, the change becomes mandatory when using
keystone v3 tokens.  Without this change, authtoken will strip the necessary
headers that s3token just added to the environment.

Change-Id: Ic7d07d869aa617ee00190a9aedf411017539b97b
2017-07-27 15:52:58 -05:00
Markos Chandras
5a3ee5489b Add support for the openSUSE Leap distributions
Add support for the openSUSE Leap distributions. Nothing special is
required for this except for adding the appropriate distro variables
file and also update the zypper cache before package installation.
Moreover, the syslog user belongs to the 'users' group instead of a
dedicated 'syslog' group so we adjust the defaults for openSUSE.

Depends-On: I96c02fb2ee26691f1d7dd449d7205baa231795fe
Change-Id: I86beac2b3e038a0a4a3bf9618218bc1e393bdf08
2017-07-14 10:37:58 +01:00
Charles Farquhar
14c57dd419 Add support for swift3 middleware
This commit adds support for the swift3 middware, which allows S3-compatible
clients to use swift for object storage.

Change-Id: I56cd63057cc771310b69c311d975e06f73c773f7
Related-Bug: 1625053
2017-06-23 18:26:46 +00:00
Jesse Pretorius
b14497f8aa Switch to Cryptography over pycrypto
The keystonemiddleware library recently switched to using the
cryptography library over pycrypto, which was unmaintained. See
Iced7f5115e49ccf4f7f5bf6813cb5988b95c248b

Change-Id: Ib492bce6df005e8b4e824352e523d2d80c1a8839
Co-Authored-By: Nolan Brubaker <nolan.brubaker@rackspace.com>
2017-06-07 17:58:32 +00:00
Jimmy McCrory
dbd37c4337 Set defaults for swift_vars and swift_proxy_vars
Use the defaults file to set the 'swift_vars' and 'swift_proxy_vars'
variables instead of set_fact tasks.

Change-Id: I52636950652180c4192ad7596dc96c397a7c78f9
2017-04-26 17:11:55 +00:00
Jenkins
6a1669fb3d Merge "Deprecate auth_plugin option" 2017-04-06 15:58:13 +00:00
Andy McCrae
78b4f6aac6 Update container-sync to use internal client
This is to match the way this is configured in upstream Swift.

Change-Id: Ifbfd4158e04f38bf3383a7c2461374a21088fdd3
2017-04-04 15:04:19 +00:00
Kevin Carter
867f3157e9 Ensure the components are isolated from the system
This creates a specific slice which all OpenStack services will operate
from. By creating an independent slice these components will be governed
away from the system slice allowing us to better optimise resource
consumption.

See the following for more information on slices:

* https://www.freedesktop.org/software/systemd/man/systemd.slice.html

See for following for more information on resource controls:

* https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html

Tools like ``systemd-cgtop`` and ``systemd-cgls`` will now give us
insight into specific processes, process groups, and resouce consumption
in ways that we've not had access to before. To enable some of this reporting
the accounting options have been added to the [Service] section of the unit
file.

Change-Id: Ibcb7d2a69ed67a99c88dc143c76aa8448d31cc9e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2017-03-30 15:16:03 +00:00
ZhongShengping
7ca4b62eb5 Deprecate auth_plugin option
Option "auth_plugin" is deprecated, use option "auth_type" instead.

Change-Id: I6ac382f3856b754291378cde335bc71cd2161023
Implements: blueprint deprecate-auth-plugin
2017-03-25 12:26:23 +08:00
Ravi Gummadi
bb9e3f5828 Cap the number of worker threads
Users can configure the number of worker threads. However when it's
not specified the calculated number of workers can get too large on
hosts with a large number of CPUs. Capping only swift proxy server
worker threads when the proxy is in a container. Not capping the
remaining swift services' workers because of the performance impact
it may cause because of the capping.

Change-Id: I12d930552558144ab49fecc0b3776747c1f02166
2017-03-06 06:39:01 -05:00
Logan V
2f1291bf55 Clean up developer mode logic
Fixes the ability to deploy a venv in cases where:
1) developer_mode is not enabled
2) A cached venv is not downloaded from the repo server

Additional cleanup to the developer_mode venv deployment
logic is implemented by adding a *_venv_download var
which is used to decouple developer_mode from the
cached venv extraction process so that a deployer
can force venv builds in-place (disable cached
venv usage) without enabling developer mode
constraints.

Change-Id: Ie19e4e035e5944a984026024aaeebf72313779d2
2017-01-13 19:52:42 -06:00
Jenkins
a0d6cdea8a Merge "Remove Trusty support from os_swift role" 2016-12-09 11:25:21 +00:00
Andy McCrae
26bd8127b6 Remove Trusty support from os_swift role
As a part of removing Trusty from OpenStack-Ansible we shall aim to
remove this from all the roles.

Testing has already been removed for Trusty in the integrated build and
all individual repositories on master (Ocata), as such we can now go
ahead and remove the support within the roles.

Change-Id: I89ba35fd15703aba2a05d11d4550690704bdf272
Implements: blueprint trusty-removal
2016-12-08 11:50:58 +00:00
Andy McCrae
db66b43dd2 Set copy as a middleware for Swift
This patch adds copy as a middleware for swift, this follows upstream's
approach and reduces errors that indicate it is being automatically
included in the pipeline.

Change-Id: I4591ff3f3464d8bfa4ffd012f117aba881b02b65
2016-12-07 14:29:52 +00:00
Jenkins
8238849e41 Merge "Allow X-History-Location versioned_writes" 2016-12-03 03:14:48 +00:00
Andy McCrae
3fb6784459 Attempt to fix swift_pypy memory issues
Move to use tempauth to resolve memory issues resulting from an AIO
swift install running in pypy.

This PR adds some options for using pypy:
* Set a pypy Garbage collection value

This PR includes some tempauth fixes to make it useable:
* Set the tempauth users based on a variable

Testing is changed as follows for pypy:
* Use only memcache within swift-proxy
* Remove galera/keystone
* Add swap for swift-storage hosts
* Use tempauth for pypy
* Reduce to 2 swift hosts

Change-Id: Ic1ed5acc9b20853d9a159035226f97fda088f035
2016-12-01 11:07:10 +00:00
Andy McCrae
de7ab0a800 Allow X-History-Location versioned_writes
Versioned Objects in Swift now use a middleware that is added to the
pipeline instead of the "allow_versions" in the container config. These
have 2 different Headers, so to prevent "X-Versions-Location" from not
working the "allow_versions" in the container config is left in place.

This patch adds the middleware by default and sets the Versioned Objects
support to be on. This is the upstream default and the deployer would
still need to set the "X-History-Location" header on a container to
utilise the feature.

Change-Id: I88811fd77fad8d2241448ca5ffb565fa7d704a00
2016-11-30 11:18:45 +00:00
Andy McCrae
4aa96a832c Set rsync to not perform reverse lookups
We don't setup hosts entries for replication addresses on storage hosts,
by default. Rsync is performing a reverse lookup on each rsync, and
failing. We should turn this reverse lookup off by default, but allow
this to be set to True.

This PR adds the "swift_rsync_reverse_lookup" boolean which is defaulted
to false.

Change-Id: Ie98113fd71a70ce9c74c3812046be6959b9a353f
2016-11-07 16:24:52 +00:00
Jesse Pretorius
ead9bc82b0 Simplify pip options/constraints mechanism
The current constraints generation for the
installation involves multiple tasks and multiple
variables.

Using multiple tasks extends the installation time
unnecessarily and the additional variables are
unnecessary.

This patch aims to simplify the mechanism and
hopes to speed it up a little.

Change-Id: Icdd5d053346e7d31dcfd4331c7b7b0f184bf376b
2016-11-02 18:07:36 +00:00
Logan V
7de60df8c3 Fix linting issues for ansible-lint 3.4.1
Preparing this role for the ansible-lint version bump

Change-Id: Ib78b5fd36dcd23d18fc13382359f5099405856d0
2016-11-02 13:03:19 +00:00
Andy McCrae
cd222e8cff Add object-reconstructor for swift EC
Swift Erasure Coding requires the object-reconstructor to be running,
this PR adds the object-reconstructor configuration and systemd/upstart
file.

Additionally, this PR adds testing for an erasure-coding policy as well
as for multiple policies in 1 deployment.

Change-Id: I71fe3d77e93112b9d27c93b37b59e6242cb3e00e
Closes-Bug: #1634859
2016-10-19 18:36:14 +01:00
Andy McCrae
033aa502e5 Fix swift init scripts w/o dedicated replication
When not using dedicated replication systemd still puts init scripts
down, which take a long time to restart/start. upstart scripts get
around this by setting blank scripts.

Now that we are using a service dict we can do better by defining an
"service_en" flag and not setting up scripts when the service isn't
enabled.

Additionally, the systemd tempfiles and init files were not using the
appropriate "program_binary" variable at all, this has been fixed.

Change-Id: Iae569bfe38a440fb09e56658b3a934799a8821e8
2016-10-14 18:19:51 +01:00
Andy McCrae
bf1ab1750a Use dictionary for service mappings
Change the 'swift_x_program_names' from a list to a dictionary
mapping of services, groups that install those services. This
brings the method into line with that used in the os_neutron role
in order to implement a more standardised method.

The init tasks have been updated to run once and loop through this
mapping rather than being included multiple times and re-run against
each host. This may potentially reduce role run times.

Currently the reload of upstart/systemd scripts may not happen if
only one script changes as the task uses a loop with only one result
register. This patch implements handlers to reload upstart/systemd
scripts to ensure that this happens when any one of the scripts
change.

The handler to reload the services now only tries to restart the
service if the host is in the group for the service according to the
service group mapping. This allows us to ensure that handler
failures are no longer ignored and that no execution time is wasted
trying to restart services which do not exist on the host.

Finally:
- Common variables shared by each service's template files have
  been updated to use the service namespaced variables.
- Unused handlers have been removed.
- Unused variables have been removed.

Change-Id: Id35de501acf6b3164221085f8f9e142234ea0d73
2016-10-13 13:30:58 +01:00
Andy McCrae
9578aa5fd7 Ensure storage/replication_ip are uniform
We need to ensure the swift services are listening on the same IP and
port as the ring is setup for. Currently we calculate these values in
different ways, which leads to the possibility that the ring is setup
for ports and IPs that the services are not listening on.

This change ensures this by calculating the storage and replication
addresses within the role - whilst still accepting overrides from
outside the role. This ensures functionality does not change, but that
it is now not possible for the swift services to not listen on the
ports/IPs defined in the ring.

As part of this change we move to set "swift_dedicated_replication"
based on whether the storage and replication addresses are the same,
rather than the specified network. This means you can run a dedicated
replication config by configuring the services, which was possible to
configure, but again would have meant the services and ring were not
configured in the same way. This is required to ensure uniformity.

Finally - we remove the ability to set a storage_port or repl_port
within the swift_vars, this was a setting that would never work in it's
current implementation. The storage/repl_port would depend entirely on
which swift service was in use, and can not be set in such a blanket
fashion. Since this was a completely not working funcitonality it has
now been removed.

Change-Id: Ibe5330cdca7fd81f379b80dc55ad06529ce4d580
2016-09-26 11:23:15 +01:00
Jimmy McCrory
98cb749139 Store dedicated replication conditions in var
Create a new default var, swift_dedicated_replication, for storing
conditions used to determine if a replication network is being used for
swift. This allows for removing redundant tasks, reducing role run
times.

Change-Id: Ieb4263035527ff069017d40b95787f934fbd308c
2016-09-21 06:32:35 -07:00
Andy McCrae
10d3ee9026 Add check for swift_hash_path_ variables
We should never change the swift_hash_path_prefix/suffix variables on a
running cluster. This PR implements a check that will fail if the
variable is different to what is already on disk.

To ensure this is still possible this PR implements a
"swift_force_change_hashes" variable which can be set to "True" in order
to force change the swift_hash_path variables regardless of whether they
are different or not.

Change-Id: Idaedc125aede22c347668afd9e98ed1823eb142c
2016-09-14 13:06:53 +00:00
Steve Lewis
1d55c377b8 Filter Gnocchi traffic optionally
When using Swift storage for Gnocchi we want to add a filter to the
Ceilometer middleware for Swift to exclude reporting traffic caused
by Gnocchi storing data into Swift to avoid feedback-loops. This
is typically done by isolating the Gnocchi in a dedicated Project in
Keystone.

The best place to filter this traffic is in the middleware
as opposed to filtering in the Ceilometer Collector.

Related-To: I67b5bff394ad35cf95d10ba32d602954799d7348
Change-Id: If20248494d4e6c127db4ffdf77ae43482f64fe58
2016-09-07 11:40:43 -07:00