The swift3 middleware has been deprecated in replacement with the
s3api middleware instead. This removes all the swift3 references
to hopefully enable someone to cleanly add s3api eventually.
Change-Id: I3a8a1ab861ec81b1f4f8dbc02a6a332d4ce495b8
The variables swift_developer_mode and swift_venv_download
no longer carry any meaning. This review changes swift to
do the equivalent of what developer_mode was all the time,
meaning that it always builds the venv and never requires
the repo server, but it will use a repo server when available.
As part of this, we move the source build out of its own file
because it's now a single task to include the venv build role.
This is just to make it easier to follow the code.
Change-Id: I9233e4daff0034339750477fd21d5cfa181afd83
In order to enable the service setup host python interpreter to
be changed easily, we make it a variable. This will be useful
when someone sets the service setup host to be the utility
container, because we'll be able to set this var by default.
Change-Id: Icb3ea9ad782218c357ec0e3c577ecbe6fbf60461
In order to radically simplify how we prepare the service
venvs, we use a common role to do the wheel builds and the
venv preparation. This makes the process far simpler to
understand, because the role does its own building and
installing. It also reduces the code maintenance burden,
because instead of duplicating the build processes in the
repo_build role and the service role - we only have it all
done in a single place.
We also change the role venv tag var to use the integrated
build's common venv tag so that we can remove the role's
venv tag in group_vars in the integrated build. This reduces
memory consumption and also reduces the duplication.
This is by no means the final stop in the simplification
process, but it is a step forward. The will be work to follow
which:
1. Replaces 'developer mode' with an equivalent mechanism
that uses the common role and is simpler to understand.
We will also simplify the provisioning of pip install
arguments when doing this.
2. Simplifies the installation of optional pip packages.
Right now it's more complicated than it needs to be due
to us needing to keep the py_pkgs plugin working in the
integrated build.
3. Deduplicates the distro package installs. Right now the
role installs the distro packages twice - just before
building the venv, and during the python_venv_build role
execution.
Depends-On: https://review.openstack.org/598957
Change-Id: Iecb64d28afe3acfbae7060af55c1a891310e5ef4
Implements: blueprint python-build-install-simplification
Signed-off-by: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
This reverts commit 6ee96f224c795e8f4d24b394dc95ccb3bb6d8920.
This variable is no longer referenced anywhere, so we can remove it.
Change-Id: I70a262020918d51b20486cf819dc9131abfce7f1
In order to reduce the packages required to pip install on to the hosts,
we allow the service setup to be delegated to a specific host, defaulting
to the deploy host. We also switch as many tasks as possible to using the
built-in Ansible modules which make use of the shade library.
The 'virtualenv' package is now installed appropriately by the openstack_hosts
role, so there's no need to install it any more. The 'httplib2' package is a
legacy Ansible requirement for the get_url/get_uri module which is no longer
needed. The keystone client library is not required any more now that we're
using the upstream modules. As there are no required packages left, the task
to install them is also removed.
Depends-On: https://review.openstack.org/582359
Depends-On: https://review.openstack.org/587376
Change-Id: I68f3a0bf2b7a3a12cbf40d7d6a853d5b4c6dd0f3
The integrated build has no value for swift_rabbitmq_password
and yet the oslomsg vars reference it. Given that the integrated
gate is quite hard to merge anything into right now - set a
default value here temporarily until we can merge a fix there.
Change-Id: I8c13fe063cd5ec454196bc630cb6362f97c4f146
There is no record for why we implement the MQ vhost/user creation
outside of the role in the playbook, when we could do it inside the
role.
Implementing it inside the role allows us to reduce the quantity of
group_vars duplicated from the role, and allows us to better document
the required variables in the role. The delegation can still be done
as it is done in the playbook too.
In this patch we implement the new variable:
- swift_oslomsg_notify_setup_host
This is used in the role to allow delegation of the MQ vhost/user
setup for each type to any host, but they default to using the first
member of the applicable oslomsg host group.
We also adjust some of the defaults to automatically inherit existing
vars set in group_vars form the integrated build so that we do not
need to do the wiring in the integrated build's group vars. We still
default them in the role too for independent role usage.
Depends-On: https://review.openstack.org/584630
Change-Id: I3d5e00e090d37ea7aa95460965749ef066b63b23
The 'httplib2' package is a legacy Ansible requirement for the
get_url/get_uri module which is no longer needed.
Change-Id: I420dfd74e78963486ed9d5a66b8c3b72372b5ae6
The following packages are required in-order to run osprofiler.
these packages will provide deployers the ability to profile
a service on demand should they choose to enable the profile
functionality.
Change-Id: If60f98fc069c040680f58658aacc63a156c4317f
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This introduces oslo.messaging variables that define the Notify transport
in place of the rabbitmq values.
This patch:
* Add oslo.messaging variable for Notify to defaults
* Update url generation
* Add oslo.messaging to inventory
* Add release note
Change-Id: I1c2e844c4c7a2256087bcc4521f970ca8e8c6b16
The keystoneclient package is being installed on the host by PIP but
that means that a whole bunch of required dependencies are being pulled
in as well.
This brings the host to a rather messed up state when installing
keystone from distro packages, since distribution and
PIP packages are being mixed together. We only need the client to
register the service with keystone so we can simply use the distro
package for that to avoid installing lots of PIP packages on the
host.
Change-Id: Id5d79db00e1a4aa4983aafd92c088ef8f13a7da0
Implements: blueprint openstack-distribution-packages
Distributions provide packages for the OpenStack services so we add
support for using these instead of the pip ones. However, functional
testing is not complete yet since it requires tempest to be in the swift
virtual environment which doesn't exist for distro installs. As such,
for functional testing to cover this method of installation it needs
to be changed quite a bit.
Change-Id: Id75e56440685df407a9991018dc07b5a75429082
Implements: blueprint openstack-distribution-packages
This removes the systemd service templates and tasks from this role and
leverages a common systemd service role instead. This change removes a
lot of code duplication across all roles all without sacrificing features
or functionality. The intention of this change is to ensure uniformity and
reduce the maintenance burden on the community when sweeping changes are
needed. The exterior role is built to be OSA compatible and may be pulled
into tree should we deem it necessary.
Change-Id: Icb7ca523cb19c560de5c84b0d60a06305029192c
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
virtualenv-tools has a bug which gets triggered in gates: it can't
change the shebang of a virtualenv python bin/ files if they
were generated with a virtualenv script whose shebang ends with
python2 instead of python.
Because we can't modify virtualenv-tools, we use shell scripts
instead.
Change-Id: I2dc9a507162fcd2323e19a3a2daa97c6db9cdcdb
Partial-Bug: #1741634
This is the time of the cycle where we update all the static
elements in the roles. PyPy has a static file version, so we
bump it.
Change-Id: I15c21c177c32dcae0a7f2710b50f91829be09586
Beginning with commit 6ffcc29 of swift3, the swift3 and s3token middlewares
must come between authtoken and keystoneauth in the swift proxy pipeline.
When 6ffcc29 was committed, reordering was optional at the expense of an
unneceessary call to keystone. However, the change becomes mandatory when using
keystone v3 tokens. Without this change, authtoken will strip the necessary
headers that s3token just added to the environment.
Change-Id: Ic7d07d869aa617ee00190a9aedf411017539b97b
Add support for the openSUSE Leap distributions. Nothing special is
required for this except for adding the appropriate distro variables
file and also update the zypper cache before package installation.
Moreover, the syslog user belongs to the 'users' group instead of a
dedicated 'syslog' group so we adjust the defaults for openSUSE.
Depends-On: I96c02fb2ee26691f1d7dd449d7205baa231795fe
Change-Id: I86beac2b3e038a0a4a3bf9618218bc1e393bdf08
This commit adds support for the swift3 middware, which allows S3-compatible
clients to use swift for object storage.
Change-Id: I56cd63057cc771310b69c311d975e06f73c773f7
Related-Bug: 1625053
The keystonemiddleware library recently switched to using the
cryptography library over pycrypto, which was unmaintained. See
Iced7f5115e49ccf4f7f5bf6813cb5988b95c248b
Change-Id: Ib492bce6df005e8b4e824352e523d2d80c1a8839
Co-Authored-By: Nolan Brubaker <nolan.brubaker@rackspace.com>
Use the defaults file to set the 'swift_vars' and 'swift_proxy_vars'
variables instead of set_fact tasks.
Change-Id: I52636950652180c4192ad7596dc96c397a7c78f9
This creates a specific slice which all OpenStack services will operate
from. By creating an independent slice these components will be governed
away from the system slice allowing us to better optimise resource
consumption.
See the following for more information on slices:
* https://www.freedesktop.org/software/systemd/man/systemd.slice.html
See for following for more information on resource controls:
* https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html
Tools like ``systemd-cgtop`` and ``systemd-cgls`` will now give us
insight into specific processes, process groups, and resouce consumption
in ways that we've not had access to before. To enable some of this reporting
the accounting options have been added to the [Service] section of the unit
file.
Change-Id: Ibcb7d2a69ed67a99c88dc143c76aa8448d31cc9e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Users can configure the number of worker threads. However when it's
not specified the calculated number of workers can get too large on
hosts with a large number of CPUs. Capping only swift proxy server
worker threads when the proxy is in a container. Not capping the
remaining swift services' workers because of the performance impact
it may cause because of the capping.
Change-Id: I12d930552558144ab49fecc0b3776747c1f02166
Fixes the ability to deploy a venv in cases where:
1) developer_mode is not enabled
2) A cached venv is not downloaded from the repo server
Additional cleanup to the developer_mode venv deployment
logic is implemented by adding a *_venv_download var
which is used to decouple developer_mode from the
cached venv extraction process so that a deployer
can force venv builds in-place (disable cached
venv usage) without enabling developer mode
constraints.
Change-Id: Ie19e4e035e5944a984026024aaeebf72313779d2
As a part of removing Trusty from OpenStack-Ansible we shall aim to
remove this from all the roles.
Testing has already been removed for Trusty in the integrated build and
all individual repositories on master (Ocata), as such we can now go
ahead and remove the support within the roles.
Change-Id: I89ba35fd15703aba2a05d11d4550690704bdf272
Implements: blueprint trusty-removal
This patch adds copy as a middleware for swift, this follows upstream's
approach and reduces errors that indicate it is being automatically
included in the pipeline.
Change-Id: I4591ff3f3464d8bfa4ffd012f117aba881b02b65
Move to use tempauth to resolve memory issues resulting from an AIO
swift install running in pypy.
This PR adds some options for using pypy:
* Set a pypy Garbage collection value
This PR includes some tempauth fixes to make it useable:
* Set the tempauth users based on a variable
Testing is changed as follows for pypy:
* Use only memcache within swift-proxy
* Remove galera/keystone
* Add swap for swift-storage hosts
* Use tempauth for pypy
* Reduce to 2 swift hosts
Change-Id: Ic1ed5acc9b20853d9a159035226f97fda088f035
Versioned Objects in Swift now use a middleware that is added to the
pipeline instead of the "allow_versions" in the container config. These
have 2 different Headers, so to prevent "X-Versions-Location" from not
working the "allow_versions" in the container config is left in place.
This patch adds the middleware by default and sets the Versioned Objects
support to be on. This is the upstream default and the deployer would
still need to set the "X-History-Location" header on a container to
utilise the feature.
Change-Id: I88811fd77fad8d2241448ca5ffb565fa7d704a00
We don't setup hosts entries for replication addresses on storage hosts,
by default. Rsync is performing a reverse lookup on each rsync, and
failing. We should turn this reverse lookup off by default, but allow
this to be set to True.
This PR adds the "swift_rsync_reverse_lookup" boolean which is defaulted
to false.
Change-Id: Ie98113fd71a70ce9c74c3812046be6959b9a353f
The current constraints generation for the
installation involves multiple tasks and multiple
variables.
Using multiple tasks extends the installation time
unnecessarily and the additional variables are
unnecessary.
This patch aims to simplify the mechanism and
hopes to speed it up a little.
Change-Id: Icdd5d053346e7d31dcfd4331c7b7b0f184bf376b
Swift Erasure Coding requires the object-reconstructor to be running,
this PR adds the object-reconstructor configuration and systemd/upstart
file.
Additionally, this PR adds testing for an erasure-coding policy as well
as for multiple policies in 1 deployment.
Change-Id: I71fe3d77e93112b9d27c93b37b59e6242cb3e00e
Closes-Bug: #1634859
When not using dedicated replication systemd still puts init scripts
down, which take a long time to restart/start. upstart scripts get
around this by setting blank scripts.
Now that we are using a service dict we can do better by defining an
"service_en" flag and not setting up scripts when the service isn't
enabled.
Additionally, the systemd tempfiles and init files were not using the
appropriate "program_binary" variable at all, this has been fixed.
Change-Id: Iae569bfe38a440fb09e56658b3a934799a8821e8
Change the 'swift_x_program_names' from a list to a dictionary
mapping of services, groups that install those services. This
brings the method into line with that used in the os_neutron role
in order to implement a more standardised method.
The init tasks have been updated to run once and loop through this
mapping rather than being included multiple times and re-run against
each host. This may potentially reduce role run times.
Currently the reload of upstart/systemd scripts may not happen if
only one script changes as the task uses a loop with only one result
register. This patch implements handlers to reload upstart/systemd
scripts to ensure that this happens when any one of the scripts
change.
The handler to reload the services now only tries to restart the
service if the host is in the group for the service according to the
service group mapping. This allows us to ensure that handler
failures are no longer ignored and that no execution time is wasted
trying to restart services which do not exist on the host.
Finally:
- Common variables shared by each service's template files have
been updated to use the service namespaced variables.
- Unused handlers have been removed.
- Unused variables have been removed.
Change-Id: Id35de501acf6b3164221085f8f9e142234ea0d73
We need to ensure the swift services are listening on the same IP and
port as the ring is setup for. Currently we calculate these values in
different ways, which leads to the possibility that the ring is setup
for ports and IPs that the services are not listening on.
This change ensures this by calculating the storage and replication
addresses within the role - whilst still accepting overrides from
outside the role. This ensures functionality does not change, but that
it is now not possible for the swift services to not listen on the
ports/IPs defined in the ring.
As part of this change we move to set "swift_dedicated_replication"
based on whether the storage and replication addresses are the same,
rather than the specified network. This means you can run a dedicated
replication config by configuring the services, which was possible to
configure, but again would have meant the services and ring were not
configured in the same way. This is required to ensure uniformity.
Finally - we remove the ability to set a storage_port or repl_port
within the swift_vars, this was a setting that would never work in it's
current implementation. The storage/repl_port would depend entirely on
which swift service was in use, and can not be set in such a blanket
fashion. Since this was a completely not working funcitonality it has
now been removed.
Change-Id: Ibe5330cdca7fd81f379b80dc55ad06529ce4d580
Create a new default var, swift_dedicated_replication, for storing
conditions used to determine if a replication network is being used for
swift. This allows for removing redundant tasks, reducing role run
times.
Change-Id: Ieb4263035527ff069017d40b95787f934fbd308c
We should never change the swift_hash_path_prefix/suffix variables on a
running cluster. This PR implements a check that will fail if the
variable is different to what is already on disk.
To ensure this is still possible this PR implements a
"swift_force_change_hashes" variable which can be set to "True" in order
to force change the swift_hash_path variables regardless of whether they
are different or not.
Change-Id: Idaedc125aede22c347668afd9e98ed1823eb142c
When using Swift storage for Gnocchi we want to add a filter to the
Ceilometer middleware for Swift to exclude reporting traffic caused
by Gnocchi storing data into Swift to avoid feedback-loops. This
is typically done by isolating the Gnocchi in a dedicated Project in
Keystone.
The best place to filter this traffic is in the middleware
as opposed to filtering in the Ceilometer Collector.
Related-To: I67b5bff394ad35cf95d10ba32d602954799d7348
Change-Id: If20248494d4e6c127db4ffdf77ae43482f64fe58