In order to reduce the packages required to pip install on to the hosts,
we allow the service setup to be delegated to a specific host, defaulting
to the deploy host. We also switch as many tasks as possible to using the
built-in Ansible modules which make use of the shade library.
The 'virtualenv' package is now installed appropriately by the openstack_hosts
role, so there's no need to install it any more. The 'httplib2' package is a
legacy Ansible requirement for the get_url/get_uri module which is no longer
needed. The keystone client library is not required any more now that we're
using the upstream modules. As there are no required packages left, the task
to install them is also removed.
Depends-On: https://review.openstack.org/582359
Depends-On: https://review.openstack.org/587376
Change-Id: I68f3a0bf2b7a3a12cbf40d7d6a853d5b4c6dd0f3
This introduces oslo.messaging variables that define the Notify transport
in place of the rabbitmq values.
This patch:
* Add oslo.messaging variable for Notify to defaults
* Update url generation
* Add oslo.messaging to inventory
* Add release note
Change-Id: I1c2e844c4c7a2256087bcc4521f970ca8e8c6b16
Distributions provide packages for the OpenStack services so we add
support for using these instead of the pip ones. However, functional
testing is not complete yet since it requires tempest to be in the swift
virtual environment which doesn't exist for distro installs. As such,
for functional testing to cover this method of installation it needs
to be changed quite a bit.
Change-Id: Id75e56440685df407a9991018dc07b5a75429082
Implements: blueprint openstack-distribution-packages
Release notes are version independent, so remove version/release
values. We've found that projects now require the service package
to be installed in order to build release notes, and this is entirely
due to the current convention of pulling in the version information.
Release notes should not need installation in order to build, so this
unnecessary version setting needs to be removed.
This is needed for new release notes publishing, see
I56909152975f731a9d2c21b2825b972195e48ee8 and the discussion starting
at
http://lists.openstack.org/pipermail/openstack-dev/2017-November/124480.html
.
Change-Id: I7df0385b62d118b535179dd42d7aa576897815ca
This changes the a-r-r with the proper version, and fixes
the repo path for role upgrades:
The role will, during test-upgrade-pre, have its current branch
checked out, and will deploy the current branch infrastructure,
including the generation of the constraints.
Then, the installation of previous branch of the role will use
the constraints for its previous version, but will have no
constraints for it, and fail.
We need to generate the constraints for the previous version too.
This should do it.
This patch also implements an initial set of jobs intended to
match the current job execution method. It does not intend to
improve how the jobs are executed - only to replicate what is
currently in openstack-infra/openstack-zuul-jobs and provide
the platform to iterate on.
Change-Id: I4aa7591c931501490e706b9d61ba0bc12c006baa
As part of the docs migration work[0] for Pike we need to switch to use the
openstackdocstheme.
[0]https://review.openstack.org/#/c/472275/
Change-Id: Ia3d468d5ed5c31314a20cf1c7b4b6a9829628064
This commit adds support for the swift3 middware, which allows S3-compatible
clients to use swift for object storage.
Change-Id: I56cd63057cc771310b69c311d975e06f73c773f7
Related-Bug: 1625053
The systemd unit 'TimeoutSec' value which controls the time
between sending a SIGTERM signal and a SIGKILL signal when
stopping or restarting the service has been reduced from 300
seconds to 120 seconds. This provides 2 minutes for long-lived
sessions to drain while preventing new ones from starting
before a restart or a stop.
The 'RestartSec' value which controls the time between the
service stop and start when restarting has been reduced from
150 seconds to 2 seconds to make the restart happen faster.
These values can be adjusted by using the *_init_config_overrides
variables which use the config_template task to change template
defaults.
Change-Id: I048b877e859ad744dc54f19a93afdd89f8ef1661
This creates a specific slice which all OpenStack services will operate
from. By creating an independent slice these components will be governed
away from the system slice allowing us to better optimise resource
consumption.
See the following for more information on slices:
* https://www.freedesktop.org/software/systemd/man/systemd.slice.html
See for following for more information on resource controls:
* https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html
Tools like ``systemd-cgtop`` and ``systemd-cgls`` will now give us
insight into specific processes, process groups, and resouce consumption
in ways that we've not had access to before. To enable some of this reporting
the accounting options have been added to the [Service] section of the unit
file.
Change-Id: Ibcb7d2a69ed67a99c88dc143c76aa8448d31cc9e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Users can configure the number of worker threads. However when it's
not specified the calculated number of workers can get too large on
hosts with a large number of CPUs. Capping only swift proxy server
worker threads when the proxy is in a container. Not capping the
remaining swift services' workers because of the performance impact
it may cause because of the capping.
Change-Id: I12d930552558144ab49fecc0b3776747c1f02166
Move to use tempauth to resolve memory issues resulting from an AIO
swift install running in pypy.
This PR adds some options for using pypy:
* Set a pypy Garbage collection value
This PR includes some tempauth fixes to make it useable:
* Set the tempauth users based on a variable
Testing is changed as follows for pypy:
* Use only memcache within swift-proxy
* Remove galera/keystone
* Add swap for swift-storage hosts
* Use tempauth for pypy
* Reduce to 2 swift hosts
Change-Id: Ic1ed5acc9b20853d9a159035226f97fda088f035
Versioned Objects in Swift now use a middleware that is added to the
pipeline instead of the "allow_versions" in the container config. These
have 2 different Headers, so to prevent "X-Versions-Location" from not
working the "allow_versions" in the container config is left in place.
This patch adds the middleware by default and sets the Versioned Objects
support to be on. This is the upstream default and the deployer would
still need to set the "X-History-Location" header on a container to
utilise the feature.
Change-Id: I88811fd77fad8d2241448ca5ffb565fa7d704a00
This PR Adds CentOS 7 support for Swift.
The following was required to get CentOS 7 to work:
* Add yum install path + packages
* Variablize rsync service name
* Gather network interface facts prior to setting storage/repl IPs
* Ensure /etc/defaults/rsync is only set for apt installations.
* Ensure the rsyslog service is started and enabled.
Change-Id: Ibaf8bc8d54b55820e8b527b52940c61c05c732d8
We don't setup hosts entries for replication addresses on storage hosts,
by default. Rsync is performing a reverse lookup on each rsync, and
failing. We should turn this reverse lookup off by default, but allow
this to be set to True.
This PR adds the "swift_rsync_reverse_lookup" boolean which is defaulted
to false.
Change-Id: Ie98113fd71a70ce9c74c3812046be6959b9a353f
The current constraints generation for the
installation involves multiple tasks and multiple
variables.
Using multiple tasks extends the installation time
unnecessarily and the additional variables are
unnecessary.
This patch aims to simplify the mechanism and
hopes to speed it up a little.
Change-Id: Icdd5d053346e7d31dcfd4331c7b7b0f184bf376b
Releasenote translation publishing is being prepared. 'locale_dirs'
needs to be defined in conf.py to generate translated version of the
release notes.
Note that this repository might not get translated release notes - or
no translations at all - but we add the entry here nevertheless to
prepare for it.
Change-Id: I582151ca1049a6e6911814448467c9831178f125
We should never change the swift_hash_path_prefix/suffix variables on a
running cluster. This PR implements a check that will fail if the
variable is different to what is already on disk.
To ensure this is still possible this PR implements a
"swift_force_change_hashes" variable which can be set to "True" in order
to force change the swift_hash_path variables regardless of whether they
are different or not.
Change-Id: Idaedc125aede22c347668afd9e98ed1823eb142c
We can simplify and refactor swift_rings.py and swift_rings_check.py by
moving to a "FULL_HOST_KEY" model where we simply compare the full
string, rather than individual elements.
To do this we need to adjust the contents template to use the same field
values as used by swift:
* rename repl_ip to replication_ip
* rename repl_port to replication_port
Additionally, this allows us the ability to change port values on the
fly, by adjusting the "DEVICE_KEY" to only be the IP and device name the
port will now automatically get changed if the service port changes.
This is a precursor to adjusting the default swift storage service ports
to match upstream defaults, and will reduce the upgrade impact of that
task.
Change-Id: I704edcba4facb2170990ebec2a67d4179a023fc2
In order to make it easier to differentiate between the lists of
python packages, distribution packages, downloaded packages,
package pins and other similar variables the variable names are
being changed to ensure that they have a more explicit suffix
that defines the purpose and makes the naming more consistent.
This is to facilitate a lookup plugin which will be able to look
up all the package lists and present them as a consolidated piece
of data which may be used for artifact preparation.
Change-Id: I2c0d397df559df1725ec541afd7616e114d49d6f
This patch adds the ability to configure the log file permissions.
This is primarily to address a gating issue where the jenkins does
not have access to the account, container and object service logs.
That said, it's generaly useful to have this option available.
Change-Id: Ibc3d66273a4494414dde1a95ec132ff83c235f9c
The current method of installing the distribution packages required is
set in the tasks and cannot be changed by a deployer.
Currently the apt task always installs the latest package. This results
in unexpected binary changes when a deployer may simply be trying to
execute a configuration change.
This patch adds the ability for a deployer to change the desired state
so that the results are predictable.
Change-Id: Ib58ccc89b868f42295425527eb1a9dbcbe21839c
Expose the "pretend_min_part_hours_passed" option when accessing
swift-ring-builder.
This PR adds a variable "swift_pretend_min_part_hours_passed" which
defaults to False.
Change-Id: Ief94f011049b57f1ab0c2c48dfa81659d7ba13f7
This patch updates the sphinx configuration for docs and releasenotes
to make it easier to replicate across repositories and to comply with
pep8 testing without exceptions.
Change-Id: I67b952afefbf28a62f33ef784f498c950763de7d
Options have been added to allow a deployer to control the
hard, soft, and fs.file-max limits on swift specific nodes.
Change-Id: Ia5dd51fa360745a9776d58d0bf49d8e1201d3139
Closes-Bug: #1595290
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Remove all tasks and variables related to toggling between installation
of swift inside or outside of a Python virtual environment.
Installing within a venv is now the only supported deployment.
Additionally, a few changes have been made to make the creation of the
venv more resistant to interruptions during a run of the role.
* unarchiving a pre-built venv will now also occur when the venv
directory is created, not only after being downloaded
* virtualenv-tools is run against both pre-built and non pre-built venvs
to account for interruptions during or prior to unarchiving
Change-Id: Id2a27c082a5eb4eddc9eacde9de30f4e61f9ec36
Implements: blueprint only-install-venvs
This commit adds support for Ubuntu 16.04.
Primarily, this commit consists of changes
necessary to support systemd.
Change-Id: Ide0d570b508be8efaa885431598c726d7baad4f6
The ansible-managed line includes date/time information, meaning that
the conf file that gets laid down on different hosts does not have the
same MD5 sum across all hosts. This causes swift-recon --md5 to fail.
This commit fixes the issue by adding a static comment
to the config file instead
Change-Id: Id00cd79636f43776aaaa51e2ae6d7c5eacff8d34
Partial-Bug: #1589532
This change adds the "staticweb" middleware to the default list of
middleware deployed.
Per http://docs.openstack.org/developer/swift/middleware.html#staticweb,
staticweb should be added to the pipeline just after any auth
middleware.
Change-Id: Ic8614ac7295751df9ad1e5ff235b84c0a0a7d252
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The intention of this change is to enable an option to increase swift
performance without changing any of the swift internals. This change is
in-line with documentation published by the good folks at swiftstack and
intel. This change will only effect swift when its installed in a
virtual environment which is a self-imposed limitation such that we're
not messing with the system's default Python.
New Options:
- A new option `swift_pypy_enabled` has been added to enable or
disable the pypy interpreter for swift. The default is "false".
- A new option `swift_pypy_archive` has been added to allow a
pre-built pypy archive to be downloaded and moved into place. This
option is a dictionary and contains the URL and SHA256 as keys.
Reference: [ https://software.intel.com/en-us/blogs/2016/05/06/doubling-the-performance-of-openstack-swift-with-no-code-changes ].
Change-Id: I045e8b97d29b96e2c2450761efa1c2668de12a75
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This PR sets up rsync module per drive for swift object servers. This
improves resource distribution, and also allows us to utilise rsync
module changes to prevent disk writes to specific drives when they begin
filling up.
Enable this option by setting "swift_rsync_module_per_drive: True", this
will still default to False to match upstream defaults. Additionally the
rsync max connections has been increased to match upstream defaults.
Additionally we include rsync.d/*.conf files by default and set this
directory up, so that we can add individual configuration to disable
specific drives.
Change-Id: I2019cade5bf5f2878497d30ce738dff07786fa64
The upstream default for fallocate_reserve has just changed to 1% with
the inclusion of a "%" option for fallocate_reserve.
This PR sets the openstack-ansible default to match this value.
Change-Id: Iaafcb881dc067fa94d37e9eae93eed16d18c2f20
This fix removes the XFS filesystem from the daily cron job.
It will help to remove unnecessary disk IO due to updatedb/mlocate
swift object indexing inside the /srv/node folders.
Change-Id: I8bfa92003ce06ee4f065663e054cd2d04f458ec6
Closes-Bug: #1572307