This change adds the "staticweb" middleware to the default list of
middleware deployed.
Per http://docs.openstack.org/developer/swift/middleware.html#staticweb,
staticweb should be added to the pipeline just after any auth
middleware.
Change-Id: Ic8614ac7295751df9ad1e5ff235b84c0a0a7d252
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Debian-specific vars and logic have been moved to tasks
that will execute only on those distributions.
Change-Id: I370621eda9e87bd19532e4e37e46097607bf4166
Implements: blueprint multi-platform-host
The swift clone was hardcoded to master and should be based on the
variable we use to specify the version of swift downloaded from git.
Change-Id: I4083482a8badcf8a83c8483c5f2937d2bcda90fe
The intention of this change is to enable an option to increase swift
performance without changing any of the swift internals. This change is
in-line with documentation published by the good folks at swiftstack and
intel. This change will only effect swift when its installed in a
virtual environment which is a self-imposed limitation such that we're
not messing with the system's default Python.
New Options:
- A new option `swift_pypy_enabled` has been added to enable or
disable the pypy interpreter for swift. The default is "false".
- A new option `swift_pypy_archive` has been added to allow a
pre-built pypy archive to be downloaded and moved into place. This
option is a dictionary and contains the URL and SHA256 as keys.
Reference: [ https://software.intel.com/en-us/blogs/2016/05/06/doubling-the-performance-of-openstack-swift-with-no-code-changes ].
Change-Id: I045e8b97d29b96e2c2450761efa1c2668de12a75
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This PR sets up rsync module per drive for swift object servers. This
improves resource distribution, and also allows us to utilise rsync
module changes to prevent disk writes to specific drives when they begin
filling up.
Enable this option by setting "swift_rsync_module_per_drive: True", this
will still default to False to match upstream defaults. Additionally the
rsync max connections has been increased to match upstream defaults.
Additionally we include rsync.d/*.conf files by default and set this
directory up, so that we can add individual configuration to disable
specific drives.
Change-Id: I2019cade5bf5f2878497d30ce738dff07786fa64
The upstream default for fallocate_reserve has just changed to 1% with
the inclusion of a "%" option for fallocate_reserve.
This PR sets the openstack-ansible default to match this value.
Change-Id: Iaafcb881dc067fa94d37e9eae93eed16d18c2f20
This PR refactors the swift testing:
Move vars into test-vars
Split out tasks
Adds replication address
Make more settings variables
This closely mirrors the changes made to testing in the os-nova
repository, and should allow us to split out the initial setup tasks far
easier.
Change-Id: I81150d88027616f494fef0eed33c5f2838634593
Currently all pip install tasks only require the package to be
present. This means that when an environment undergoes a minor
upgrade the package is not upgraded to the same version that
was tested with. This ultimately results in a deployed
environment that does not match the tested environment.
While for the services installed into venvs this is not an
issue, it does affect those which do not use venvs and any
packages which are installed outside of a venv or on top
of a venv.
This patch changes the behaviour to ensure that the install
task will always use the latest available package. In
developer_mode this will mean using the version specified
in upper-constraints, and in an integrated build this will
mean the version which is available in the wheel repo's
folder for the tag.
Change-Id: Id04b2f74831e3422b036308c638be5428509e57a
Paramiko version 2.0 has been released. It now uses the Python library
cryptography. Installing this requires additional system packages. This
commit adds in the appropriate packages required by cryptography based
on its documentation [1].
An alternative approach would have been to constrain the version of
Paramiko however the project describes the 1.x versions as relying on
insecure dependencies [2].
[1] https://cryptography.io/en/latest/installation/
[2] http://www.paramiko.org/installing.html
Change-Id: Ieecf0e0ec7cc1cd534a1a97adc14c1bbc5f2f8f2
This fix removes the XFS filesystem from the daily cron job.
It will help to remove unnecessary disk IO due to updatedb/mlocate
swift object indexing inside the /srv/node folders.
Change-Id: I8bfa92003ce06ee4f065663e054cd2d04f458ec6
Closes-Bug: #1572307
This is a first pass at adding functional tests for swift using the
.functests within the upstream swift repository.
User specific tests, as defined in /etc/swift/test.conf, are enabled
with the exception of keystone v3 domain tests for which further work
will be required.
Change-Id: If0bcc96ce3b4bb2f80f08f4523dad3fd89023e77
This version of ansible is broken due to the following upstream
Ansible issue:
* https://github.com/ansible/ansible-modules-extras/issues/2042
Change-Id: Ifdaa67f1e1a96c56c96732d236b5c6af267219e5
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
When swift disks fill up completely it becomes difficult to perform
operations since delete operations require a write.
By setting this value we can ensure that PUT requests won't happen when
the disk space is below the threshold set (in bytes). This will not
govern rsyncs, which can still happen to fill up the disks, but assuming
a reasonably uniform distribution of data this should help prevent disks
from filling up.
Change-Id: I753de5e5ed09a21571575ef51c38752edc0730cb
Closes-Bug: #1545018
This PR fixes the following minor test issues:
* The swift ring and swift-service addresses are now consistent.
* Memcache now listens on the appropriate IP.
* The openrc file is now populated correctly.
* Rsyslog restarts appropriately, meaning logs are generated.
Change-Id: Ie4f9672eba5936232ccfc613bdcc03a7fdeb090b
When ceilometer is enabled the middleware for swift proxy can be
valuable in allowing additional metrics to be collected, beyond
those available through the swift poller.
Change-Id: I78b45acdbe98f225a5090832660d62792bf41027
The following warning appears in the swift proxy logs:
"Using the in-process token cache is deprecated as of the 4.2.0
release and may be removed in the 5.0.0 release or the 'O'
development cycle. The in-process cache causes inconsistent results
and high memory usage."
Specifying the location of a memcached server will not only clean up
a deprecation warning in the logs but should provide a performance
improvement.
This is caused by the absence of a memcached_servers key in the
proxy configuration. While this variable is defined as a group var
in the integrated release we insulate against the absense of this
value in stand-alone role usage or Swift Stand-Alone deploys which
may not include memcached yet to prevent upgrade impacts.
Change-Id: I0a16ef0030d1425f52b6c07214d7c8effef51c13
The name of the reseller_admin_role in the proxy config needs
to map to the correct role within the deploy. It was specified
manually and inconsistently though a variable exists in the role
defaults.
Change-Id: Ib93fc57f1e26e8a0943a7bd7c2212ce94be54f7c
Closes-Bug: #1542391
The pip_install role is depended on by a lot of other roles, and
therefore sometimes gets processed prior to the pip_lock_down
role resulting in the pip, setuptools and wheels packages being
installed from a source other than the repo server once the repo
server is available. This is not the intended behaviour - the
repo server should always be a the primary source once it's
available.
This patch ensures that the pip_lock_down role is applied before
all the other dependent roles to ensure that the expected
behaviour is followed.
Change-Id: I4cecb48a9ff9015905f3d65b312ebad4efcd9085
The git source for python2_lxc was necessary before the package was
available on pypi, which it now is:
- https://pypi.python.org/pypi/lxc-python2
This patch removes the py_from_git role from the functional testing
playbook.
Change-Id: I69e105becbcb4483391ed44292ae861500b8baef
This will allow the specification of container-sync realms via the
swift_container_sync_realms variable (documented in the
defaults/main.yml within the os_swift role).
Creating a conf file that is then used to enable and utilise
container-sync Realms within Swift.
Change-Id: Icf71d008765ff83743f6ab28ef0cea29943e362e
This updates the repository SHA's to use stable/mitaka where
available and updated SHA's where not.
Change-Id: I542d0b91edc10edd1946251c0045555534779017
We want to improve the gate speed by running the swift-role once on AIO,
since we don't need to split the sync tasks out when not using MR swift.
This means for MR swift we need to differentiate between the tasks that
should be run on remote hosts and those that should be run on local
hosts.
Change-Id: I0a8436ebbab0226f6c4fb499c938f760429aad39
This pacakge is causing system breakage due to a new version being
released that is otherwise incompatible with requirements as found
within OpenStack. After reviewing the package usage and the reason
it's been included but the only thing I could find that actually
uses this resource was cinder and only when using the EMC VMAX
driver: http://paste.openstack.org/show/491185/ -- being that this
package is not even listed in global requirements and is not in
support of anything we could say is a common backend within Cinder
that we know we're currently supporting I am of the opinion that
this package should be removed.
Change-Id: I152fa0130e64ad5756b413ed8f9714d28e75d0bb
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This commit changes the pip_install_options fact name in
swift_install.yml to pip_install_options_fact. This allows us to
maintain the existing pip_install_options variable without overwriting
it with options when in developer mode, which ultimately means we can
have multiple services running in a container use a combination of
*_developer_mode: true and *_developer_mode: false. At the moment,
if a service writes pip_install_options fact with the constraints
options, those options will persist to other services running in the
container even if *_developer_mode: false.
Change-Id: I248aeecafe9ac4e06e3e24796dd279f67c6589dd
Now that auth token usage is deprecated, prefer the admin
user and password for all swift setup tasks run against
keystone.
Change-Id: I1c87dc6976c26e0c00e740ce1e4105bcb30cc32f
This patch sets the test to check if the test is running in
OpenStack-CI, and if it is - then make the containers use
the host's pip.conf and some of the apt configuration.
All containers should make use of the hosts's apt sources
configuration. The tasks detect and implement this.
The intent is to make use of OpenStack-CI resources better
and consequently reduce the test run time.
Change-Id: Iab63f245c9bb892170cc623a7002398f31ae314b
The repo split out has meant that having multiple roles for swift has
become inconvenient. We can merge the os-swift-sync role back into
os-swift and utilise variables to ensure that the swift-install,
swift-setup and swift-sync playbooks work in the same way.
This will avoid code duplication and the requirement for duplicated
changes to happen to both repositories.
Change-Id: If01d8a3aa29fc7637aecb92e61929523af08103d
Setup variables for the account, object & container rsync max
connections.
Defaults to the current default of 2, and allows it to be specified on a
per service (account, object, container) basis.
Change-Id: I837f7ddabbc012d1607c897bb45ae58ad9f8540a
Gate was failing due to a race condition whereby the partition is not ready
and so the first mounted partitions fails to mount - causing hosts to fail.
Adding a retry/delay on failures in the mount task should resolve this issue.
ansible-lint failures were occurring due to a recent release of ansible-lint
Since the gate failure cannot be merged without the linting fix, this change
also contains I0cc973867d9c279eb6ea05decc5ebccfaa19c7b0 to break the co-
dependency between this and the other patch.
Change-Id: I3f4daaed764c4cad8a07408fedf8d0cba570a344
Pre-empting a situation where we run into issues installing pip packages
due to unspecified upper constraints files.
We ran into this issue with the os_keystone repo which was patched in
this way: https://review.openstack.org/#/c/290446
Change-Id: I0ee1471a65ad92323e4a274f5ec2968969e291fe
Adds a functional/convergence tests for os-swift including keystone,
galera, memcache and rabbit.
Swift testing infrastructure consists of:
1 service container /w keystone, galera, rabbit, memcached.
4 storage hosts (each with 2 disks)
1 proxy container.
This fixes a few issues to ensure the functional tests will run, such as
delegating sysctl net.ipv4.tcp_tw_reuse task to physical host, since we
are running swift storage inside containers for the purposes of this
test, and the containers won't have access to make this change.
Fixes the developer_mode settings which were incorrect.
Change-Id: Ie1dfaf666cbbac9fe78edd0a9218cec790c8c4d2
Closes-Bug: #1553967
This patch adds configuration for swift's drive-audit script with
a cron job to run it hourly at 15 minutes past the hour.
Closes-Bug: 1433619
Change-Id: I1a5cfbfffef007a0d8e9b56dd5936aad7c38916d
