Though a `swift_operator_role` variable exists to allow specifying an
operator role other than `swiftoperator`, it is not applied to all
uses of the role, eg the proxy-server.conf template.
Replace all remaining hard-coded references to the `swiftoperator`
role with the `swift_operator_role` variable.
Change-Id: Ie6db872cc2b7a1b1a90d9a690ee08937a9cab785
Signed-off-by: Corey Wright <corey.wright@rackspace.com>
This introduces oslo.messaging variables that define the Notify transport
in place of the rabbitmq values.
This patch:
* Add oslo.messaging variable for Notify to defaults
* Update url generation
* Add oslo.messaging to inventory
* Add release note
Change-Id: I1c2e844c4c7a2256087bcc4521f970ca8e8c6b16
This removes the systemd service templates and tasks from this role and
leverages a common systemd service role instead. This change removes a
lot of code duplication across all roles all without sacrificing features
or functionality. The intention of this change is to ensure uniformity and
reduce the maintenance burden on the community when sweeping changes are
needed. The exterior role is built to be OSA compatible and may be pulled
into tree should we deem it necessary.
Change-Id: Icb7ca523cb19c560de5c84b0d60a06305029192c
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Option auth_uri from group keystone_authtoken is deprecated[1].
Use option www_authenticate_uri from group keystone_authtoken.
[1]https://review.openstack.org/#/c/508522/
Change-Id: I7897412e22d91c6ab786652ff95ca44f7d10dedc
Swift should mirror other roles and use a list instead of a dictionary
for it's filtered_services.
This patch makes that change.
Change-Id: Ie6bf282a36ed63d73996447a88c3c5f6056465a7
This commit adds support for the swift3 middware, which allows S3-compatible
clients to use swift for object storage.
Change-Id: I56cd63057cc771310b69c311d975e06f73c773f7
Related-Bug: 1625053
The systemd unit 'TimeoutSec' value which controls the time
between sending a SIGTERM signal and a SIGKILL signal when
stopping or restarting the service has been reduced from 300
seconds to 120 seconds. This provides 2 minutes for long-lived
sessions to drain while preventing new ones from starting
before a restart or a stop.
The 'RestartSec' value which controls the time between the
service stop and start when restarting has been reduced from
150 seconds to 2 seconds to make the restart happen faster.
These values can be adjusted by using the *_init_config_overrides
variables which use the config_template task to change template
defaults.
Change-Id: I048b877e859ad744dc54f19a93afdd89f8ef1661
This creates a specific slice which all OpenStack services will operate
from. By creating an independent slice these components will be governed
away from the system slice allowing us to better optimise resource
consumption.
See the following for more information on slices:
* https://www.freedesktop.org/software/systemd/man/systemd.slice.html
See for following for more information on resource controls:
* https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html
Tools like ``systemd-cgtop`` and ``systemd-cgls`` will now give us
insight into specific processes, process groups, and resouce consumption
in ways that we've not had access to before. To enable some of this reporting
the accounting options have been added to the [Service] section of the unit
file.
Change-Id: Ibcb7d2a69ed67a99c88dc143c76aa8448d31cc9e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Utilize the "ringbuilder.devs_changed" option to ensure that the ring
needs a rebalance. This will prevent unnecessary rebalances that cause
failures due to "min_part_hours" not being passed even though no changes
were required.
Additionally, we can now return a correct Ansible repsonse when the ring
has changed/rebalanced - and return "OK" when it hasn't returned at all.
Change-Id: I1fb4b3544a50ab5f566b3846d616107a84ff29c9
Users can configure the number of worker threads. However when it's
not specified the calculated number of workers can get too large on
hosts with a large number of CPUs. Capping only swift proxy server
worker threads when the proxy is in a container. Not capping the
remaining swift services' workers because of the performance impact
it may cause because of the capping.
Change-Id: I12d930552558144ab49fecc0b3776747c1f02166
The old version of this scipt used to interface to the
ringbuilder cli interface. This meant we did some crazy
threading. That was complicated.
This patch changes that to use the RingBuilder and RingData
classes, which makes things much simpler, and we can remove
all the threading stuff.
Change-Id: I94004db3b2b772644d89e20c1201d7f403f3eb86
The statsd.j2 include approach is great, but it is hitting an ansible
bug with Jinja2==2.9.5 which hasn't been fixed with Ansible and doens't
seem to be fixed anytime soon.
Here is an example bug:
https://github.com/ansible/ansible/issues/20494
This patch also refactors the statsd.j2 import parts, a lot of
if/else statements were not required.
Change-Id: Ib78ac0a8891874b1c2e777fac8f3fb89304e6872
The swift_rings.py script creates a thread and calls out to
swift's ringbuilder cli interface. It wasn't failing if
ringbuilder failed.
This change changes the threading to capture the threads exit
code and sys.exit on a bad one.
Change-Id: Ic2199ccc393b25a60af82af3aa638f21f19a6418
The old limits config was still running on the upstart setup. While the
directories within the ubuntu exist they are ignored in Ubuntu 16.04 and
CentOS 7. This change removes the old upstart config and adds the
required systemd config.
Change-Id: Ic75d6cfe32678f4205d6f8ea991f393526d0a082
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
As a part of removing Trusty from OpenStack-Ansible we shall aim to
remove this from all the roles.
Testing has already been removed for Trusty in the integrated build and
all individual repositories on master (Ocata), as such we can now go
ahead and remove the support within the roles.
Change-Id: I89ba35fd15703aba2a05d11d4550690704bdf272
Implements: blueprint trusty-removal
This patch adds copy as a middleware for swift, this follows upstream's
approach and reduces errors that indicate it is being automatically
included in the pipeline.
Change-Id: I4591ff3f3464d8bfa4ffd012f117aba881b02b65
Move to use tempauth to resolve memory issues resulting from an AIO
swift install running in pypy.
This PR adds some options for using pypy:
* Set a pypy Garbage collection value
This PR includes some tempauth fixes to make it useable:
* Set the tempauth users based on a variable
Testing is changed as follows for pypy:
* Use only memcache within swift-proxy
* Remove galera/keystone
* Add swap for swift-storage hosts
* Use tempauth for pypy
* Reduce to 2 swift hosts
Change-Id: Ic1ed5acc9b20853d9a159035226f97fda088f035
Versioned Objects in Swift now use a middleware that is added to the
pipeline instead of the "allow_versions" in the container config. These
have 2 different Headers, so to prevent "X-Versions-Location" from not
working the "allow_versions" in the container config is left in place.
This patch adds the middleware by default and sets the Versioned Objects
support to be on. This is the upstream default and the deployer would
still need to set the "X-History-Location" header on a container to
utilise the feature.
Change-Id: I88811fd77fad8d2241448ca5ffb565fa7d704a00
We don't setup hosts entries for replication addresses on storage hosts,
by default. Rsync is performing a reverse lookup on each rsync, and
failing. We should turn this reverse lookup off by default, but allow
this to be set to True.
This PR adds the "swift_rsync_reverse_lookup" boolean which is defaulted
to false.
Change-Id: Ie98113fd71a70ce9c74c3812046be6959b9a353f
Performance testing of an OSA deployed Swift cluster hit several issues.
Comparison of the proxy conf with the Swift Auth docu [1] found
two settings that affect performance that that were not set in the template.
include_service_catalog - This defaults to True which makes the proxy server
fetch the service catalog on every request in the authtoken middleware. This
hammers Keystone.
cache - When using memcache this should be set so the authtoken middleware
uses the correct cache and not go back to Keystone on every request.
[1] http://docs.openstack.org/developer/swift/overview_auth.html
Change-Id: Iddaf097e3ff5c1b7118b11cfc598a40a91d6f96f
Swift Erasure Coding requires the object-reconstructor to be running,
this PR adds the object-reconstructor configuration and systemd/upstart
file.
Additionally, this PR adds testing for an erasure-coding policy as well
as for multiple policies in 1 deployment.
Change-Id: I71fe3d77e93112b9d27c93b37b59e6242cb3e00e
Closes-Bug: #1634859
When not using dedicated replication systemd still puts init scripts
down, which take a long time to restart/start. upstart scripts get
around this by setting blank scripts.
Now that we are using a service dict we can do better by defining an
"service_en" flag and not setting up scripts when the service isn't
enabled.
Additionally, the systemd tempfiles and init files were not using the
appropriate "program_binary" variable at all, this has been fixed.
Change-Id: Iae569bfe38a440fb09e56658b3a934799a8821e8
Change the 'swift_x_program_names' from a list to a dictionary
mapping of services, groups that install those services. This
brings the method into line with that used in the os_neutron role
in order to implement a more standardised method.
The init tasks have been updated to run once and loop through this
mapping rather than being included multiple times and re-run against
each host. This may potentially reduce role run times.
Currently the reload of upstart/systemd scripts may not happen if
only one script changes as the task uses a loop with only one result
register. This patch implements handlers to reload upstart/systemd
scripts to ensure that this happens when any one of the scripts
change.
The handler to reload the services now only tries to restart the
service if the host is in the group for the service according to the
service group mapping. This allows us to ensure that handler
failures are no longer ignored and that no execution time is wasted
trying to restart services which do not exist on the host.
Finally:
- Common variables shared by each service's template files have
been updated to use the service namespaced variables.
- Unused handlers have been removed.
- Unused variables have been removed.
Change-Id: Id35de501acf6b3164221085f8f9e142234ea0d73
This PR ensures that if a string is entered as the var it will be forced
to an "int" type, which is expected by the script.
Change-Id: I8d7f3c4894c2b0975893ebb30b734ae7fccfbc22
We need to ensure the swift services are listening on the same IP and
port as the ring is setup for. Currently we calculate these values in
different ways, which leads to the possibility that the ring is setup
for ports and IPs that the services are not listening on.
This change ensures this by calculating the storage and replication
addresses within the role - whilst still accepting overrides from
outside the role. This ensures functionality does not change, but that
it is now not possible for the swift services to not listen on the
ports/IPs defined in the ring.
As part of this change we move to set "swift_dedicated_replication"
based on whether the storage and replication addresses are the same,
rather than the specified network. This means you can run a dedicated
replication config by configuring the services, which was possible to
configure, but again would have meant the services and ring were not
configured in the same way. This is required to ensure uniformity.
Finally - we remove the ability to set a storage_port or repl_port
within the swift_vars, this was a setting that would never work in it's
current implementation. The storage/repl_port would depend entirely on
which swift service was in use, and can not be set in such a blanket
fashion. Since this was a completely not working funcitonality it has
now been removed.
Change-Id: Ibe5330cdca7fd81f379b80dc55ad06529ce4d580
Move to use the openstack-ansible-tests repository for Swift.
Make changes use Ansible 2.1.1 (These are required for gating to work
properly, since the majority of repositories have already changed).
Additionally, lets set this up to work with 1 infra host in the MR swift
gate to improve efficiency.
Change-Id: Ic0d35bcf8bafb90a986c7cdda3987f70580af165
Create a new default var, swift_dedicated_replication, for storing
conditions used to determine if a replication network is being used for
swift. This allows for removing redundant tasks, reducing role run
times.
Change-Id: Ieb4263035527ff069017d40b95787f934fbd308c
Regression introduced by Ic2129bd6be85f6a51feea95d47cfbb349c44f363.
Ansible setups Jinja2 in such a way that a newline after an ``{% endfor %}``
is stripped. Therefore, we insert an additional newline.
Change-Id: I6c72c82bea0c9b93b7ef18ce89ccd7c18066b802
When using Swift storage for Gnocchi we want to add a filter to the
Ceilometer middleware for Swift to exclude reporting traffic caused
by Gnocchi storing data into Swift to avoid feedback-loops. This
is typically done by isolating the Gnocchi in a dedicated Project in
Keystone.
The best place to filter this traffic is in the middleware
as opposed to filtering in the Ceilometer Collector.
Related-To: I67b5bff394ad35cf95d10ba32d602954799d7348
Change-Id: If20248494d4e6c127db4ffdf77ae43482f64fe58
We can simplify and refactor swift_rings.py and swift_rings_check.py by
moving to a "FULL_HOST_KEY" model where we simply compare the full
string, rather than individual elements.
To do this we need to adjust the contents template to use the same field
values as used by swift:
* rename repl_ip to replication_ip
* rename repl_port to replication_port
Additionally, this allows us the ability to change port values on the
fly, by adjusting the "DEVICE_KEY" to only be the IP and device name the
port will now automatically get changed if the service port changes.
This is a precursor to adjusting the default swift storage service ports
to match upstream defaults, and will reduce the upgrade impact of that
task.
Change-Id: I704edcba4facb2170990ebec2a67d4179a023fc2
When running dedicated replication network we need the auditor service
to point at the configuration file with the replicator.
Without this we will get errors in the various servers indicating the
replicator service can not be found by the auditor service:
object-auditor: STDOUT: Unable to find object-replicator config section
in /etc/swift/object-server/object-server.conf
Change-Id: Idd5e6ddab66a7855959e93139d2c7934762c6ba9