107 Commits

Author SHA1 Message Date
Zuul
b216283ed0 Merge "Update auth_uri option to www_authenticate_uri" 2018-09-08 21:39:31 +00:00
Zuul
6061ad9d9b Merge "Replace swiftoperator role with swift_operator_role variable" 2018-07-02 15:22:11 +00:00
Corey Wright
6654e4835d Replace swiftoperator role with swift_operator_role variable
Though a `swift_operator_role` variable exists to allow specifying an
operator role other than `swiftoperator`, it is not applied to all
uses of the role, eg the proxy-server.conf template.

Replace all remaining hard-coded references to the `swiftoperator`
role with the `swift_operator_role` variable.

Change-Id: Ie6db872cc2b7a1b1a90d9a690ee08937a9cab785
Signed-off-by: Corey Wright <corey.wright@rackspace.com>
2018-06-21 19:44:45 -05:00
Andrew Smith
91af246bdf Update to use oslo.messaging service for Notify
This introduces oslo.messaging variables that define the Notify transport
in place of the rabbitmq values.

This patch:
* Add oslo.messaging variable for Notify to defaults
* Update url generation
* Add oslo.messaging to inventory
* Add release note

Change-Id: I1c2e844c4c7a2256087bcc4521f970ca8e8c6b16
2018-06-11 16:45:16 -04:00
Kevin Carter
8f1cb4dde5 Convert role to use a common systemd service role
This removes the systemd service templates and tasks from this role and
leverages a common systemd service role instead. This change removes a
lot of code duplication across all roles all without sacrificing features
or functionality. The intention of this change is to ensure uniformity and
reduce the maintenance burden on the community when sweeping changes are
needed. The exterior role is built to be OSA compatible and may be pulled
into tree should we deem it necessary.

Change-Id: Icb7ca523cb19c560de5c84b0d60a06305029192c
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-04-22 14:00:27 +00:00
melissaml
407e2cc5ed Update auth_uri option to www_authenticate_uri
Option auth_uri from group keystone_authtoken is deprecated[1].
Use option www_authenticate_uri from group keystone_authtoken.

[1]https://review.openstack.org/#/c/508522/

Change-Id: I7897412e22d91c6ab786652ff95ca44f7d10dedc
2018-04-17 04:58:02 +08:00
Andy McCrae
d860153400 Use a list instead of a dict for filtered_services
Swift should mirror other roles and use a list instead of a dictionary
for it's filtered_services.

This patch makes that change.

Change-Id: Ie6bf282a36ed63d73996447a88c3c5f6056465a7
2017-07-31 17:24:34 +01:00
Charles Farquhar
14c57dd419 Add support for swift3 middleware
This commit adds support for the swift3 middware, which allows S3-compatible
clients to use swift for object storage.

Change-Id: I56cd63057cc771310b69c311d975e06f73c773f7
Related-Bug: 1625053
2017-06-23 18:26:46 +00:00
Jesse Pretorius
22a4a72afb Reduce init restart/kill times
The systemd unit 'TimeoutSec' value which controls the time
between sending a SIGTERM signal and a SIGKILL signal when
stopping or restarting the service has been reduced from 300
seconds to 120 seconds. This provides 2 minutes for long-lived
sessions to drain while preventing new ones from starting
before a restart or a stop.

The 'RestartSec' value which controls the time between the
service stop and start when restarting has been reduced from
150 seconds to 2 seconds to make the restart happen faster.

These values can be adjusted by using the *_init_config_overrides
variables which use the config_template task to change template
defaults.

Change-Id: I048b877e859ad744dc54f19a93afdd89f8ef1661
2017-04-26 17:04:04 +00:00
Jenkins
6a1669fb3d Merge "Deprecate auth_plugin option" 2017-04-06 15:58:13 +00:00
Andy McCrae
78b4f6aac6 Update container-sync to use internal client
This is to match the way this is configured in upstream Swift.

Change-Id: Ifbfd4158e04f38bf3383a7c2461374a21088fdd3
2017-04-04 15:04:19 +00:00
Kevin Carter
867f3157e9 Ensure the components are isolated from the system
This creates a specific slice which all OpenStack services will operate
from. By creating an independent slice these components will be governed
away from the system slice allowing us to better optimise resource
consumption.

See the following for more information on slices:

* https://www.freedesktop.org/software/systemd/man/systemd.slice.html

See for following for more information on resource controls:

* https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html

Tools like ``systemd-cgtop`` and ``systemd-cgls`` will now give us
insight into specific processes, process groups, and resouce consumption
in ways that we've not had access to before. To enable some of this reporting
the accounting options have been added to the [Service] section of the unit
file.

Change-Id: Ibcb7d2a69ed67a99c88dc143c76aa8448d31cc9e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2017-03-30 15:16:03 +00:00
ZhongShengping
7ca4b62eb5 Deprecate auth_plugin option
Option "auth_plugin" is deprecated, use option "auth_type" instead.

Change-Id: I6ac382f3856b754291378cde335bc71cd2161023
Implements: blueprint deprecate-auth-plugin
2017-03-25 12:26:23 +08:00
Andy McCrae
57de3c8817 Check if changes have made before attempting a rebalance
Utilize the "ringbuilder.devs_changed" option to ensure that the ring
needs a rebalance. This will prevent unnecessary rebalances that cause
failures due to "min_part_hours" not being passed even though no changes
were required.

Additionally, we can now return a correct Ansible repsonse when the ring
has changed/rebalanced - and return "OK" when it hasn't returned at all.

Change-Id: I1fb4b3544a50ab5f566b3846d616107a84ff29c9
2017-03-20 13:03:32 +00:00
Jenkins
0dad1e4fbd Merge "Cap the number of worker threads" 2017-03-10 14:16:03 +00:00
Ravi Gummadi
bb9e3f5828 Cap the number of worker threads
Users can configure the number of worker threads. However when it's
not specified the calculated number of workers can get too large on
hosts with a large number of CPUs. Capping only swift proxy server
worker threads when the proxy is in a container. Not capping the
remaining swift services' workers because of the performance impact
it may cause because of the capping.

Change-Id: I12d930552558144ab49fecc0b3776747c1f02166
2017-03-06 06:39:01 -05:00
Matthew Oliver
3c09d9bfd5 Rework swift_rings.py to use the RingBuilder class
The old version of this scipt used to interface to the
ringbuilder cli interface. This meant we did some crazy
threading. That was complicated.

This patch changes that to use the RingBuilder and RingData
classes, which makes things much simpler, and we can remove
all the threading stuff.

Change-Id: I94004db3b2b772644d89e20c1201d7f403f3eb86
2017-02-28 14:36:46 +00:00
Jenkins
56190b63f5 Merge "Make swift_rings threading react to bad return codes" 2017-02-17 14:43:45 +00:00
Andy McCrae
352969e2d1 Move away from include statsd.j2
The statsd.j2 include approach is great, but it is hitting an ansible
bug with Jinja2==2.9.5 which hasn't been fixed with Ansible and doens't
seem to be fixed anytime soon.

Here is an example bug:
https://github.com/ansible/ansible/issues/20494

This patch also refactors the statsd.j2 import parts, a lot of
if/else statements were not required.

Change-Id: Ib78ac0a8891874b1c2e777fac8f3fb89304e6872
2017-02-13 15:39:17 +00:00
Matthew Oliver
0dd92296cd Make swift_rings threading react to bad return codes
The swift_rings.py script creates a thread and calls out to
swift's ringbuilder cli interface. It wasn't failing if
ringbuilder failed.

This change changes the threading to capture the threads exit
code and sys.exit on a bad one.

Change-Id: Ic2199ccc393b25a60af82af3aa638f21f19a6418
2017-02-10 15:00:34 +11:00
Kevin Carter
195621208a Converted the limits config to use systemd
The old limits config was still running on the upstart setup. While the
directories within the ubuntu exist they are ignored in Ubuntu 16.04 and
CentOS 7. This change removes the old upstart config and adds the
required systemd config.

Change-Id: Ic75d6cfe32678f4205d6f8ea991f393526d0a082
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2017-01-30 18:56:52 +00:00
Jenkins
a0d6cdea8a Merge "Remove Trusty support from os_swift role" 2016-12-09 11:25:21 +00:00
Andy McCrae
26bd8127b6 Remove Trusty support from os_swift role
As a part of removing Trusty from OpenStack-Ansible we shall aim to
remove this from all the roles.

Testing has already been removed for Trusty in the integrated build and
all individual repositories on master (Ocata), as such we can now go
ahead and remove the support within the roles.

Change-Id: I89ba35fd15703aba2a05d11d4550690704bdf272
Implements: blueprint trusty-removal
2016-12-08 11:50:58 +00:00
Andy McCrae
db66b43dd2 Set copy as a middleware for Swift
This patch adds copy as a middleware for swift, this follows upstream's
approach and reduces errors that indicate it is being automatically
included in the pipeline.

Change-Id: I4591ff3f3464d8bfa4ffd012f117aba881b02b65
2016-12-07 14:29:52 +00:00
Jenkins
8238849e41 Merge "Allow X-History-Location versioned_writes" 2016-12-03 03:14:48 +00:00
Andy McCrae
3fb6784459 Attempt to fix swift_pypy memory issues
Move to use tempauth to resolve memory issues resulting from an AIO
swift install running in pypy.

This PR adds some options for using pypy:
* Set a pypy Garbage collection value

This PR includes some tempauth fixes to make it useable:
* Set the tempauth users based on a variable

Testing is changed as follows for pypy:
* Use only memcache within swift-proxy
* Remove galera/keystone
* Add swap for swift-storage hosts
* Use tempauth for pypy
* Reduce to 2 swift hosts

Change-Id: Ic1ed5acc9b20853d9a159035226f97fda088f035
2016-12-01 11:07:10 +00:00
Andy McCrae
de7ab0a800 Allow X-History-Location versioned_writes
Versioned Objects in Swift now use a middleware that is added to the
pipeline instead of the "allow_versions" in the container config. These
have 2 different Headers, so to prevent "X-Versions-Location" from not
working the "allow_versions" in the container config is left in place.

This patch adds the middleware by default and sets the Versioned Objects
support to be on. This is the upstream default and the deployer would
still need to set the "X-History-Location" header on a container to
utilise the feature.

Change-Id: I88811fd77fad8d2241448ca5ffb565fa7d704a00
2016-11-30 11:18:45 +00:00
Pedro Perez
3e90d83252 Fix hardcorded drive-audit path
Change-Id: I447e48ec704c4a3a0bfe1dc1cee46f0b0daf539e
2016-11-15 19:23:21 +01:00
Jenkins
bb9c63bdc2 Merge "Set rsync to not perform reverse lookups" 2016-11-09 14:03:40 +00:00
Andy McCrae
4aa96a832c Set rsync to not perform reverse lookups
We don't setup hosts entries for replication addresses on storage hosts,
by default. Rsync is performing a reverse lookup on each rsync, and
failing. We should turn this reverse lookup off by default, but allow
this to be set to True.

This PR adds the "swift_rsync_reverse_lookup" boolean which is defaulted
to false.

Change-Id: Ie98113fd71a70ce9c74c3812046be6959b9a353f
2016-11-07 16:24:52 +00:00
Marc Gariepy
d277ccc437 Fix tmpfiles.d when multiple service are running
This fix tmpfile when multiple services runs in the same host with systemd.

Change-Id: Ibbac2bfd94affe27375cce13f5f91adcf37cb194
2016-11-04 14:53:36 -04:00
Jenkins
6fc3caee2b Merge "Swift proxy performance fixes" 2016-10-28 15:39:06 +00:00
Samuel Matzek
b0f330e7b2 Swift proxy performance fixes
Performance testing of an OSA deployed Swift cluster hit several issues.
Comparison of the proxy conf with the Swift Auth docu [1] found
two settings that affect performance that that were not set in the template.

include_service_catalog - This defaults to True which makes the proxy server
fetch the service catalog on every request in the authtoken middleware. This
hammers Keystone.

cache - When using memcache this should be set so the authtoken middleware
uses the correct cache and not go back to Keystone on every request.

[1] http://docs.openstack.org/developer/swift/overview_auth.html

Change-Id: Iddaf097e3ff5c1b7118b11cfc598a40a91d6f96f
2016-10-28 15:58:40 +02:00
Andy McCrae
cd222e8cff Add object-reconstructor for swift EC
Swift Erasure Coding requires the object-reconstructor to be running,
this PR adds the object-reconstructor configuration and systemd/upstart
file.

Additionally, this PR adds testing for an erasure-coding policy as well
as for multiple policies in 1 deployment.

Change-Id: I71fe3d77e93112b9d27c93b37b59e6242cb3e00e
Closes-Bug: #1634859
2016-10-19 18:36:14 +01:00
Andy McCrae
033aa502e5 Fix swift init scripts w/o dedicated replication
When not using dedicated replication systemd still puts init scripts
down, which take a long time to restart/start. upstart scripts get
around this by setting blank scripts.

Now that we are using a service dict we can do better by defining an
"service_en" flag and not setting up scripts when the service isn't
enabled.

Additionally, the systemd tempfiles and init files were not using the
appropriate "program_binary" variable at all, this has been fixed.

Change-Id: Iae569bfe38a440fb09e56658b3a934799a8821e8
2016-10-14 18:19:51 +01:00
Andy McCrae
bf1ab1750a Use dictionary for service mappings
Change the 'swift_x_program_names' from a list to a dictionary
mapping of services, groups that install those services. This
brings the method into line with that used in the os_neutron role
in order to implement a more standardised method.

The init tasks have been updated to run once and loop through this
mapping rather than being included multiple times and re-run against
each host. This may potentially reduce role run times.

Currently the reload of upstart/systemd scripts may not happen if
only one script changes as the task uses a loop with only one result
register. This patch implements handlers to reload upstart/systemd
scripts to ensure that this happens when any one of the scripts
change.

The handler to reload the services now only tries to restart the
service if the host is in the group for the service according to the
service group mapping. This allows us to ensure that handler
failures are no longer ignored and that no execution time is wasted
trying to restart services which do not exist on the host.

Finally:
- Common variables shared by each service's template files have
  been updated to use the service namespaced variables.
- Unused handlers have been removed.
- Unused variables have been removed.

Change-Id: Id35de501acf6b3164221085f8f9e142234ea0d73
2016-10-13 13:30:58 +01:00
Andy McCrae
a894972d6a Type min_part_hours,part_power,repl_number as int
This PR ensures that if a string is entered as the var it will be forced
to an "int" type, which is expected by the script.

Change-Id: I8d7f3c4894c2b0975893ebb30b734ae7fccfbc22
2016-09-26 15:12:23 +01:00
Andy McCrae
9578aa5fd7 Ensure storage/replication_ip are uniform
We need to ensure the swift services are listening on the same IP and
port as the ring is setup for. Currently we calculate these values in
different ways, which leads to the possibility that the ring is setup
for ports and IPs that the services are not listening on.

This change ensures this by calculating the storage and replication
addresses within the role - whilst still accepting overrides from
outside the role. This ensures functionality does not change, but that
it is now not possible for the swift services to not listen on the
ports/IPs defined in the ring.

As part of this change we move to set "swift_dedicated_replication"
based on whether the storage and replication addresses are the same,
rather than the specified network. This means you can run a dedicated
replication config by configuring the services, which was possible to
configure, but again would have meant the services and ring were not
configured in the same way. This is required to ensure uniformity.

Finally - we remove the ability to set a storage_port or repl_port
within the swift_vars, this was a setting that would never work in it's
current implementation. The storage/repl_port would depend entirely on
which swift service was in use, and can not be set in such a blanket
fashion. Since this was a completely not working funcitonality it has
now been removed.

Change-Id: Ibe5330cdca7fd81f379b80dc55ad06529ce4d580
2016-09-26 11:23:15 +01:00
Andy McCrae
2257044bc0 Use openstack-ansible-tests repository for Swift
Move to use the openstack-ansible-tests repository for Swift.
Make changes use Ansible 2.1.1 (These are required for gating to work
properly, since the majority of repositories have already changed).

Additionally, lets set this up to work with 1 infra host in the MR swift
gate to improve efficiency.

Change-Id: Ic0d35bcf8bafb90a986c7cdda3987f70580af165
2016-09-22 15:59:32 +00:00
Jimmy McCrory
98cb749139 Store dedicated replication conditions in var
Create a new default var, swift_dedicated_replication, for storing
conditions used to determine if a replication network is being used for
swift. This allows for removing redundant tasks, reducing role run
times.

Change-Id: Ieb4263035527ff069017d40b95787f934fbd308c
2016-09-21 06:32:35 -07:00
Paulo Matias
4ea2993fd5 Fix typo in ceilometer ignore_projects filter
Change-Id: I62b2151c8c6ed3dd728953d9981c0d7cb0b59aef
2016-09-17 14:12:43 -03:00
Jenkins
11c9c3a019 Merge "Filter Gnocchi traffic optionally" 2016-09-13 09:29:53 +00:00
Paulo Matias
7adb8b2876 Fix missing newline in ceilometermiddleware config
Regression introduced by Ic2129bd6be85f6a51feea95d47cfbb349c44f363.
Ansible setups Jinja2 in such a way that a newline after an ``{% endfor %}``
is stripped. Therefore, we insert an additional newline.

Change-Id: I6c72c82bea0c9b93b7ef18ce89ccd7c18066b802
2016-09-10 23:09:44 -03:00
Steve Lewis
1d55c377b8 Filter Gnocchi traffic optionally
When using Swift storage for Gnocchi we want to add a filter to the
Ceilometer middleware for Swift to exclude reporting traffic caused
by Gnocchi storing data into Swift to avoid feedback-loops. This
is typically done by isolating the Gnocchi in a dedicated Project in
Keystone.

The best place to filter this traffic is in the middleware
as opposed to filtering in the Ceilometer Collector.

Related-To: I67b5bff394ad35cf95d10ba32d602954799d7348
Change-Id: If20248494d4e6c127db4ffdf77ae43482f64fe58
2016-09-07 11:40:43 -07:00
Andy McCrae
750ccaa9e8 Refactor and improve swift-rings.py
We can simplify and refactor swift_rings.py and swift_rings_check.py by
moving to a "FULL_HOST_KEY" model where we simply compare the full
string, rather than individual elements.

To do this we need to adjust the contents template to use the same field
values as used by swift:
* rename repl_ip to replication_ip
* rename repl_port to replication_port

Additionally, this allows us the ability to change port values on the
fly, by adjusting the "DEVICE_KEY" to only be the IP and device name the
port will now automatically get changed if the service port changes.

This is a precursor to adjusting the default swift storage service ports
to match upstream defaults, and will reduce the upgrade impact of that
task.

Change-Id: I704edcba4facb2170990ebec2a67d4179a023fc2
2016-09-02 05:10:17 +00:00
Jenkins
a7f0e1bd7c Merge "Point auditor service at the replicator config" 2016-08-31 18:55:55 +00:00
Andy McCrae
599232af30 Point auditor service at the replicator config
When running dedicated replication network we need the auditor service
to point at the configuration file with the replicator.

Without this we will get errors in the various servers indicating the
replicator service can not be found by the auditor service:

object-auditor: STDOUT: Unable to find object-replicator config section
in /etc/swift/object-server/object-server.conf

Change-Id: Idd5e6ddab66a7855959e93139d2c7934762c6ba9
2016-08-31 15:26:41 +00:00
Andy McCrae
71122a3184 Namespace the telemetry vars for swift
Change-Id: Ic2129bd6be85f6a51feea95d47cfbb349c44f363
2016-08-31 14:49:31 +00:00
Kevin Carter
702026d3e0 remove the path from the auth_url
Based on the upstream documentation for swift it would seem
that the full auth path "http://localhsot:35357/v3" is no
longer needed.

DOC Link: http://docs.openstack.org/mitaka/install-guide-ubuntu/swift-controller-install.html

Change-Id: Ie9986e89d5e6198f21e768c57ce18a4e7cbce153
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2016-08-31 11:29:58 +00:00
Jenkins
9d81f521d2 Merge "Properly default pretend_min_part_hours_passed" 2016-08-05 09:32:08 +00:00