53 Commits

Author SHA1 Message Date
df4c66af1e Update reno for stable/rocky
Change-Id: I1c4cb0844a7cf50112bf11e84437853f6e5e1c87
2018-08-10 16:48:26 +00:00
Jesse Pretorius
fc6c1935d7 Execute service setup against a delegated host using Ansible built-in modules
In order to reduce the packages required to pip install on to the hosts,
we allow the service setup to be delegated to a specific host, defaulting
to the deploy host. We also switch as many tasks as possible to using the
built-in Ansible modules which make use of the shade library.

The 'virtualenv' package is now installed appropriately by the openstack_hosts
role, so there's no need to install it any more. The 'httplib2' package is a
legacy Ansible requirement for the get_url/get_uri module which is no longer
needed. The keystone client library is not required any more now that we're
using the upstream modules. As there are no required packages left, the task
to install them is also removed.

Depends-On: https://review.openstack.org/582359
Depends-On: https://review.openstack.org/587376
Change-Id: I68f3a0bf2b7a3a12cbf40d7d6a853d5b4c6dd0f3
2018-07-31 17:37:02 +00:00
Andrew Smith
91af246bdf Update to use oslo.messaging service for Notify
This introduces oslo.messaging variables that define the Notify transport
in place of the rabbitmq values.

This patch:
* Add oslo.messaging variable for Notify to defaults
* Update url generation
* Add oslo.messaging to inventory
* Add release note

Change-Id: I1c2e844c4c7a2256087bcc4521f970ca8e8c6b16
2018-06-11 16:45:16 -04:00
Markos Chandras
51ea328b7c Add support for using distribution packages for OpenStack services
Distributions provide packages for the OpenStack services so we add
support for using these instead of the pip ones. However, functional
testing is not complete yet since it requires tempest to be in the swift
virtual environment which doesn't exist for distro installs. As such,
for functional testing to cover this method of installation it needs
to be changed quite a bit.

Change-Id: Id75e56440685df407a9991018dc07b5a75429082
Implements: blueprint openstack-distribution-packages
2018-06-04 08:41:19 +01:00
ea1f94f748 Update reno for stable/queens
Change-Id: If010ba4ab0f4ad238390ec01378804ba34670ccf
2018-02-14 15:50:34 +00:00
Andreas Jaeger
5ac157cf39 Remove setting of version/release from releasenotes
Release notes are version independent, so remove version/release
values. We've found that projects now require the service package
to be installed in order to build release notes, and this is entirely
due to the current convention of pulling in the version information.

Release notes should not need installation in order to build, so this
unnecessary version setting needs to be removed.

This is needed for new release notes publishing, see
I56909152975f731a9d2c21b2825b972195e48ee8 and the discussion starting
at
http://lists.openstack.org/pipermail/openstack-dev/2017-November/124480.html
.

Change-Id: I7df0385b62d118b535179dd42d7aa576897815ca
2017-11-17 08:04:47 +01:00
Jean-Philippe Evrard
0147d155d4 Update upgrade role for Queens from P
This changes the a-r-r with the proper version, and fixes
the repo path for role upgrades:

The role will, during test-upgrade-pre, have its current branch
checked out, and will deploy the current branch infrastructure,
including the generation of the constraints.

Then, the installation of previous branch of the role will use
the constraints for its previous version, but will have no
constraints for it, and fail.

We need to generate the constraints for the previous version too.
This should do it.

This patch also implements an initial set of jobs intended to
match the current job execution method. It does not intend to
improve how the jobs are executed - only to replicate what is
currently in openstack-infra/openstack-zuul-jobs and provide
the platform to iterate on.

Change-Id: I4aa7591c931501490e706b9d61ba0bc12c006baa
2017-10-17 20:52:25 +00:00
b4330eaa56 Update reno for stable/pike
Change-Id: I1ff7d69550a8080441906704130499052efea4ba
2017-08-21 10:34:17 +01:00
ZhongShengping
52228ba571 Fix openstackdocstheme settings
To use openstackdocstheme 1.11.0 properly, this patch fixes
some settings according to follow[0].

[0]https://docs.openstack.org/openstackdocstheme/latest/

Change-Id: I7ef76fd6e6fe79d12bc57eabe0b120ab10d6d429
2017-07-03 16:22:24 +08:00
ZhongShengping
0c1373609b Switch from oslosphinx to openstackdocstheme
As part of the docs migration work[0] for Pike we need to switch to use the
openstackdocstheme.

[0]https://review.openstack.org/#/c/472275/

Change-Id: Ia3d468d5ed5c31314a20cf1c7b4b6a9829628064
2017-06-26 11:39:11 +08:00
Charles Farquhar
14c57dd419 Add support for swift3 middleware
This commit adds support for the swift3 middware, which allows S3-compatible
clients to use swift for object storage.

Change-Id: I56cd63057cc771310b69c311d975e06f73c773f7
Related-Bug: 1625053
2017-06-23 18:26:46 +00:00
Jesse Pretorius
5bf8389219 Rename release note to unique hash
Change-Id: Icf5fda2c7c0ff0b8b04f2b1f01620180f3ce2f16
2017-04-28 11:40:02 +01:00
Jesse Pretorius
22a4a72afb Reduce init restart/kill times
The systemd unit 'TimeoutSec' value which controls the time
between sending a SIGTERM signal and a SIGKILL signal when
stopping or restarting the service has been reduced from 300
seconds to 120 seconds. This provides 2 minutes for long-lived
sessions to drain while preventing new ones from starting
before a restart or a stop.

The 'RestartSec' value which controls the time between the
service stop and start when restarting has been reduced from
150 seconds to 2 seconds to make the restart happen faster.

These values can be adjusted by using the *_init_config_overrides
variables which use the config_template task to change template
defaults.

Change-Id: I048b877e859ad744dc54f19a93afdd89f8ef1661
2017-04-26 17:04:04 +00:00
Andy McCrae
e54538cd17 Rename reno to avoid conflict on integrated repo
Change-Id: I3230eae9838c72058d29936345247ce402b46e57
2017-04-11 12:12:39 +01:00
Jenkins
6a1669fb3d Merge "Deprecate auth_plugin option" 2017-04-06 15:58:13 +00:00
Andy McCrae
78b4f6aac6 Update container-sync to use internal client
This is to match the way this is configured in upstream Swift.

Change-Id: Ifbfd4158e04f38bf3383a7c2461374a21088fdd3
2017-04-04 15:04:19 +00:00
Kevin Carter
867f3157e9 Ensure the components are isolated from the system
This creates a specific slice which all OpenStack services will operate
from. By creating an independent slice these components will be governed
away from the system slice allowing us to better optimise resource
consumption.

See the following for more information on slices:

* https://www.freedesktop.org/software/systemd/man/systemd.slice.html

See for following for more information on resource controls:

* https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html

Tools like ``systemd-cgtop`` and ``systemd-cgls`` will now give us
insight into specific processes, process groups, and resouce consumption
in ways that we've not had access to before. To enable some of this reporting
the accounting options have been added to the [Service] section of the unit
file.

Change-Id: Ibcb7d2a69ed67a99c88dc143c76aa8448d31cc9e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2017-03-30 15:16:03 +00:00
ZhongShengping
7ca4b62eb5 Deprecate auth_plugin option
Option "auth_plugin" is deprecated, use option "auth_type" instead.

Change-Id: I6ac382f3856b754291378cde335bc71cd2161023
Implements: blueprint deprecate-auth-plugin
2017-03-25 12:26:23 +08:00
Ravi Gummadi
bb9e3f5828 Cap the number of worker threads
Users can configure the number of worker threads. However when it's
not specified the calculated number of workers can get too large on
hosts with a large number of CPUs. Capping only swift proxy server
worker threads when the proxy is in a container. Not capping the
remaining swift services' workers because of the performance impact
it may cause because of the capping.

Change-Id: I12d930552558144ab49fecc0b3776747c1f02166
2017-03-06 06:39:01 -05:00
Cao Xuan Hoang
42cc0a4212 Use https instead of http for git.openstack.org
Trivialfix

Change-Id: If7efc916e31cea7ff3afb2da55339232e89babbf
2017-02-06 17:09:07 +07:00
47454fb824 Update reno for stable/ocata
Change-Id: I702e2eafac590cd2fd22d54c4f5ad3bcffea644c
2017-02-03 18:58:49 +00:00
Andy McCrae
c7b7e862a3 Fix erroneous release note
The use of `here` as a link causes gate failures in the integrated
build.

Change-Id: Iee48f045a1a2c50fc19d1844522155ad556103a0
2017-02-02 15:48:47 +00:00
Jenkins
8238849e41 Merge "Allow X-History-Location versioned_writes" 2016-12-03 03:14:48 +00:00
Andy McCrae
3fb6784459 Attempt to fix swift_pypy memory issues
Move to use tempauth to resolve memory issues resulting from an AIO
swift install running in pypy.

This PR adds some options for using pypy:
* Set a pypy Garbage collection value

This PR includes some tempauth fixes to make it useable:
* Set the tempauth users based on a variable

Testing is changed as follows for pypy:
* Use only memcache within swift-proxy
* Remove galera/keystone
* Add swap for swift-storage hosts
* Use tempauth for pypy
* Reduce to 2 swift hosts

Change-Id: Ic1ed5acc9b20853d9a159035226f97fda088f035
2016-12-01 11:07:10 +00:00
Andy McCrae
de7ab0a800 Allow X-History-Location versioned_writes
Versioned Objects in Swift now use a middleware that is added to the
pipeline instead of the "allow_versions" in the container config. These
have 2 different Headers, so to prevent "X-Versions-Location" from not
working the "allow_versions" in the container config is left in place.

This patch adds the middleware by default and sets the Versioned Objects
support to be on. This is the upstream default and the deployer would
still need to set the "X-History-Location" header on a container to
utilise the feature.

Change-Id: I88811fd77fad8d2241448ca5ffb565fa7d704a00
2016-11-30 11:18:45 +00:00
Jenkins
1ff581f6b9 Merge "Add CentOS7 support for Swift" 2016-11-15 14:30:34 +00:00
Andy McCrae
42acde5ee9 Add CentOS7 support for Swift
This PR Adds CentOS 7 support for Swift.
The following was required to get CentOS 7 to work:
* Add yum install path + packages
* Variablize rsync service name
* Gather network interface facts prior to setting storage/repl IPs
* Ensure /etc/defaults/rsync is only set for apt installations.
* Ensure the rsyslog service is started and enabled.

Change-Id: Ibaf8bc8d54b55820e8b527b52940c61c05c732d8
2016-11-14 17:03:02 +00:00
Andy McCrae
4aa96a832c Set rsync to not perform reverse lookups
We don't setup hosts entries for replication addresses on storage hosts,
by default. Rsync is performing a reverse lookup on each rsync, and
failing. We should turn this reverse lookup off by default, but allow
this to be set to True.

This PR adds the "swift_rsync_reverse_lookup" boolean which is defaulted
to false.

Change-Id: Ie98113fd71a70ce9c74c3812046be6959b9a353f
2016-11-07 16:24:52 +00:00
Jesse Pretorius
ead9bc82b0 Simplify pip options/constraints mechanism
The current constraints generation for the
installation involves multiple tasks and multiple
variables.

Using multiple tasks extends the installation time
unnecessarily and the additional variables are
unnecessary.

This patch aims to simplify the mechanism and
hopes to speed it up a little.

Change-Id: Icdd5d053346e7d31dcfd4331c7b7b0f184bf376b
2016-11-02 18:07:36 +00:00
Andreas Jaeger
4c1fbcd8e5 Enable release notes translation
Releasenote translation publishing is being prepared. 'locale_dirs'
needs to be defined in conf.py to generate translated version of the
release notes.

Note that this repository might not get translated release notes - or
no translations at all - but we add the entry here nevertheless to
prepare for it.

Change-Id: I582151ca1049a6e6911814448467c9831178f125
2016-10-06 20:38:11 +02:00
Major Hayden
4447c245ff Update reno for stable/newton
Change-Id: I73f4443554e591d61483b157804bff5cbac784be
2016-09-29 14:24:12 -05:00
Andy McCrae
10d3ee9026 Add check for swift_hash_path_ variables
We should never change the swift_hash_path_prefix/suffix variables on a
running cluster. This PR implements a check that will fail if the
variable is different to what is already on disk.

To ensure this is still possible this PR implements a
"swift_force_change_hashes" variable which can be set to "True" in order
to force change the swift_hash_path variables regardless of whether they
are different or not.

Change-Id: Idaedc125aede22c347668afd9e98ed1823eb142c
2016-09-14 13:06:53 +00:00
Andy McCrae
750ccaa9e8 Refactor and improve swift-rings.py
We can simplify and refactor swift_rings.py and swift_rings_check.py by
moving to a "FULL_HOST_KEY" model where we simply compare the full
string, rather than individual elements.

To do this we need to adjust the contents template to use the same field
values as used by swift:
* rename repl_ip to replication_ip
* rename repl_port to replication_port

Additionally, this allows us the ability to change port values on the
fly, by adjusting the "DEVICE_KEY" to only be the IP and device name the
port will now automatically get changed if the service port changes.

This is a precursor to adjusting the default swift storage service ports
to match upstream defaults, and will reduce the upgrade impact of that
task.

Change-Id: I704edcba4facb2170990ebec2a67d4179a023fc2
2016-09-02 05:10:17 +00:00
Jesse Pretorius
e31bec0e04 Rename package lists (and related vars) appropriately
In order to make it easier to differentiate between the lists of
python packages, distribution packages, downloaded packages,
package pins and other similar variables the variable names are
being changed to ensure that they have a more explicit suffix
that defines the purpose and makes the naming more consistent.

This is to facilitate a lookup plugin which will be able to look
up all the package lists and present them as a consolidated piece
of data which may be used for artifact preparation.

Change-Id: I2c0d397df559df1725ec541afd7616e114d49d6f
2016-08-31 10:23:09 +01:00
Jesse Pretorius
8256654ece Allow swift log file permissions to be set
This patch adds the ability to configure the log file permissions.

This is primarily to address a gating issue where the jenkins does
not have access to the account, container and object service logs.

That said, it's generaly useful to have this option available.

Change-Id: Ibc3d66273a4494414dde1a95ec132ff83c235f9c
2016-08-04 06:35:49 +01:00
Jesse Pretorius
9790f9eb29 Add ability to change apt/yum package state
The current method of installing the distribution packages required is
set in the tasks and cannot be changed by a deployer.

Currently the apt task always installs the latest package. This results
in unexpected binary changes when a deployer may simply be trying to
execute a configuration change.

This patch adds the ability for a deployer to change the desired state
so that the results are predictable.

Change-Id: Ib58ccc89b868f42295425527eb1a9dbcbe21839c
2016-08-02 15:18:51 +01:00
Jenkins
9f6e7ebc6b Merge "Add pretend_min_part_hours_passed to ring build" 2016-07-13 17:39:16 +00:00
Andy McCrae
cce0410334 Add pretend_min_part_hours_passed to ring build
Expose the "pretend_min_part_hours_passed" option when accessing
swift-ring-builder.

This PR adds a variable "swift_pretend_min_part_hours_passed" which
defaults to False.

Change-Id: Ief94f011049b57f1ab0c2c48dfa81659d7ba13f7
2016-07-13 09:27:54 +01:00
Jesse Pretorius
5f9c83a83c Update sphinx configuration
This patch updates the sphinx configuration for docs and releasenotes
to make it easier to replicate across repositories and to comply with
pep8 testing without exceptions.

Change-Id: I67b952afefbf28a62f33ef784f498c950763de7d
2016-07-12 16:54:14 +01:00
Kevin Carter
022b62fc8d Add options for configuring file limits in swift
Options have been added to allow a deployer to control the
hard, soft, and fs.file-max limits on swift specific nodes.

Change-Id: Ia5dd51fa360745a9776d58d0bf49d8e1201d3139
Closes-Bug: #1595290
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2016-07-12 12:12:41 +01:00
Anton Khaldin
9d30bb30b9 Only install to virtual environment
Remove all tasks and variables related to toggling between installation
of swift inside or outside of a Python virtual environment.
Installing within a venv is now the only supported deployment.

Additionally, a few changes have been made to make the creation of the
venv more resistant to interruptions during a run of the role.
* unarchiving a pre-built venv will now also occur when the venv
  directory is created, not only after being downloaded
* virtualenv-tools is run against both pre-built and non pre-built venvs
  to account for interruptions during or prior to unarchiving

Change-Id: Id2a27c082a5eb4eddc9eacde9de30f4e61f9ec36
Implements: blueprint only-install-venvs
2016-07-08 03:15:37 -07:00
Michael Gugino
d0c001c5e6 Implement Xenial Support
This commit adds support for Ubuntu 16.04.
Primarily, this commit consists of changes
necessary to support systemd.

Change-Id: Ide0d570b508be8efaa885431598c726d7baad4f6
2016-06-28 09:20:11 -04:00
Darren Birkett
2dc3514da5 Make ansible managed line in swift.conf static
The ansible-managed line includes date/time information, meaning that
the conf file that gets laid down on different hosts does not have the
same MD5 sum across all hosts. This causes swift-recon --md5 to fail.

This commit fixes the issue by adding a static comment
to the config file instead

Change-Id: Id00cd79636f43776aaaa51e2ae6d7c5eacff8d34
Partial-Bug: #1589532
2016-06-10 15:48:07 +00:00
Kevin Carter
38b510c0d4 Add staticweb to the default middleware list
This change adds the "staticweb" middleware to the default list of
middleware deployed.

Per http://docs.openstack.org/developer/swift/middleware.html#staticweb,
staticweb should be added to the pipeline just after any auth
middleware.

Change-Id: Ic8614ac7295751df9ad1e5ff235b84c0a0a7d252
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2016-05-24 14:10:59 -05:00
Jenkins
c171e57c35 Merge "Enable pypy to be used with swift" 2016-05-18 07:58:13 +00:00
Kevin Carter
60b7b834f9 Enable pypy to be used with swift
The intention of this change is to enable an option to increase swift
performance without changing any of the swift internals. This change is
in-line with documentation published by the good folks at swiftstack and
intel. This change will only effect swift when its installed in a
virtual environment which is a self-imposed limitation such that we're
not messing with the system's default Python.

New Options:
  - A new option `swift_pypy_enabled` has been added to enable or
    disable the pypy interpreter for swift. The default is "false".
  - A new option `swift_pypy_archive` has been added to allow a
    pre-built pypy archive to be downloaded and moved into place. This
    option is a dictionary and contains the URL and SHA256 as keys.

Reference: [ https://software.intel.com/en-us/blogs/2016/05/06/doubling-the-performance-of-openstack-swift-with-no-code-changes ].

Change-Id: I045e8b97d29b96e2c2450761efa1c2668de12a75
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2016-05-17 04:41:34 +00:00
Andy McCrae
53027de8c8 Setup rsync module per object server drive
This PR sets up rsync module per drive for swift object servers. This
improves resource distribution, and also allows us to utilise rsync
module changes to prevent disk writes to specific drives when they begin
filling up.

Enable this option by setting "swift_rsync_module_per_drive: True", this
will still default to False to match upstream defaults. Additionally the
rsync max connections has been increased to match upstream defaults.

Additionally we include rsync.d/*.conf files by default and set this
directory up, so that we can add individual configuration to disable
specific drives.

Change-Id: I2019cade5bf5f2878497d30ce738dff07786fa64
2016-05-16 11:57:46 +01:00
Andy McCrae
292c443163 Set fallocate_reserve to match upstream default
The upstream default for fallocate_reserve has just changed to 1% with
the inclusion of a "%" option for fallocate_reserve.
This PR sets the openstack-ansible default to match this value.

Change-Id: Iaafcb881dc067fa94d37e9eae93eed16d18c2f20
2016-05-13 11:34:53 +00:00
Jenkins
bf85beb999 Merge "Remove XFS filesystem from the daily mlocate cron job" 2016-04-26 21:16:10 +00:00
Bjoern Teipel
db4ad90762 Remove XFS filesystem from the daily mlocate cron job
This fix removes the XFS filesystem from the daily cron job.
It will help to remove unnecessary disk IO due to updatedb/mlocate
swift object indexing inside the /srv/node folders.

Change-Id: I8bfa92003ce06ee4f065663e054cd2d04f458ec6
Closes-Bug: #1572307
2016-04-22 08:11:57 -05:00