594 Commits

Author SHA1 Message Date
Jenkins
4dbd443f31 Merge "Adjust key distribution mechanism for Swift" 2015-07-17 17:26:35 +00:00
Jenkins
8a4fa2a080 Merge "Adjust swift_rings.py to work on specified regions" 2015-07-16 01:37:39 +00:00
Jenkins
24b4ad523a Merge "Cleanup unused functions in swift_rings.py" 2015-07-15 22:19:05 +00:00
Jenkins
38f045d80b Merge "Add md5sum check for swift rings after ring-sync" 2015-07-15 21:04:42 +00:00
Andy McCrae
ca5e780800 Add md5sum check for swift rings after ring-sync
Moving towards multi-region swift there is a chance that 2 regions will
attempt to update the ring at the same time. Whilst measures are in
place to ensure a region only updates its own region entries in the
ring it would still be possible, if the 2 runs happened simultaneously,
that some ring inconsistencies could happen. For example, if a region A
updates at the same time as region B but the sync order is different
some nodes could have region A's "updated" ring and some with region
B's "updated" ring.

To ensure this hasn't happened (without our knowledge) this patch adds
another md5sum check which will report if the rings are inconsistent
across the nodes.

Change-Id: Id88dfebcaa0553437953f92235bf63363f750797
Partially-Implements: blueprint multi-region-swift
2015-07-14 14:05:49 +01:00
Andy McCrae
0a1174c695 Adjust the network rings are syncd on
Swift nodes don't have to be able to talk to each other on the
ansible_ssh_host, but will always have to talk on the storage_network
specified.

This will allow us to let remote or local hosts, that can't connect to
each other on the ansible_ssh_host address to still be able to sync
their rings.

In order to achieve this we set a swift_storage_address fact which then
simplifies the "ring_contents" file, to avoid performing the same logic
twice.

Change-Id: Ic1f2a915244101ad4fbbe52496dd2b991915d01d
Partially-Implements: blueprint multi-region-swift
2015-07-14 13:29:34 +01:00
Andy McCrae
307160b892 Cleanup unused functions in swift_rings.py
Removes 2 unused functions in swift_rings.py (check_section &
has_section), these are not called at all and should be removed for
clarity.

Change-Id: Id56654df92834f7a48ce21e70b372f04e920653e
Closes-Bug: #1474334
2015-07-14 13:06:31 +01:00
Ian Cordasco
4a3ede3175 Upgrade the Keystone library to use v3
In order to enable and deploy federated Keystone, we need to use version
3 of the Keystone API and the v3 Keystone Client. This work begins that
transition by having a set of backwards compatible library commands.

Specifically, this commit updates the keystone library to use v3
Keystone Client and the usage of ensure_tenant in the os_keystone tasks
to use the v3 admin url.

In version 3 of Keystone's Endpoints (Catalog) API each endpoint only
has one URL and has separate interface types (public, internal, admin).
This change updates all uses of ensure_endpoint to structure the
endpoint data in a better way for the ensure_endpoint command in the
keystone module. As a result, some incidents where internalurl and
adminurl were swapped have been fixed.

Note:
In new deployments the endpoints will be created using the v3 API and
will therefore not be available via the v2 API. This will be a breaking
change to legacy CLI clients. The openstack CLI should be used instead.

DocImpact
Related-Bug: #1470635
Partially-implements: blueprint keystone-federation
Change-Id: I2cd4f505e850b4b113452abc25ee00d486b1637d
2015-07-13 19:41:44 -07:00
Andy McCrae
126f3f2837 Adjust swift_rings.py to work on specified regions
This patch allows the swift_ring.py to only adjust/add/remove nodes from
a specified region, leaving the other regions that are already in the
ring unmodified.

This will allow multi-region swift to be managed by separate locations
each managing their own region's nodes and leaving other regions to
handle their own nodes.

The default is to manage all regions, so not specifying a region will
work the same as it does now and the script's functionality remains
unchanged.

Change-Id: I1cf73be20f27c437450c0181bb247c86e0f46bc6
Partially-Implements: blueprint multi-region-swift
2015-07-13 10:36:28 -07:00
Andy McCrae
e0c8cfc43c Adjust key distribution mechanism for Swift
In preparation for Swift global cluster we need a solution that moves
away from a centralised "memcache" server as a distribution mechanism,
since we can't guarantee that all swift hosts will have access to one
memcache server.

This patch uses ansible variables (which were already being set) as the
distribution mechanism so that the authorized_keys file can be generated
by using the ansible variables set for each host as part of the tasks.

Additionally this moves away from using "shell" commands to generate the
key and adjust the authorized_keys file and instead uses the built in
ansible "authorized_keys" module as well as the built in
"generate_ssh_key" option for the "user" module.

Finally this adds a "swift_recreate_keys" option which can be set to
"True" in order to recreate all the swift keys, and recreate the
authorized_keys file. This was happening on every run and will now not
happen by default, unless the variable is set to "True".

Change-Id: Ic4eb05042244c32050233e1445633d5731f9603b
Partially-Implements: blueprint multi-region-swift
2015-07-12 10:25:22 +01:00
Jenkins
7d9120327c Merge "Enable all services to use Keystone 'insecurely'" 2015-07-10 16:53:24 +00:00
Jesse Pretorius
a15ccc9327 Enable all services to use Keystone 'insecurely'
This patch introduces an insecure flag for the Keystone internal
 and admin endpoints:

* keystone_service_adminuri_insecure
* keystone_service_internaluri_insecure

Both values default to false. If you have setup SSL endpoints
for Keystone using an untrusted certificate then you should
set the appropriate flag to true in your user_variables.

This patch is used to enable testing and development with
Keystone SSL endpoints without having to make use of SSL
certificates signed by a trusted, public CA.

The patch introduces a new optional argument (insecure) to the
keystone, glance and neutron Ansible libraries. This is a
boolean value which, when true, enables these libraries to
access Keystone endpoints 'insecurely'. When these libraries
are used in plays, the appropriate value is set automatically
as per the above conditions.

Implements: blueprint keystone-federation
Change-Id: Ia07e7e201f901042dd06a86efe5c6f6725e9ce13
2015-07-10 14:06:25 +01:00
Miguel Cantu
9ece6ddfb9 Implement Ceilometer
This patch implements the implement-ceilometer blueprint.
It addes the necessary role/variables to deploy ceilometer
with a Mongodb backend. The Monogdb backend is assumed to
be up and configured and the playbooks only require a few
values to be set in user_variables to establish a connection.

Change-Id: I2164a1f27f632ce254cc2711ada2c449a9961fed
Implements: blueprint implement-ceilometer
2015-07-07 19:31:04 +00:00
git-harry
7b237fc5fc Remove unused swift vars
The following variables are defined as os_swift role defaults but are
unused:

    swift_service_admin_tenant_name
    swift_service_admin_username

This commit removes these unused variables.

Change-Id: I8272f4e398030e04c69a7092d4a770675e3c6df0
Closes-bug: #1460497
2015-07-01 17:54:48 +00:00
Andy McCrae
7b9cc594a3 Make swift_proxy_vars not a required variable
The swift_proxy_vars settings per swift-proxy_hosts entry should not be
a required variable. This patch ensures that if it isn't specified the
plays will still run as normal.

Change-Id: I0ce8c3781c6fccb0fd757498222d1dab6124313f
Closes-Bug: #1469134
2015-06-26 12:56:20 +01:00
Jenkins
858a733f73 Merge "Add global endpoint_type_proto options" 2015-06-22 15:02:08 +00:00
Jenkins
250a5fa651 Merge "Add read/write_affinity settings for Swift" 2015-06-21 01:34:52 +00:00
Andy McCrae
5750386a59 Add global endpoint_type_proto options
This allows you to set the endpoint-type protocol globally for all
services, e.g. internaluri can be http, and publicuri can be https.

You will no longer have to specify it per service, although those
settings already exist and have not changed.

This patch changes no functionality for existing installs or deployments
and the values are defaulted to be the same as before, but allows these
values to be adjusted on a per-endpoint type basis.

Change-Id: I4854216726491f6ea4e265694e702f980fddc5a6
Closes-Bug: #1399383
2015-06-19 15:29:30 +01:00
Andy McCrae
15df49cd3c Add read/write_affinity settings for Swift
Allow the setting of read/write_affinity and write_affinity_node_count
on a per proxy_host basis.

This allows the deployer to set preferences for which region to
read/write to, which can increase the efficiency of a multi-region
swift cluster.

Sample swift.yml has been updated, as well as the aio swift.yml to
ensure these settings are setup as part of the gate, but this shouldn't
change the functionality of swift at all (since there is only 1 region).

Change-Id: I95b456672f419fcc331d6739ce259b022d350472
Closes-Bug: #1415172
2015-06-19 12:30:38 +01:00
kevin
a68baf28e9 Added apt update tasks to everything using apt
This change adds a specific update task to all tasks that all the
apt ansible module. This change was done to ensure that the cache
is updated as expected when instructed to do so. The reason that
the cache update is being removed from the grouping is because
there is an upstream bug that is effecting the process by which
the apt cache is updated when there is a package list to process
within the same task. The work around to make this function as
expected is to move the update into its own task without a package
list.

Upstream Ansible bug:
  - https://github.com/ansible/ansible-modules-core/issues/1497

Change-Id: Ic06d89a76d772c12888b4bc4bbf147be58b0c150
Related-Bug: 1464771
2015-06-16 13:13:58 -05:00
Andy McCrae
e1eeaaed2a Allow protocol to be set per endpoint-type
If services are running behind an SSL terminating LB you will want to
differentiate between protocol on internalURL and publicURL endpoints.
This patch allows you to set the values of protocol per endpoint type,
but doesn't change the default behaviour which is to have it set in one
var.

Change-Id: I7a74c85a8841499623746586ae27103a71c6fec0
Partial-Bug: #1399383
2015-06-16 08:57:17 +00:00
Jenkins
050dc40163 Merge "Fix spelling of runlevel in init scripts" 2015-06-16 04:49:02 +00:00
git-harry
3a41b20943 Fix spelling of runlevel in init scripts
A number of the init templates in the OpenStack roles have the word
runlevel incorrectly spelt as runlelvl. This commit correct those
spelling mistakes.

Change-Id: I0d1b7d5c5cf088fecf07cf0e1bd676b4e4088e2b
Closes-bug: #1464603
2015-06-13 10:21:17 +01:00
Matt Thompson
17d3c68874 Move swift_allow_all_users to role defaults
We currently default swift_allow_all_users in
etc/openstack_deploy/user_variables.yml instead of
playbooks/roles/os_swift/defaults/main.yml.  If a deployer removes this
variable from etc/openstack_deploy/user_variables.yml, the swift
playbook will fail.  This commit moves the variable default to
playbooks/roles/os_swift/defaults/main.yml.

Change-Id: I9a73eda990327bf427f40a13965484fde00cbe21
Closes-Bug: #1424981
2015-06-12 10:16:16 +01:00
Steve Lewis
ea12187051 Configure DB addresses for each service
To enable partitioning of DB traffic by-service, each service needs to
use a custom connection string. Defaulting the service address to a
common galera_address makes things continue to work by default.

While the galera_address could be overridden on a container or host
basis this requires repeating that behavior across each infra node in
the inventory. Providing service-specific connection address variables
simplifies the management somewhat for large deployments and may reduce
error rates.

The service install playbooks now default the service-specific variables
instead of galera_address to the internal lb vip from inventory to
maintain the ease-of-use currently available.

Any value for a service-specific variable set in user_variables.yml will
override the value in the playbook's vars to provide selective
customization as needed.

Change-Id: I4c98bf906a0c1cb11ddd41277a855dce22ff646a
Closes-Bug: 1462529
2015-06-10 02:07:38 +00:00
Jesse Pretorius
67ad05dbbb Add handler flushing to roles that need it
This patch adds handler flushing as the last task in each role to ensure
that there are log files present when the rsyslog client configuration
task is executed a little later in the playbook that consumes the role.

Closes-Bug: #1458822
Change-Id: I92a26b620aa7bc0fbe33175594d37da7d5aca7df
2015-05-26 18:13:55 +01:00
Jenkins
4a693d9bee Merge "added role to pin packages" 2015-05-13 15:59:01 +00:00
Kevin Carter
091e60bb1c added role to pin packages
This new role is now providing the ability for a user to pin apt
packages as they see fit. The idea is to allow someone to implement
pinning in a generic way that can be represented as a global variable
or as a hostvar. The new role has been added to all install roles as
a dependency which will allow it to ensure that packages are pinned
everywhere as would be expected.

Change-Id: I354e8515570fa7174366ba57d57aece3c304568e
2015-05-08 13:22:42 -05:00
Jenkins
691c9e7b35 Merge "Add erasure code policy support to swift" 2015-05-08 14:17:09 +00:00
Matthew Oliver
54f11a1fd3 Add erasure code policy support to swift
This patch is version 1 (not tested) of adding erasure code support
to swift. It adds the following:

  - Add policy_type, ec_type, ec_num_data_fragments,
    ec_num_parity_fragments and ec_object_segment_size to the
    policy definition.

  - Update the ring.contents.j2 to set replica count for the ring
    to ec_num_parity_fragments + ec_object_segment_size, if using
    the erasure code policy_type.

  - Adds extra EC options to swift.conf for EC policies.

I may have missed something and again this hasn't been tested yet.

NOTE: EC in Swift is strictly _BETA_ and shouldn't be run in
      production, however, we do need to test it!

Change-Id: If2069a95e6ea92e34fb329cb6e0027188f15f0bb
2015-05-07 15:09:26 +00:00
Andy McCrae
ba8af72c6f Managed /etc/swift/backups using ansible
If the permissions on /etc/swift/backups are not swift.swift it can
cause the copying of the ring files to fail.

This patch adds /etc/swift/backups to the list of directories managed by
ansible when setting up/configuring swift.

Change-Id: I3937efc54f03e25f504937213a99bde19789aa59
Closes-Bug: #1452743
2015-05-07 15:16:22 +01:00
Andy McCrae
d1cd6a9617 Set tcp_tw_reuse for swift storage hosts
For swift storage hosts we are seeing a lot of connections in TIME WAIT
status, violating nf_conntrack. Setting tcp_tw_reuse should help
alleviate this.

Additionally, in order for tcp_tw_reuse to be set safely we need to
ensure nf_conntrack is loaded.

Change-Id: I4392c4022a9a5a884d07eb6fbf27093f0b16f914
Closes-Bug: #1441363
2015-04-24 11:36:05 -05:00
Kevin Carter
3360e5f2a8 updated swift deps for use in stable/kilo
swift now depends on the package pyeclib which has a library dep
of liberasurecode1, liberasurecode-dev and this commit adds the
libs to the swift install as well as the repo server.

Change-Id: I36ff6354b78faedcfd716f31c53627c1bcb54d78
Partially-Implements: blueprint master-kilofication
2015-04-17 21:46:15 -05:00
Matthew Kassawara
7e258828a4 Update keystone middleware in swift for Kilo
Update keystone authentication middleware in swift to
support the v3 API in Kilo.

Partially implements: blueprint master-kilofication

Change-Id: I28420dbb1cc0da958791c5e23c13eb38689028dd
2015-04-16 08:12:57 +00:00
Andy McCrae
59bff1bd3a Distribute rings to build dir on all swift_hosts
This prevents a situation where swift_hosts[0] changes as a host, or is
removed (meaning a new swift_hosts[0]) which would prevent the rings
from being built. Additionally it will act as a backup for the ring
build files.

To clarify the purpose of the rings directory, the directory was renamed
to "ring_build_files", clarifying that the active rings are in
/etc/swift/

Change-Id: Id878d219504db195353286b28d7c038e5e659263
Closes-Bug: #1444529
2015-04-15 16:30:58 +01:00
Kevin Carter
3e361977c8 Flake8 update - swift_rings.py
This update fixes issues with linting such that it can now pass
OpenStack hacking/flake8 checks.

Change-Id: I564ac9729381ee068743b8fb70969b492b46b018
Partial-Bug: 1440462
2015-04-07 09:29:18 +00:00
Andy McCrae
7a4f1c7d9e Fix dedicated replication network logic
* Adjust the init scripts to be dropped
* Ensure dedicated server is started for dedicated replication network
* Remove the dedicated replication conf when no dedicated replication
network is in use.

Closes-Bug: 1435802
Change-Id: I8ccc32eb7d3cdeb36b3cae94ea51d8b0a1fd3e71
2015-03-24 12:43:47 +00:00
Kevin Carter
3f89385863 Removed all rackspace related logging parts
This commit removes all of the rackspace related logging components.
This change is part 1 of 3 to update all of the logging bits within
the stack such that they're made more generic and community
consumable.

Plays removed:
* rsyslog-install.yml

Roles removed:
* rsyslog_setup

Variable changes:
* The default kibana and elasticsreach variables were removed.

Example config changes:
* The environment map was updated with the removed logging comonents.

Gate changes:
* rsyslog-install has been removed from the gating script as it no longer
  serves the same purpose.
* The kibana variable override was removed.
* Kibana entries in `haproxy_config.yml` have been removed.

DocImpact
Implements: blueprint rsyslog-update

Change-Id: Icd25653a29c9936cecc63ba5dc82aeb1cfb7ebd8
2015-03-14 22:35:59 -05:00
Jenkins
87805b8ab0 Merge "Allow swift object versioning to be enabled" 2015-03-05 09:41:41 +00:00
Andy McCrae
c306a1308c Handler should only restart relevent swift services
* Adjust the handler to include a "restart" handler for each of account,
container, object and proxy service groups.
* Add a variable in defaults listing program names for each swift
service group.
* Remove the over-arching "all swift program_names" variable.
* Change the storage and proxy host tasks to call the appropriate
* handler.

Change-Id: I25adfa152fc7a3da83ca7c12d57977eec8b51d7b
Closes-Bug: #1427601
2015-03-04 09:53:08 +00:00
Andy McCrae
474d3ba232 Ensure replication_network is not required for swift
* Adjust config location variables for account, container, object
variables, to name them more clearly.
* Adjust config placement order, to happen after init scripts are
created so that the service restart handler will work appropriately.
* Ensure the init scripts for the replicator service are created
appropriately whether a dedicated replication_network is or isn't used.
* Ensure the dedicated replication network configuration is only created
when replication_network is specified and different to storage_network.
* Ensure the appropriate configuration section in the server.conf is
only added when the replication_network is specified and different to
the storage_network.

Change-Id: I4b204a974bb0a217f5222b6aa0fa36aa8c23b999
Closes-Bug: #1427599
2015-03-03 19:19:28 +00:00
Matt Thompson
186c8efaba Allow swift object versioning to be enabled
By default, allowed_versions (swift object versioning) is disabled.
This change allows the deployer to determine if this feature should be
enabled or disabled.  The default value in the os_swift role for this
feature is True (enabled).

Change-Id: I837e0c0d063cacf911ba2461af21df3da5791afb
Closes-Bug: 1427154
2015-03-02 12:01:50 +00:00
Matt Thompson
c1ab8e5635 Update auth_uri / identity_uri
While testing glance + swift, I noticed that the glance-api.conf and
glance-registry.conf being dropped had incorrect auth_uri /
identity_uri values set.  This change updates auth_uri / identity_uri
throughout to point to the correct keystone_service_adminuri /
keystone_service_internalurl variables.

Change-Id: I3cbbfefe7da54b08bb9a55e4a2ca3a8bd786577d
Closes-Bug: 1425099
2015-02-24 16:07:55 +00:00
Kevin Carter
64b7659015 Convert existing roles into galaxy roles
This change implements the blueprint to convert all roles and plays into
a more generic setup, following upstream ansible best practices.

Items Changed:
* All tasks have tags.
* All roles use namespaced variables.
* All redundant tasks within a given play and role have been removed.
* All of the repetitive plays have been removed in-favor of a more
  simplistic approach. This change duplicates code within the roles but
  ensures that the roles only ever run within their own scope.
* All roles have been built using an ansible galaxy syntax.
* The `*requirement.txt` files have been reformatted follow upstream
  Openstack practices.
* Dynamically generated inventory is now more organized, this should assist
  anyone who may want or need to dive into the JSON blob that is created.
  In the inventory a properties field is used for items that customize containers
  within the inventory.
* The environment map has been modified to support additional host groups to
  enable the seperation of infrastructure pieces. While the old infra_hosts group
  will still work this change allows for groups to be divided up into seperate
  chunks; eg: deployment of a swift only stack.
* The LXC logic now exists within the plays.
* etc/openstack_deploy/user_variables.yml has all password/token
  variables extracted into the separate file
  etc/openstack_deploy/user_secrets.yml in order to allow seperate
  security settings on that file.

Items Excised:
* All of the roles have had the LXC logic removed from within them which
  should allow roles to be consumed outside of the `os-ansible-deployment`
  reference architecture.

Note:
* the directory rpc_deployment still exists and is presently pointed at plays
  containing a deprecation warning instructing the user to move to the standard
  playbooks directory.
* While all of the rackspace specific components and variables have been removed
  and or were refactored the repository still relies on an upstream mirror of
  Openstack built python files and container images. This upstream mirror is hosted
  at rackspace at "http://rpc-repo.rackspace.com" though this is
  not locked to and or tied to rackspace specific installations. This repository
  contains all of the needed code to create and/or clone your own mirror.

DocImpact
Co-Authored-By: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
Closes-Bug: #1403676
Implements: blueprint galaxy-roles
Change-Id: I03df3328b7655f0cc9e43ba83b02623d038d214e
2015-02-18 10:56:25 +00:00