Commit Graph

221 Commits

Author SHA1 Message Date
Major Hayden
d4d6b042df Adds group support to inventory-manage.py
Added -g for showing groups per instance and -G for showing instances
per group.

Also fixed up some pep8 and pylint issues.

Closes-Bug: 1482141

Change-Id: I698384467268336ecc74f865ee35f3e39cad5186
2015-09-21 08:10:57 -05:00
Nolan Brubaker
0c6f849288 Remove Juno to Kilo logic in upgrade script
This is a barebones start to the upgrade script for the Liberty cycle.
This change simply cleans it out, more work will come later.

Change-Id: I86515970164a000d321ac4375497bfeba860ba8e
Implements: blueprint liberty-upgrade-path
2015-09-12 04:28:57 +00:00
Kevin Carter
35ec780515 Rename everything for the osad to osa name change
This change updates all fo the names that we were using to the post
openstack migration name for openstack-ansible.

Change-Id: I6524af53ed02e19a0f56908e42a65d2dae8b71e3
2015-09-12 02:31:21 +01:00
Major Hayden
059cae3d76 Add profiling for Ansible tasks
This should help find the slowest Ansible tasks that need to be optimized.

It is only enabled by default for gate jobs.

The profiling callback code is MIT licensed.  Per legal-discuss, I've added
the MIT license to the LICENSE.txt at the root of the repository along with
a note about which code is MIT licensed.

Thread on legal-discuss:

http://lists.openstack.org/pipermail/legal-discuss/2015-September/000398.html

Closes-Bug: #1488639

Change-Id: I46a2c495d9f8f1131ec08be99696efc39133f291
2015-09-09 15:40:24 +00:00
Jenkins
4da86250b7 Merge "Set the Ubuntu mirror used based on the environment" 2015-09-09 06:19:54 +00:00
Jesse Pretorius
b940e7b7de Set the Ubuntu mirror used based on the environment
Apt updates/installs are the most common cause of build timeouts
within HP Cloud gate checks.

This patch changes both the AIO and the gate check from using the
Rackspace Ubuntu mirror for everything to:

1) The AIO bootstrap will use whichever mirror is already defined
   on the host operating system.

2) The gate check will use a known mirror which is defined per
   cloud provider. If the cloud provider is not known, or the
   script is being used outside of OpenStack-CI, then the gate
   checkwill fall back to using the AIO's mechanism.

The idea is to make use of an apt mirror that is as close to the
instance as possible in order to speed up apt updates and installs.

Closes-Bug: #1491749
Change-Id: Ia37100bc198b64b74f27ea98ec5956e7cf084883
2015-09-08 10:38:03 +01:00
Jenkins
35a8dc8cf8 Merge "Update Ansible to v1.9.3-1" 2015-09-06 18:48:48 +00:00
Matthew Kassawara
6f6a37fce8 Disable scatter-gather offload on host bridges
Disable scatter-gather offload on host bridges to eliminate
kernel traces that may impact container connectivity. Only
addressing AIO interfaces for now as host configuration for
actual deployments resides in documentation.

Change-Id: Ia66b2bb64b9ace66f5fa3ca8edcc9909af54a4f2
Partial-Bug: #1488815
Co-Authored-By: Evan Callicoat <apsu@propter.net>
2015-09-04 13:57:02 +00:00
kevin
87a4c06c7a
Update Ansible to v1.9.3-1
The change updates ansible to the latest stable release within the
1.9.x series.

Change-Id: I6c50fcb056d9130d71123b62ddda28685c7af639
2015-09-03 21:58:07 -05:00
Jesse Pretorius
614b37621b Disable python buffering for gate checks
In order to ensure that the ansible output in the jenkins console
log is output as it comes it is necessary to turn off the python
output buffering.

Change-Id: I0f8f11cd04fdfd6c26d5a5afc0e8647648edb1bb
2015-09-02 09:02:06 +00:00
Jenkins
6863e67e1d Merge "Remove temporary upgrade task that removes profile" 2015-08-27 11:12:36 +00:00
Ian Cordasco
999cdf52d7 Remove temporary upgrade task that removes profile
When performing an upgrade, this project strives to have minimal
downtime for VMs that are running. By removing the apparmor profile as a
precondition for upgrades, when the container create role runs, the
profile will default to contained (the most restrictive profile). This
causes instance downtime since neutron can not create network
namespaces.

Related-bug: 1487130
Closes-bug: 1489144
Change-Id: Ife7aab044c7cb882a89c6b108b2d66f5e39aa10c
2015-08-26 17:09:37 -05:00
Jesse Pretorius
80e7086ea3 Enable tempest testing of ceilometer
This patch enables tempest testing of ceilometer in the gate
check.

Change-Id: Ic1e432f004dc8060f8d8a83521ce9c4c60e9abf7
Closes-Bug: #1481822
2015-08-25 11:43:48 +01:00
Jenkins
24d0bd83d3 Merge "Enable/disable Swift/OpenStack deployment properly" 2015-08-24 16:08:41 +00:00
Jenkins
0b01f94b55 Merge "Remove read only disks from lvm candidates" 2015-08-24 16:08:37 +00:00
Jesse Pretorius
f0b46125d2 Remove upstream repo dependency
This patch removes the upstream repo dependency so that
all builds executing setup-infrastructure now execute a
local repository build instead of an rcync clone from
an upstream source.

Change-Id: Idf9beb9ea396d241758d4e970a4f88fef96cdf46
Implements: blueprint remove-upstream-repo-dependency
2015-08-24 08:32:07 +01:00
Jenkins
0923c17da8 Merge "Implement /usr/bin/env as the shebang in all bash scripts" 2015-08-22 01:59:26 +00:00
David Stanek
6351214ca5 Fixes loops for bashate
The 'do' is expected to be on the same line as the 'for' or 'while' in a
loop declaration.

Change-Id: I87591aac700af858e174c88cca03d06fe9688c59
2015-08-21 15:44:05 +00:00
David Stanek
65d620cbb7 Fixes function declarations for bashate
Change-Id: Ie631377de130cf656717fb3c8c74bdff2b8e2120
2015-08-21 11:03:12 +00:00
David Stanek
46a1638480 Fixes deprecated arithmetic expansion for bashate
$[...] is deprecated for $((...))

Change-Id: I432b6af3a04e4bb2196c53be84375a2653ae6a30
2015-08-21 10:02:02 +01:00
David Stanek
928ffa071d Removes tab indents for bashate
Change-Id: Ic874ca88ea14dfd53f38f202545786cd56f56c28
2015-08-21 06:37:02 +00:00
Jesse Pretorius
a515e835bb Set iptables-persistent install execution to append to log
This patch sets the command to install the iptables-persistent
package in the run-playbooks script to append its output to the
existing log instead of overwriting it.

This is to ensure that outputs from both commands are captured,
not just the second one.

Change-Id: I4d45382cb44c5f811818fe1229a63e6a2c2ecdee
Partial-Bug: #1485917
2015-08-20 16:48:02 +00:00
Jesse Pretorius
979c876378 Remove read only disks from lvm candidates
This patch removes read only disks from the candidates to be used
to store the lxc and cinder lvm volume groups.

Change-Id: I8fc0842f3629ac3c9fd01b01ecf37ffc68e3b871
Closes-Bug: #1487055
2015-08-20 15:37:05 +01:00
Jesse Pretorius
0d46f3f18c Enable/disable Swift/OpenStack deployment properly
This patch properly enables or disables Ceilometer, general
OpenStack and Swift deployment properly.

For the moment the containers will still be created, but
none of the related software will be installed, configured
or tested.

A fix to limit the containers created will be implemented
after a revision of the haproxy playbook/role to accommodate
this is implemented.

Closes-Bug: #1485945
Implements: blueprint split-aio-gates
Change-Id: Ia6657a02a6d1c53a4d76d7a17f74748ec9d2a2ee
2015-08-19 09:31:40 +01:00
Christian Berendt
d1f410d08a Implement /usr/bin/env as the shebang in all bash scripts
In order to be more consistent, and also account for different
environments and platforms, this patch implements the use of
#!/usr/bin/env as the shebang for all bash scripts in the
environment.

Closes-Bug: #1485557
Change-Id: I93e6a47fd6be816620682eddb2880e38f11bf675
2015-08-17 20:14:27 +00:00
Jesse Pretorius
91ac17211a Enable debug logging for gate checks
This patch enables debug logging for all OpenStack services
in the gate checks to ensure that the maximum amount of
information is available for debugging gate check failures.

Change-Id: Ide583fa6dc06008641bbc10b1abcaa816e662337
Closes-Bug: #1482572
2015-08-15 08:36:59 +00:00
Jenkins
2c4ea74f95 Merge "Add iptables rule to neutron agents containers" 2015-08-15 08:29:55 +00:00
Ian Cordasco
58d0daef9b Use playbook status to report upgrade failure
Previously, we simply checked $? which at that point would be the exit
status of echo, not openstack-ansible. By recording the actual
openstack-ansible exit status, we can properly report failures of the
upgrade script.

Closes-bug: 1480342
Change-Id: Icf43bea84660e4160a2dfcdb4ac93055340b3573
(cherry picked from commit 8a106d184a)
2015-08-13 18:18:20 +00:00
Matt Thompson
d879994e13 Add iptables rule to neutron agents containers
When running in an AIO environment, we need to drop an iptables rule to
ensure that communication between instances and the neutron metadata
service works.

Change-Id: Icc081fe83712ce883baa88f99db60c52dcc4c1ae
Closes-Bug: #1483603
2015-08-13 13:08:44 +01:00
Jesse Pretorius
7f91978a46 Move ansible logging to gate-check script only
Enacting the log link creation and the ansible.cfg change has
resulted in polluted patch reviews by developers making use of
AIO's for dev/test purposes.

This patch moves the Ansible logging changes to the
gate-check-script only as that's the only time that it's
actually required.

Change-Id: I4a1accad94ae153bf363b53fda0905e814c15173
Closes-Bug: #1479824
2015-08-09 09:58:46 +00:00
Miguel Grinberg
27831a026f Wrapper script to perform K2K federated login
This patch adds a small script that automates the process of accessing a
service provider (SP) cloud using credentials from a identity provider
cloud (IdP), where both clouds use Keystone based authentication. The
script performs the complete authentication flow and displays the token
and endpoints to use with the openstack command line client.

Implements: blueprint keystone-federation
Change-Id: I4b8113d0aef9c754fb55497d44138df660332bb8
2015-08-08 14:17:51 +00:00
Ian Cordasco
ef3e743113 Properly parse and quote arguments in upgrade script
This patch fixes the following:

1. Properly quote arguments to run_lock function
2. Properly parse out the playbook filename in run_lock

Specifically the upgrade steps where we were using

   "-e 'rabbitmq_upgrade=true' setup-infrastructure.yml"
   "/tmp/fix_container_interfaces.yml || true"

Were causing issues and this patch resolves them.

Closes-bug: 1479916
Change-Id: I809085d6da493f7f7d545547a0d984c0e7b1bf45
(cherry picked from commit 560fbbdb07)
2015-08-07 08:41:32 +00:00
Jenkins
6bf8a7b0f4 Merge "Change ansible forks used" 2015-08-03 17:59:49 +00:00
Serge van Ginderachter
b878370a0b Add Ceph/RBD support to playbooks
Currently the playbooks do not allow Ceph to be configured as a backend
for Cinder, Glance or Nova. This commit adds a new role called
ceph_client to do the required configuration of the hosts and updates
the service roles to include the required configuration file changes.
This commit requires that a Ceph cluster already exists and does not
make any changes to that cluster.

ceph_client role, run on the OpenStack service hosts
  - configures the Ceph apt repo
  - installs any required Ceph dependencies
  - copies the ceph.conf file and appropriate keyring file to /etc/ceph
  - creates the necessary libvirt secrets

os_glance role
glance-api.conf will set the following variables for Ceph:
  - [DEFAULT]/show_image_direct_url
  - [glance_store]/stores
  - [glance_store]/rbd_store_pool
  - [glance_store]/rbd_store_user
  - [glance_store]/rbd_store_ceph_conf
  - [glance_store]/rbd_store_chunk_size

os_nova role
nova.conf will set the following variables for Ceph:
  - [libvirt]/rbd_user
  - [libvirt]/rbd_secret_uuid
  - [libvirt]/images_type
  - [libvirt]/images_rbd_pool
  - [libvirt]/images_rbd_ceph_conf
  - [libvirt]/inject_password
  - [libvirt]/inject_key
  - [libvirt]/inject_partition
  - [libvirt]/live_migration_flag

os_cinder is not updated because ceph is defined as a backend and that
is generated from a dictionary of the config, for an example backend
config, see etc/openstack_deploy/openstack_user_config.yml.example

pw-token-gen.py is updated so that variables ending in uuid are assigned
a UUID.

DocImpact
Implements: blueprint ceph-block-devices
Closes-Bug: #1455238
Change-Id: Ie484ce0bbb93adc53c30be32f291aa5058b20028
2015-08-01 19:49:00 +01:00
Jesse Pretorius
7b10c64007 Change ansible forks used
This patch changes the number of forks used by ansible when
using any of the convenience (and thus gate check) scripts
to the number of processors available on the deployment
system.

The previous values used were found to cause ssh connection
errors and it was found that reducing the number improved
the chances of success.

This patch also removes the forks setting from ansible.conf
so that ansible will use the default value when run in any
other way. This leaves the decision of setting the number
of forks to the deployer, as it should be.

Change-Id: I31ad7353344f7994063127ecfce8f4733769234c
Closes-Bug: #1479812
2015-07-30 15:11:14 +01:00
Jenkins
cf93a8de54 Merge "Remove reference to missing user_group_vars file" 2015-07-29 22:52:28 +00:00
Nolan Brubaker
928af8a9ac Remove reference to missing user_group_vars file
The in-tree version of user_group_vars.yml was removed in
30f9443c5d, but the corresponding
reference in the upgrade script was not also updated.

This commit changes the behavior to remove the file from /etc/ if found.

Change-Id: I9f5b061289c5f43e32983845469f5123cc9f209d
Closes-Bug: #1479501
2015-07-29 17:15:24 -04:00
Darren Birkett
fccefaf3a5 update teardown.sh to remove mongodb
If mongodb has been installed as a backend for ceilometer (as it is by
the bootstrap-aio script), we want it removed in the teardown.  This
commit updates the teardown.sh to do just that.

Also s/remote_pip_pacakges/remove_pip_packages/

Change-Id: I909feac8321feaf66f44642cac1b729ded10d2fb
2015-07-29 13:28:14 -05:00
kevin
a86d89f1a7 Added gate process and more logs
This adds a forced apt update for packages and keys which is intended to help
with the HPB4 gate issues. This also creates logs for all ansible commands
run in the gate process.

Change-Id: I79b64567e615d83c52c1a8727082345ffe265c87
2015-07-23 00:28:01 +00:00
Jenkins
86a376e4c2 Merge "Fix rabbitmq playbook to allow upgrades" 2015-07-22 15:36:26 +00:00
git-harry
6ea86e6274
Fix rabbitmq playbook to allow upgrades
The rabbitmq playbook is designed to run in parallel across the cluster.
This causes an issue when upgrading rabbitmq to a new major or minor
version because RabbitMQ does not support doing an online migration of
datasets between major versions. while a minor release can be upgrade
while online it is recommended to bring down the cluster to do any
upgrade actions. The current configuration takes no account of this.

Reference:
https://www.rabbitmq.com/clustering.html#upgrading for further details.

* A new variable has been added called `rabbitmq_upgrade`. This is set to
  false by default to prevent a new version being installed unintentionally.
  To run the upgrade, which will shutdown the cluster, the variable can be
  set to true on the commandline:

  Example:
    openstack-ansible -e rabbitmq_upgrade=true \
    rabbitmq-install.yml

* A new variable has been added called `rabbitmq_ignore_version_state`
  which can be set "true" to ignore the package and version state tasks.
  This has been provided to allow a deployer to rerun the plays in an
  environment where the playbooks have been upgraded and the default
  version of rabbitmq has changed within the role and the deployer has
  elected to upgraded the installation at that time. This will ensure a
  deployer is able to recluster an environment as needed without
  effecting the package state.

  Example:
    openstack-ansible -e rabbitmq_ignore_version_state=true \
    rabbitmq-install.yml

* A new variable has been added `rabbitmq_primary_cluster_node` which
  allows a deployer to elect / set the primary cluster node in an
  environment. This variable is used to determine the restart order
  of RabbitMQ nodes. IE this will be the last node down and first one
  up in an environment. By default this variable is set to:
  rabbitmq_primary_cluster_node: "{{ groups['rabbitmq_all'][0] }}"

scripts/run-upgrade.sh has been modified to pass 'rabbitmq_upgrade=true'
on the command line so that RabbitMQ can be upgraded as part of the
upgrade between OpenStack versions.

DocImpact
Change-Id: I17d4429b9b94d47c1578dd58a2fb20698d1fe02e
Closes-bug: #1474992
2015-07-21 18:32:52 -05:00
Jenkins
a3cc484deb Merge "Fix general upgrade issues for Juno > Kilo" 2015-07-21 21:53:49 +00:00
Hugh Saunders
39409db9b9 Read affinity from environment
This patch allows a user to set the number of containers per host for
each type of container that has a default affinity greater than 1 for an
AIO build.

Currently this list is:
 * keystone
 * galera
 * rabbit_mq
 * horizon
 * repo

For example, one could set NUM_keystone_CONTAINERS=1 before running an
AIO to ensure that only one keystone container is built.

Change-Id: If90704a50414150ca37584e1f823ba0e85e287ac
Partially-Implements: blueprint split-aio-gates
2015-07-20 19:46:24 +00:00
kevin
2badb5341f Fix general upgrade issues for Juno > Kilo
This change adds a container task to ensure that container networks are up
and using the new configs as written by the lxc-container-create play. This
should resolve an issue where the container networks could be in a down
state after an upgrade due to a configuration file change.

A run function was also added to make it possible for a deployer to know
where in the upgrade process something might have failed and the order in
which the tasks may need to be rerun to continue the upgrade.

Change-Id: If02c4e269375368b6f613c5a9e3c947dddbd27f9
Closes-Bug: #1474585
Partial-Bug: #1475727
2015-07-18 17:50:44 +00:00
kevin
c81cea9698
Adds retries
The `successerator` has been re-enabled using a single retry which will also help
the gate process to ensure that we're not having to deal with so many transient
failures.

Change-Id: I7ca374ce18b7a78b0a1563d5244b1d9ac52c4d92
2015-07-17 10:00:06 -05:00
Jenkins
6aad682f00 Merge "Restart mysql when config changed" 2015-07-15 15:38:48 +00:00
Jenkins
8cb06d4412 Merge "Wait until mongo responds after restart" 2015-07-13 15:58:21 +00:00
git-harry
4a860f715c Restart mysql when config changed
Currently mysql is only (re)started when creating a cluster, adding a
node or the service isn't running. This means changes to configuration
files are not used.

This commit moves the restart functionality to a handler and sets
notifies on the the appropriate tasks so that when the config is changes
mysql is restarted.

This commit modifies the galera play to run in serial so that only one
instance of mysql can be restarted at a time. This is to prevent the
possibility of them all being restarted in one go. An additional play
'Check galera cluster status' has been added, this makes an attempt to
check that the cluster is healthy before proceeding with any possible
config changes. The checks can be overridden by setting
'galera_ignore_cluster_state=true' when running the playbook.

A facts module has been added to gather information about the status of
the cluster for use by the galera-install playbook.

DocImpact
Closes-bug: #1449958

Change-Id: Id83ba642c114d1f5cac04cc219be151a82a1023f
2015-07-12 18:04:10 +01:00
Jenkins
f4ca1c4fd6 Merge "Remove haproxy galera monitoring user" 2015-07-10 21:39:12 +00:00
Jenkins
ecc0e4ad98 Merge "Fix scripts/run-upgrade.sh" 2015-07-10 15:35:26 +00:00