Apt updates/installs are the most common cause of build timeouts
within HP Cloud gate checks.
This patch changes both the AIO and the gate check from using the
Rackspace Ubuntu mirror for everything to:
1) The AIO bootstrap will use whichever mirror is already defined
on the host operating system.
2) The gate check will use a known mirror which is defined per
cloud provider. If the cloud provider is not known, or the
script is being used outside of OpenStack-CI, then the gate
checkwill fall back to using the AIO's mechanism.
The idea is to make use of an apt mirror that is as close to the
instance as possible in order to speed up apt updates and installs.
Closes-Bug: #1491749
Change-Id: Ia37100bc198b64b74f27ea98ec5956e7cf084883
Disable scatter-gather offload on host bridges to eliminate
kernel traces that may impact container connectivity. Only
addressing AIO interfaces for now as host configuration for
actual deployments resides in documentation.
Change-Id: Ia66b2bb64b9ace66f5fa3ca8edcc9909af54a4f2
Partial-Bug: #1488815
Co-Authored-By: Evan Callicoat <apsu@propter.net>
This patch properly enables or disables Ceilometer, general
OpenStack and Swift deployment properly.
For the moment the containers will still be created, but
none of the related software will be installed, configured
or tested.
A fix to limit the containers created will be implemented
after a revision of the haproxy playbook/role to accommodate
this is implemented.
Closes-Bug: #1485945
Implements: blueprint split-aio-gates
Change-Id: Ia6657a02a6d1c53a4d76d7a17f74748ec9d2a2ee
When running in an AIO environment, we need to drop an iptables rule to
ensure that communication between instances and the neutron metadata
service works.
Change-Id: Icc081fe83712ce883baa88f99db60c52dcc4c1ae
Closes-Bug: #1483603
Enacting the log link creation and the ansible.cfg change has
resulted in polluted patch reviews by developers making use of
AIO's for dev/test purposes.
This patch moves the Ansible logging changes to the
gate-check-script only as that's the only time that it's
actually required.
Change-Id: I4a1accad94ae153bf363b53fda0905e814c15173
Closes-Bug: #1479824
This patch allows a user to set the number of containers per host for
each type of container that has a default affinity greater than 1 for an
AIO build.
Currently this list is:
* keystone
* galera
* rabbit_mq
* horizon
* repo
For example, one could set NUM_keystone_CONTAINERS=1 before running an
AIO to ensure that only one keystone container is built.
Change-Id: If90704a50414150ca37584e1f823ba0e85e287ac
Partially-Implements: blueprint split-aio-gates
Changed because some gate runs failed trying to connect to mongo after a
restart.
Also fixes a couple of incorrect conditionals.
Change-Id: I8d14be62617976ef97bb1247f30afc6bb8a3f103
This patch implements the implement-ceilometer blueprint.
It addes the necessary role/variables to deploy ceilometer
with a Mongodb backend. The Monogdb backend is assumed to
be up and configured and the playbooks only require a few
values to be set in user_variables to establish a connection.
Change-Id: I2164a1f27f632ce254cc2711ada2c449a9961fed
Implements: blueprint implement-ceilometer
This patch introduces galera_wsrep_provider_options as a list of
options instead of only catering for changing the gcache.size
option. This allows the deployer to change any number of the
wsrep_provider_options available.
Note: The variable galera_gcache_size is respected in this patch
but if galera_wsrep_provider_options is customised by a deployer
then the custom list must include reference to galera_gcache_size
for it to be respected.
DocImpact
Closes-Bug: #1464699
Co-Authored-By: Ian Cordasco <ian.cordasco@rackspace.com>
Change-Id: I78a5a1d344729fdc06636b1c6dade63a7ba59e94
Currently each service has an individual service_region var
which needs to be set when setting up a region that is
different to the default.
This patch implements a user_group_var called 'service_region'
which can be used to set the region for all services.
The AIO bootstrap script is also altered to set the
service_region in the user config.
Change-Id: I5ed216916f667506ffed900a01744dad973221bf
Closes-Bug: #1463366
The bootstrap-aio.sh script generates a password by reading a set number
of characters from urandom then removing those that aren't alphanumeric.
The problem with this is that its possible for 0 characters to remain.
This patch changes the order so that tr (translate) reads urandom
directly and removes non alphanumeric characters then that output is
truncated to the required length. This ensures the generated password is
always the desired length.
Change-Id: I03984ab891f88fe4b064f7f7fcb71edf48096493
Closes-Bug: #1465684
In the bootstrap-aio.sh and osad-aio-heat-template.yml we use a default
password of "secrete". As a minor security concern, this patch adjusts
this to be random.
Change-Id: I54b9a085aba7845b7a9ad435c60604359921fc09
Closes-Bug: #1462000
Moved all of the group_vars/all.yml file into its own variable file
This change was done to allow a user to override basic options
without having to modify the default group variable files. While
the group_vars/all.yml file is still present it is only holding the
revision information that is used for release data and the minimal
required kernel that allows the system to function using VXLAN.
The upgrade script was modified to support the new "default"
user_group_vars.yml file.
tempest_swift_enabled was set to true in group_vars, so this has
now been set as a default in the role instead.
Commit 1bd2bc052a36c7f8c28e473d0a37fd29827198b0 implemented the
package URL update for rabbitmq, but not the corresponding sha256
update. This was not noticed due to group_vars overriding the URL
to a previous version, resulting in the above-mentioned commit
taking no effect. This patch therefore also corrects the sha256.
Closes-Bug: #1460516
Closes-Bug: #1460992
Change-Id: I8e42bb124827bb276134d662c9a171db8e4c017e
This change updates all of the external URLs to use HTTPS where
possible. the change will create a more secure deployment by default.
Change-Id: I4d53c021904208bffb0d597c7ae53bbd00f40209
Closes-Bug: 1411331
fallocate is faster than using dd to create a thick swapfile with
zeros, but it is not supported on all filesystems (specifically ext3).
This commit adds a fallback to using dd if fallocate fails
Closes-Bug: #1458841
Change-Id: Ie4adf625d85f84a0d89a108ef0438622ab763b9d
The default values should use the local swift installation, since these
are only used when the glance backend is set to "swift" it won't matter
if there is no local swift when the backend is set to "file".
Adjust AIO script to use the defaults.
Move the configuration values from user_secrets.yml into
user_variables.yml since these are not passwords that should be set.
Additionally comment them out by default.
Change-Id: I579500a0287bc29f27fdbdb4f810212a2e194dea
Closes-Bug: #1450117
This commit allows tempest to be configured to test the cinder backup
service.
Additionally, we:
- add a new function to openstack_tempest_gate.sh so we can include
some cinder-backup-related API tests in our test list
- update bootstrap-aio.sh to enable cinder backups and the ability to
test cinder backups
- update run-tempest.sh to include the cinder backup tests in the
default test list
Change-Id: I2b8b0255ef5fdf747bbc136c7b1152896161f8a0
Closes-Bug: #1449427
Most of master is transitioning to the liberty cycle, to allow us
to continue gating the service install versions is being set back
to the head of the stable/kilo branch. This should be updated once
all upstream OpenStack projects fully transition to Liberty.
This adds a wait condition when creating the containers to ensure
that the containers are created correctly and that ssh is available
within them before actually continuing with the installation process
This change forces the apt mirrors to use a set that is known good.
The previous changes to set the mirrors to the same as what was
already on the host has proven itself to be a greater issue by not
using a set that is known to be in good working order.
Change-Id: I1872a32b53b9caecb8ac0803db304fa0ee254475
Closes-Bug: 1441148
The swift installation process requires access to the Ubuntu backports
repository to install the liberasurecode1 and liberasurecode-dev packages. In
the hpcloud environments used in OpenStack-CI the backport repository is not
implemented, so we need to ensure that it's there if it hasn't already been
configured. This patch does that as part of the AIO bootstrap process.
Additionally, heat now requires python-saharaclient >= 0.9.0 and we bump here
to allow heat to install successfully.
Note that both issues are causing the gate to fail and need to be addressed
together to get a successful gate.
Closes-Bug: #1448126
Closes-Bug: #1448152
Change-Id: I5d8c6f4eb7182129ab22defd70a50d78e7ffe86f
As both Ubuntu and the MariaDB repositories carry packages with the same
names, apt package pinning is required to ensure that the packages we install
come from the right repository. This patch adds a pinning preference file to
ensure that the packages we install come from the MariaDB repository.
Due to the OpenStack CI images having conflicting packages installed in the
base image, this patch also includes the removal of these packages during the
AIO bootstrap to ensure that the deployment succeeds.
For environments where the same two packages are installed on the hosts prior
to the deployment of os-ansible-deployment, they would need to ensure that the
packages are removed first. This has been seen in environments where
monitoring tooling has installed the packages as a dependancy.
Closes-Bug: #1446772
Change-Id: I5ad0b9010e49832c4f866c0b141e1c035d5f9268
Implements extra diagnostic information for gate check failure analysis and
also appends DNS settings instead of replacing them for the AIO bootstrap.
Some extra settings are added to the DNS to try to catch edge cases for DNS
resolution. The DNS `resolv.conf` was being overwritten on every run using
google name servers however we need to add google name servers to the existing
resolver not overwrite the whole conf so a conditional was added.
The gate scripts were attempting to create a symlink to logs on every run.
While this is fine for the gate which is creates a new node on every run
the link will cause failures on recheck when doing more local testing.
The ansible log entry was being appended to the `ansible.cfg` file on every
run which should also be limited to one entry. The conditional added is
helpful when doing local testing with the gate scripts.
Closes-Bug: #1442630
Change-Id: I25aac98d7408ba32f94befea22da4471bd1697b6
The AIO example files have been renamed such that they are no longer
creating an issue where the deployer "could" cause problems in
deployment when the AIO configuration files are being used by
default. The issue is ever present when the deployer does a blind
copy of the etc/openstack_deloy directory into /etc/.
This change is to enhance the Kilo gating / deployment process.
Partially implements: blueprint master-kilofication
Change-Id: I0c76ae9012aeafcc8a39a03c0e11b68b2ee5ca9c
The md5sum check doesn't currently solve any issues we have, and with
the addition of a custom "env.d" directory this will only cause
frustration as the md5sum changes.
The initial idea was to prompt a user to update their
user_config/user_variables/environment files between updates in the
os-a-d repository but the md5sum check never really solved this problem,
since both the files being checked won't have changed if you update the
repo and don't copy the new versions over.
Change-Id: Id1e7d307aa5dbffe069d6d4fa4569dd13ad4e8c1
Partial-Bug: #1399430
The AIO setup used for the commit checks in the gate is highly resource
constrained. Currently the swap space available is 1GB and the RAM available
is 8GB. The swap size would be better set to 4G if the RAM amount is 8GB or
less, or 8G if the RAM amount is more than 8GB.
The patch switches from using dd to using fallocate for the thick loopback
file creation in order to prevent memory exhaustion when trying to create
files that are larger than the available memory.
Change-Id: I9cdb145780fb867b933890111f642a48ed7add70
Closes-Bug: #1442028
This change modifies the cache sizes for galera when bootstrapping
an AIO. This change lowers the cache sizes to allow the for normal
operations on hosts that may be resource constrained.
Change-Id: Ie7b9c12a3231b92e6414e97f535ec128d13bf2d2
Closes-Bug: #1442028
This patch sets the AIO to use the same apt repository as the host in order to
reduce intermittent failures exhibited in environments other than RAX Cloud.
Change-Id: Id4069efa51f6198504d0eb1cfcd9c8beb9607075
Closes-Bug: #1441148
The gate-check-commit.sh script defined the SYMLINK_DIR variable, but it
was only used in bootstrap-aio.sh. This meant that using
bootstrap-aio.sh by itself would fail due to the undefined variable.
This change moves the declaration to boostrap-aio.sh so that an AIO can
be boostrapped without necessitating a gate check.
Change-Id: Id1b20379c5837cdd4b3df03de42f0ea612e3fbb5
Closes-Bug: #1434315
We're currently get a number of gate failures due to timeouts
downloading from http://www.rabbitmq.com/. This commit temporarily
overrides rabbitmq_package_url so we can clear backlog of reviews.
Change-Id: I7fa91501049e22184707787b7125017e5b63859b
Partial-Bug: #1436240
As stated in neutron's default conf file:
# This feature is experimental until issues are addressed and testing
# has been
# enabled for various plugins for compatibility.
This change has been shown to be reliable in manual testing of the gate
jobs. These jobs had been seeing transient failures possibly leading
back to this value. The changes here have been set within the
`bootstrap-aio.sh` script such that gating is using a consistent
environment even if these options change their values in the
future.
Thanks to Evan Callicoat for doing the work to isolate the failures.
Change-Id: I98116eef94a7240addfdf449d116ec1c24260c59
Co-Authored-By: Evan Callicoat <evan.callicoat@rackspace.com>
Closes-Bug: #1425255
Introduced namespaced variables for all OpenStack services supporting
this setting as defined through oslo libraries. Default value is False
in each case. Gating commit checks now enable the fatal_deprecations
setting for each supporting service.
Closes Bug: 1428412
Change-Id: I5f41d3fdfa1cc876efc0c33c657c9dad18a8ba51
This commit removes all of the rackspace related logging components.
This change is part 1 of 3 to update all of the logging bits within
the stack such that they're made more generic and community
consumable.
Plays removed:
* rsyslog-install.yml
Roles removed:
* rsyslog_setup
Variable changes:
* The default kibana and elasticsreach variables were removed.
Example config changes:
* The environment map was updated with the removed logging comonents.
Gate changes:
* rsyslog-install has been removed from the gating script as it no longer
serves the same purpose.
* The kibana variable override was removed.
* Kibana entries in `haproxy_config.yml` have been removed.
DocImpact
Implements: blueprint rsyslog-update
Change-Id: Icd25653a29c9936cecc63ba5dc82aeb1cfb7ebd8
Currently the verbose shell builtin is used, resulting in the entire
scripts contents being output in the logs. The instance information is
also output to stdout. This is unnecessarily verbose and prone to
failure. To ensure that the instance information is correctly rendered
failures from within the instance information commands will now always
return true and be logged within the `/openstack/log/ansible-logging`
directory. Each time that the function is called a new log will be
created tagged with the datecode in seconds. This commit also removes
the verbose builtin from the scripts, removes the unused
os-ansible-aio-check script.
Partial-Bug: #1425482
Change-Id: I59952d077a1a8dda12278e57a165d4bac272ed5f
This commit adds the ansible log file to the aio bootstrap script
and creates a vanilla logging directory that will allow us to collect
additional information on what is happening within a given ansible run.
Partial-Bug: #1425482
Partial-Bug: #1428744
Change-Id: I2786f73334b99bfa98d344c50337c86f75fae273
This patch creates the /openstack/log directory within the AIO bootstrap
script to prevent 'unstable' results in openstack-infra's jenkins.
It also moves the symlink to that directory to much earlier in the
gate-check script for the same purpose.
Co-Authored-By: Ian Cordasco <ian.cordasco@rackspace.com>
Closes-Bug: #1428744
Partial-Bug: #1425482
Change-Id: Ib74b5fd216d43e74ca264c14818cfef6e2dc0c70
In the Juno branch the required kernel is set in user_variables.yml
to match the running kernel if the vxlan module is loadable. This
functionality was never added into the master branch and this commit
adds that functionalty back into the AIO stack.
Change-Id: Ie3b72964a32ed3cfe6ff616d6c1fa935ddfad18b
Closes-Bug: #1424921
This patch adds some logic that determines what the primary interface is based
on the default route and uses that as the default rather than eth0.
Closes-Bug: 1424828
Change-Id: I4e1cebec60275c57768a0ef4281035c21f69dbbf
This change implements the blueprint to convert all roles and plays into
a more generic setup, following upstream ansible best practices.
Items Changed:
* All tasks have tags.
* All roles use namespaced variables.
* All redundant tasks within a given play and role have been removed.
* All of the repetitive plays have been removed in-favor of a more
simplistic approach. This change duplicates code within the roles but
ensures that the roles only ever run within their own scope.
* All roles have been built using an ansible galaxy syntax.
* The `*requirement.txt` files have been reformatted follow upstream
Openstack practices.
* Dynamically generated inventory is now more organized, this should assist
anyone who may want or need to dive into the JSON blob that is created.
In the inventory a properties field is used for items that customize containers
within the inventory.
* The environment map has been modified to support additional host groups to
enable the seperation of infrastructure pieces. While the old infra_hosts group
will still work this change allows for groups to be divided up into seperate
chunks; eg: deployment of a swift only stack.
* The LXC logic now exists within the plays.
* etc/openstack_deploy/user_variables.yml has all password/token
variables extracted into the separate file
etc/openstack_deploy/user_secrets.yml in order to allow seperate
security settings on that file.
Items Excised:
* All of the roles have had the LXC logic removed from within them which
should allow roles to be consumed outside of the `os-ansible-deployment`
reference architecture.
Note:
* the directory rpc_deployment still exists and is presently pointed at plays
containing a deprecation warning instructing the user to move to the standard
playbooks directory.
* While all of the rackspace specific components and variables have been removed
and or were refactored the repository still relies on an upstream mirror of
Openstack built python files and container images. This upstream mirror is hosted
at rackspace at "http://rpc-repo.rackspace.com" though this is
not locked to and or tied to rackspace specific installations. This repository
contains all of the needed code to create and/or clone your own mirror.
DocImpact
Co-Authored-By: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
Closes-Bug: #1403676
Implements: blueprint galaxy-roles
Change-Id: I03df3328b7655f0cc9e43ba83b02623d038d214e
This patch implements the following:
- scripts-library.sh which includes commonly used functions, variables
and other preparation commands for all other scripts
- bootstrap-ansible.sh which only deploys a selected version of ansible
and ensures that any other requirements are prepared on the
deployment host
- bootstrap-aio.sh which runs all host preparation actions for an
all-in-one build
- gate-check-lint.sh which runs a lint and syntax check
- gate-check-commit.sh which runs all actions required for a gate
commit check, utilising the other scripts where required
- run-smoke-test.sh which runs tempest from inside the utility container
- run-playbooks.sh which runs the playbooks
- the existing conf.d/swift.yml is renamed to be an example
configuration - the example configurations can be used as
documentation
- etc/network/interfaces.d/aio_interfaces.cfg,
etc/rpc_deploy/conf.d/swift.yml and
etc/rpc_deploy/rpc_user_config.yml are now configurations used for
the AIO deployment
- a workaround for https://bugs.launchpad.net/bugs/1244589 to ensure
that DHCP checksums are implemented by the host which is required for
the smoke tests to work
- the removal of the rpc heat templates as they're unusable in their
current state
- setting MAX_RETRIES to 0, ensuring that any failures cause an
immediate commit check failure in the gate - this prevents the
masking of failures by retry attempts
DocImpact
Co-Authored-By: Kevin Carter <kevin.carter@rackspace.com>
Closes-Bug: #1415883
Closes-Bug: #1417999
Closes-Bug: #1419807
Change-Id: I95242d48ad0fb055f16510803c8aa14dc183ac17