When running 'overcloud deploy' command look for dynamic
defaults file for these options:
--roles-file, --network-file, --vip-file and
--baremetal-deployment
When the option is set by the user, use the user provided
file and make sure a copy is created in the working
directory. If the argument is not set look in the working
directory for the file used previously and use that file.
overclod node, and overcloud network commands require the
user input. But will place a copy in the working_dir for
overcloud deploy.
The depends-on creates these "defaults" by running the
different nova-less/network-v2 export commands when
upgrading the undercloud. With this change the next
'overcloud deploy' after the undercloud upgrade will use
the correct files (unless the user set the args ...)
Depends-On: https://review.opendev.org/795773
Change-Id: I53ba631dc80428c6f1fe71c2bbfb0b5a36dd8f01
MD5 hash, is no longer considered sufficient in security contexts,
as it is susceptible to collisions.[0][1][2]
Since Glance offers multiple hashing algorithms and all other uses
of the function are internal to the tripleoclient, the call can be replaced.
Function is now able to work with multiple hash algorithms,
provided their names are known to python hashlib and specified as compliant
in the tripleoclient constants.
Tests were adjusted to work with new hash algorithm,
and expanded to one compliant, and one non-compliant, alternative.
Docstrings now describe where is the information about image coming from.
In order to simplify potential future work on the related functionality.
[0] - https://csrc.nist.gov/projects/hash-functions
[1] - https://csrc.nist.gov/publications/detail/fips/180/4/final
[2] - https://www.win.tue.nl/~bdeweger/CollidingCertificates/
Signed-off-by: Jiri Podivin <jpodivin@redhat.com>
Change-Id: Iee5184755365d94f3b85073ed689079966c8bfcc
The new "openstack overcloud ceph deploy" is used to deploy
Ceph after the hardware has been provisioned with networking
and before the overcloud is deployed. The command takes the
output of "openstack overcloud node provision" as input and
returns a Heat enviornment file, e.g. deployed_ceph.yaml,
as output. The deployed_ceph.yaml file may then be passed
to the "openstack overcloud deploy" command as input.
Change-Id: Ie0032190f0c07fd47a36a1915c02f0ba1a9ae2a4
When using ephemeral Heat, the Neutron service is not available to Heat.
This change adds a check on the resource_registry to see if any
resources map to Neutron types. If any are found, an exception is
raised.
Change-Id: I0836f8c0e8265a1b3f273d445c4eb41850028efd
TripleO recently concluded the migration to the new ceph deployment
tool (cephadm). The ceph-ansible.yaml environment file, however, is
still relevant in an upgrade context, where ceph-ansible rolling
update playbook is executed.
For this reason, this patch introduces a new validation to make sure
this environment file and the related registry resources can be used
during the update/upgrade context but not when a regular overcloud
deploy or stack updates are executed.
Change-Id: Iec23801a9eec1830469a53ab5ffc641125b5d3ff
This command is used for debugging and reproducability, we should always
create it. Also, it's not presently possible to toggle the creation, so
for now just default to always creating it.
Change-Id: If45b1820f9840a50b26afce82a1780124c684a4e
Signed-off-by: James Slagle <jslagle@redhat.com>
When using the oc deploy command with the options
--baremetal-provision, --vip-file and network-v2
version networks definition the heat environment
files are added to user environments internally.
Including the legacy environment files, such as
`network-isolation.yaml` would result in overriding
the internally defined resource types and a failed
deployment.
This patch implements a check that will detect
resource type conflicts and raise error if a
protected resource type is overriden.
For users that may still want/need to override the
resource types, the protection can be disabled by
setting: `--disable-protected-resource-types`
NOTE: Parameter's are left unprotected since
traditionally THT interfaces always allowed
overriding anything and everything.
Also refactor the process_multiple_environments
method by splitting part of method to the new
rewrite_env_path method, so that it can be used
also when checking for prohibited overrides.
Depends-On: I8008344f215be6a54e00d7d27b697375b7f88f0f
Change-Id: I8e0f2762d744b21ec1555faa1e9bbe6e2d00f67b
Just like cli-overcloud-node-network-config.yaml,
cli-overcloud-node-growvols.yaml will be run by the "openstack
overcloud node provision" directly after provisioning so that whole
disk images will have their /var volume grown to 100%.
This default growvols behaviour can be overridden by adding an
ansible_playbooks entry which sets different arguments:
- count: 3
name: Controller
ansible_playbooks:
- playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-growvols.yaml
extra_vars:
growvols_args: /var=50% /srv=50%
Other changes in this commit include:
- Always call run_role_playbooks and pass network_config as an
argument
- Full unit test coverage on run_role_playbooks
- Fix run_role_playbook using os.path.basename instead of os.path.dirname
Change-Id: I085ab9da30e1e1a7d2b9a9f230dd0275bd40480d
Blueprint: whole-disk-default
Depends-On: I0a847ad4077a02a02ad817dd189a6a31c3637a93
Tests cover branches of the __init__ and __exit__
methods that were not covered by existing tests.
Signed-off-by: Jiri Podivin <jpodivin@redhat.com>
Change-Id: I1944a5a9dbd115f49e3fe0514d82af6f98f4556f
As per [1] RGW is now deployed by default when the cephadm envs
are included. However, during the upgrade procedure we should
warn users if both services are enabled, failing early during
the update or upgrade prepare step and suggesting that the
cephadm-rbd-only env should be added to make sure Swift is kept
and RGW is not automatically included in the heat stack.
This change introduces this kind of check, and an exception is
raised if the stack already exists, Ceph is deployed by TripleO
and swift is found in the previosly created stack.
[1] https://review.opendev.org/786827
Change-Id: I32387f19346697655355e15b3cf4ff41bd303b08
Instead of using tmp dirs for ansible-runner, use the consistent working
directory, which allows for saving the output for debugging and
reproducability.
Change-Id: I83ad02817ace364eb4bc596127cfd7d6699c32aa
Signed-off-by: James Slagle <jslagle@redhat.com>
In the depends-on change the Redis and OvnDBs VIP is
refactored to the respective service templates.
This adds a validation to raise a ConfigurationError
in case the resource registry contains the deprecated
resoirce definitions:
OS::TripleO::Network::Ports::RedisVipPort
OS::TripleO::Network::Ports::OVNDBsVipPort
Depends-On: https://review.opendev.org/777259
Change-Id: Id5415a94f71cbed5fad9856fd68e109979ffd491
Adds a new cli arg --working-dir, which defaults to:
$HOME/overcloud-deploy-<stack>
The working directory will be used for all files created by the
overcloud deploy command, instead of using tmp dirs and files directly
in $HOME. The working dir provides a single dir for all state associated
with the deployment, which is needed with the transition to using
ephemeral Heat, and especially when combined with
multi-stack/multi-overcloud.
This patch addresses:
- deployment status
- templates
- config-download
- heat-launcher
- overcloudrc
Further patches will address other uses of files outside of working-dir
and migrate them over.
Change-Id: I0d803f695c725c58ef2e6b655753b6c8248d1b2f
Signed-off-by: James Slagle <jslagle@redhat.com>
This changes to update the stack without using
the plan and also enables server side env merging
as we don't use the plan-environment.
Also makes changes to call derive params playbooks
without plan.
Depends-On: https://review.opendev.org/c/openstack/tripleo-ansible/+/772197
Change-Id: I8caad3e9185f1c6d23b0941b966192957ca8320b
As we've moved to a new way of generating nic configs
with only ansible, this would ensure that we check
if there is any custom heat nic config mapping and
only allow if user sets the 'NetworkConfigWithAnsible'
parameter as false.
Depends-On: https://review.opendev.org/758333
Change-Id: Ief2e6bb41687233d226ab5cb186fc6dbae191ce2
There have been cases where operators inadvertenly changed the
CephClusterFSID on a stack update, which is unsupported by both
Ceph and openstack.
For this reason we need to check that the existing deployed Ceph
cluster ID present in the stack is consistent with the value of
the environment, raising an InvalidConfiguration exception if
they are different.
Change-Id: I6aca5c701cb00c82c6b3f92db72b5547799a10bf
Closes-Bug: #1882548
When using the ironic "direct" deploy interface RAW images
are streamed to the target node. In comparison 'qcow2' images
is transferred to the baremtal node and the image is then
converted in to 'raw' in RAM. This put's a high RAM requirement
on the baremetal nodes.
This change updates the image upload/update code to convert
the 'qcow2' image to a 'raw' image prior to upload/update.
Related-Bug: #1893912
Change-Id: I4774e6afc3844ee7c1e8900f2509a2c402abf490
Don't use return code from ansible_runner and manage with a flag.
This also removes the fail_on_rc flag from run_ansible_playbook()
and makes it consistent to raise RuntimeError if rc !=0
everywhere including _standalone_deploy().
Change-Id: Ia5971af601d1d9500f33045768b38ac7937117f5
Closes-Bug: #1889394
This change will ensure that an ansible.cfg file stored in the config-download
directory is used and persists across runs.
Change-Id: I3b546921689d00b2cc1bde5a4d09363e65df79b5
Signed-off-by: Kevin Carter <kecarter@redhat.com>
This change will provide the operator the ability to better control
a given deployment or operational task while leveraging the
tripleoclient.
A utility has been added to sanitize user input. This will ensure
the parsed string is in valid ansible limit format.
Change-Id: I190f6efe8d728f124c18ce80be715ae7c5c0da01
Depends-On: I0056fdbe3d9807e6baf4a1645a632ab9eb1b2668
Signed-off-by: Luke Short <ekultails@gmail.com>
Co-Authored-By: Kevin Carter <kecarter@redhat.com>
This change will provide the ability to load extra vars from files, instead
of having to pass options through the CLI parser. By loading vars from file
we can ensure options are made more safe and better handled, especially in
cases when a given option may be massive, as is the case with
`parameter_defaults`.
> A new argument has been added to the ansible playbook runner which will
allow us to pass options into the method that will be stored in an
extravars file, which is then dynamically loaded by ansible-runner.
Information on extravars files can be seen here: [0].
> A test has been added to exercise the new extravars file capability.
[0]: https://ansible-runner.readthedocs.io/en/latest/intro.html#env-extravars
Closes-Bug: #1871338
Change-Id: I9675e587abf3f07e91319a40620a8f4c67fbf97b
Signed-off-by: Kevin Carter <kecarter@redhat.com>
Fix misused ansible connection timeout and deployment timeout passed in
config download and ansible runner utility.
Allow ansible runner utility to be given a job_timeout as well.
Also fix the misuse of timeout parameters in related worklows. Add
--overcloud-ssh-port-timeout and use it to configure ansible connection
timeout for the DeleteNode interace of the involved
workflows. Then use the timeout parameter as real timeout instead of
mistakingly passing it as a connection timeout.
Add new unit test for ansible timeout in config_download. Add missing
coverage for the existing timeout-related params in other unit tests.
Closes-Bug: #1868063
Co-authored-by: Kevin Carter <kevin@cloudnull.com>
Change-Id: I2a4d151bcb83074af5bcf7d1b8c68d81d3c0400d
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
This used to be created earlier by the mistral workflows running before
setting the status. As we've removing those, it has to be created,
if does not exist.
Change-Id: I9600f8b08391b36eae02a051713967b932fa06d3
This change adds a switch that will enable or disable raising an
exception when a playbook executes. This will allow some methods to
return the RC and status information when a playbook is run, even when
there's a failure. The default behaviour is to raise an exception on
failure, but when fail_on_rc is set to False the run_ansible_playbook
method will return the status and rc information regardless of any
failures.
To ensure we're not raising an exception from the ansible runner library
its been changed to RuntimeError.
Change-Id: I3af652615b5227144256074c05170d148f19bc1d
Closes-Bug: #1859182
Signed-off-by: Kevin Carter <kecarter@redhat.com>
This change removes the use of mistral from the update static methods. These
methods will now call the required functions directly using
`deployment.config_download`, which will save time and improve reliability.
* The run_update_ansible_action method has been updated to ensure we're supporting
multi-playbook execution properly.
Story: 2007212
Task: 38435
Task: 38438
Change-Id: Ic324847341142829e986128d502fdcab2cbddcd8
Signed-off-by: Kevin Carter <kecarter@redhat.com>
This change Converts plan status checks to using a direct call instead of
executing via mistral.
* A new method was added allowing us to update the deployment status object
when required.
* Tests have been added for the new static method update_deployment_status.
Story: 2007212
Task: 38430
Change-Id: Ie19be2078e2f349bf06e5b99ab93ca843e367463
Signed-off-by: Kevin Carter <kecarter@redhat.com>
This change adds a raised exception when the return code from a given
playbook run is not 0. This will ensure that any playbook failure is
captured, raising an exception from within the executing method. Prior
to this change exception handling was expected to be done in the calling
method, however, doing that would create lots of unnecessary code
duplication.
Change-Id: Ic742ec4653eb45a66c0d5c86d3d0ff31947be5c4
Signed-off-by: Kevin Carter <kecarter@redhat.com>
returning and raising exceptions are mutually exclusive. we need to
return rc properly from run_ansible_playbook() for it to be used
later in tripleo deploy.
Change-Id: Ia07433fb6886931530afebad49c8b6bf1f062af5
Closes-Bug: #1859182
This change replaces all of the ansible shell commands with the
python library, ansible-runner. This library is supported by
upstream ansible, is approved by the openstack foundation, is
supported in global requirements, and provides a better, more
programatic interface into running ansible playbooks.
All tests that interacted with the old shell commands have been
updated to now test using the library.
Change-Id: I8db50da826e2fbc074f4e7986d6fd00f6d488648
Signed-off-by: Kevin Carter <kecarter@redhat.com>
In a rare case like httpd reload by logrotate, heat-api returns
500 code. If this happens, trpleoclient can't get the status of
the stack even though the process is still on-going.
To handle this situation, tipleoclient should retry when it can't
get the stack information by 500 code.
Change-Id: I97a6825f4ff9f125eb597e5b7bd0c553c37e49e7
Closes-Bug: #1855633
This patch adds a new parameter called 'gathering_policy' (Defaults to
None) to the 'run_ansible_playbook' function. This parameter will
control the default policy of the Ansible fact gathering. Sets to None
by default, it will use the default policy for Ansible (ie. 'implicit').
Change-Id: I0668241a1675dd4e344cc24b6ff2cbb8f93b7a45
Signed-off-by: Gael Chamoulaud <gchamoul@redhat.com>
This patch adds a new parameter called 'plan' (Defaults to "overcloud")
to the 'run_ansible_playbook' function. It will allow to execute
Validations with Ansible in a different plan or stack that the default
one through the TripleO CLI. Note That it was already possible only
while using Mistral but not for Ansible.
This patch also brings:
- Change the default values of the 'tags' and 'skip_tags' arguments
to 'None' and fixes their non Pythonic tests.
- Add '--stack' alias to the '--plan' argument for 'validator run'
command.
Change-Id: I6f8f55963f3f5261ec1497b650e0ca509d31dd32
Signed-off-by: Gael Chamoulaud <gchamoul@redhat.com>
The 'openstack tripleo validator list' subcommand can now get only the
available parameters for the validations using the new --parameters
argument.
```
$ openstack tripleo validator list \
--parameters \
[--validation-name <validation_id>[,<validation_id>,...] |
--group <group>[,<group>,...]]
```
Here is an output example:
```
Waiting for messages on queue 'tripleo' with no timeout.
{
"undercloud-cpu": {
"parameters": {
"min_undercloud_cpu_count": 8
}
},
"undercloud-ram": {
"parameters": {
"min_undercloud_ram_gb": 24
}
}
}
```
The --create-vars-file allow the operator to generate either a JSON or a
YAML file containing only the parameters of one or multiple validations.
This file will be available to pass as extra vars to the validations
execution.
```
$ openstack tripleo validator list \
--parameters \
--create-vars-file [json|yaml] /home/stack/myvars \
[--validation-name <validation_id>[,<validation_id>,...] |
--group <group>[,<group>,...]]
```
Change-Id: I6e2255c0d490ee8105f0757d02f5d8fba1d4fa20
Signed-off-by: Gael Chamoulaud <gchamoul@redhat.com>
This patch adds a new parameter called 'extra_vars (Defaults to None) to
the 'run_ansible_playbook' function. This parameter will set additional
variables to the 'ansible-playbook' command. It will accept either a
dict or the absolute path of a file (YAML or JSON format).
Change-Id: Ib25ee9593528ad680b14ca09c62addbbd0b773a3
Signed-off-by: Gael Chamoulaud <gchamoul@redhat.com>
This patch switch over Ansible to run validations by names by
default. The --use-mistral argument will have to be used in order to
execute them through Mistral.
Co-Authored-By: Gaël Chamoulaud <gchamoul@redhat.com>
Change-Id: Ia393f4d776ab2c09439e7772b5596ddbb47e0a5e
This patch adds a new parameter called 'log_path_dir' (Defaults to None)
to the 'run_ansible_playbook' function. The Ansible log file will be
created in the location of the playbook by default, otherwise in the given
directory path.
Change-Id: I7222a116974458b9149771cb44f7d5f7bc51bc79
Signed-off-by: Gael Chamoulaud <gchamoul@redhat.com>
If the heat api is overloaded or temporarily unavailable, we might get a
503 or 504 from haproxy during the deployment. We should retry polling
for events in this case as to not prematurely exit the deployment.
Change-Id: I947cd0f9bf4a97e46c3d2bf3e9b986f7d38e9357
Closes-Bug: #1833452
Ironic can use HTTP links or local files, and we already put the images to
a location accessible inside of ironic containers (for introspection).
This change switches to using file images for IPA. The existing Glance
images are not deleted since some nodes may be using them. Multi-arch
layout of [[PLATFORM-]ARCH/]agent.EXT is reused from the unit tests
of the `image upload` command, assuming that's what people are using.
Change-Id: Ie6fa04112e3348f429dc42b28442f8996ab03f29
Implements: blueprint nova-less-deploy
Depends-On: https://review.opendev.org/#/c/663897/
Alex Schultz
Harald Jensås
Jiri Podivin
John Fulton
James Slagle
Francesco Pantano
Steve Baker
ramishra
Mathieu Bultel
Kevin Carter
Luke Short
Bogdan Dobrelya
Emilien Macchi
Keigo Noha
Dougal Matthews
Gael Chamoulaud
Carlos Camacho
Dmitry Tantsur