174 Commits

Author SHA1 Message Date
James Slagle
e0d26441f1 Add ServiceNetMap to global_vars
This adds the ServiceNetMap value to global_vars for config-download.
This will make the value consumable from ansible tasks when running
config-download.

Additional values can be added in a similar fashion in the future to
allow for less hardcoded data coming out of Heat, and instead using
jinja expressions to consume data from Ansible variables when
config-download runs.

Change-Id: I8c442caac140f1c96123c1be47e858949419fd8f
2019-04-24 14:10:30 -04:00
James Slagle
bfd3fea2ef Add Keystone admin/public to enabled services list
These services need to be added to the ServiceNames resources list of
enabled services per role if we want groupings for them to be created in
the ansible inventory.

This same logic already exists in network/ports/net_ip_list_map.j2.yaml,
where the hieradata for <service>_node_[names,ips] are created. We also
need these service groupings to exist in the ansible inventory if we
want to make this hieradata generic instead of hardcoded coming out of
the heat templates.

Change-Id: Ie4c221a850a9018b4eb62bfd3d54ac22395b7a9c
2019-04-24 14:10:30 -04:00
Zuul
2add17b409 Merge "Make krb-service-principal metadata per-Role" 2019-04-16 20:17:17 +00:00
Oliver Walsh
e096a93e6d Do not set the cell endpoints for regular split-controlplane child stacks
When we deploy a nova cell child stack we have to redirect some endpoints to
the local instance instead of the central instance.
For non-nova cell child stacks we must disable this.

Closes-bug: 1823992
Change-Id: If39def80959ad91d96d92882f60e0e4c23b9b85e
2019-04-11 14:03:24 +01:00
Harald Jensås
d5ecc1f651 Make krb-service-principal metadata per-Role
Not all roles are connected to all networks, there is no
need to create metadata for networks not associated with
the role.

In edge/spine-and-leaf deployments the total number of
composable networks used can be high. Passing all the
networks we quickly go beyond the nova metadata fields
size limit (each field cannot exceed 256 bytes).

Also update tools/check-up-to-date.sh script to use the
simple yaml-diff.py instead of diff. The env generator
code will sort data, while jinja rendered environments
are not sorted, thus need to diff the data in yaml not
the text.

Closes-Bug: #1821377
Change-Id: I5ae3bc845b0a6ad6986d44b14ff4b0737a9b033b
2019-04-05 14:22:20 +00:00
Martin Schuppert
ffa6810e49 Add novnc proxy to cellsv2 multicell controller
With cellsv2 multicell in each cell there needs to be a novnc proxy as the
console token is stored in the cell conductor database. This change adds
the NovaVncProxy service to the CellController role and configures the
endpoint to the local public address of the cell.

Closes-Bug: #1822607
Depends-On: https://review.openstack.org/649265

Change-Id: Ia3a36d369fdc18685f4c965a9e371ca3143967bf
2019-04-02 11:52:35 +02:00
Rabi Mishra
95362173c2 Don't look for primary_role ips in AllNodesValidationConfig
We changed the AllNodesValidationConfig to be role specific.
However, we still use primary_role_name ips.

Change-Id: I0aa1174992f6f049f1e64faea6d88e377d357bad
Closes-Bug: #1817087
2019-02-21 21:00:12 +05:30
Oliver Walsh
dc9a76aa23 cell_v2 multi-cell
- uses split-control-plane
- adds a new CellController role
  - nova-conductor, message rpc (not notifications) and db
- move nova dbsync from nova-api to nova-conductor
  - nova db is more tightly coupled to conductor/computes
  - we don't have a nova-api services on a CellController
  - super-conductor on Controller will sync cell0 db
- new 'magic' MysqlCellInternal endpoint
  - always refers the to local MysqlInternal endpoint
  - identical to MysqlInternal for regular deployment
  - but doesn't get overridden when inheriting EndpointMap from parent
    control-plane stack
- duplicate service node name hiera for transport_urls on cell stack
  - nova -> cell oslo messaging rpc nodes
  - neutron agent -> global messaging rpc nodes
- run cell host discovery only on default cell, for additional cells
the cell needs to be created first

bp tripleo-multicell-basic

Co-Authored-By: Martin Schuppert <mschuppert@redhat.com>

Change-Id: Ife9bf12d3a6011906fa8d9f97f7524b51aef906a
Depends-On: I79c1080605611c5c7748a28d2afcc9c7275a2e5d
2019-02-15 12:16:48 +01:00
Thomas Herve
c95f315ef0 Remove RoleConfig
Now that config-download is the default, RoleConfig and the associated
deployment isn't used anymore, let's remove it.

Change-Id: I0fbaccfea8f583101b03c6ee645ff01dac11b7af
2019-02-11 23:20:07 +00:00
James Slagle
2634ffaa5d Add GlobalConfigExtraMapData
Adds a new GlobalConfigExtraMapData parameter that can be used to inject
global_config_settings hieradata into the deployment. Any values generated
in the stack will override those passed in by the parameter value.

This will be used for the distributed compute node when deploying with separate
stacks and data from the control plane stack needs to be injected into the
compute stack.

Change-Id: Id3e52e272bae67ee4036c81b3d7640255e0349ae
2019-02-08 10:29:04 -05:00
Oliver Walsh
8a0ddc7f09 Export global_config for compute-only stack
Change-Id: Ib52c8bec82158055f4dfd9c778c80bcbb3e80f89
2019-02-05 08:39:02 +01:00
Harald Jensås
53027484ae Skip templating disabled networks
Ignore disabled networks when rendering templates.

Add's the ctlplane network to maps to ensure we don't
end up with no keys/values in map_replace functions.

Also some Jinja cleanup:
 - Reduce the number of times we iterate over networks
   where we can.
 - Add's indentation to make the code easier to read.

Related-Bug: #1809313
Depends-On: I2e8135bc9389d3bf1a6ef01e273515af5c488a9a
Change-Id: Ifeb2d2d1acb43c16a5bf29e95965776494d61fef
2019-01-21 19:35:37 +01:00
Zuul
aa624468b5 Merge "Move cellv2 discovery from control plane services to compute services" 2019-01-18 10:27:02 +00:00
Harald Jensås
8665a0d97b Make NetCidrMapValue contain list of cidrs in each net
Prior to routed networks we only had one subnet per network.
With routed networks each network can have multiple subnets.
The NetCidrMapValue should contain a list storing the cidr
of each subnet for each network.

Ceph:
  list_join is used to make a comma separated list of
  cidrs for public_network, monitor_address_block,
  cluster_network and radosgw_address_block.

Partial: blueprint tripleo-routed-networks-templates
Depends-On: Ia8e219b30d4f8b199b882e95fe2834252a92c15a
Depends-On: I1ace0a02e6aa2610559fee0d8576e6f1bc98d699
Change-Id: I68e064d23ec5d43f59146d974cae604d2c5fdb52
2019-01-06 18:20:27 +01:00
Harald Jensås
2f2d8183e6 L3 routed networks - subnet fixed_ips (3/3)
When using neutron routed networks we need to specify
either the subnet or a ip address in the fixed-ips-request
when creating neutron ports.

a) For the Vip's:

Adds VipSubnetMap and VipSubnetMapDefaults parameters in
service_net_map.yaml. The two maps are merged, so that the
operator can override the subnet where VIP port should be
hosted. For example:

parameter_defaults:
  VipSubnetMap:
    ctlplane: ctlplane-leaf1
    InternalApi: internal_api_leaf1
    Storage: storage_leaf1
    redis: internal_api_leaf1

b) For overcloud node ports:

Enrich 'networks' in roles defenition to include both
network and subnet data. Changes the list to a map
instead of a list of strings. New schema:

- name: <role_name>
  networks:
    <network_name>
      subnet: <subnet_name>

For backward compatibility a conditional is used to check
if the data is a map or not. In either case the internal
list of role networks is created as '_role_networks' in
the jinja2 templates.

When the data is a map, and the map contains the 'subnet'
key the subnet specified in roles_data.yaml is used as
the subnet in the fixed-ips-reqest when ports are created.
If subnet is not set (or role.networks is not a map) the
default will be {{network.name_lower}}_subnet.

Also, since the fixed_ips request passed to Vip ports are no
longer [] by default, the conditinal has been updated to
test for 'ip_address' entries in the request.

Partial: blueprint tripleo-routed-networks-templates
Depends-On: I773a38fd903fe287132151a4d178326a46890969
Change-Id: I77edc82723d00bfece6752b5dd2c79137db93443
2019-01-03 19:07:20 +01:00
Oliver Walsh
e0e885b8ca Move cellv2 discovery from control plane services to compute services
If compute nodes are deployed without deploying/updating the controllers then
the computes will not have cellv2 mappings as this is run in the controller
deploy steps (nova-api).
This can happen if the controller nodes are blacklisted during a compute scale
out. It's also likely to be an issue going forward if the deployment is staged
(e.g split control plane).

This change moves the cell_v2 discovery logic to the nova-compute/nova-ironic
deploy step.

Closes-bug: 1786961
Change-Id: I12a02f636f31985bc1b71bff5b744d346286a95f
2018-12-20 11:23:06 +05:30
Zuul
707cfc90c6 Merge "Remove deploy steps on empty roles" 2018-12-14 22:38:31 +00:00
Zuul
2ee47591ef Merge "Allow for service_bootstrap_ips empty list in IpListMap" 2018-12-14 14:03:55 +00:00
Rabi Mishra
571a764cba Allow for service_bootstrap_ips empty list in IpListMap
It seems in some cases we get an empty list for services in
service_bootstrap_ips and the yaql expression fails. Though
there can be better solution to not look for EnabledServices
for roles that has zero count, this would probably fix the
immediate scenario008 failures.

Change-Id: Ife1fc3f7736ed5743c80fa3748a75cb0bb52b817
Closes-Bug: #1808240
2018-12-14 03:48:56 +00:00
Thomas Herve
eb3efe7133 Remove deploy steps on empty roles
When a role count is 0, we can create the deployment resources
conditionally.

Closes-Bug: #1671859
Change-Id: I467b9ded1a1b33d520cb69aa86b253a0552643f7
2018-12-12 09:50:36 +01:00
Thomas Herve
862f52cce0 Put user data in the main stack
We create user data per instance, but two are global for all, and the
last one per role, so we can move it up the stack.

Change-Id: I1330e54744adef9be159edd8f01aefa3db85a480
2018-12-07 15:45:10 +01:00
Zuul
2485978bee Merge "Add SERVICE_bootstrap_node_ip values to allNodesConfig" 2018-12-06 11:49:36 +00:00
Steven Hardy
a77d045663 Add SERVICE_bootstrap_node_ip values to allNodesConfig
This can be used to replace the per-role bootstrap_nodeid_ip,
and the redis-base template is updated to use the new hiera
key.

The old bootstrap_nodeid_ip appears to only be used for redis,
so the old key is removed, with an upgrade release note added
should any out-of-tree services reference this value.

Partial-Bug: #1792613
Change-Id: I830d5b9bae3e9d65c2c393e3dcdf70bffdb1ac7b
2018-11-26 17:01:24 +00:00
Rabi Mishra
5d275fb922 Check for available networks for a role
For network isolation, we specifcy available networks for role.
Therefore, there is no point in creating noop network resources for
networks that are not available/connected. This results in redundant
host entries for not available networks on overcloud nodes.

If a network is not available for a role we don't need to create
those extra noop resources.

For Undercloud/Standalone role we keep all networks in roles data
as the default ServiceNetMap specifies non ctlplane networks though
they map to ctlplane.

Change-Id: I07822ec0cba7eed352c0010eb893b5e5a522e95c
Closes-Bug: #1800811
2018-11-19 10:14:34 +05:30
Juan Antonio Osorio Robles
cb3c72f37d Remove references to logging_source
This has been unused for a while, and even deprecation was scheduled
(although the patch never merged [1]). So, in order to stop folks
getting confused with this, it's being removed.

[1] https://review.openstack.org/#/c/543871/

Change-Id: Iada64874432146ef311682f26af5990469790ed2
2018-10-08 13:43:47 +03:00
Juan Antonio Osorio Robles
90234f4f2a Remove references to logging_group
This has been unused for a while, and even deprecation was scheduled
(although the patch never merged [1]). So, in order to stop folks
getting confused with this, it's being removed.

[1] https://review.openstack.org/#/c/543871/

Change-Id: Icc6b51044ccc826f5b629eb1abd3342813ed84c0
2018-08-29 13:43:30 +03:00
Zuul
85abf46ada Merge "Add BlacklistedIpAddresses stack output" 2018-08-17 11:30:36 +00:00
Zuul
9928adca3b Merge "Make Horizon's SECURITY_KEY 64 characters long" 2018-08-13 23:00:11 +00:00
James Slagle
aeded3e428 Add BlacklistedIpAddresses stack output
The output is a list of blacklisted server ip addresses on the ctlplane
network and will be used by the enable_ssh_admin workflow so that the
workflow does not operate on any blacklisted servers.

Change-Id: Ie96acf29a857e4801f5823f26a7de6bc989f39e2
Partial-Bug: #1785680
2018-08-06 14:17:14 -04:00
Zuul
dfc09b6ff1 Merge "ControlPlaneSubnetCidr using get_attr" 2018-07-24 01:34:20 +00:00
Harald Jensås
6ab86a3ebe ControlPlaneSubnetCidr using get_attr
Use get_attr on the server resource to resolve attribute
value from the subnet(s) and pass it to the parameter
'ControlPlaneSubnetCidr' used in the THT/network/config/*
templates.

As the value is now resolved from resource attributes,
this changes the default for 'ControlPlaneSubnetCidr' to ''
as well as the comment that these value should be overriden
in parameters_defaults. It also removes the parameter from
network-environment templates.

A conditinal is used in  puppet/role.role.j2.yaml so that
the parameter value is used whenever it is not '' (the
default) to provide backwards compatibility in case the user
set a different value (different from the one used in
undercloud.conf) for this parameter in
network-environment.yaml.

When deploying a routed control plane the network config
templates would previously need to be updated to carry
'ControlPlaneXSubnetCidr' parameter (in case the subnet
mask is not the same for all the routed network leafs).
With 8 Leafs in addition to the network local to the
undercloud that is 8 parameters less to place in the
configuration. By getting the value to pass from the
server resource this change reduces the required nic-config
template customisation (reduces the risk of user error).

Partial: blueprint tripleo-routed-networks-templates
Change-Id: I92ee0f9a2107cdf1ca5903d3756a235a79c36c73
2018-07-14 09:11:28 +02:00
Radomir Dopieralski
d5bfa09b4d Make Horizon's SECURITY_KEY 64 characters long
Our own security guide recommends it:
https://docs.openstack.org/security-guide/dashboard/secret-key.html

Change-Id: I7c85e9ff8b3bc92b80a3d0728f299ed1e4cb436c
2018-07-10 10:56:44 +02:00
James Slagle
7f42272024 Add AllNodesExtraMapData parameter
The AllNodesExtraMapData parameter is used to inject additional
hieradata into the all_nodes hierdata file on each node. The injected
data will be deeploy merged with the calculated all_nodes data for the
stack.

The parameter can be taken advantage of for split-controlplane use cases
where the hieradata from the control stack needs to be populated into
the separate compute stacks.

To easily get the hieradata out of the control stack, a new stack output
is added, AllNodesConfig.

Partially Implements: blueprint split-controlplane

Change-Id: I7b865bf82520006eef3ac2f36df34b1f3c34e642
2018-07-09 12:22:32 -04:00
Zuul
ddc6b91b5e Merge "Add default value for name_lower in network_data.yaml to update ServiceNetMap" 2018-06-30 09:22:04 +00:00
Bob Fournier
d3eb296e19 Add default value for name_lower in network_data.yaml to update ServiceNetMap
In Pike and later, the name_lower field in network_data.yaml can be
re-defined to contain a custom network name.  When this is done the
ServiceNetMap field must be overridden to reflect the new name in all
places.  This changes adds a new optional field to network_data.yaml
that should be set to the original default name_lower value.
ServiceNetMap will then be automatically updated and will not need
to be overridden.

This also fixes the VipPort naming for the StorageManagement network
to not use a static value.

Change-Id: I8a238038122288899cef49faf38ea2c2ffc2176b
2018-06-28 10:17:28 -04:00
Jill Rouleau
c16167f3d9 Enable Ansible error handling per role
Enable any_errors_fatal and max_fail_percentage Ansible options
to be set per TripleO role.  This change also provides a
structure by which future per-role Ansible options can readily
be added to group_vars.

Closes-Bug: 1760989
Change-Id: I47954717f42f14bae8d9fd2bd17cd8ea1fd787b3
2018-06-21 09:40:29 -07:00
Zuul
4fb30dd5ec Merge "Add BlacklistedHostnames stack output" 2018-06-14 19:04:30 +00:00
Alex Schultz
7c97320334 Use str_replace for known_hosts
There is a limit to how long input data can be for the heat script hook.
It turns out that data longer than 131072, will return an Argument list
to long error. To get around this, we need to pass this data in a
different way so that the heat script hook will work.

Change-Id: Ie3bd17ca9863e7687721e8c2628e485ea1849321
Closes-Bug: #1772071
2018-05-21 14:32:49 +00:00
Zuul
a7857d6dfc Merge "Revert "Switch public endpoints to use FQDNs by default"" 2018-05-17 21:56:29 +00:00
James Slagle
f254a2169d Revert "Switch public endpoints to use FQDNs by default"
This reverts commit 8e104b3c549118727b53c9825a438e799715b7f9.

https://review.openstack.org/#/c/559926/ introduced requiring CloudName.
This broke the documented deployment process. I also don't see how
CloudName can be required, but CloudDomain can not.

I don't see a technical reason why we can't keep the default as
localdomain. If necessary, we can instead add a parameter
validation instead of requiring the parameter.

Closes-Bug: #1771627
Depends-On: Ia86842b0b1f42512f25390d6bdb695e0f8133c6d
Change-Id: I2c5b511df50f29c63aa613899c2bebb506360bf4
2018-05-16 21:45:42 +00:00
Zuul
2e1e3fcd7f Merge "Make the KeystoneURL stack output versionless" 2018-05-14 23:37:13 +00:00
Zuul
f5152da976 Merge "Switch public endpoints to use FQDNs by default" 2018-05-14 23:27:04 +00:00
Carlos Camacho
44ef2a3ec1 Change template names to rocky
The new master branch should point now to rocky.

So, HOT templates should specify that they might contain features
for rocky release [1]

Also, this submission updates the yaml validation to use only latest
heat_version alias. There are cases in which we will need to set
the version for specific templates i.e. mixed versions, so there
is added a variable to assign specific templates to specific heat_version
aliases, avoiding the introductions of error by bulk replacing the
the old version in new releases.

[1]: https://docs.openstack.org/heat/latest/template_guide/hot_spec.html#rocky
Change-Id: Ib17526d9cc453516d99d4659ee5fa51a5aa7fb4b
2018-05-09 08:28:42 +02:00
Juan Antonio Osorio Robles
8e104b3c54 Switch public endpoints to use FQDNs by default
This is in preparation for TLS by default, since the TLS certificate will
use FQDNs for the SubjectAltName, and that will be verified.
This required for us to change both CloudDomain and CloudName to be
required parameters, and not default them to use localdomain. This is to
avoid folks in real deployments using them in their clouds.

Change-Id: Ic70dd323b33596eaa3fc18bdc69a7c011ccd7fa1
2018-05-08 18:16:27 +03:00
Juan Antonio Osorio Robles
9926359131 Make the KeystoneURL stack output versionless
Given that we have now moved all of the places where the keystone auth
URL is used to be versionless. We now make the KeystoneURL output to be
versionless as well.

Story: #2001897
Change-Id: I8c9fbfc77fe47e3ed2e58eac27119f86a045483c
2018-05-08 08:19:30 +00:00
James Slagle
262c0b4b86 Add BlacklistedHostnames stack output
The BacklistedHostnames stack output will be used as input into the
config-download-deploy workflow so that the hostnames can be excluded
from the Ansible deployment with config-download.

Change-Id: I4705be446756869ba3d04fc59daffa4d4748e12c
2018-05-01 16:42:32 -04:00
Zuul
1e2cdd60aa Merge "Support SshKnownHostsDeployment with config-download" 2018-03-29 21:45:09 +00:00
Zuul
3eb0c62e47 Merge "Remove unused minor update code" 2018-03-19 12:34:21 +00:00
James Slagle
088d5c12f0 Support SshKnownHostsDeployment with config-download
Add support for the SshKnownHostsDeployment resources to
config-download. Since the deployment resources relied on Heat outputs,
they were not supported with the default handling from tripleo-common
that relies on the group_vars mechanism.

Instead, this patch refactors the templates to add the known hosts
entries as global_vars to deploy_steps_playbook.yaml, and then includes
the new tripleo-ssh-known-hosts role from tripleo-common to apply the
same configuration that the Heat deployment did.

Since these deployments no longer need to be triggered when including
config-download-environment.yaml, a mapping is added that can be
overridden to OS::Heat::None to disable the deployment resources when
using config-download.

The default behavior when not using config-download remains unchanged.

Closes-Bug: #1746336
Change-Id: Ia334fe6adc9a8ab228f75cb1d0c441c1344e2bd9
2018-03-19 07:50:06 -04:00
Jiri Stransky
a782462a1a Remove unused minor update code
Since Pike, minor updates are done via the composable services
framework. The old shell script approach hasn't been used/tested for 2
releases now, and should be dropped.

Also drop the UpdateWorkflow interface. Before we started doing
upgrades via Ansible, we used this pluggable resource interface to
perform oneshot operations like migrations to WSGI or AODH
services. Nowadays this interface is not referenced from anywhere and
we'd probably rather do similar operations via Ansible tasks.

Change-Id: I6c5eafe76eb53bc38d100a9ba132dd8fe6dd2d5f
2018-03-15 18:27:14 +01:00