config loader: Prefer cli/env over clouds.yaml args for some args

If a user specifies a project or domain on the cli, that should take
precedence over the value set in clouds.yaml. This fixes enables workflows
that rely on domain- or cloud-wide credentials in clouds.yaml.

The cli args that are reprioritized in this patch are:

 domain-id
 domain-name
 user-domain-id
 user-domain-name
 project-domain-id
 project-domain-name
 auth-token
 project-id
 tenant-id
 project-name
 tenant-name

Story: 2010784
Change-Id: I45e7cff6579e6686d790bd3bb3e3ab9955885a64

doc/source/user/config/configuration.rst

@@ -450,3 +450,50 @@ region.
           - name: inap-17037-WAN1654
             routes_externally: true
           - name: inap-17037-LAN6745
+
+
+Setting Precedence
+------------------
+
+Some settings are redundant, e.g. ``project-name`` and ``project-id`` both
+specify the project. In a conflict between redundant settings, the
+``_name`` ``clouds.yaml`` option (or equivalent ``-name`` CLI option and ``_NAME`` environment variable) will be used.
+
+Some environment variables or commandline flags can override the settings from
+clouds.yaml. These are:
+
+- ``--domain-id`` (``OS_DOMAIN_ID``)
+- ``--domain-name`` (``OS_DOMAIN_NAME``)
+- ``--user-domain-id`` (``OS_USER_DOMAIN_ID``)
+- ``--user-domain-name`` (``OS_USER_DOMAIN_NAME``)
+- ``--project-domain-id`` (``OS_PROJECT_DOMAIN_ID``)
+- ``--project-domain-name`` (``OS_PROJECT_DOMAIN_NAME``)
+- ``--auth-token`` (``OS_AUTH_TOKEN``)
+- ``--project-id`` (``OS_PROJECT_ID``)
+- ``--project-name`` (``OS_PROJECT_NAME``)
+- ``--tenant-id`` (``OS_TENANT_ID``) (deprecated for ``--project-id``)
+- ``--tenant-name`` (``OS_TENANT_NAME``) (deprecated for ``--project-name``)
+
+Similarly, if one of the above settings is specified in ``clouds.yaml`` as
+part of the ``auth`` section as well as the main section, the ``auth`` settings
+will be overridden. For example in this config section, note that project is
+specified multiple times:
+
+.. code-block:: yaml
+
+  clouds:
+    mtvexx:
+      profile: https://vexxhost.com
+      auth:
+        username: mordred@inaugust.com
+        password: XXXXXXXXX
+        project_name: mylessfavoriteproject
+        project_id: 0bedab75-898c-4521-a038-0b4b71c41bed
+      region_name: ca-ymq-1
+      project_name: myfavoriteproject
+      project_id: 2acf9403-25e8-479e-a3c6-d67540c424a4
+
+In the above example, the ``project_id`` configuration values will be ignored
+in favor of the ``project_name`` configuration values, and the higher-level
+project will be chosen over the auth-specified project. So the actual project
+used will be ```myfavoriteproject```.

openstack/config/loader.py

@@ -777,12 +777,15 @@ class OpenStackConfig:
         for target_key, possible_values in mappings.items():
             target = None
             for key in possible_values:
-                if key in cloud:
-                    target = str(cloud[key])
-                    del cloud[key]
+                # Prefer values from the 'auth' section
+                # as they may contain cli or environment overrides.
+                # See story 2010784 for context.
                 if key in cloud['auth']:
                     target = str(cloud['auth'][key])
                     del cloud['auth'][key]
+                if key in cloud:
+                    target = str(cloud[key])
+                    del cloud[key]
             if target:
                 cloud['auth'][target_key] = target
         return cloud

openstack/tests/unit/config/test_config.py

@@ -1471,6 +1471,49 @@ class TestBackwardsCompatibility(base.TestCase):
         }
         self.assertEqual(expected, result)
 
+    def test_project_conflict_priority(self):
+        """The order of priority should be
+        1: env or cli settings
+        2: setting from 'auth' section of clouds.yaml
+
+        The ordering of #1 is important so that operators can use domain-wide
+        inherited credentials in clouds.yaml.
+        """
+
+        c = config.OpenStackConfig(
+            config_files=[self.cloud_yaml], vendor_files=[self.vendor_yaml]
+        )
+        cloud = {
+            'auth_type': 'password',
+            'auth': {
+                'project_id': 'my_project_id',
+            },
+        }
+        result = c._fix_backwards_project(cloud)
+        expected = {
+            'auth_type': 'password',
+            'auth': {
+                'project_id': 'my_project_id',
+            },
+        }
+        self.assertEqual(expected, result)
+
+        cloud = {
+            'auth_type': 'password',
+            'auth': {
+                'project_id': 'my_project_id',
+            },
+            'project_id': 'different_project_id',
+        }
+        result = c._fix_backwards_project(cloud)
+        expected = {
+            'auth_type': 'password',
+            'auth': {
+                'project_id': 'different_project_id',
+            },
+        }
+        self.assertEqual(expected, result)
+
     def test_backwards_network_fail(self):
         c = config.OpenStackConfig(
             config_files=[self.cloud_yaml], vendor_files=[self.vendor_yaml]

releasenotes/notes/story-2010784-21d23043155497f5.yaml

@@ -0,0 +1,33 @@
+---
+upgrade:
+  - |
+    Many cloud administrators use universal cloud-wide credentials. This is
+    supported in keystone via 'inherited' roles that can be applied cloud-
+    or domain-wide.
+
+    In previous releases, these credentials could not be usefully defined
+    within ```clouds.yaml``` because ```clouds.yaml``` supports only
+    specifying a single domain and project for auth purposes. This project
+    or domain could not be overridden on the commandline.
+fixes:
+  - |
+    When some config settings are specified multiple times, the order of
+    precendence has been changed to prefer command-line or env settings over
+    those found in ```clouds.yaml```. The same reordering has been done when
+    a setting is specified multiple times within ```clouds.yaml```; now a
+    higher-level setting will take precedence over that specified within
+    the auth section.
+
+    Affected settings are:
+
+    - ``domain_id``
+    - ``domain_name``
+    - ``user_domain_id``
+    - ``user_domain_name``
+    - ``project_domain_id``
+    - ``project_domain_name``
+    - ``auth-token``
+    - ``project_id``
+    - ``tenant_id``
+    - ``project_name``
+    - ``tenant_name``