[goal] Deprecate the JSON formatted policy file

As per the community goal of migrating the policy file
the format from JSON to YAML[1], we need to do two things:

1. Change the default value of '[oslo_policy] policy_file''
config option from 'policy.json' to 'policy.yaml' with
upgrade checks.

2. Deprecate the JSON formatted policy file on the project side
via warning in doc and releasenotes.

Also replace policy.json to policy.yaml ref from doc and tests.

[1]https://governance.openstack.org/tc/goals/selected/wallaby/migrate-policy-format-from-json-to-yaml.html

Change-Id: Ic65d2fd6ce7215b4a47a6fb41b9cbf991f27773b

doc/source/admin/cli-manage-projects-users-and-roles.rst

@@ -10,8 +10,8 @@ define which actions users can perform. You assign roles to
 user-project pairs.
 
 You can define actions for OpenStack service roles in the
-``/etc/PROJECT/policy.json`` files. For example, define actions for
-Compute service roles in the ``/etc/nova/policy.json`` file.
+``/etc/PROJECT/policy.yaml`` files. For example, define actions for
+Compute service roles in the ``/etc/nova/policy.yaml`` file.
 
 You can manage projects, users, and roles independently from each other.
 

doc/source/admin/identity-concepts.rst

@@ -122,9 +122,9 @@ Identity user management examples:
      Individual services assign meaning to roles, typically through
      limiting or granting access to users with the role to the
      operations that the service supports. Role access is typically
-     configured in the service's ``policy.json`` file. For example,
+     configured in the service's ``policy.yaml`` file. For example,
      to limit Compute access to the ``compute-user`` role, edit the
-     Compute service's ``policy.json`` file to require this role for
+     Compute service's ``policy.yaml`` file to require this role for
      Compute operations.
 
 The Identity service assigns a project and a role to a user. You might
@@ -139,25 +139,25 @@ A user can have different roles in different projects. For example, Alice
 might also have the ``admin`` role in the ``Cyberdyne`` project. A user
 can also have multiple roles in the same project.
 
-The ``/etc/[SERVICE_CODENAME]/policy.json`` file controls the
+The ``/etc/[SERVICE_CODENAME]/policy.yaml`` file controls the
 tasks that users can perform for a given service. For example, the
-``/etc/nova/policy.json`` file specifies the access policy for the
-Compute service, the ``/etc/glance/policy.json`` file specifies
+``/etc/nova/policy.yaml`` file specifies the access policy for the
+Compute service, the ``/etc/glance/policy.yaml`` file specifies
 the access policy for the Image service, and the
-``/etc/keystone/policy.json`` file specifies the access policy for
+``/etc/keystone/policy.yaml`` file specifies the access policy for
 the Identity service.
 
-The default ``policy.json`` files in the Compute, Identity, and
+The default ``policy.yaml`` files in the Compute, Identity, and
 Image services recognize only the ``admin`` role. Any user with
 any role in a project can access all operations that do not require the
 ``admin`` role.
 
 To restrict users from performing operations in, for example, the
 Compute service, you must create a role in the Identity service and
-then modify the ``/etc/nova/policy.json`` file so that this role
+then modify the ``/etc/nova/policy.yaml`` file so that this role
 is required for Compute operations.
 
-For example, the following line in the ``/etc/cinder/policy.json``
+For example, the following line in the ``/etc/cinder/policy.yaml``
 file does not restrict which users can create volumes:
 
 .. code-block:: none

doc/source/admin/upgrading.rst

@@ -42,7 +42,7 @@ Plan your upgrade:
   to re-read the release notes for the previous release (or two!).
 
 * Prepare your new configuration files, including ``keystone.conf``,
-  ``logging.conf``, ``policy.json``, ``keystone-paste.ini``, and anything else
+  ``logging.conf``, ``policy.yaml``, ``keystone-paste.ini``, and anything else
   in ``/etc/keystone/``, by customizing the corresponding files from the next
   release.
 

doc/source/configuration/policy.rst

@@ -2,6 +2,15 @@
 Policy configuration
 ====================
 
+.. warning::
+
+   JSON formatted policy file is deprecated since Keystone 19.0.0 (Wallaby).
+   This `oslopolicy-convert-json-to-yaml`__ tool will migrate your existing
+   JSON-formatted policy file to YAML in a backward-compatible way.
+
+.. __: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html
+
+
 Configuration
 ~~~~~~~~~~~~~
 

doc/source/getting-started/policy_mapping.rst

@@ -2,7 +2,7 @@
 Mapping of policy target to API
 ===============================
 
-The following table shows the target in the policy.json file for each API.
+The following table shows the target in the policy.yaml file for each API.
 
 =========================================================  ===
 Target                                                     API

keystone/cmd/status.py

@@ -12,6 +12,7 @@
 
 from oslo_policy import _checks
 from oslo_policy import policy
+from oslo_upgradecheck import common_checks
 from oslo_upgradecheck import upgradecheck
 
 from keystone.common import driver_hints
@@ -86,6 +87,8 @@ class Checks(upgradecheck.UpgradeCommands):
          check_trust_policies_are_not_empty),
         ("Check default roles are immutable",
          check_default_roles_are_immutable),
+        ("Policy File JSON to YAML Migration",
+         (common_checks.check_policy_json, {'conf': CONF})),
     )
 
 

keystone/common/rbac_enforcer/enforcer.py

@@ -14,6 +14,7 @@ import functools
 
 import flask
 from oslo_log import log
+from oslo_policy import opts
 from oslo_policy import policy as common_policy
 from oslo_utils import strutils
 
@@ -39,6 +40,13 @@ _POSSIBLE_TARGET_ACTIONS = frozenset([
 _ENFORCEMENT_CHECK_ATTR = 'keystone:RBAC:enforcement_called'
 
 
+# TODO(gmann): Remove setting the default value of config policy_file
+# once oslo_policy change the default value to 'policy.yaml'.
+# https://github.com/openstack/oslo.policy/blob/a626ad12fe5a3abd49d70e3e5b95589d279ab578/oslo_policy/opts.py#L49
+DEFAULT_POLICY_FILE = 'policy.yaml'
+opts.set_defaults(CONF, DEFAULT_POLICY_FILE)
+
+
 class RBACEnforcer(object):
     """Enforce RBAC on API calls."""
 

keystone/conf/__init__.py

@@ -18,6 +18,7 @@ from oslo_log import log
 from oslo_log import versionutils
 import oslo_messaging
 from oslo_middleware import cors
+from oslo_policy import opts as policy_opts
 from osprofiler import opts as profiler
 
 from keystone.conf import application_credential
@@ -185,6 +186,12 @@ def set_external_opts_defaults():
     # configure OSprofiler options
     profiler.set_defaults(CONF, enabled=False, trace_sqlalchemy=False)
 
+    # TODO(gmann): Remove setting the default value of config policy_file
+    # once oslo_policy change the default value to 'policy.yaml'.
+    # https://github.com/openstack/oslo.policy/blob/a626ad12fe5a3abd49d70e3e5b95589d279ab578/oslo_policy/opts.py#L49
+    DEFAULT_POLICY_FILE = 'policy.yaml'
+    policy_opts.set_defaults(cfg.CONF, DEFAULT_POLICY_FILE)
+
     # Oslo.cache is always enabled by default for request-local caching
     # TODO(morganfainberg): Fix this to not use internal interface when
     # oslo.cache has proper interface to set defaults added. This is

keystone/tests/unit/test_policy.py

@@ -197,7 +197,7 @@ class PolicyJsonTestCase(unit.TestCase):
     def test_all_targets_documented(self):
         policy_keys = self._get_default_policy_rules()
 
-        # These keys are in the policy.json but aren't targets.
+        # These keys are in the policy.yaml but aren't targets.
         policy_rule_keys = [
             'admin_or_owner', 'admin_or_token_subject', 'admin_required',
             'owner', 'service_admin_or_token_subject', 'service_or_admin',

keystone/tests/unit/test_v3.py

@@ -247,7 +247,7 @@ class RestfulTestCase(unit.SQLDriverOverrides, rest.RestfulTestCase,
             domain_id=DEFAULT_DOMAIN_ID)
         self.default_domain_user_id = self.default_domain_user['id']
 
-        # create & grant policy.json's default role for admin_required
+        # create & grant policy.yaml's default role for admin_required
         self.role = unit.new_role_ref(name='admin')
         self.role_id = self.role['id']
         PROVIDERS.role_api.create_role(self.role_id, self.role)

lower-constraints.txt

@@ -25,16 +25,16 @@ oauthlib==0.6.2
 os-api-ref==1.4.0
 oslo.cache==1.26.0
 oslo.concurrency==3.26.0
-oslo.config==5.2.0
+oslo.config==6.8.0
 oslo.context==2.22.0
 oslo.db==6.0.0
 oslo.i18n==3.15.3
 oslo.log==3.44.0
 oslo.messaging==5.29.0
 oslo.middleware==3.31.0
-oslo.policy==3.0.2
+oslo.policy==3.6.0
 oslo.serialization==2.18.0
-oslo.upgradecheck==0.1.0
+oslo.upgradecheck==1.3.0
 oslo.utils==3.33.0
 oslotest==3.2.0
 osprofiler==1.4.0

releasenotes/notes/deprecate-json-formatted-policy-file-95f6307f88358f58.yaml

@@ -0,0 +1,20 @@
+---
+upgrade:
+  - |
+    The default value of ``[oslo_policy] policy_file`` config option has
+    been changed from ``policy.json`` to ``policy.yaml``.
+    Operators who are utilizing customized or previously generated
+    static policy JSON files (which are not needed by default), should
+    generate new policy files or convert them in YAML format. Use the
+    `oslopolicy-convert-json-to-yaml
+    <https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html>`_
+    tool to convert a JSON to YAML formatted policy file in
+    backward compatible way.
+deprecations:
+  - |
+    Use of JSON policy files was deprecated by the ``oslo.policy`` library
+    during the Victoria development cycle. As a result, this deprecation is
+    being noted in the Wallaby cycle with an anticipated future removal of support
+    by ``oslo.policy``. As such operators will need to convert to YAML policy
+    files. Please see the upgrade notes for details on migration of any
+    custom policy files.

requirements.txt

@@ -16,16 +16,16 @@ keystonemiddleware>=7.0.0 # Apache-2.0
 bcrypt>=3.1.3 # Apache-2.0
 scrypt>=0.8.0 # BSD
 oslo.cache>=1.26.0 # Apache-2.0
-oslo.config>=5.2.0 # Apache-2.0
+oslo.config>=6.8.0 # Apache-2.0
 oslo.context>=2.22.0 # Apache-2.0
 oslo.messaging>=5.29.0 # Apache-2.0
 oslo.db>=6.0.0 # Apache-2.0
 oslo.i18n>=3.15.3 # Apache-2.0
 oslo.log>=3.44.0 # Apache-2.0
 oslo.middleware>=3.31.0 # Apache-2.0
-oslo.policy>=3.0.2 # Apache-2.0
+oslo.policy>=3.6.0 # Apache-2.0
 oslo.serialization!=2.19.1,>=2.18.0 # Apache-2.0
-oslo.upgradecheck>=0.1.0 # Apache-2.0
+oslo.upgradecheck>=1.3.0 # Apache-2.0
 oslo.utils>=3.33.0 # Apache-2.0
 oauthlib>=0.6.2 # BSD
 pysaml2>=5.0.0