[goal] Deprecate the JSON formatted policy file

As per the community goal of migrating the policy file
the format from JSON to YAML[1], we need to do two things:

1. Change the default value of '[oslo_policy] policy_file''
config option from 'policy.json' to 'policy.yaml' with
upgrade checks.

2. Deprecate the JSON formatted policy file on the project side
via warning in doc and releasenotes.

Also replace policy.json to policy.yaml ref from doc and tests.

[1]https://governance.openstack.org/tc/goals/selected/wallaby/migrate-policy-format-from-json-to-yaml.html

Change-Id: I0dbb8484e749e645627756e88ec79c1b26a6414a

doc/source/admin/archives/adv-features.rst

@@ -418,7 +418,7 @@ VMware NSX QoS extension
 The VMware NSX QoS extension rate-limits network ports to guarantee a
 specific amount of bandwidth for each port. This extension, by default,
 is only accessible by a project with an admin role but is configurable
-through the ``policy.json`` file. To use this extension, create a queue
+through the ``policy.yaml`` file. To use this extension, create a queue
 and specify the min/max bandwidth rates (kbps) and optionally set the
 QoS Marking and DSCP value (if your network fabric uses these values to
 make forwarding decisions). Once created, you can associate a queue with

doc/source/admin/archives/auth.rst

@@ -31,7 +31,7 @@ Networking handles two kind of authorization policies:
    The actual authorization policies enforced in Networking might vary
    from deployment to deployment.
 
-The policy engine reads entries from the ``policy.json`` file. The
+The policy engine reads entries from the ``policy.yaml`` file. The
 actual location of this file might vary from distribution to
 distribution. Entries can be updated while the system is running, and no
 service restart is required. Every time the policy file is updated, the
@@ -84,7 +84,7 @@ terminal rules:
    in the resource is equal to the project identifier of the user
    submitting the request.
 
-This extract is from the default ``policy.json`` file:
+This extract is from the default ``policy.yaml`` file:
 
 -  A rule that evaluates successfully if the current user is an
    administrator or the owner of the resource specified in the request
@@ -92,49 +92,42 @@ This extract is from the default ``policy.json`` file:
 
    .. code-block:: none
 
-      {
+      "admin_or_owner": "role:admin or tenant_id:%(tenant_id)s"
-       "admin_or_owner": "role:admin",
+      "admin_or_network_owner": "role:admin or tenant_id:%(network_tenant_id)s"
-       "tenant_id:%(tenant_id)s",
+      "admin_only": "role:admin"
-       "admin_or_network_owner": "role:admin",
+      "regular_user": ""
-       "tenant_id:%(network_tenant_id)s",
+      "shared": "field:networks:shared=True"
-       "admin_only": "role:admin",
-       "regular_user": "",
-       "shared":"field:networks:shared=True",
-       "default":
 
 -  The default policy that is always evaluated if an API operation does
-   not match any of the policies in ``policy.json``.
+   not match any of the policies in ``policy.yaml``.
 
    .. code-block:: none
 
-        "rule:admin_or_owner",
+      "default": "rule:admin_or_owner"
-        "create_subnet": "rule:admin_or_network_owner",
+      "create_subnet": "rule:admin_or_network_owner"
-        "get_subnet": "rule:admin_or_owner",
+      "get_subnet": "rule:admin_or_owner or rule:shared"
-        "rule:shared",
+      "update_subnet": "rule:admin_or_network_owner"
-        "update_subnet": "rule:admin_or_network_owner",
+      "delete_subnet": "rule:admin_or_network_owner"
-        "delete_subnet": "rule:admin_or_network_owner",
+      "create_network": ""
-        "create_network": "",
-        "get_network": "rule:admin_or_owner",
 
 -  This policy evaluates successfully if either *admin_or_owner*, or
    *shared* evaluates successfully.
 
    .. code-block:: none
 
-         "rule:shared",
+       "get_network": "rule:admin_or_owner or rule:shared"
-         "create_network:shared": "rule:admin_only"
+       "create_network:shared": "rule:admin_only"
 
 -  This policy restricts the ability to manipulate the *shared*
    attribute for a network to administrators only.
 
    .. code-block:: none
 
-         ,
+       "update_network": "rule:admin_or_owner"
-         "update_network": "rule:admin_or_owner",
+       "delete_network": "rule:admin_or_owner"
-        "delete_network": "rule:admin_or_owner",
+       "create_port": ""
-        "create_port": "",
+       "create_port:mac_address": "rule:admin_or_network_owner"
-        "create_port:mac_address": "rule:admin_or_network_owner",
+       "create_port:fixed_ips": "rule:admin_or_network_owner"
-        "create_port:fixed_ips":
 
 -  This policy restricts the ability to manipulate the *mac_address*
    attribute for a port only to administrators and the owner of the
@@ -142,11 +135,9 @@ This extract is from the default ``policy.json`` file:
 
    .. code-block:: none
 
-         "rule:admin_or_network_owner",
+       "get_port": "rule:admin_or_owner"
-        "get_port": "rule:admin_or_owner",
+       "update_port": "rule:admin_or_owner"
-        "update_port": "rule:admin_or_owner",
+       "delete_port": "rule:admin_or_owner"
-         "delete_port": "rule:admin_or_owner"
-       }
 
 In some cases, some operations are restricted to administrators only.
 This example shows you how to modify a policy file to permit project to
@@ -155,21 +146,20 @@ perform all other operations:
 
 .. code-block:: none
 
-    {
+            "admin_or_owner": "role:admin or tenant_id:%(tenant_id)s"
-            "admin_or_owner": "role:admin", "tenant_id:%(tenant_id)s",
+            "admin_only": "role:admin"
-            "admin_only": "role:admin", "regular_user": "",
+            "regular_user": ""
-            "default": "rule:admin_only",
+            "default": "rule:admin_only"
-            "create_subnet": "rule:admin_only",
+            "create_subnet": "rule:admin_only"
-            "get_subnet": "rule:admin_or_owner",
+            "get_subnet": "rule:admin_or_owner"
-            "update_subnet": "rule:admin_only",
+            "update_subnet": "rule:admin_only"
-            "delete_subnet": "rule:admin_only",
+            "delete_subnet": "rule:admin_only"
-            "create_network": "",
+            "create_network": ""
-            "get_network": "rule:admin_or_owner",
+            "get_network": "rule:admin_or_owner"
-            "create_network:shared": "rule:admin_only",
+            "create_network:shared": "rule:admin_only"
-            "update_network": "rule:admin_or_owner",
+            "update_network": "rule:admin_or_owner"
-            "delete_network": "rule:admin_or_owner",
+            "delete_network": "rule:admin_or_owner"
-            "create_port": "rule:admin_only",
+            "create_port": "rule:admin_only"
-            "get_port": "rule:admin_or_owner",
+            "get_port": "rule:admin_or_owner"
-            "update_port": "rule:admin_only",
+            "update_port": "rule:admin_only"
             "delete_port": "rule:admin_only"
-    }

doc/source/admin/archives/config-agents.rst

@@ -372,11 +372,11 @@ correctly using these
       > cd C:\OpenStack\neutron\
       > python setup.py install
 
-#. Copy the ``policy.json`` file:
+#. Copy the ``policy.yaml`` file:
 
    .. code-block:: console
 
-      > xcopy C:\OpenStack\neutron\etc\policy.json C:\etc\
+      > xcopy C:\OpenStack\neutron\etc\policy.yaml C:\etc\
 
 #. Create the ``C:\etc\neutron-hyperv-agent.conf`` file and add the proper
    configuration options and the `Hyper-V related
@@ -386,7 +386,7 @@ correctly using these
 
       [DEFAULT]
       control_exchange = neutron
-      policy_file = C:\etc\policy.json
+      policy_file = C:\etc\policy.yaml
       rpc_backend = neutron.openstack.common.rpc.impl_kombu
       rabbit_host = IP_ADDRESS
       rabbit_port = 5672

doc/source/admin/config-logging.rst

@@ -80,16 +80,16 @@ To enable the logging service, follow the below steps.
        - If we don't specify ``local_output_log_base``, logged packets will be
          stored in system journal like ``/var/log/syslog`` by default.
 
-Trusted projects policy.json configuration
+Trusted projects policy.yaml configuration
 ----------------------------------------------
 
-With the default ``/etc/neutron/policy.json``, administrators must set up
+With the default ``/etc/neutron/policy.yaml``, administrators must set up
 resource logging on behalf of the cloud projects.
 
 If projects are trusted to administer their own loggable resources  in their
-cloud, neutron's policy file ``policy.json`` can be modified to allow this.
+cloud, neutron's policy file ``policy.yaml`` can be modified to allow this.
 
-Modify ``/etc/neutron/policy.json`` entries as follows:
+Modify ``/etc/neutron/policy.yaml`` entries as follows:
 
 .. code-block:: none
 

doc/source/admin/config-qos.rst

@@ -248,13 +248,13 @@ not automatically copied to the outer header.
    If the ``dscp_inherit`` option is set to true, the previous ``dscp`` option
    is overwritten.
 
-Trusted projects policy.json configuration
+Trusted projects policy.yaml configuration
 ------------------------------------------
 
 If projects are trusted to administrate their own QoS policies in
-your cloud, neutron's file ``policy.json`` can be modified to allow this.
+your cloud, neutron's file ``policy.yaml`` can be modified to allow this.
 
-Modify ``/etc/neutron/policy.json`` policy entries as follows:
+Modify ``/etc/neutron/policy.yaml`` policy entries as follows:
 
 .. code-block:: none
 
@@ -294,12 +294,12 @@ To enable minimum bandwidth rule:
 User workflow
 ~~~~~~~~~~~~~
 
-QoS policies are only created by admins with the default ``policy.json``.
+QoS policies are only created by admins with the default ``policy.yaml``.
 Therefore, you should have the cloud operator set them up on
 behalf of the cloud projects.
 
 If projects are trusted to create their own policies, check the trusted
-projects ``policy.json`` configuration section.
+projects ``policy.yaml`` configuration section.
 
 First, create a QoS policy and its bandwidth limit rule:
 

doc/source/admin/config-rbac.rst

@@ -701,11 +701,11 @@ as any other RBAC ``access_as_external`` policy.
 Preventing regular users from sharing objects with each other
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-The default ``policy.json`` file will not allow regular
+The default ``policy.yaml`` file will not allow regular
 users to share objects with every other project using a wildcard;
 however, it will allow them to share objects with specific project
 IDs.
 
 If an operator wants to prevent normal users from doing this, the
-``"create_rbac_policy":`` entry in ``policy.json`` can be adjusted
+``"create_rbac_policy":`` entry in ``policy.yaml`` can be adjusted
 from ``""`` to ``"rule:admin_only"``.

doc/source/admin/intro-os-networking.rst

@@ -71,7 +71,7 @@ at the cost of flexibility. By default only administrators can create or
 update provider networks because they require configuration of physical
 network infrastructure. It is possible to change the user who is allowed to
 create or update provider networks with the following parameters of
-``policy.json``:
+``policy.yaml``:
 
 * ``create_network:provider:physical_network``
 * ``update_network:provider:physical_network``

doc/source/configuration/policy.rst

@@ -2,6 +2,14 @@
 Policy Reference
 ================
 
+.. warning::
+
+   JSON formatted policy file is deprecated since Neutron 18.0.0 (Wallaby).
+   This `oslopolicy-convert-json-to-yaml`__ tool will migrate your existing
+   JSON-formatted policy file to YAML in a backward-compatible way.
+
+.. __: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html
+
 Neutron, like most OpenStack projects, uses a policy language to restrict
 permissions on REST API actions.
 

doc/source/contributor/contribute.rst

@@ -615,7 +615,7 @@ Other repo-split items
 
 (These are still TBD.)
 
-* Splitting policy.json? **ToDo** Armando will investigate.
+* Splitting policy.yaml? **ToDo** Armando will investigate.
 
 * Generic instructions (or a template) for installing an out-of-tree plugin or
   driver for Neutron. Possibly something for the networking guide, and/or a

doc/source/contributor/internals/policy.rst

@@ -86,7 +86,7 @@ The ``_build_match_rule`` routine returns a ``oslo_policy.RuleCheck`` instance
 built in the following way:
 
 * Always add a check for the action being performed. This will match
-  a policy like create_network in ``policy.json``;
+  a policy like create_network in ``policy.yaml``;
 * Return for ``GET`` operations; more detailed checks will be performed anyway
   when building the response;
 * For each attribute which has been explicitly specified in the request
@@ -226,7 +226,7 @@ keyword, and provides a way to perform fine grained checks on resource
 attributes. For instance, using this class of rules it is possible to specify
 a rule for granting every project read access to shared resources.
 
-In policy.json, a FieldCheck rules is specified in the following way::
+In policy.yaml, a FieldCheck rules is specified in the following way::
 
 > field: <resource>:<field>=<value>
 
@@ -291,7 +291,7 @@ Notes
 * There is no way at the moment to specify an ``OR`` relationship between two
   attributes of a given resource (eg.: ``port.name == 'meh' or
   port.status == 'DOWN'``), unless the rule with the or condition is explicitly
-  added to the policy.json file.
+  added to the policy.yaml file.
 * ``OwnerCheck`` performs a plugin access; this will likely require a database
   access, but since the behaviour is implementation specific it might also
   imply a round-trip to the backend. This class of checks, when involving

doc/source/contributor/internals/quota.rst

@@ -112,7 +112,7 @@ an attribute is added for every resource managed by the quota engine.
 Request authorisation is performed in this controller, and only 'admin' users
 are allowed to modify quotas for projects. As the neutron policy engine is not
 used, it is not possible to configure which users should be allowed to manage
-quotas using policy.json.
+quotas using policy.yaml.
 
 The driver operations dealing with quota management are:
 

doc/source/contributor/policies/bugs.rst

@@ -378,7 +378,7 @@ more will be added over time if needed.
 +-------------------------------+-----------------------------------------+--------------------------+
 | Tag                           | Description                             | Contact                  |
 +===============================+=========================================+==========================+
-| access-control_               | A bug affecting RBAC and policy.json    | Miguel Lavalle           |
+| access-control_               | A bug affecting RBAC and policy.yaml    | Miguel Lavalle           |
 +-------------------------------+-----------------------------------------+--------------------------+
 | api_                          | A bug affecting the API layer           | Akihiro Motoki           |
 +-------------------------------+-----------------------------------------+--------------------------+

lower-constraints.txt

@@ -61,7 +61,7 @@ os-vif==1.15.1
 osc-lib==1.8.0
 oslo.cache==1.26.0
 oslo.concurrency==3.26.0
-oslo.config==6.0.0
+oslo.config==6.8.0
 oslo.context==2.22.0
 oslo.db==4.44.0
 oslo.i18n==3.20.0
@@ -74,8 +74,8 @@ oslo.reports==1.18.0
 oslo.rootwrap==5.8.0
 oslo.serialization==2.25.0
 oslo.service==1.31.0
-oslo.upgradecheck==0.1.0
+oslo.upgradecheck==1.3.0
-oslo.utils==4.4.0
+oslo.utils==4.5.0
 oslo.versionedobjects==1.35.1
 oslotest==3.2.0
 osprofiler==2.3.0
@@ -112,9 +112,8 @@ pytz==2013.6
 PyYAML==5.3.1
 reno==3.1.0
 repoze.lru==0.7
-requests==2.14.2
+requests==2.18.0
 requestsexceptions==1.2.0
-rfc3986==0.3.1
 Routes==2.3.1
 simplejson==3.5.1
 smmap==0.9.0

neutron/cmd/upgrade_checks/checks.py

@@ -17,6 +17,7 @@ from neutron_lib import context
 from neutron_lib.db import model_query
 from oslo_config import cfg
 from oslo_serialization import jsonutils
+from oslo_upgradecheck import common_checks
 from oslo_upgradecheck import upgradecheck
 from sqlalchemy import or_
 
@@ -97,6 +98,8 @@ class CoreChecks(base.BaseChecks):
              self.nic_switch_agent_min_kernel_check),
             (_("VLAN allocations valid segmentation ID check"),
              self.vlan_allocations_segid_check),
+            (_('Policy File JSON to YAML Migration'),
+             (common_checks.check_policy_json, {'conf': cfg.CONF})),
         ]
 
     @staticmethod

neutron/common/config.py

@@ -26,6 +26,7 @@ from oslo_config import cfg
 from oslo_log import log as logging
 import oslo_messaging
 from oslo_middleware import cors
+from oslo_policy import opts
 from oslo_service import wsgi
 
 from neutron._i18n import _
@@ -138,6 +139,11 @@ def set_config_defaults():
     """This method updates all configuration default values."""
     set_cors_middleware_defaults()
 
+    # TODO(gmann): Remove setting the default value of config policy_file
+    # once oslo_policy change the default value to 'policy.yaml'.
+    # https://github.com/openstack/oslo.policy/blob/a626ad12fe5a3abd49d70e3e5b95589d279ab578/oslo_policy/opts.py#L49
+    opts.set_defaults(cfg.CONF, policy.DEFAULT_POLICY_FILE)
+
 
 def set_cors_middleware_defaults():
     """Update default configuration options for oslo.middleware."""

neutron/policy.py

@@ -28,6 +28,7 @@ from neutron_lib.services import constants as service_const
 from oslo_config import cfg
 from oslo_db import exception as db_exc
 from oslo_log import log as logging
+from oslo_policy import opts
 from oslo_policy import policy
 from oslo_utils import excutils
 import stevedore
@@ -51,6 +52,13 @@ _RESOURCE_FOREIGN_KEYS = {
 }
 
 
+# TODO(gmann): Remove setting the default value of config policy_file
+# once oslo_policy change the default value to 'policy.yaml'.
+# https://github.com/openstack/oslo.policy/blob/a626ad12fe5a3abd49d70e3e5b95589d279ab578/oslo_policy/opts.py#L49
+DEFAULT_POLICY_FILE = 'policy.yaml'
+opts.set_defaults(cfg.CONF, DEFAULT_POLICY_FILE)
+
+
 def reset():
     global _ENFORCER
     if _ENFORCER:
@@ -226,7 +234,7 @@ def _build_match_rule(action, target, pluralized):
 # This check is registered as 'tenant_id' so that it can override
 # GenericCheck which was used for validating parent resource ownership.
 # This will prevent us from having to handling backward compatibility
-# for policy.json
+# for policy.yaml
 # TODO(salv-orlando): Reinstate GenericCheck for simple tenant_id checks
 @policy.register('tenant_id')
 class OwnerCheck(policy.Check):

neutron/tests/fullstack/resources/config.py

@@ -100,7 +100,7 @@ class NeutronConfigFixture(ConfigFixture):
             self.config['DEFAULT']['api_paste_config'] = (
                 self._generate_api_paste())
 
-        policy_file = self._generate_policy_json()
+        policy_file = self._generate_policy_yaml()
         if policy_file:
             self.config['oslo_policy'] = {'policy_file': policy_file}
 
@@ -154,8 +154,8 @@ class NeutronConfigFixture(ConfigFixture):
     def _generate_api_paste(self):
         return c_helpers.find_sample_file('api-paste.ini')
 
-    def _generate_policy_json(self):
+    def _generate_policy_yaml(self):
-        return c_helpers.find_sample_file('policy.json')
+        return c_helpers.find_sample_file('policy.yaml')
 
     def get_host(self):
         return self.config['DEFAULT']['host']

neutron/tests/unit/test_policy.py

@@ -157,7 +157,7 @@ class DefaultPolicyTestCase(base.BaseTestCase):
 
     def setUp(self):
         super(DefaultPolicyTestCase, self).setUp()
-        tmpfilename = self.get_temp_file_path('policy.json')
+        tmpfilename = self.get_temp_file_path('policy.yaml')
         self.rules = {
             "default": '',
             "example:exist": '!',

releasenotes/notes/deprecate-json-formatted-policy-file-4be7b9394a2ae40f.yaml

@@ -0,0 +1,20 @@
+---
+upgrade:
+  - |
+    The default value of ``[oslo_policy] policy_file`` config option has
+    been changed from ``policy.json`` to ``policy.yaml``.
+    Operators who are utilizing customized or previously generated
+    static policy JSON files (which are not needed by default), should
+    generate new policy files or convert them in YAML format. Use the
+    `oslopolicy-convert-json-to-yaml
+    <https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html>`_
+    tool to convert a JSON to YAML formatted policy file in
+    backward compatible way.
+deprecations:
+  - |
+    Use of JSON policy files was deprecated by the ``oslo.policy`` library
+    during the Victoria development cycle. As a result, this deprecation is
+    being noted in the Wallaby cycle with an anticipated future removal of support
+    by ``oslo.policy``. As such operators will need to convert to YAML policy
+    files. Please see the upgrade notes for details on migration of any
+    custom policy files.

requirements.txt

@@ -11,7 +11,7 @@ decorator>=3.4.0 # BSD
 eventlet>=0.22.1 # MIT
 pecan>=1.3.2 # BSD
 httplib2>=0.9.1 # MIT
-requests>=2.14.2 # Apache-2.0
+requests>=2.18.0 # Apache-2.0
 Jinja2>=2.10 # BSD License (3 clause)
 keystonemiddleware>=5.1.0 # Apache-2.0
 netaddr>=0.7.18 # BSD
@@ -26,7 +26,7 @@ alembic>=0.9.6 # MIT
 stevedore>=1.20.0 # Apache-2.0
 oslo.cache>=1.26.0 # Apache-2.0
 oslo.concurrency>=3.26.0 # Apache-2.0
-oslo.config>=6.0.0 # Apache-2.0
+oslo.config>=6.8.0 # Apache-2.0
 oslo.context>=2.22.0 # Apache-2.0
 oslo.db>=4.44.0 # Apache-2.0
 oslo.i18n>=3.20.0 # Apache-2.0
@@ -39,8 +39,8 @@ oslo.reports>=1.18.0 # Apache-2.0
 oslo.rootwrap>=5.8.0 # Apache-2.0
 oslo.serialization>=2.25.0 # Apache-2.0
 oslo.service>=1.31.0 # Apache-2.0
-oslo.upgradecheck>=0.1.0 # Apache-2.0
+oslo.upgradecheck>=1.3.0 # Apache-2.0
-oslo.utils>=4.4.0 # Apache-2.0
+oslo.utils>=4.5.0 # Apache-2.0
 oslo.versionedobjects>=1.35.1 # Apache-2.0
 osprofiler>=2.3.0 # Apache-2.0
 os-ken >= 0.3.0 # Apache-2.0

setup.cfg

@@ -162,7 +162,7 @@ oslo.config.opts =
     neutron.ovn.metadata.agent = neutron.conf.agent.ovn.metadata.config:list_metadata_agent_opts
     nova.auth = neutron.opts:list_auth_opts
 oslo.config.opts.defaults =
-    neutron = neutron.common.config:set_cors_middleware_defaults
+    neutron = neutron.common.config:set_config_defaults
 oslo.policy.enforcer =
     neutron = neutron.policy:get_enforcer
 oslo.policy.policies =