[goal] Deprecate the JSON formatted policy file

As per the community goal of migrating the policy file
format from JSON to YAML[1], we need to do two things:

1. Change the default value of '[oslo_policy] policy_file''
config option from 'policy.json' to 'policy.yaml' with
upgrade checks.

2. Deprecate the JSON formatted policy file on project side
via warning in doc and format releasenotes.

1st item if already done by cinder in Stein so this commit
only cover the 2nd item.

Since oslo.policy 3.6.0, by default oslo policy will
fallback to existing policy.json file to give operator some
time to migrate it to new default policy.yaml. But cinder
already changed the default value to policy.yaml long back
since Stein so no need to fallback to default JSON file. To
do that it disable this fallback via flag to oslo.policy.

Also convert the tests/unit/policy.json to policy.yaml file.

Additionally, made some corrections to outdated documentation
when removing references to a "policy.json" file.

[1]https://governance.openstack.org/tc/goals/selected/wallaby/migrate-policy-format-from-json-to-yaml.html

Co-Authored-By: Brian Rosmaita <rosmaita.fossdev@gmail.com>

Change-Id: Iaf8a454e60d1e4b66981b61175f89203cc15e439

cinder/policy.py

@@ -51,7 +51,8 @@ def init(use_conf=True):
     if not _ENFORCER:
         _ENFORCER = policy.Enforcer(
             CONF,
-            use_conf=use_conf)
+            use_conf=use_conf,
+            fallback_to_json_file=False)
         register_rules(_ENFORCER)
         _ENFORCER.load_rules()
 

cinder/tests/unit/policy.json

@@ -1,54 +0,0 @@
-{
-    "admin_api": "is_admin:True",
-    "admin_or_owner":  "is_admin:True or project_id:%(project_id)s",
-
-    "volume:create_snapshot": "",
-    "volume:delete_snapshot": "",
-    "volume:get_snapshot": "",
-    "volume:get_all_snapshots": "",
-    "volume:update_snapshot": "",
-    "volume:get_snapshot_metadata": "",
-    "volume:delete_snapshot_metadata": "",
-    "volume:update_snapshot_metadata": "",
-    "volume:revert_to_snapshot": "",
-    "volume_extension:volume_actions:upload_image": "",
-    "volume_extension:types_manage": "",
-    "volume_extension:types_extra_specs:create": "",
-    "volume_extension:types_extra_specs:delete": "",
-    "volume_extension:types_extra_specs:index": "",
-    "volume_extension:types_extra_specs:show": "",
-    "volume_extension:types_extra_specs:update": "",
-    "volume_extension:volume_type_access": "",
-    "volume_extension:extended_snapshot_attributes": "",
-    "volume_extension:services:index": "",
-    "volume_extension:services:update" : "rule:admin_api",
-
-    "limits_extension:used_limits": "",
-
-    "volume:create_transfer": "",
-    "volume:delete_transfer": "",
-    "volume:get_transfer": "",
-    "volume:get_all_transfers": "",
-
-    "backup:delete": "",
-    "backup:get": "",
-    "backup:get_all": "",
-    "backup:restore": "",
-
-    "group:delete": "",
-    "group:update": "",
-    "group:get": "",
-    "group:get_all": "",
-
-    "group:delete_group_snapshot": "",
-    "group:update_group_snapshot": "",
-    "group:get_group_snapshot": "",
-    "group:get_all_group_snapshots": "",
-    "group:reset_group_snapshot_status":"",
-    "group:reset_status":"",
-    "group:enable_replication": "",
-    "group:disable_replication": "",
-    "group:failover_replication": "",
-    "group:list_replication_targets": ""
-
-}

cinder/tests/unit/policy.yaml

@@ -0,0 +1,198 @@
+# Default rule for most non-Admin APIs.
+"admin_or_owner": "is_admin:True or project_id:%(project_id)s"
+
+# Default rule for most Admin APIs.
+"admin_api": "is_admin:True"
+
+# Show snapshot's metadata or one specified metadata with a given key.
+# GET  /snapshots/{snapshot_id}/metadata
+# GET  /snapshots/{snapshot_id}/metadata/{key}
+"volume:get_snapshot_metadata": ""
+
+# Update snapshot's metadata or one specified metadata with a given
+# key.
+# PUT  /snapshots/{snapshot_id}/metadata
+# PUT  /snapshots/{snapshot_id}/metadata/{key}
+"volume:update_snapshot_metadata": ""
+
+# Delete snapshot's specified metadata with a given key.
+# DELETE  /snapshots/{snapshot_id}/metadata/{key}
+"volume:delete_snapshot_metadata": ""
+
+# List snapshots.
+# GET  /snapshots
+# GET  /snapshots/detail
+"volume:get_all_snapshots": ""
+
+# List or show snapshots with extended attributes.
+# GET  /snapshots/{snapshot_id}
+# GET  /snapshots/detail
+"volume_extension:extended_snapshot_attributes": ""
+
+# Create snapshot.
+# POST  /snapshots
+"volume:create_snapshot": ""
+
+# Show snapshot.
+# GET  /snapshots/{snapshot_id}
+"volume:get_snapshot": ""
+
+# Update snapshot.
+# PUT  /snapshots/{snapshot_id}
+"volume:update_snapshot": ""
+
+# Delete snapshot.
+# DELETE  /snapshots/{snapshot_id}
+"volume:delete_snapshot": ""
+
+# List backups.
+# GET  /backups
+# GET  /backups/detail
+"backup:get_all": ""
+
+# Show backup.
+# GET  /backups/{backup_id}
+"backup:get": ""
+
+# Delete backup.
+# DELETE  /backups/{backup_id}
+"backup:delete": ""
+
+# Restore backup.
+# POST  /backups/{backup_id}/restore
+"backup:restore": ""
+
+# List groups.
+# GET  /groups
+# GET  /groups/detail
+"group:get_all": ""
+
+# Show group.
+# GET  /groups/{group_id}
+"group:get": ""
+
+# Update group.
+# PUT  /groups/{group_id}
+"group:update": ""
+
+# List group snapshots.
+# GET  /group_snapshots
+# GET  /group_snapshots/detail
+"group:get_all_group_snapshots": ""
+
+# Show group snapshot.
+# GET  /group_snapshots/{group_snapshot_id}
+"group:get_group_snapshot": ""
+
+# Delete group snapshot.
+# DELETE  /group_snapshots/{group_snapshot_id}
+"group:delete_group_snapshot": ""
+
+# Update group snapshot.
+# PUT  /group_snapshots/{group_snapshot_id}
+"group:update_group_snapshot": ""
+
+# Reset status of group snapshot.
+# POST  /group_snapshots/{g_snapshot_id}/action (reset_status)
+"group:reset_group_snapshot_status": ""
+
+# Delete group.
+# POST  /groups/{group_id}/action (delete)
+"group:delete": ""
+
+# Reset status of group.
+# POST  /groups/{group_id}/action (reset_status)
+"group:reset_status": ""
+
+# Enable replication.
+# POST  /groups/{group_id}/action (enable_replication)
+"group:enable_replication": ""
+
+# Disable replication.
+# POST  /groups/{group_id}/action (disable_replication)
+"group:disable_replication": ""
+
+# Fail over replication.
+# POST  /groups/{group_id}/action (failover_replication)
+"group:failover_replication": ""
+
+# List failover replication.
+# POST  /groups/{group_id}/action (list_replication_targets)
+"group:list_replication_targets": ""
+
+# List all services.
+# GET  /os-services
+"volume_extension:services:index": ""
+
+# Update service, including failover_host, thaw, freeze, disable,
+# enable, set-log and get-log actions.
+# PUT  /os-services/{action}
+#"volume_extension:services:update": "rule:admin_api"
+
+# Show limits with used limit attributes.
+# GET  /limits
+"limits_extension:used_limits": ""
+
+# Create, update and delete volume type.
+# POST  /types
+# PUT  /types
+# DELETE  /types
+"volume_extension:types_manage": ""
+
+# Volume type access related APIs.
+# GET  /types
+# GET  /types/detail
+# GET  /types/{type_id}
+# POST  /types
+"volume_extension:volume_type_access": ""
+
+# Revert a volume to a snapshot.
+# POST  /volumes/{volume_id}/action (revert)
+"volume:revert_to_snapshot": ""
+
+# Upload a volume to image.
+# POST  /volumes/{volume_id}/action (os-volume_upload_image)
+"volume_extension:volume_actions:upload_image": ""
+
+# List volume transfer.
+# GET  /os-volume-transfer
+# GET  /os-volume-transfer/detail
+# GET  /volume_transfers
+# GET  /volume-transfers/detail
+"volume:get_all_transfers": ""
+
+# Create a volume transfer.
+# POST  /os-volume-transfer
+# POST  /volume_transfers
+"volume:create_transfer": ""
+
+# Show one specified volume transfer.
+# GET  /os-volume-transfer/{transfer_id}
+# GET  /volume-transfers/{transfer_id}
+"volume:get_transfer": ""
+
+# Delete volume transfer.
+# DELETE  /os-volume-transfer/{transfer_id}
+# DELETE  /volume-transfers/{transfer_id}
+"volume:delete_transfer": ""
+
+# List type extra specs.
+# GET  /types/{type_id}/extra_specs
+"volume_extension:types_extra_specs:index": ""
+
+# Create type extra specs.
+# POST  /types/{type_id}/extra_specs
+"volume_extension:types_extra_specs:create": ""
+
+# Show one specified type extra specs.
+# GET  /types/{type_id}/extra_specs/{extra_spec_key}
+"volume_extension:types_extra_specs:show": ""
+
+# Update type extra specs.
+# PUT  /types/{type_id}/extra_specs/{extra_spec_key}
+"volume_extension:types_extra_specs:update": ""
+
+# Delete type extra specs.
+# DELETE  /types/{type_id}/extra_specs/{extra_spec_key}
+"volume_extension:types_extra_specs:delete": ""
+

cinder/tests/unit/test.py

@@ -106,7 +106,7 @@ class TestCase(testtools.TestCase):
             os.path.dirname(__file__),
             '../../../'))
     POLICY_PATH = os.path.join(SOURCE_TREE_ROOT,
-                               'cinder/tests/unit/policy.json')
+                               'cinder/tests/unit/policy.yaml')
     RESOURCE_FILTER_FILENAME = 'etc/cinder/resource_filters.json'
     RESOURCE_FILTER_PATH = os.path.join(SOURCE_TREE_ROOT,
                                         RESOURCE_FILTER_FILENAME)

doc/source/admin/blockstorage-consistency-groups.rst

@@ -11,9 +11,12 @@ group operations can be performed using the Block Storage command line.
 
 .. note::
 
-   Block Storage API supports consistency groups since V2 version. You can
+   The Consistency Group APIs have been deprecated since the Queens release.
-   specify ``--os-volume-api-version 2`` when using Block Storage
+   Use the Generic Volume Group APIs instead.
-   command line for consistency group operations.
+
+   The Consistency Group APIs are governed by the same policies as the
+   Generic Volume Group APIs.  For information about configuring cinder
+   policies, see :ref:`policy-configuration`.
 
 Before using consistency groups, make sure the Block Storage driver that
 you are running has consistency group support by reading the Block
@@ -22,46 +25,6 @@ number of drivers that have implemented this feature. The default LVM
 driver does not support consistency groups yet because the consistency
 technology is not available at the storage level.
 
-Before using consistency groups, you must change policies for the
-consistency group APIs in the ``/etc/cinder/policy.json`` file.
-By default, the consistency group APIs are disabled.
-Enable them before running consistency group operations.
-
-Here are existing policy entries for consistency groups:
-
-.. code-block:: json
-
-   {
-   "consistencygroup:create": "group:nobody",
-   "consistencygroup:delete": "group:nobody",
-   "consistencygroup:update": "group:nobody",
-   "consistencygroup:get": "group:nobody",
-   "consistencygroup:get_all": "group:nobody",
-   "consistencygroup:create_cgsnapshot" : "group:nobody",
-   "consistencygroup:delete_cgsnapshot": "group:nobody",
-   "consistencygroup:get_cgsnapshot": "group:nobody",
-   "consistencygroup:get_all_cgsnapshots": "group:nobody",
-   }
-
-Remove ``group:nobody`` to enable these APIs:
-
-.. code-block:: json
-
-   {
-   "consistencygroup:create": "",
-   "consistencygroup:delete": "",
-   "consistencygroup:update": "",
-   "consistencygroup:get": "",
-   "consistencygroup:get_all": "",
-   "consistencygroup:create_cgsnapshot" : "",
-   "consistencygroup:delete_cgsnapshot": "",
-   "consistencygroup:get_cgsnapshot": "",
-   "consistencygroup:get_all_cgsnapshots": "",
-   }
-
-
-Restart Block Storage API service after changing policies.
-
 The following consistency group operations are supported:
 
 -  Create a consistency group, given volume types.

doc/source/admin/blockstorage-volume-multiattach.rst

@@ -58,8 +58,10 @@ volume type the following way:
 
 .. note::
 
-   Creating a new volume type is an admin-only operation by default, you can
+   Creating a new volume type is an admin-only operation by default.  You can
-   change the settings in the 'policy.json' configuration file if needed.
+   change the settings in the cinder policy file if needed.  For more
+   information about configuring cinder policies, see
+   :ref:`policy-configuration`.
 
 To create the volume you need to use the volume type you created earlier, like
 this:
@@ -101,9 +103,10 @@ information on which back end provides the functionality.
 Policy rules
 ~~~~~~~~~~~~
 
-You can control the availability of volume multi-attach through policies. We
+You can control the availability of volume multi-attach through policies that
-describe the default values in this documentation, you need to modify the
+you can configure in the cinder policy file.  For more information about the
-'policy.json' configuration file if you would like to changes these settings.
+cinder policy file, including how to generate a sample file so you can view
+the default policy settings, see :ref:`policy-configuration`.
 
 Multiattach policy
 ------------------
@@ -111,8 +114,6 @@ Multiattach policy
 The general policy rule to allow the creation or retyping of multiattach
 volumes is named  ``volume:multiattach``.
 
-The default setting of this policy is ``rule:admin_or_owner``.
-
 Multiattach policy for bootable volumes
 ---------------------------------------
 
@@ -120,8 +121,6 @@ This is a policy to disallow the ability to create multiple attachments on a
 volume that is marked as bootable with the name
 ``volume:multiattach_bootable_volume``.
 
-This is an attachment policy with a default setting of ``rule:admin_or_owner``.
-
 Known issues and limitations
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

doc/source/configuration/block-storage/block-storage-overview.rst

@@ -51,11 +51,16 @@ The Block Storage service contains the following components:
    Roles control the actions that a user is allowed to perform. In the
    default configuration, most actions do not require a particular role,
    but this can be configured by the system administrator in the
-   appropriate ``policy.json`` file that maintains the rules. A user's
+   cinder policy file that maintains the rules.
-   access to particular volumes is limited by tenant, but the user name
+
-   and password are assigned per user. Key pairs granting access to a
+   .. note::
-   volume are enabled per user, but quotas to control resource
+      For more information about configuring cinder policies, see
-   consumption across available hardware resources are per tenant.
+      :ref:`policy-configuration`.
+
+   A user's access to particular volumes is limited by tenant, but the user
+   name and password are assigned per user. Key pairs granting access to a
+   volume are enabled per user, but quotas to control resource consumption
+   across available hardware resources are per tenant.
 
    For tenants, quota controls are available to limit:
 

doc/source/configuration/block-storage/policy-config-HOWTO.rst

@@ -44,6 +44,13 @@ model Cinder employs and how it can be modified by adjusting policies.
   <https://opendev.org/openstack/cinder>`_ (or its `github mirror
   <https://github.com/openstack/cinder>`_).
 
+* OpenStack has deprecated the use of a JSON policy file since the Wallaby
+  release (Cinder 18.0.0).  If you are still using the JSON format, there
+  is a `oslopolicy-convert-json-to-yaml`__ tool that will migrate your
+  existing JSON-formatted policy file to YAML in a backward-compatible way.
+
+  .. __: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html
+
 Vocabulary Note
 ~~~~~~~~~~~~~~~
 

doc/source/configuration/block-storage/policy.rst

@@ -1,3 +1,5 @@
+.. _policy-configuration:
+
 ====================
 Policy configuration
 ====================
@@ -5,7 +7,9 @@ Policy configuration
 Configuration
 ~~~~~~~~~~~~~
 
-The following is an overview of all available policies in Cinder.
+The following is an overview of all available policies in Cinder.  For
+information on how to write a custom policy file to modify these policies,
+see :ref:`policy-file` in the Cinder configuration documentation.
 
 .. show-policy::
    :config-file: tools/config/cinder-policy-generator.conf

doc/source/configuration/block-storage/samples/policy.yaml.rst

@@ -1,3 +1,5 @@
+.. _policy-file:
+
 ===========
 policy.yaml
 ===========
@@ -12,10 +14,17 @@ run Cinder.  From the Queens release onward, the following hold:
   default values are defined in the code.
 
 * If you wish to run Cinder with policies different from the default, you may
-  write a policy file in either JSON or YAML.
+  write a policy file.
 
   * Given that JSON does not allow comments, we recommend using YAML to write
-    a custom policy file.
+    a custom policy file.  (Also, see next item.)
+
+  * OpenStack has deprecated the use of a JSON policy file since the Wallaby
+    release (Cinder 18.0.0).  If you are still using the JSON format, there
+    is a `oslopolicy-convert-json-to-yaml`__ tool that will migrate your
+    existing JSON-formatted policy file to YAML in a backward-compatible way.
+
+    .. __: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html
 
 * If you supply a custom policy file, you only need to supply entries for the
   policies you wish to change from their default values.  For instance, if you
@@ -27,6 +36,12 @@ run Cinder.  From the Queens release onward, the following hold:
   ``policy_file`` configuration option in the ``[oslo_policy]`` section of the
   the Cinder configuration file.
 
+* Instructions for generating a sample ``policy.yaml`` file directly from the
+  Cinder source code can be found in the file ``README-policy.generate.md``
+  in the ``etc/cinder`` directory in the Cinder `source code repository
+  <https://opendev.org/openstack/cinder>`_ (or its `github mirror
+  <https://github.com/openstack/cinder>`_).
+
 The following provides a listing of the default policies. It is not recommended
 to copy this file into ``/etc/cinder`` unless you are planning on providing a
 different policy for an operation that is not the default.

lower-constraints.txt

@@ -62,7 +62,7 @@ oslo.i18n==5.0.1
 oslo.log==4.4.0
 oslo.messaging==12.5.0
 oslo.middleware==4.1.1
-oslo.policy==3.5.0
+oslo.policy==3.6.0
 oslo.privsep==2.4.0
 oslo.reports==2.2.0
 oslo.rootwrap==6.2.0

releasenotes/notes/deprecate-json-formatted-policy-file-dc3441a7b1dbfb47.yaml

@@ -0,0 +1,12 @@
+---
+deprecations:
+  - |
+    Use of JSON formatted policy files was deprecated by the ``oslo.policy``
+    library during the Victoria development cycle. As a result, this
+    deprecation is being noted in the Wallaby cycle with an anticipated future
+    removal of JSON formatted file support by ``oslo.policy``. As such
+    operators will need to convert to YAML policy files.
+    Use the `oslopolicy-convert-json-to-yaml
+    <https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html>`_
+    tool to convert the existing JSON formatted policy file to YAML in a backward
+    compatible way.

requirements.txt

@@ -21,7 +21,7 @@ oslo.db>=8.4.0 # Apache-2.0
 oslo.log>=4.4.0 # Apache-2.0
 oslo.messaging>=12.5.0 # Apache-2.0
 oslo.middleware>=4.1.1 # Apache-2.0
-oslo.policy>=3.5.0 # Apache-2.0
+oslo.policy>=3.6.0 # Apache-2.0
 oslo.privsep>=2.4.0 # Apache-2.0
 oslo.reports>=2.2.0 # Apache-2.0
 oslo.rootwrap>=6.2.0 # Apache-2.0