Browse Source

Refactor Access Rules APIs

- Pull up policy check to beginning of the APIs.
- Avoid making access rules changes when one or
  more instances of the share are in an invalid state.
- Add back the per rule share instance access status.
  This restoration provides better visibility for which
  rules were applied successfully.
- Remove 'updating' and 'updating_multiple' as valid
  states for the share instance access rules status.
- Deprecate the access rule state 'new' in favor of
  'queued_to_apply' and the share instance access rules
  status 'out_of_sync' in favor of 'syncing'.

In a new API micro-version:
- Allow access rule changes irrespective of the share's
   access_rules_status.
- Expose new access rule states and share's
  access_rules_status values.

Access rules for each share instance now transition
from 'queued_to_apply' to 'applying' to 'active' or 'error';
and from 'active', 'queued_to_apply', 'applying' or 'error'
to 'queued_to_deny' to 'denying' to 'deleted'.

APIImpact
DocImpact

Partially-implements: bp fix-and-improve-access-rules
Co-Authored-By: Mike Rooney <rooneym@netapp.com>
Change-Id: Ic25e63215b5ba723cbc8cab7c51789c698e76f28
changes/68/369668/28
Goutham Pacha Ravi 5 years ago
parent
commit
64a73b1419
  1. 21
      api-ref/source/parameters.yaml
  2. 68
      api-ref/source/share-actions.inc
  3. 8
      manila/api/openstack/api_version_request.py
  4. 9
      manila/api/openstack/rest_api_version_history.rst
  5. 26
      manila/api/v1/shares.py
  6. 12
      manila/api/v2/shares.py
  7. 15
      manila/api/views/share_accesses.py
  8. 8
      manila/api/views/shares.py
  9. 28
      manila/common/constants.py
  10. 41
      manila/data/helper.py
  11. 68
      manila/db/api.py
  12. 15
      manila/db/migrations/alembic/versions/344c1ac4747f_add_share_instance_access_rules_status.py
  13. 99
      manila/db/migrations/alembic/versions/54667b9cade7_restore_share_instance_access_map_state.py
  14. 146
      manila/db/sqlalchemy/api.py
  15. 57
      manila/db/sqlalchemy/models.py
  16. 660
      manila/share/access.py
  17. 143
      manila/share/api.py
  18. 73
      manila/share/driver.py
  19. 2
      manila/share/drivers/cephfs/cephfs_native.py
  20. 8
      manila/share/drivers/huawei/v3/connection.py
  21. 2
      manila/share/drivers/netapp/dataontap/cluster_mode/lib_base.py
  22. 2
      manila/share/drivers/zfsonlinux/driver.py
  23. 144
      manila/share/manager.py
  24. 46
      manila/share/migration.py
  25. 31
      manila/share/rpcapi.py
  26. 3
      manila/tests/api/contrib/stubs.py
  27. 31
      manila/tests/api/v1/test_shares.py
  28. 140
      manila/tests/api/v2/test_shares.py
  29. 11
      manila/tests/api/views/test_share_accesses.py
  30. 59
      manila/tests/data/test_helper.py
  31. 136
      manila/tests/db/migrations/alembic/migrations_data_checks.py
  32. 135
      manila/tests/db/sqlalchemy/test_api.py
  33. 43
      manila/tests/db/sqlalchemy/test_models.py
  34. 13
      manila/tests/db_utils.py
  35. 2
      manila/tests/fake_share.py
  36. 30
      manila/tests/share/drivers/cephfs/test_cephfs_native.py
  37. 15
      manila/tests/share/drivers/huawei/test_huawei_nas.py
  38. 4
      manila/tests/share/drivers/netapp/dataontap/cluster_mode/test_lib_base.py
  39. 15
      manila/tests/share/drivers/zfsonlinux/test_driver.py
  40. 951
      manila/tests/share/test_access.py
  41. 244
      manila/tests/share/test_api.py
  42. 235
      manila/tests/share/test_manager.py
  43. 47
      manila/tests/share/test_migration.py
  44. 25
      manila/tests/share/test_rpcapi.py
  45. 20
      manila/tests/test_utils.py
  46. 3
      manila/utils.py
  47. 2
      manila_tempest_tests/config.py
  48. 24
      releasenotes/notes/bug-1626249-reintroduce-per-share-instance-access-rule-state-7c08a91373b21557.yaml

21
api-ref/source/parameters.yaml

@ -506,19 +506,13 @@ access_rule_updated_at:
type: string
access_rules_status:
description: |
The share instance access rules status. Valid value are ``active``,
``error``, and ``out_of_sync``.
The share instance access rules status. A valid value is ``active``,
``error``, or ``syncing``. In versions prior to 2.28, ``syncing`` was
represented with status ``out_of_sync``.
in: body
required: true
type: string
min_version: 2.10
access_rules_status_1:
description: |
(Since API v2.10) The share access rules status.
Valid values are ``active``, ``error``, and ``out_of_sync``.
in: body
required: true
type: string
access_share_id:
description: |
The UUID of the share to which you are granted
@ -3088,8 +3082,13 @@ source_cgsnapshot_member_id:
type: string
state:
description: |
The access rule state. A valid value is
``active`` or ``error``.
Prior to versions 2.28, the state of all access rules of a given share
is the same at all times. This could be ``new``, ``active`` or
``error``. Since 2.28, the state of each access rule of a share is
independent of the others and can be ``queued_to_apply``,
``applying``, ``active``, ``error``, ``queued_to_deny`` or ``denying``.
A new rule starts out in ``queued_to_apply`` state and is successfully
applied if it transitions to ``active`` state.
in: body
required: true
type: string

68
api-ref/source/share-actions.inc

@ -4,35 +4,9 @@
Share actions
=============
Grants or revokes share access, lists the permissions for a share,
and explicitly updates the state of a share.
To grant or revoke share access, specify one of these supported
share access levels:
- ``rw``. Read and write (RW) access.
- ``ro``. Read-only (RO) access.
You must also specify one of these supported authentication
methods:
- ``ip``. Authenticates an instance through its IP address. A valid
format is ``XX.XX.XX.XX`` or ``XX.XX.XX.XX/XX``. For example
``0.0.0.0/0``.
- ``cert``. Authenticates an instance through a TLS certificate.
Specify the TLS identity as the IDENTKEY. A valid value is any
string up to 64 characters long in the common name (CN) of the
certificate. The meaning of a string depends on its
interpretation.
- ``user``. Authenticates by a user or group name. A valid value is
an alphanumeric string that can contain some special characters
and is from 4 to 32 characters long.
To verify that the access rules (ACL) were configured correctly for
a share, you list permissions for a share.
Share actions include granting or revoking share access, listing the
available access rules for a share, explicitly updating the state of a
share, resizing a share and un-managing a share.
As administrator, you can reset the state of a share and force-
delete a share in any state. Use the ``policy.json`` file to grant
@ -61,11 +35,38 @@ access_list": null} is valid for v1.0-2.6
Grant access
============
All manila shares begin with no access. Clients must be provided with
explicit access via this API.
To grant access, specify one of these supported share access levels:
- ``rw``. Read and write (RW) access.
- ``ro``. Read-only (RO) access.
You must also specify one of these supported authentication
methods:
- ``ip``. Authenticates an instance through its IP address. A valid
format is ``XX.XX.XX.XX`` or ``XX.XX.XX.XX/XX``. For example
``0.0.0.0/0``.
- ``cert``. Authenticates an instance through a TLS certificate.
Specify the TLS identity as the IDENTKEY. A valid value is any
string up to 64 characters long in the common name (CN) of the
certificate. The meaning of a string depends on its
interpretation.
- ``user``. Authenticates by a user or group name. A valid value is
an alphanumeric string that can contain some special characters
and is from 4 to 32 characters long.
.. rest_method:: POST /v2/{tenant_id}/shares/{share_id}/action
Grants access to a share.
Normal response codes: 202
Error response codes: badRequest(400), unauthorized(401), forbidden(403),
itemNotFound(404)
@ -114,9 +115,12 @@ Revoke access
.. rest_method:: POST /v2/{tenant_id}/shares/{share_id}/action
Revokes access from a share.
The shared file systems service stores each access rule in its database and
assigns it a unique ID. This ID can be used to revoke access after access
has been requested.
Normal response codes: 202
Error response codes: badRequest(400), unauthorized(401), forbidden(403),
itemNotFound(404)
@ -142,9 +146,11 @@ List access rules
.. rest_method:: POST /v2/{tenant_id}/shares/{share_id}/action
Lists access rules for a share.
Lists access rules for a share. The Access ID returned is necessary to deny
access.
Normal response codes: 200
Error response codes: badRequest(400), unauthorized(401), forbidden(403),
itemNotFound(404)

8
manila/api/openstack/api_version_request.py

@ -84,15 +84,19 @@ REST_API_VERSION_HISTORY = """
spec. Also made the 'snapshot_support' extra spec optional.
* 2.25 - Added quota-show detail API.
* 2.26 - Removed 'nova_net_id' parameter from share_network API.
* 2.27 - Added share revert to snapshot API.
* 2.28 - Added transitional states to access rules and replaced all
transitional access_rules_status values of
shares (share_instances) with 'syncing'. Share action API
'access_allow' now accepts rules even when a share or any of
its instances may have an access_rules_status set to 'error'.
"""
# The minimum and maximum versions of the API supported
# The default api version request is defined to be the
# minimum version of the API supported.
_MIN_API_VERSION = "2.0"
_MAX_API_VERSION = "2.27"
_MAX_API_VERSION = "2.28"
DEFAULT_API_VERSION = _MIN_API_VERSION

9
manila/api/openstack/rest_api_version_history.rst

@ -163,3 +163,12 @@ user documentation.
snapshot. The share is reverted in place, and the snapshot must be the most
recent one known to manila. The feature is controlled by a new standard
optional extra spec, revert_to_snapshot_support.
2.28
----
Added transitional states ('queued_to_apply' - was previously 'new',
'queued_to_deny', 'applying' and 'denying') to access rules.
'updating', 'updating_multiple' and 'out_of_sync' are no longer valid
values for the 'access_rules_status' field of shares, they have
been collapsed into the transitional state 'syncing'. Access rule changes
can be made independent of a share's 'access_rules_status'.

26
manila/api/v1/shares.py

@ -30,6 +30,7 @@ from manila.api import common
from manila.api.openstack import wsgi
from manila.api.views import share_accesses as share_access_views
from manila.api.views import shares as share_views
from manila.common import constants
from manila import db
from manila import exception
from manila.i18n import _, _LI
@ -401,12 +402,34 @@ class ShareMixin(object):
raise webob.exc.HTTPBadRequest(explanation=_(
'Ceph IDs may not contain periods'))
def _allow_access(self, req, id, body, enable_ceph=False):
@staticmethod
def _any_instance_has_errored_rules(share):
for instance in share['instances']:
access_rules_status = instance['access_rules_status']
if access_rules_status == constants.SHARE_INSTANCE_RULES_ERROR:
return True
return False
@wsgi.Controller.authorize('allow_access')
def _allow_access(self, req, id, body, enable_ceph=False,
allow_on_error_status=False):
"""Add share access rule."""
context = req.environ['manila.context']
access_data = body.get('allow_access', body.get('os-allow_access'))
share = self.share_api.get(context, id)
if (not allow_on_error_status and
self._any_instance_has_errored_rules(share)):
msg = _("Access rules cannot be added while the share or any of "
"its replicas or migration copies has its "
"access_rules_status set to %(instance_rules_status)s. "
"Deny any rules in %(rule_state) state and try "
"again.") % {
'instance_rules_status': constants.SHARE_INSTANCE_RULES_ERROR,
'rule_state': constants.ACCESS_STATE_ERROR,
}
raise webob.exc.HTTPBadRequest(explanation=msg)
access_type = access_data['access_type']
access_to = access_data['access_to']
if access_type == 'ip':
@ -435,6 +458,7 @@ class ShareMixin(object):
return self._access_view_builder.view(req, access)
@wsgi.Controller.authorize('deny_access')
def _deny_access(self, req, id, body):
"""Remove share access rule."""
context = req.environ['manila.context']

12
manila/api/v2/shares.py

@ -323,10 +323,14 @@ class ShareController(shares.ShareMixin,
@wsgi.action('allow_access')
def allow_access(self, req, id, body):
"""Add share access rule."""
if req.api_version_request < api_version.APIVersionRequest("2.13"):
return self._allow_access(req, id, body)
else:
return self._allow_access(req, id, body, enable_ceph=True)
args = (req, id, body)
kwargs = {}
if req.api_version_request >= api_version.APIVersionRequest("2.13"):
kwargs['enable_ceph'] = True
if req.api_version_request >= api_version.APIVersionRequest("2.28"):
kwargs['allow_on_error_status'] = True
return self._allow_access(*args, **kwargs)
@wsgi.Controller.api_version('2.0', '2.6')
@wsgi.action('os-deny_access')

15
manila/api/views/share_accesses.py

@ -14,6 +14,8 @@
# under the License.
from manila.api import common
from manila.common import constants
from manila.share import api as share_api
class ViewBuilder(common.ViewBuilder):
@ -22,6 +24,7 @@ class ViewBuilder(common.ViewBuilder):
_collection_name = 'share_accesses'
_detail_version_modifiers = [
"add_access_key",
"translate_transitional_statuses",
]
def list_view(self, request, accesses):
@ -59,3 +62,15 @@ class ViewBuilder(common.ViewBuilder):
@common.ViewBuilder.versioned_method("2.21")
def add_access_key(self, context, access_dict, access):
access_dict['access_key'] = access.get('access_key')
@common.ViewBuilder.versioned_method("1.0", "2.27")
def translate_transitional_statuses(self, context, access_dict, access):
"""In 2.28, the per access rule status was (re)introduced."""
api = share_api.API()
share = api.get(context, access['share_id'])
if (share['access_rules_status'] ==
constants.SHARE_INSTANCE_RULES_SYNCING):
access_dict['state'] = constants.STATUS_NEW
else:
access_dict['state'] = share['access_rules_status']

8
manila/api/views/shares.py

@ -14,6 +14,7 @@
# under the License.
from manila.api import common
from manila.common import constants
class ViewBuilder(common.ViewBuilder):
@ -31,6 +32,7 @@ class ViewBuilder(common.ViewBuilder):
"add_user_id",
"add_create_share_from_snapshot_support_field",
"add_revert_to_snapshot_support_field",
"translate_access_rules_status",
]
def summary_list(self, request, shares):
@ -154,6 +156,12 @@ class ViewBuilder(common.ViewBuilder):
share_dict['revert_to_snapshot_support'] = share.get(
'revert_to_snapshot_support')
@common.ViewBuilder.versioned_method("2.10", "2.27")
def translate_access_rules_status(self, context, share_dict, share):
if (share['access_rules_status'] ==
constants.SHARE_INSTANCE_RULES_SYNCING):
share_dict['access_rules_status'] = constants.STATUS_OUT_OF_SYNC
def _list_view(self, func, request, shares):
"""Provide a view for a list of shares."""
shares_list = [func(request, share)['share'] for share in shares]

28
manila/common/constants.py

@ -13,18 +13,14 @@
# License for the specific language governing permissions and limitations
# under the License.
STATUS_NEW = 'new'
# SHARE AND GENERAL STATUSES
STATUS_CREATING = 'creating'
STATUS_DELETING = 'deleting'
STATUS_DELETED = 'deleted'
STATUS_ERROR = 'error'
STATUS_ERROR_DELETING = 'error_deleting'
STATUS_AVAILABLE = 'available'
STATUS_ACTIVE = 'active'
STATUS_INACTIVE = 'inactive'
STATUS_OUT_OF_SYNC = 'out_of_sync'
STATUS_UPDATING = 'updating'
STATUS_UPDATING_MULTIPLE = 'updating_multiple'
STATUS_MANAGING = 'manage_starting'
STATUS_MANAGE_ERROR = 'manage_error'
STATUS_UNMANAGING = 'unmanage_starting'
@ -44,6 +40,23 @@ STATUS_RESTORING = 'restoring'
STATUS_REVERTING = 'reverting'
STATUS_REVERTING_ERROR = 'reverting_error'
# Access rule states
ACCESS_STATE_QUEUED_TO_APPLY = 'queued_to_apply'
ACCESS_STATE_QUEUED_TO_DENY = 'queued_to_deny'
ACCESS_STATE_APPLYING = 'applying'
ACCESS_STATE_DENYING = 'denying'
ACCESS_STATE_ACTIVE = 'active'
ACCESS_STATE_ERROR = 'error'
# Share instance "access_rules_status" field values
SHARE_INSTANCE_RULES_SYNCING = 'syncing'
SHARE_INSTANCE_RULES_ERROR = 'error'
# States/statuses for multiple resources
STATUS_NEW = 'new'
STATUS_OUT_OF_SYNC = 'out_of_sync'
STATUS_ACTIVE = 'active'
TASK_STATE_MIGRATION_STARTING = 'migration_starting'
TASK_STATE_MIGRATION_IN_PROGRESS = 'migration_in_progress'
TASK_STATE_MIGRATION_COMPLETING = 'migration_completing'
@ -87,9 +100,8 @@ TRANSITIONAL_STATUSES = (
STATUS_RESTORING, STATUS_REVERTING,
)
UPDATING_RULES_STATUSES = (
STATUS_UPDATING,
STATUS_UPDATING_MULTIPLE,
INVALID_SHARE_INSTANCE_STATUSES_FOR_ACCESS_RULE_UPDATES = (
TRANSITIONAL_STATUSES + (STATUS_ERROR,)
)
SUPPORTED_SHARE_PROTOCOLS = (

41
manila/data/helper.py

@ -22,6 +22,7 @@ from oslo_log import log
from manila.common import constants
from manila import exception
from manila.i18n import _, _LW
from manila.share import access as access_manager
from manila.share import rpcapi as share_rpc
from manila import utils
@ -66,14 +67,13 @@ class DataServiceHelper(object):
self.share = share
self.context = context
self.share_rpc = share_rpc.ShareAPI()
self.access_helper = access_manager.ShareInstanceAccess(self.db, None)
self.wait_access_rules_timeout = (
CONF.data_access_wait_access_rules_timeout)
def deny_access_to_data_service(self, access_ref_list, share_instance):
for access_ref in access_ref_list:
self._change_data_access_to_instance(
share_instance, access_ref, allow=False)
self._change_data_access_to_instance(
share_instance, access_ref_list, deny=True)
# NOTE(ganso): Cleanup methods do not throw exceptions, since the
# exceptions that should be thrown are the ones that call the cleanup
@ -114,16 +114,22 @@ class DataServiceHelper(object):
'instance_id': share_instance_id,
'share_id': self.share['id']})
def _change_data_access_to_instance(
self, instance, access_ref, allow=False):
def _change_data_access_to_instance(self, instance, accesses, deny=False):
if not isinstance(accesses, list):
accesses = [accesses]
self.db.share_instance_update_access_status(
self.context, instance['id'], constants.STATUS_OUT_OF_SYNC)
self.access_helper.get_and_update_share_instance_access_rules_status(
self.context, status=constants.SHARE_INSTANCE_RULES_SYNCING,
share_instance_id=instance['id'])
if allow:
self.share_rpc.allow_access(self.context, instance, access_ref)
else:
self.share_rpc.deny_access(self.context, instance, access_ref)
if deny:
access_filters = {'access_id': [a['id'] for a in accesses]}
updates = {'state': constants.ACCESS_STATE_QUEUED_TO_DENY}
self.access_helper.get_and_update_share_instance_access_rules(
self.context, filters=access_filters, updates=updates,
share_instance_id=instance['id'])
self.share_rpc.update_access(self.context, instance)
utils.wait_for_access_update(
self.context, self.db, instance, self.wait_access_rules_timeout)
@ -166,20 +172,19 @@ class DataServiceHelper(object):
self.context, self.share['id'], access['access_type'],
access['access_to'])
for old_access in old_access_list:
self._change_data_access_to_instance(
share_instance, old_access, allow=False)
# Create new access rule and deny all old ones corresponding to
# the mapping. Since this is a bulk update, all access changes
# are made in one go.
access_ref = self.db.share_instance_access_create(
self.context, values, share_instance['id'])
self._change_data_access_to_instance(
share_instance, access_ref, allow=True)
share_instance, old_access_list, deny=True)
if allow_access_to_destination_instance:
access_ref = self.db.share_instance_access_create(
self.context, values, dest_share_instance['id'])
self._change_data_access_to_instance(
dest_share_instance, access_ref, allow=True)
dest_share_instance, access_ref)
access_ref_list.append(access_ref)

68
manila/db/api.py

@ -392,45 +392,22 @@ def share_access_create(context, values):
return IMPL.share_access_create(context, values)
def share_instance_access_create(context, values, share_instance_id):
"""Allow access to share instance."""
return IMPL.share_instance_access_create(
context, values, share_instance_id)
def share_instance_access_copy(context, share_id, instance_id):
"""Maps the existing access rules for the share to the instance in the DB.
Adds the instance mapping to the share's access rules and
returns the share's access rules.
"""
return IMPL.share_instance_access_copy(context, share_id, instance_id)
def share_access_get(context, access_id):
"""Get share access rule."""
return IMPL.share_access_get(context, access_id)
def share_instance_access_get(context, access_id, instance_id):
"""Get access rule mapping for share instance."""
return IMPL.share_instance_access_get(context, access_id, instance_id)
def share_access_get_all_for_share(context, share_id):
"""Get all access rules for given share."""
return IMPL.share_access_get_all_for_share(context, share_id)
def share_access_update_access_key(context, access_id, access_key):
"""Update the access_key field of a share access mapping."""
return IMPL.share_access_update_access_key(context, access_id, access_key)
def share_access_get_all_for_instance(context, instance_id, session=None):
def share_access_get_all_for_instance(context, instance_id, filters=None,
with_share_access_data=True):
"""Get all access rules related to a certain share instance."""
return IMPL.share_access_get_all_for_instance(
context, instance_id, session=None)
context, instance_id, filters=filters,
with_share_access_data=with_share_access_data)
def share_access_get_all_by_type_and_access(context, share_id, access_type,
@ -440,21 +417,38 @@ def share_access_get_all_by_type_and_access(context, share_id, access_type,
context, share_id, access_type, access)
def share_access_delete(context, access_id):
"""Deny access to share."""
return IMPL.share_access_delete(context, access_id)
def share_instance_access_create(context, values, share_instance_id):
"""Allow access to share instance."""
return IMPL.share_instance_access_create(
context, values, share_instance_id)
def share_instance_access_delete(context, mapping_id):
"""Deny access to share instance."""
return IMPL.share_instance_access_delete(context, mapping_id)
def share_instance_access_copy(context, share_id, instance_id):
"""Maps the existing access rules for the share to the instance in the DB.
Adds the instance mapping to the share's access rules and
returns the share's access rules.
"""
return IMPL.share_instance_access_copy(context, share_id, instance_id)
def share_instance_access_get(context, access_id, instance_id,
with_share_access_data=True):
"""Get access rule mapping for share instance."""
return IMPL.share_instance_access_get(
context, access_id, instance_id,
with_share_access_data=with_share_access_data)
def share_instance_access_update(context, access_id, instance_id, updates):
"""Update the access mapping row for a given share instance and access."""
return IMPL.share_instance_access_update(
context, access_id, instance_id, updates)
def share_instance_update_access_status(context, share_instance_id, status):
"""Update access rules status of share instance."""
return IMPL.share_instance_update_access_status(context, share_instance_id,
status)
def share_instance_access_delete(context, mapping_id):
"""Deny access to share instance."""
return IMPL.share_instance_access_delete(context, mapping_id)
####################

15
manila/db/migrations/alembic/versions/344c1ac4747f_add_share_instance_access_rules_status.py

@ -42,14 +42,6 @@ upgrade_data_mapping = {
'error': 'error',
}
downgrade_data_mapping = {
'active': 'active',
# NOTE(u_glide): We cannot determine is it applied rule or not in Manila,
# so administrator should manually handle such access rules.
'out_of_sync': 'error',
'error': 'error',
}
def upgrade():
"""Transform individual access rules states to 'access_rules_status'.
@ -120,7 +112,12 @@ def downgrade():
for instance in connection.execute(instances_query):
state = downgrade_data_mapping[instance['access_rules_status']]
# NOTE(u_glide): We cannot determine if a rule is applied or not in
# Manila, so administrator should manually handle such access rules.
if instance['access_rules_status'] == 'active':
state = 'active'
else:
state = 'error'
op.execute(
instance_access_table.update().where(

99
manila/db/migrations/alembic/versions/54667b9cade7_restore_share_instance_access_map_state.py

@ -0,0 +1,99 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""add_share_instance_access_map_state
Revision ID: 54667b9cade7
Revises: 87ce15c59bbe
Create Date: 2016-09-02 10:18:07.290461
"""
# revision identifiers, used by Alembic.
revision = '54667b9cade7'
down_revision = '87ce15c59bbe'
from alembic import op
from sqlalchemy import Column, String
from manila.common import constants
from manila.db.migrations import utils
# Mapping for new value to be assigned as ShareInstanceAccessMapping's state
access_rules_status_to_state_mapping = {
constants.STATUS_ACTIVE: constants.ACCESS_STATE_ACTIVE,
constants.STATUS_OUT_OF_SYNC: constants.ACCESS_STATE_QUEUED_TO_APPLY,
'updating': constants.ACCESS_STATE_QUEUED_TO_APPLY,
'updating_multiple': constants.ACCESS_STATE_QUEUED_TO_APPLY,
constants.STATUS_ERROR: constants.ACCESS_STATE_ERROR,
}
# Mapping for changes to Share Instance's access_rules_status
access_rules_status_upgrade_mapping = {
constants.STATUS_ACTIVE: constants.STATUS_ACTIVE,
constants.STATUS_OUT_OF_SYNC: constants.SHARE_INSTANCE_RULES_SYNCING,
'updating': constants.SHARE_INSTANCE_RULES_SYNCING,
'updating_multiple': constants.SHARE_INSTANCE_RULES_SYNCING,
constants.STATUS_ERROR: constants.STATUS_ERROR,
}
def upgrade():
op.add_column('share_instance_access_map',
Column('state', String(length=255),
default=constants.ACCESS_STATE_QUEUED_TO_APPLY))
connection = op.get_bind()
share_instances_table = utils.load_table('share_instances', connection)
instance_access_map_table = utils.load_table('share_instance_access_map',
connection)
instances_query = (
share_instances_table.select().where(
share_instances_table.c.status ==
constants.STATUS_AVAILABLE).where(
share_instances_table.c.deleted == 'False')
)
for instance in connection.execute(instances_query):
access_rule_status = instance['access_rules_status']
op.execute(
instance_access_map_table.update().where(
instance_access_map_table.c.share_instance_id == instance['id']
).values({
'state': access_rules_status_to_state_mapping[
access_rule_status],
})
)
op.execute(
share_instances_table.update().where(
share_instances_table.c.id == instance['id']
).values({
'access_rules_status': access_rules_status_upgrade_mapping[
access_rule_status],
})
)
def downgrade():
op.drop_column('share_instance_access_map', 'state')
connection = op.get_bind()
share_instances_table = utils.load_table('share_instances', connection)
op.execute(
share_instances_table.update().where(
share_instances_table.c.access_rules_status ==
constants.SHARE_INSTANCE_RULES_SYNCING).values({
'access_rules_status': constants.STATUS_OUT_OF_SYNC})
)

146
manila/db/sqlalchemy/api.py

@ -38,7 +38,6 @@ from oslo_log import log
from oslo_utils import timeutils
from oslo_utils import uuidutils
import six
from sqlalchemy import and_
from sqlalchemy import MetaData
from sqlalchemy import or_
from sqlalchemy.orm import joinedload
@ -1084,25 +1083,25 @@ def reservation_expire(context):
################
def _extract_instance_values(values_to_extract, fields):
values = copy.deepcopy(values_to_extract)
instance_values = {}
def _extract_subdict_by_fields(source_dict, fields):
dict_to_extract_from = copy.deepcopy(source_dict)
sub_dict = {}
for field in fields:
field_value = values.pop(field, None)
field_value = dict_to_extract_from.pop(field, None)
if field_value:
instance_values.update({field: field_value})
sub_dict.update({field: field_value})
return instance_values, values
return sub_dict, dict_to_extract_from
def _extract_share_instance_values(values):
share_instance_model_fields = [
'status', 'host', 'scheduled_at', 'launched_at', 'terminated_at',
'share_server_id', 'share_network_id', 'availability_zone',
'replica_state', 'share_type_id', 'share_type',
'replica_state', 'share_type_id', 'share_type', 'access_rules_status',
]
share_instance_values, share_values = (
_extract_instance_values(values, share_instance_model_fields)
_extract_subdict_by_fields(values, share_instance_model_fields)
)
return share_instance_values, share_values
@ -1110,7 +1109,7 @@ def _extract_share_instance_values(values):
def _extract_snapshot_instance_values(values):
fields = ['status', 'progress', 'provider_location']
snapshot_instance_values, snapshot_values = (
_extract_instance_values(values, fields)
_extract_subdict_by_fields(values, fields)
)
return snapshot_instance_values, snapshot_values
@ -1756,8 +1755,9 @@ def share_instance_access_copy(context, share_id, instance_id, session=None):
"""Copy access rules from share to share instance."""
session = session or get_session()
share_access_rules = share_access_get_all_for_share(
context, share_id, session=session)
share_access_rules = _share_access_get_query(
context, session, {'share_id': share_id}).all()
for access_rule in share_access_rules:
values = {
'share_instance_id': instance_id,
@ -1777,9 +1777,9 @@ def _share_instance_access_create(values, session):
@require_context
def share_access_get(context, access_id):
def share_access_get(context, access_id, session=None):
"""Get access record."""
session = get_session()
session = session or get_session()
access = _share_access_get_query(
context, session, {'id': access_id}).first()
@ -1790,43 +1790,63 @@ def share_access_get(context, access_id):
@require_context
def share_instance_access_get(context, access_id, instance_id):
def share_instance_access_get(context, access_id, instance_id,
with_share_access_data=True):
"""Get access record."""
session = get_session()
access = _share_instance_access_query(context, session, access_id,
instance_id).first()
if access:
return access
else:
if access is None:
raise exception.NotFound()
if with_share_access_data:
access = _set_instances_share_access_data(context, access, session)[0]
return access
@require_context
def share_access_get_all_for_share(context, share_id, session=None):
session = session or get_session()
return _share_access_get_query(context, session,
{'share_id': share_id}).all()
return _share_access_get_query(
context, session, {'share_id': share_id}).filter(
models.ShareAccessMapping.instance_mappings.any()).all()
@require_context
def share_access_get_all_for_instance(context, instance_id, session=None):
def share_access_get_all_for_instance(context, instance_id, filters=None,
with_share_access_data=True,
session=None):
"""Get all access rules related to a certain share instance."""
session = get_session()
return _share_access_get_query(context, session, {}).join(
models.ShareInstanceAccessMapping,
models.ShareInstanceAccessMapping.access_id ==
models.ShareAccessMapping.id).filter(and_(
models.ShareInstanceAccessMapping.share_instance_id ==
instance_id, models.ShareInstanceAccessMapping.deleted ==
"False")).all()
session = session or get_session()
filters = copy.deepcopy(filters) if filters else {}
filters.update({'share_instance_id': instance_id})
legal_filter_keys = ('id', 'share_instance_id', 'access_id', 'state')
query = _share_instance_access_query(context, session)
query = exact_filter(
query, models.ShareInstanceAccessMapping, filters, legal_filter_keys)
@require_context
def share_instance_access_get_all(context, access_id, session=None):
if not session:
session = get_session()
return _share_instance_access_query(context, session, access_id).all()
instance_accesses = query.all()
if with_share_access_data:
instance_accesses = _set_instances_share_access_data(
context, instance_accesses, session)
return instance_accesses
def _set_instances_share_access_data(context, instance_accesses, session):
if instance_accesses and not isinstance(instance_accesses, list):
instance_accesses = [instance_accesses]
for instance_access in instance_accesses:
share_access = share_access_get(
context, instance_access['access_id'], session=session)
instance_access.set_share_access_data(share_access)
return instance_accesses
@require_context
@ -1839,22 +1859,6 @@ def share_access_get_all_by_type_and_access(context, share_id, access_type,
'access_to': access}).all()
@require_context
def share_access_delete(context, access_id):
session = get_session()
with session.begin():
mappings = share_instance_access_get_all(context, access_id, session)
if len(mappings) > 0:
msg = (_("Access rule %s has mappings"
" to share instances.") % access_id)
raise exception.InvalidShareAccess(msg)
session.query(models.ShareAccessMapping).\
filter_by(id=access_id).soft_delete()
@require_context
def share_access_delete_all_by_share(context, share_id):
session = get_session()
@ -1863,17 +1867,6 @@ def share_access_delete_all_by_share(context, share_id):
filter_by(share_id=share_id).soft_delete()
@require_context
def share_access_update_access_key(context, access_id, access_key):
session = get_session()
with session.begin():
mapping = (session.query(models.ShareAccessMapping).
filter_by(id=access_id).first())
mapping.update({'access_key': access_key})
mapping.save(session=session)
return mapping
@require_context
def share_instance_access_delete(context, mapping_id):
session = get_session()
@ -1885,10 +1878,11 @@ def share_instance_access_delete(context, mapping_id):
if not mapping:
exception.NotFound()
mapping.soft_delete(session)
mapping.soft_delete(session, update_status=True,
status_field_name='state')
other_mappings = share_instance_access_get_all(
context, mapping['access_id'], session)
other_mappings = _share_instance_access_query(
context, session, mapping['access_id']).all()
# NOTE(u_glide): Remove access rule if all mappings were removed.
if len(other_mappings) == 0:
@ -1901,15 +1895,27 @@ def share_instance_access_delete(context, mapping_id):
@require_context
@oslo_db_api.wrap_db_retry(max_retries=5, retry_on_deadlock=True)
def share_instance_update_access_status(context, share_instance_id, status):
def share_instance_access_update(context, access_id, instance_id, updates):
session = get_session()
share_access_fields = ('access_type', 'access_to', 'access_key',
'access_level')
share_access_map_updates, share_instance_access_map_updates = (
_extract_subdict_by_fields(updates, share_access_fields)
)
with session.begin():
mapping = session.query(models.ShareInstance).\
filter_by(id=share_instance_id).first()
mapping.update({'access_rules_status': status})
mapping.save(session=session)
return mapping
share_access = _share_access_get_query(
context, session, {'id': access_id}).first()
share_access.update(share_access_map_updates)
share_access.save(session=session)
access = _share_instance_access_query(
context, session, access_id, instance_id).first()
access.update(share_instance_access_map_updates)
access.save(session=session)
return access
###################

57
manila/db/sqlalchemy/models.py

@ -364,17 +364,13 @@ class ShareInstance(BASE, ManilaBase):
ACCESS_STATUS_PRIORITIES = {
constants.STATUS_ACTIVE: 0,
constants.STATUS_OUT_OF_SYNC: 1,
constants.STATUS_UPDATING: 2,
constants.STATUS_UPDATING_MULTIPLE: 3,
constants.STATUS_ERROR: 4,
constants.SHARE_INSTANCE_RULES_SYNCING: 1,
constants.SHARE_INSTANCE_RULES_ERROR: 2,
}
access_rules_status = Column(Enum(constants.STATUS_ACTIVE,
constants.STATUS_OUT_OF_SYNC,
constants.STATUS_UPDATING,
constants.STATUS_UPDATING_MULTIPLE,
constants.STATUS_ERROR),
constants.SHARE_INSTANCE_RULES_SYNCING,
constants.SHARE_INSTANCE_RULES_ERROR),
default=constants.STATUS_ACTIVE)
scheduled_at = Column(DateTime)
@ -546,6 +542,28 @@ class ShareAccessMapping(BASE, ManilaBase):
access_level = Column(Enum(*constants.ACCESS_LEVELS),
default=constants.ACCESS_LEVEL_RW)
@property
def state(self):
"""Get the aggregated 'state' from all the instance mapping states.
An access rule is supposed to be truly 'active' when it has been
applied across all of the share instances of the parent share object.
"""
state = None
if len(self.instance_mappings) > 0:
order = (constants.ACCESS_STATE_ERROR,
constants.ACCESS_STATE_DENYING,
constants.ACCESS_STATE_QUEUED_TO_DENY,
constants.ACCESS_STATE_QUEUED_TO_APPLY,
constants.ACCESS_STATE_APPLYING,
constants.ACCESS_STATE_ACTIVE)
sorted_instance_mappings = sorted(
self.instance_mappings, key=lambda x: order.index(x['state']))
state = sorted_instance_mappings[0].state
return state
instance_mappings = orm.relationship(
"ShareInstanceAccessMapping",
lazy='immediate',
@ -557,28 +575,23 @@ class ShareAccessMapping(BASE, ManilaBase):
)
)
@property
def state(self):
instances = [im.instance for im in self.instance_mappings]
access_rules_status = get_access_rules_status(instances)
if access_rules_status in (
constants.STATUS_OUT_OF_SYNC,
constants.STATUS_UPDATING,
constants.STATUS_UPDATING_MULTIPLE):
return constants.STATUS_NEW
else:
return access_rules_status
class ShareInstanceAccessMapping(BASE, ManilaBase):
"""Represents access to individual share instances."""
__tablename__ = 'share_instance_access_map'
_proxified_properties = ('share_id', 'access_type', 'access_key',
'access_to', 'access_level')
def set_share_access_data(self, share_access):
for share_access_attr in self._proxified_properties:
setattr(self, share_access_attr, share_access[share_access_attr])
id = Column(String(36), primary_key=True)
deleted = Column(String(36), default='False')
share_instance_id = Column(String(36), ForeignKey('share_instances.id'))
access_id = Column(String(36), ForeignKey('share_access_map.id'))
state = Column(String(255), default=constants.ACCESS_STATE_QUEUED_TO_APPLY)
instance = orm.relationship(
"ShareInstance",
@ -1019,7 +1032,7 @@ def get_access_rules_status(instances):
share_access_status):
share_access_status = instance_access_status
if share_access_status == constants.STATUS_ERROR:
if share_access_status == constants.SHARE_INSTANCE_RULES_ERROR:
break
return share_access_status

660
manila/share/access.py

@ -13,218 +13,482 @@
# License for the specific language governing permissions and limitations
# under the License.
import copy
from oslo_log import log
import six
from manila.common import constants
from manila import exception
from manila.i18n import _, _LI
from manila import utils
LOG = log.getLogger(__name__)
class ShareInstanceAccess(object):
def locked_access_rules_operation(operation):
"""Lock decorator for access rules operations.
Takes a named lock prior to executing the operation. The lock is
named with the ID of the share instance to which the access rule belongs.
Intended use:
If an database operation to retrieve or update access rules uses this
decorator, it will block actions on all access rules of the share
instance until the named lock is free. This is used to avoid race
conditions while performing access rules updates on a given share instance.
"""
def wrapped(*args, **kwargs):
instance_id = kwargs.get('share_instance_id')
@utils.synchronized(
"locked_access_rules_operation_by_share_instance_%s" % instance_id,
external=True)
def locked_operation(*_args, **_kwargs):
return operation(*_args, **_kwargs)
return locked_operation(*args, **kwargs)
return wrapped
class ShareInstanceAccessDatabaseMixin(object):
@locked_access_rules_operation
def get_and_update_share_instance_access_rules_status(
self, context, status=None, conditionally_change={},
share_instance_id=None):
"""Get and update the access_rules_status of a share instance.
:param status: Set this parameter only if you want to
omit the conditionally_change parameter; i.e, if you want to
force a state change on the share instance regardless of the prior
state.
:param conditionally_change: Set this parameter to a dictionary of rule
state transitions to be made. The key is the expected
access_rules_status and the value is the state to transition the
access_rules_status to. If the state is not as expected,
no transition is performed. Default is {}, which means no state
transitions will be made.
:returns share_instance: if an update was made.
"""
if status is not None:
updates = {'access_rules_status': status}
else:
share_instance = self.db.share_instance_get(
context, share_instance_id)
access_rules_status = share_instance['access_rules_status']
try:
updates = {
'access_rules_status':
conditionally_change[access_rules_status],
}
except KeyError:
updates = {}
if updates:
share_instance = self.db.share_instance_update(
context, share_instance_id, updates, with_share_data=True)
return share_instance
@locked_access_rules_operation
def get_and_update_share_instance_access_rules(self, context,
filters={}, updates={},
conditionally_change={},
share_instance_id=None):
"""Get and conditionally update all access rules of a share instance.
:param updates: Set this parameter to a dictionary of key:value
pairs corresponding to the keys in the ShareInstanceAccessMapping
model. Include 'state' in this dictionary only if you want to
omit the conditionally_change parameter; i.e, if you want to
force a state change on all filtered rules regardless of the prior
state. This parameter is always honored, regardless of whether
conditionally_change allows for a state transition as desired.
Example::
{
'access_key': 'bob007680048318f4239dfc1c192d5',
'access_level': 'ro',
}
:param conditionally_change: Set this parameter to a dictionary of rule
state transitions to be made. The key is the expected state of
the access rule the value is the state to transition the
access rule to. If the state is not as expected, no transition is
performed. Default is {}, which means no state transitions
will be made.
Example::
{
'queued_to_apply': 'applying',
'queued_to_deny': 'denying',
}
"""
instance_rules = self.db.share_access_get_all_for_instance(
context, share_instance_id, filters=filters)
if instance_rules and (updates or conditionally_change):
for rule in instance_rules:
mapping_state = rule['state']
rule_updates = copy.deepcopy(updates)
try:
rule_updates['state'] = conditionally_change[mapping_state]
except KeyError:
pass
if rule_updates:
self.db.share_instance_access_update(
context, rule['access_id'], share_instance_id,
rule_updates)
# Refresh the rules after the updates
rules_to_get = {
'access_id': tuple([i['access_id'] for i in instance_rules]),
}
instance_rules = self.db.share_access_get_all_for_instance(
context, share_instance_id, filters=rules_to_get)
return instance_rules
@locked_access_rules_operation
def get_and_update_share_instance_access_rule(self, context, rule_id,
updates={},
share_instance_id=None,
conditionally_change={}):
"""Get and conditionally update a given share instance access rule.
:param updates: Set this parameter to a dictionary of key:value
pairs corresponding to the keys in the ShareInstanceAccessMapping
model. Include 'state' in this dictionary only if you want to
omit the conditionally_change parameter; i.e, if you want to
force a state change regardless of the prior state.
:param conditionally_change: Set this parameter to a dictionary of rule
state transitions to be made. The key is the expected state of
the access rule the value is the state to transition the
access rule to. If the state is not as expected, no transition is
performed. Default is {}, which means no state transitions
will be made.
Example::
{
'queued_to_apply': 'applying',
'queued_to_deny': 'denying',
}
"""
instance_rule_mapping = self.db.share_instance_access_get(
context, rule_id, share_instance_id)
if conditionally_change:
mapping_state = instance_rule_mapping['state']
try:
updated_state = conditionally_change[mapping_state]
updates.update({'state': updated_state})
except KeyError:
msg = ("The state of the access rule %(rule_id)s (allowing "
"access to share instance %(si)s) was not updated "
"because its state was modified by another operation.")
msg_payload = {
'si': share_instance_id,
'rule_id': rule_id,
}
LOG.debug(msg, msg_payload)
if updates:
self.db.share_instance_access_update(
context, rule_id, share_instance_id, updates)
# Refresh the rule after update
instance_rule_mapping = self.db.share_instance_access_get(
context, rule_id, share_instance_id)
return instance_rule_mapping
@locked_access_rules_operation
def delete_share_instance_access_rules(self, context, access_rules,
share_instance_id=None):
for rule in access_rules:
self.db.share_instance_access_delete(context, rule['id'])
class ShareInstanceAccess(ShareInstanceAccessDatabaseMixin):
def __init__(self, db, driver):
self.db = db
self.driver = driver
def update_access_rules(self, context, share_instance_id, add_rules=None,
delete_rules=None, share_server=None):
"""Update access rules in driver and database for given share instance.
:param context: current context
:param share_instance_id: Id of the share instance model
:param add_rules: list with ShareAccessMapping models or None - rules
which should be added
:param delete_rules: list with ShareAccessMapping models, "all", None
- rules which should be deleted. If "all" is provided - all rules will
be deleted.
def update_access_rules(self, context, share_instance_id,
delete_all_rules=False, share_server=None):
"""Update access rules for a given share instance.
:param context: request context
:param share_instance_id: ID of the share instance
:param delete_all_rules: set this parameter to True if all
existing access rules must be denied for a given share instance
:param share_server: Share server model or None
"""
share_instance = self.db.share_instance_get(
context, share_instance_id, with_share_data=True)
share_id = share_instance["share_id"]
@utils.synchronized(
"update_access_rules_for_share_%s" % share_id, external=True)
def _update_access_rules_locked(*args, **kwargs):
return self._update_access_rules(*args, **kwargs)
_update_access_rules_locked(
context=context,
share_instance_id=share_instance_id,
add_rules=add_rules,
delete_rules=delete_rules,
share_server=share_server,
)
msg_payload = {
'si': share_instance_id,
'shr': share_instance['share_id'],
}
if delete_all_rules:
updates = {
'state': constants.ACCESS_STATE_QUEUED_TO_DENY,
}
self.get_and_update_share_instance_access_rules(
context, updates=updates, share_instance_id=share_instance_id)
# Is there a sync in progress? If yes, ignore the incoming request.
rule_filter = {
'state': (constants.ACCESS_STATE_APPLYING,
constants.ACCESS_STATE_DENYING),
}
syncing_rules = self.get_and_update_share_instance_access_rules(
context, filters=rule_filter, share_instance_id=share_instance_id)
if syncing_rules:
msg = ("Access rules are being synced for share instance "
"%(si)s belonging to share %(shr)s, any rule changes will "
"be applied shortly.")
LOG.debug(msg, msg_payload)
else:
rules_to_apply_or_deny = (
self._update_and_get_unsynced_access_rules_from_db(
context, share_instance_id)
)
if rules_to_apply_or_deny:
msg = ("Updating access rules for share instance %(si)s "
"belonging to share %(shr)s.")
LOG.debug(msg, msg_payload)
self._update_access_rules(context, share_instance_id,
share_server=share_server)
else:
msg = ("All access rules have been synced for share instance "
"%(si)s belonging to share %(shr)s.")
LOG.debug(msg, msg_payload)
def _update_access_rules(self, context, share_instance_id, add_rules=None,
delete_rules=None, share_server=None):
# Reget share instance
def _update_access_rules(self, context, share_instance_id,
share_server=None):
# Refresh the share instance model
share_instance = self.db.share_instance_get(
context, share_instance_id, with_share_data=True)
# NOTE (rraja): preserve error state to trigger maintenance mode
if share_instance['access_rules_status'] != constants.STATUS_ERROR:
self.db.share_instance_update_access_status(
context,
share_instance_id,
constants.STATUS_UPDATING)
add_rules = add_rules or []
delete_rules = delete_rules or []
remove_rules = None
if six.text_type(delete_rules).lower() == "all":
# NOTE(ganso): if we are deleting an instance or clearing all
# the rules, we want to remove only the ones related
# to this instance.
delete_rules = self.db.share_access_get_all_for_instance(
context, share_instance['id'])
rules = []
else:
_rules = self.db.share_access_get_all_for_instance(
context, share_instance['id'])
rules = _rules
if delete_rules:
delete_ids = [rule['id'] for rule in delete_rules]
rules = list(filter(lambda r: r['id'] not in delete_ids,
rules))
# NOTE(ganso): trigger maintenance mode
if share_instance['access_rules_status'] == (
constants.STATUS_ERROR):
remove_rules = [
rule for rule in _rules
if rule["id"] in delete_ids]
delete_rules = []
# NOTE(ganso): up to here we are certain of the rules that we are
# supposed to pass to drivers. 'rules' variable is used for validating
# the refresh mechanism later, according to the 'supposed' rules.
driver_rules = rules
conditionally_change = {
constants.STATUS_ACTIVE: constants.SHARE_INSTANCE_RULES_SYNCING,
}
share_instance = (
self.get_and_update_share_instance_access_rules_status(
context, conditionally_change=conditionally_change,
share_instance_id=share_instance_id) or share_instance
)
rules_to_be_removed_from_db = []
# Populate rules to send to the driver
(access_rules_to_be_on_share, add_rules, delete_rules) = (
self._get_rules_to_send_to_driver(context, share_instance)
)
if share_instance['status'] == constants.STATUS_MIGRATING:
# NOTE(ganso): If the share instance is the source in a migration,
# it should have all its rules cast to read-only.
readonly_support = self.driver.configuration.safe_get(
'migration_readonly_rules_support')