Add helper scripts for generating policy info

This adds two helper scripts that consuming projects can use to get
information that helps deployers.

The oslopolicy-policy-generator script looks at an entry_point for a
configured policy.Enforcer and outputs a yaml formatted policy file for
that configuration. This is a merge of registered rules and configured
rules.

The oslopolicy_list_redundant script looks at an entry_point for a
configured policy.Enforcer and outputs a yaml formatted policy file with
a list of policies where the registered default matches the project
configuration. These are policies that can be removed from the
configuration file(s) without affecting policy.

Change-Id: Ibe4e6c9288768bcc8f532e384524580c57e58275
Implements: bp policy-sample-generation

doc/source/usage.rst

@@ -52,10 +52,16 @@ benefits.
   policies used are registered. The signature of Enforcer.authorize matches
   Enforcer.enforce.
 
-* More will be documented as capabilities are added.
 * A sample policy file can be generated based on the registered policies
   rather than needing to manually maintain one.
 
+* A policy file can be generated which is a merge of registered defaults and
+  policies loaded from a file. This shows the effective policy in use.
+
+* A list can be generated which contains policies defined in a file which match
+  defaults registered in code. These are candidates for removal from the file
+  in order to keep it small and understandable.
+
 How to register
 ---------------
 
@@ -106,3 +112,71 @@ where policy-generator.conf looks like::
     namespace = nova.compute.api
 
 If output_file is ommitted the sample file will be sent to stdout.
+
+Merged file generation
+----------------------
+
+This will output a policy file which includes all registered policy defaults
+and all policies configured with a policy file. This file shows the effective
+policy in use by the project.
+
+In setup.cfg of a project using oslo.policy::
+
+    [entry_points]
+    oslo.policy.enforcer =
+        nova = nova.policy:get_enforcer
+
+where get_enforcer is a method that returns a configured
+oslo_policy.policy.Enforcer object. This object should be setup exactly as it
+is used for actual policy enforcement, if it differs the generated policy file
+may not match reality.
+
+Run the oslopolicy-policy-generator script with some configuration options::
+
+    oslopolicy-policy-generator --namespace nova --output-file policy-merged.yaml
+
+or::
+
+    oslopolicy-policy-generator --config-file policy-merged-generator.conf
+
+where policy-merged-generator.conf looks like::
+
+    [DEFAULT]
+    output_file = policy-merged.yaml
+    namespace = nova
+
+If output_file is ommitted the file will be sent to stdout.
+
+List of redundant configuration
+-------------------------------
+
+This will output a list of matches for policy rules that are defined in a
+configuration file where the rule does not differ from a registered default
+rule. These are rules that can be removed from the policy file with no change
+in effective policy.
+
+In setup.cfg of a project using oslo.policy::
+
+    [entry_points]
+    oslo.policy.enforcer =
+        nova = nova.policy:get_enforcer
+
+where get_enforcer is a method that returns a configured
+oslo_policy.policy.Enforcer object. This object should be setup exactly as it
+is used for actual policy enforcement, if it differs the generated policy file
+may not match reality.
+
+Run the oslopolicy-list-redundant script::
+
+    oslopolicy-list-redundant --namespace nova
+
+or::
+
+    oslopolicy-list-redundant --config-file policy-redundant.conf
+
+where policy-redundant.conf looks like::
+
+    [DEFAULT]
+    namespace = nova
+
+Output will go to stdout.

oslo_policy/generator.py

@@ -17,17 +17,29 @@ import textwrap
 from oslo_config import cfg
 import stevedore
 
+from oslo_policy import policy
+
 LOG = logging.getLogger(__name__)
 
 _generator_opts = [
     cfg.StrOpt('output-file',
                help='Path of the file to write to. Defaults to stdout.'),
+]
+
+_rule_opts = [
     cfg.MultiStrOpt('namespace',
                     required=True,
                     help='Option namespace(s) under "oslo.policy.policies" in '
                          'which to query for options.'),
 ]
 
+_enforcer_opts = [
+    cfg.StrOpt('namespace',
+               required=True,
+               help='Option namespace under "oslo.policy.enforcer" in '
+                    'which to look for a policy.Enforcer.'),
+]
+
 
 def _get_policies_dict(namespaces):
     """Find the options available via the given namespaces.
@@ -47,6 +59,23 @@ def _get_policies_dict(namespaces):
     return opts
 
 
+def _get_enforcer(namespace):
+    """Find a policy.Enforcer via an entry point with the given namespace.
+
+    :param namespace: a namespace under oslo.policy.enforcer where the desired
+                      enforcer object can be found.
+    :returns: a policy.Enforcer object
+    """
+    mgr = stevedore.named.NamedExtensionManager(
+        'oslo.policy.enforcer',
+        names=[namespace],
+        on_load_failure_callback=on_load_failure_callback,
+        invoke_on_load=True)
+    enforcer = mgr[namespace].obj
+
+    return enforcer
+
+
 def _format_help_text(description):
     """Format a comment for a policy based on the description provided.
 
@@ -117,6 +146,51 @@ def _generate_sample(namespaces, output_file=None):
         output_file.write(section)
 
 
+def _generate_policy(namespace, output_file=None):
+    """Generate a policy file showing what will be used.
+
+    This takes all registered policies and merges them with what's defined in
+    a policy file and outputs the result. That result is the effective policy
+    that will be honored by policy checks.
+
+    :param output_file: The path of a file to output to. stdout used if None.
+    """
+    enforcer = _get_enforcer(namespace)
+    # Ensure that files have been parsed
+    enforcer.load_rules()
+
+    file_rules = [policy.RuleDefault(name, default.check_str)
+                  for name, default in enforcer.file_rules.items()]
+    registered_rules = [policy.RuleDefault(name, default.check_str)
+                        for name, default in enforcer.registered_rules.items()
+                        if name not in enforcer.file_rules]
+    policies = {'rules': file_rules + registered_rules}
+
+    output_file = (open(output_file, 'w') if output_file
+                   else sys.stdout)
+
+    for section in _sort_and_format_by_section(policies, include_help=False):
+        output_file.write(section)
+
+
+def _list_redundant(namespace):
+    """Generate a list of configured policies which match defaults.
+
+    This checks all policies loaded from policy files and checks to see if they
+    match registered policies. If so then it is redundant to have them defined
+    in a policy file and operators should consider removing them.
+    """
+    enforcer = _get_enforcer(namespace)
+    # Ensure that files have been parsed
+    enforcer.load_rules()
+
+    for name, file_rule in enforcer.file_rules.items():
+        reg_rule = enforcer.registered_rules.get(name, None)
+        if reg_rule:
+            if file_rule == reg_rule:
+                print(reg_rule)
+
+
 def on_load_failure_callback(*args, **kwargs):
     raise
 
@@ -124,7 +198,25 @@ def on_load_failure_callback(*args, **kwargs):
 def generate_sample(args=None):
     logging.basicConfig(level=logging.WARN)
     conf = cfg.ConfigOpts()
-    conf.register_cli_opts(_generator_opts)
+    conf.register_cli_opts(_generator_opts + _rule_opts)
-    conf.register_opts(_generator_opts)
+    conf.register_opts(_generator_opts + _rule_opts)
     conf(args)
     _generate_sample(conf.namespace, conf.output_file)
+
+
+def generate_policy(args=None):
+    logging.basicConfig(level=logging.WARN)
+    conf = cfg.ConfigOpts()
+    conf.register_cli_opts(_generator_opts + _enforcer_opts)
+    conf.register_opts(_generator_opts + _enforcer_opts)
+    conf(args)
+    _generate_policy(conf.namespace, conf.output_file)
+
+
+def list_redundant(args=None):
+    logging.basicConfig(level=logging.WARN)
+    conf = cfg.ConfigOpts()
+    conf.register_cli_opts(_enforcer_opts)
+    conf.register_opts(_enforcer_opts)
+    conf(args)
+    _list_redundant(conf.namespace)

oslo_policy/tests/test_generator.py

@@ -9,12 +9,14 @@
 #    License for the specific language governing permissions and limitations
 #    under the License.
 
+import operator
 import sys
 
 import fixtures
 import mock
 from oslo_config import cfg
 from six import moves
+import stevedore
 import testtools
 
 from oslo_policy import generator
@@ -148,3 +150,118 @@ class GeneratorRaiseErrorTestCase(testtools.TestCase):
         with mock.patch('sys.argv', testargs):
             self.assertRaises(cfg.RequiredOptError, generator.generate_sample,
                               [])
+
+
+class GeneratePolicyTestCase(base.PolicyBaseTestCase):
+    def setUp(self):
+        super(GeneratePolicyTestCase, self).setUp()
+
+    def test_merged_rules(self):
+        extensions = []
+        for name, opts in OPTS.items():
+            ext = stevedore.extension.Extension(name=name, entry_point=None,
+                                                plugin=None, obj=opts)
+            extensions.append(ext)
+        test_mgr = stevedore.named.NamedExtensionManager.make_test_instance(
+            extensions=extensions, namespace=['base_rules', 'rules'])
+
+        # Write the policy file for an enforcer to load
+        sample_file = self.get_config_file_fullname('policy-sample.yaml')
+        with mock.patch('stevedore.named.NamedExtensionManager',
+                        return_value=test_mgr):
+            generator._generate_sample(['base_rules', 'rules'], sample_file)
+
+        enforcer = policy.Enforcer(self.conf, policy_file='policy-sample.yaml')
+        # register an opt defined in the file
+        enforcer.register_default(policy.RuleDefault('admin',
+                                                     'is_admin:False'))
+        # register a new opt
+        enforcer.register_default(policy.RuleDefault('foo', 'role:foo'))
+
+        # Mock out stevedore to return the configured enforcer
+        ext = stevedore.extension.Extension(name='testing', entry_point=None,
+                                            plugin=None, obj=enforcer)
+        test_mgr = stevedore.named.NamedExtensionManager.make_test_instance(
+            extensions=[ext], namespace='testing')
+
+        # Generate a merged file
+        merged_file = self.get_config_file_fullname('policy-merged.yaml')
+        with mock.patch('stevedore.named.NamedExtensionManager',
+                        return_value=test_mgr) as mock_ext_mgr:
+            generator._generate_policy(namespace='testing',
+                                       output_file=merged_file)
+            mock_ext_mgr.assert_called_once_with(
+                'oslo.policy.enforcer', names=['testing'],
+                on_load_failure_callback=generator.on_load_failure_callback,
+                invoke_on_load=True)
+
+        # load the merged file with a new enforcer
+        merged_enforcer = policy.Enforcer(self.conf,
+                                          policy_file='policy-merged.yaml')
+        merged_enforcer.load_rules()
+        for rule in ['admin', 'owner', 'admin_or_owner', 'foo']:
+            self.assertIn(rule, merged_enforcer.rules)
+
+        self.assertEqual('is_admin:True', str(merged_enforcer.rules['admin']))
+        self.assertEqual('role:foo', str(merged_enforcer.rules['foo']))
+
+
+class ListRedundantTestCase(base.PolicyBaseTestCase):
+    def setUp(self):
+        super(ListRedundantTestCase, self).setUp()
+
+    def _capture_stdout(self):
+        self.useFixture(fixtures.MonkeyPatch('sys.stdout', moves.StringIO()))
+        return sys.stdout
+
+    def test_matched_rules(self):
+        extensions = []
+        for name, opts in OPTS.items():
+            ext = stevedore.extension.Extension(name=name, entry_point=None,
+                                                plugin=None, obj=opts)
+            extensions.append(ext)
+        test_mgr = stevedore.named.NamedExtensionManager.make_test_instance(
+            extensions=extensions, namespace=['base_rules', 'rules'])
+
+        # Write the policy file for an enforcer to load
+        sample_file = self.get_config_file_fullname('policy-sample.yaml')
+        with mock.patch('stevedore.named.NamedExtensionManager',
+                        return_value=test_mgr):
+            generator._generate_sample(['base_rules', 'rules'], sample_file)
+
+        enforcer = policy.Enforcer(self.conf, policy_file='policy-sample.yaml')
+        # register opts that match those defined in policy-sample.yaml
+        enforcer.register_default(policy.RuleDefault('admin', 'is_admin:True'))
+        enforcer.register_default(
+            policy.RuleDefault('owner', 'project_id:%(project_id)s'))
+        # register a new opt
+        enforcer.register_default(policy.RuleDefault('foo', 'role:foo'))
+
+        # Mock out stevedore to return the configured enforcer
+        ext = stevedore.extension.Extension(name='testing', entry_point=None,
+                                            plugin=None, obj=enforcer)
+        test_mgr = stevedore.named.NamedExtensionManager.make_test_instance(
+            extensions=[ext], namespace='testing')
+
+        stdout = self._capture_stdout()
+        with mock.patch('stevedore.named.NamedExtensionManager',
+                        return_value=test_mgr) as mock_ext_mgr:
+            generator._list_redundant(namespace='testing')
+            mock_ext_mgr.assert_called_once_with(
+                'oslo.policy.enforcer', names=['testing'],
+                on_load_failure_callback=generator.on_load_failure_callback,
+                invoke_on_load=True)
+
+        matches = [line.split(': ', 1) for
+                   line in stdout.getvalue().splitlines()]
+        matches.sort(key=operator.itemgetter(0))
+
+        # Should be 'admin'
+        opt0 = matches[0]
+        self.assertEqual('"admin"', opt0[0])
+        self.assertEqual('"is_admin:True"', opt0[1])
+
+        # Should be 'owner'
+        opt1 = matches[1]
+        self.assertEqual('"owner"', opt1[0])
+        self.assertEqual('"project_id:%(project_id)s"', opt1[1])

setup.cfg

@@ -33,6 +33,8 @@ oslo.config.opts =
 console_scripts =
     oslopolicy-checker = oslo_policy.shell:main
     oslopolicy-sample-generator = oslo_policy.generator:generate_sample
+    oslopolicy-policy-generator = oslo_policy.generator:genarate_policy
+    oslopolicy-list-redundant = oslo_policy.generator:list_redundant
 
 [build_sphinx]
 source-dir = doc/source