Adding auditing support for API request using audit middleware

This change adds audit filter in paste ini file which can be used to add
this filter in existing request processing pipeline.  Added new pipeline
with audit filter after keystone_authtoken filter.

Adding documentation about how to use this.

This change provides auditing on REST API side. Worker side changes will
be done later in another review and are not dependent on this.

Change-Id: I308796f5804aa55bc3d6496ded0504b469f00301
Partially-Implements: blueprint audit-cadf-events

doc/source/setup/audit.rst

@@ -0,0 +1,139 @@
+Using Audit Middleware with Barbican
+====================================
+
+
+Background
+----------
+
+`Audit middleware`_ is a python middleware logic which is added in service
+request processing pipeline via paste deploy filters. Audit middleware
+constructs audit event data in `CADF format`_.
+
+Audit middleware supports delivery of CADF audit events via Oslo messaging
+notifier capability. Based on `notification_driver` configuration, audit events
+can be routed to messaging infrastructure (notification_driver = messagingv2)
+or can be routed to a log file (notification_driver = log).
+
+Audit middleware creates two events per REST API interaction. First event has
+information extracted from request data and the second one has request outcome
+(response).
+
+.. _Audit middleware: http://docs.openstack.org/developer/keystonemiddleware/audit.html
+.. _CADF format: http://www.dmtf.org/sites/default/files/standards/documents/DSP2038_1.0.0.pdf
+
+
+Enabling Audit for API Requests
+-------------------------------
+
+Audit middleware is available as part of `keystonemiddleware`_ (>= 1.6) library.
+Assuming a barbican deployment is already using keystone for token validation,
+auditing support requires only configuration changes. It has Oslo messaging
+library dependency as it uses this for audit event delivery. pyCADF library is
+used for creating events in CADF format.
+
+* Enable Middleware : `Enabling Middleware Link`_  . Change is primarily in
+  service paste deploy configuration.
+* Configure Middleware : `Configuring Middleware Link`_ . Can use provided
+  audit mapping file. If there are no custom mapping for actions or path, then
+  related mapping values are derived from taxonomy defined in pyCADF library.
+
+
+.. _keystonemiddleware: https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/audit.py
+.. _Enabling Middleware Link: http://docs.openstack.org/developer/keystonemiddleware/audit.html#enabling-audit-middleware
+.. _Configuring Middleware Link: http://docs.openstack.org/developer/keystonemiddleware/audit.html#configure-audit-middleware
+
+
+.. note::
+   Audit middleware filter should be included after Keystone middleware’s keystone_authtoken
+   middleware in request pipeline. This is needed so that audit middleware can utilize
+   environment variables set by keystone_authtoken middleware.
+
+Steps
+#####
+
+1. Turn off any active instances of Barbican.
+
+#. Copy *api_audit_map.conf* to ``/etc/barbican`` directory.
+
+#. Edit ``/etc/barbican/barbican-api-paste.ini``
+
+       Replace the /v1 app pipeline from ``barbican_api`` to
+       ``barbican-api-keystone-audit`` pipeline
+
+       [pipeline:barbican-api-keystone-audit] pipeline =
+       keystone_authtoken context audit apiapp
+
+#. Edit ``barbican.conf`` to update *notification_driver* value.
+
+#. Start Barbican ``{barbican_home}/bin/barbican.sh start``
+
+
+Sample Audit Event
+------------------
+
+Following is the sample of audit event for symmetric key create request
+
+.. code-block:: json
+
+    {
+       "priority":"INFO",
+       "event_type":"audit.http.request",
+       "timestamp":"2015-12-11 00:44:26.412076",
+       "publisher_id":"uwsgi",
+       "payload":{
+          "typeURI":"http://schemas.dmtf.org/cloud/audit/1.0/event",
+          "eventTime":"2015-12-11T00:44:26.410768+0000",
+          "target":{
+             "typeURI":"service/security/keymanager/secrets",
+             "addresses":[
+                {
+                   "url":"http://{barbican_admin_host}:9311",
+                   "name":"admin"
+                },
+                {
+                   "url":"http://{barbican_internal_host}:9311",
+                   "name":"private"
+                },
+                {
+                   "url":"https://{barbican_public_host}:9311",
+                   "name":"public"
+                }
+             ],
+             "name":"barbican_service_user",
+             "id":"barbican"
+          },
+          "observer":{
+             "id":"target"
+          },
+          "tags":[
+             "correlation_id?value=openstack:7e0fe4a6-e258-477e-a1c9-0fd0921a8435"
+          ],
+          "eventType":"activity",
+          "initiator":{
+             "typeURI":"service/security/account/user",
+             "name":"cinder_user",
+             "credential":{
+                "token":"***",
+                "identity_status":"Confirmed"
+             },
+             "host":{
+                "agent":"curl/7.38.0",
+                "address":"192.168.245.2"
+             },
+             "project_id":"8eabee0a4c4e40f882df8efbce695526",
+             "id":"513e8682f23446ceb598b6b0f5c4482b"
+          },
+          "action":"create",
+          "outcome":"pending",
+          "id":"openstack:3a6a961c-9ada-4b81-9095-90968d896c41",
+          "requestPath":"/v1/secrets"
+       },
+       "message_id":"afc3fd93-51e9-4c80-b330-983e66962265"
+    }
+
+
+`Ceilometer audit wiki`_ can be referred to identify meaning of different fields
+in audit event to **7 "W"s of Audit and Compliance**.
+
+.. _Ceilometer audit wiki: https://wiki.openstack.org/wiki/Ceilometer/blueprints/
+    support-standard-audit-formats#CADF_Model_is_designed_to_answer_all_Audit_and_Compliance_Questions

doc/source/setup/index.rst

@@ -1,5 +1,5 @@
 Setting up Barbican
-=====================
+===================
 
 .. toctree::
    :maxdepth: 1
@@ -10,3 +10,4 @@ Setting up Barbican
    certificate
    troubleshooting
    noauth
+   audit

etc/barbican/api_audit_map.conf

@@ -0,0 +1,26 @@
+[DEFAULT]
+# default target endpoint type
+# should match the endpoint type defined in service catalog
+target_endpoint_type = key-manager
+
+# map urls ending with specific text to a unique action
+# Don't need custom mapping for other resource operations
+# Note: action should match action names defined in CADF taxonomy
+[custom_actions]
+acl/get = read
+
+
+# path of api requests for CADF target typeURI
+# Just need to include top resource path to identify class of resources
+[path_keywords]
+secrets=
+containers=
+orders=
+cas=None
+quotas=
+project-quotas=
+
+
+# map endpoint type defined in service catalog to CADF typeURI
+[service_endpoints]
+key-manager = service/security/keymanager

etc/barbican/barbican-api-paste.ini

@@ -21,6 +21,10 @@ pipeline = unauthenticated-context egg:Paste#cgitb egg:Paste#httpexceptions prof
 [pipeline:barbican-api-keystone]
 pipeline = keystone_authtoken context apiapp
 
+#Use this pipeline for keystone auth with audit feature
+[pipeline:barbican-api-keystone-audit]
+pipeline = keystone_authtoken context audit apiapp
+
 [app:apiapp]
 paste.app_factory = barbican.api.app:create_main_app
 
@@ -36,6 +40,10 @@ paste.filter_factory = barbican.api.middleware.context:UnauthenticatedContextMid
 [filter:context]
 paste.filter_factory = barbican.api.middleware.context:ContextMiddleware.factory
 
+[filter:audit]
+paste.filter_factory = keystonemiddleware.audit:filter_factory
+audit_map_file = /etc/barbican/api_audit_map.conf
+
 [filter:keystone_authtoken]
 paste.filter_factory = keystonemiddleware.auth_token:filter_factory
 #need ability to re-auth a token, thus admin url

etc/barbican/barbican.conf

@@ -131,6 +131,13 @@ rabbit_hosts=localhost:5672
 #   DO NOT USE THIS, due to '# FIXME(markmc): support multiple hosts' in oslo/messaging/_drivers/amqpdriver.py
 # transport_url = rabbit://guest@localhost:5672/
 
+# oslo notification driver for sending audit events via audit middleware.
+# Meaningful only when middleware is enabled in barbican paste ini file.
+# This is oslo config MultiStrOpt so can be defined multiple times in case
+# there is need to route audit event to messaging as well as log.
+# notification_driver = messagingv2
+# notification_driver = log
+
 # ======== OpenStack policy - oslo_policy ===============
 
 [oslo_policy]

requirements.txt

@@ -21,6 +21,7 @@ Paste # MIT
 PasteDeploy>=1.5.0 # MIT
 pbr>=1.6 # Apache-2.0
 pecan>=1.0.0 # BSD
+pycadf>=1.1.0,!=2.0.0  # Apache-2.0
 pycrypto>=2.6 # Public Domain
 pyOpenSSL>=0.14 # Apache-2.0
 ldap3>=0.9.8.2 # LGPLv3