openstack/deb-congress
Code Issues Proposed changes

Edit docs for kilo release

Browse Source 
Change-Id: I4183a4cbd113461309efcfd28f538578e5e9cd9f
This commit is contained in:
Tim Hinrichs 2015-04-15 14:25:11 -07:00
parent f15e27a090
commit 140925d70f
2 changed files with 317 additions and 96 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

72
doc/source/architecture.rst
View File

@ -7,8 +7,8 @@
Architecture
============
Congress consists of the Congress policy engine and a driver for each
of the cloud services acting as a data source.::
Congress consists of the Congress policy engine and a driver for any number of
other cloud services that act as sources of information about the cloud::
Policy Engine
|
@ -30,8 +30,11 @@ relational databases are also services.
Congress uses a driver to connect each service to the policy engine.
A driver fetches cloud state from its respective cloud service, and
then feeds that state to the policy engine as table data. For
example, the Nova driver periodically makes API calls to Nova to fetch
then feeds that state to the policy engine in the form of tables.
A table is a collection of rows; each row is a collection of columns;
each row-column entry stores simple data like numbers or strings.
For example, the Nova driver periodically makes API calls to Nova to fetch
the list of virtual machines in the cloud, and the properties
associated with each VM. The Nova driver then populates a table in
the policy engine with the Nova state. For example, the Nova driver
@ -53,36 +56,55 @@ results of all its most recent scans are the state. The
more detail.
2. Policy Language
2. Policy
------------------
Congress uses Datalog as its policy language. Datalog is table-based
and is similar in many ways to SQL, Prolog, and first-oreder logic.
Congress uses Datalog and a table-based data model to leverage 50
years of research and development into the language and its
implementations.
A Congress policy defines all those states of the cloud that are permitted:
all those combinations of service tables that are possible when the cloud is
behaving as intended. Since listing the permitted states explicitly is an
insurmountable task, policy authors describe the permitted states implicitly
by writing a collection of if-then statements that are always true when the
cloud is behaving as intended.
Once the cloud operator gives Congress a policy (a description of the
permitted states of the cloud), Congress will monitor the actual state
of the cloud, compare it to policy, and warn the cloud operator about
policy violations (when the cloud's actual state is one that a policy
does not permit). In the future, Congress will go farther and take
action to change the state of the cloud (enforcement) and help us
understand the history of policy and its violations (auditing). The
:ref:`Policy <policy>` section describes policies in more detail.
More precisely, Congress uses Datalog as its policy language. Datalog is a
declarative language and is similar in many ways to SQL, Prolog, and
first-order logic. Datalog has been the subject of research and
development for the past 50 years, which means there is
a wealth of tools, algorithms, and deployment experience surrounding it.
The :ref:`Policy <policy>` section describes policies in more detail.
3. Capabilities
----------------
Once Congress is given a policy, it has three
capabilities:
* monitoring the cloud for policy violations
* preventing violations before they occur
* correcting violations after the fact
In the future, Congress will also record the history of policy and its
violations for the purpose of audit.
The :ref:`Monitoring and Enforcement <enforcement>` section describes
these capabilities in more detail.
3. Congress Server and API
4. Congress Server and API
--------------------------
Congress runs as a standalone server process and presents a RESTful
API for clients; drivers run as part of the server. The API allows
clients to create policies, read policies, read input data sources,
and read state tables (including policy violation tables).
Instructions for starting the Congress server can be found in the
:ref:`Readme <readme>` file. The API section describves the API in
more detail.
API for clients; drivers run as part of the server.
Instructions for installing and starting the Congress server can be
found in the :ref:`Readme <readme>` file.
The API allows clients to perform the following operations:
* insert and delete policy statements
* check for policy violations
* ask hypothetical questions: if the cloud were to undergo these changes,
would that cause any policy violations?
* execute actions
The :ref:`API <api>` section describes the API in more detail.

341
doc/source/cloudservices.rst
View File

@ -10,9 +10,8 @@ Cloud Services
Congress will work with any cloud service, as long as Congress can
represent the service's state in *table* format. A table is a
collection of rows, where each row is a collection of columns, and a
each row-column entry contains a simple data element (e.g. string,
number).
collection of rows, where each row is a collection of columns, and
each row-column entry contains a string or a number.
For example, Neutron contains a mapping between IP addresses and the
ports they are assigned to; neutron represents this state as the
@ -29,82 +28,64 @@ following table.::
2. Drivers
----------
To plug a new service into Congress, we need to write a small piece of code,
To plug a new service into Congress, you write a small piece of code,
called a *driver*, that queries the new service (usually through API calls)
and translates the service state into tables of data.
Then we tell Congress where to find that driver, how to
configure it (e.g. with IP address/port/username), and what name we'll use in
policy to refer to the tables generated by that driver.
and translates the service state into tables of data. Out of the box
Congress includes drivers for a number of common services (see below).
For example, the driver for Neutron invokes the Neutron API calls that list
networks, ports, security groups, and routers. The driver translates each of
the JSON objects that the API calls returns into tables (which in Python we
represent as a list of tuples). The Neutron driver is implemented here::
the JSON objects that the API calls return into tables (where in Python a table
is a list of tuples). The Neutron driver is implemented here::
congress/congress/datasources/neutron_driver.py
congress/datasources/neutronv2_driver.py
Out of the box this driver is given the name 'neutron'. When writing policy,
we use the name 'neutron:ports' to reference the 'ports' table generated by
the Neutron driver, and we use 'neutron:networks' to reference the 'networks'
table.
Once the driver is available, you install it into Congress,
you configure it (such as with an IP address/port/username), and you
write policy that references the tables populated by that driver.
3. Driver Code
--------------
Currently, every datasource driver is written in Python and invokes
API calls on a service (whose connection details are provided
as parameters), converts the results of those API calls into tables, and
returns the tables (as a list of tuples).
Datasource drivers are currently stored in ::
congress/congress/datasources/
4. Driver Configuration
2.1 Driver installation
-----------------------
To install a new driver, you must add its location to the Congress
configuration file and restart the server. Congress has a single
configuration parameter (called `drivers`) that is a list of all the
installed drivers. To install a new driver, simply add to this list
and restart.
.. ### Is the following paragraph still true?
For example, to install the Neutron driver, you add the following to the
list of drivers in the configuration file::
Once a driver is in place, you can use it to create a service whose
data is available to Congress policies. To create a service, you use the API and
congress.datasources.neutronv2_driver.NeutronV2Driver
If you have Nova and Neutron installed, you configure Congress as::
drivers = congress.datasources.neutronv2_driver.NeutronV2Driver,congress.datasources.nova_driver.NovaDriver
2.2 Driver configuration and writing policy
---------------------------------------------
Once the driver code is in place, you can use it to create a `datasource` whose
data is available to Congress policies. To create a datasource, you use the API and
provide a name (the name you will use in policy to refer to the service) and
additional connection details needed by your service (e.g. an IP and a
username/password). The same driver can be used to create multiple services
(e.g. if you have 2 instances of Neutron, you can create 2 services named say
'neutron_dev' and 'neutron_prod' using the same Python driver).
additional connection details needed by your service (such as an IP and a
username/password).
To configure datasources, you will create a standard Python configuration
file and include a section for each datasource you would like to reference
in policy. For example, the following config file will create an instance
of Nova and an instance of Neutron::
For example, you can create a datasource named 'neutron' using the
Neutron driver. And if you had a second instance of Neutron running to manage
your production network, you could
create a second datasource using the Neutron driver named 'neutron_prod'.
[neutron]
module: datasources/neutron_driver.py
username: demo
password: password
auth_url: http://127.0.0.1:5000/v2.0
tenant_name: demo
[nova]
module: datasources/nova_driver.py
username: demo
password: password
auth_url: http://127.0.0.1:5000/v2.0
tenant_name: demo
This particular sample is included in the source code and is installed into
/etc/congress/datasources.conf when using the devstack installation procedure.
The original can be found at::
congress/congress/etc/datasources.conf.sample
In a future release, it will be possible to add new datasources at run-time,
but in the current release, adding a new datasource or changing and existing
datasource's configuration requires restarting the server.
When you write policy, you use the name 'neutron:ports' to reference the 'ports'
table generated by the 'neutron' datasource, and you use 'neutron:networks' to
reference the 'networks' table generated by the 'neutron' datasource. Similarly,
you use 'neutron_prod:ports' and 'neutron_prod:networks' to reference the
tables populated by the 'neutron_prod' datasource.
(More details about writing policy can be found in the
:ref:`Policy <policy>` section.)
5. Currently Supported Drivers
3. Currently Supported Drivers
------------------------------
The datasources currently shipping with Congress expose the following tables.
@ -202,17 +183,20 @@ There are tables representing routers and security groups::
.. _datasource_driver:
6. Writing a Datasource Driver
4. Writing a Datasource Driver
------------------------------
This section is a tutorial for those of you interested in writing your own
datasource driver. It can be safely skipped otherwise.
6.1 Implementing a Datasource Driver
4.1 Implementing a Datasource Driver
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
All the Datasource drivers extend the code found in
``congress/datasources/datasource_driver.py``. Typically, you will create
All the Datasource drivers extend the code found in::
congress/datasources/datasource_driver.py
Typically, you will create
a subclass of DataSourceDriver; each instance of that class will correspond to
a different service using that driver.
@ -259,13 +243,13 @@ The following steps detail how to implement a datasource driver.
``main`` that calls update_from_datasource, and prints out the raw
API results along with the tables that were generated.
6.2 Converting API results into Tables
4.2 Converting API results into Tables
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Since Congress requires the state of each dataservice to be represented as
tables, we must convert the results of each API call (which may be comprised
of dictionaries, lists, dictionaries embedded within lists, etc.) into tables.
6.2.1 Convenience translators
4.2.1 Convenience translators
*****************************
Congress provides a translation method to make the translation from API
@ -311,7 +295,7 @@ networks_translator and then passes the API response objects to
translate_objs() which is defined in congress/datasources/datasource_driver.py
See congress/datasources/neutron_driver.py as an example.
6.2.2 Custom data conversion
4.2.2 Custom data conversion
****************************
The convenience translators may be insufficient in some cases, for example,
@ -440,3 +424,218 @@ flatten that subobject into tables.
]
*Note* : uuid* are congress generated uuids
4.3 Writing a Datasource driver test
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Once you've written a driver, you'll want to add a unit test for it. To help, this section describes how the unit test for the Glance driver works. Here are the relevant files.
* Driver code: congress/datasources/glance_v2driver.py
* Test code: congress/tests/datasources/test_glancev2_driver.py (appearing in full at the end of this section)
The test code has two methods: setUp() and test_update_from_datasource().
4.3.1 Glance setup
***********************
We begin our description with the setUp() method of the test.
.. code-block:: python
def setUp(self):
First the test creates a fake (actually a mock) Keystone. Most clients talk to Keystone, so having a fake one seems to be necessary to make the Glance client work properly.
.. code-block:: python
self.keystone_client_p = mock.patch(
"keystoneclient.v2_0.client.Client")
self.keystone_client_p.start()
Next the test creates a fake Glance client. Glance is an OpenStack service that stores (among other things) operating system Images that you can use to create a new VM. The Glance datasource driver makes a call to <glance-client>.images.list() to retrieve the list of those images, and then turns that list of images into tables. The test creates a fake Glance client so it can control the return value of <glance-client>.images.list().
.. code-block:: python
self.glance_client_p = mock.patch("glanceclient.v2.client.Client")
self.glance_client_p.start()
Next the test instantiates the GlanceV2Driver class, which contains the code for the Glance driver. Passing 'poll_time' as 0 is probably unnecessary here, but it tells the driver not to poll automatically. Passing 'client' is important because it tells the GlanceV2Driver class to use a mocked version of the Glance client instead of creating its own.
.. code-block:: python
args = helper.datasource_openstack_args()
args['poll_time'] = 0
args['client'] = mock.MagicMock()
self.driver = glancev2_driver.GlanceV2Driver(args=args)
Next the test defines which value it wants <glance-client>.images.list() to return. The test itself will check if the Glance driver code properly translates this return value into tables. So this is the actual input to the test. Either you can write this by hand, or you can run the heat-client and print out the results.
.. code-block:: python
self.mock_images = {'images': [
{u'checksum': u'9e486c3bf76219a6a37add392e425b36',
u'container_format': u'bare',
u'created_at': u'2014-10-01T20:28:08Z’,
...
4.3.2 Glance test
*****************************************
test_update_from_datasource() is the actual test, where we have the datasource driver grab the list of Glance images and translate them to tables. The test runs the update_from_datasource() method like normal except it ensures the return value of <glance-client>.images.list() is self.mock_images.
.. code-block:: python
def test_update_from_datasource(self):
The first thing the method does is set the return value of self.driver.glance.images.list() to self.mock_images[‘images’]. Then it calls update_from_datasource() in the usual way, which translates self.mock_images['images'] into tables and stores the result into the driver's self.state dictionary.
.. code-block:: python
with mock.patch.object(self.driver.glance.images, "list") as img_list:
img_list.return_value = self.mock_images['images']
self.driver.update_from_datasource()
Next the test defines the tables that update_from_datasource() should construct. Actually, the test defines the expected value of Glance’s self.state when update_from_datasource() finishes. Remember that self.state is a dictionary mapping a table name to the set of tuples that belong to the table. For Glance, there’s just one table: ‘images’, and so the expected self.state is a dictionary with one key ‘images’ and one value: a set of tuples.
.. code-block:: python
expected = {'images': set([
(u'6934941f-3eef-43f7-9198-9b3c188e4aab',
u'active',
u'cirros-0.3.2-x86_64-uec',
u'ami',
u'2014-10-01T20:28:06Z',
u'2014-10-01T20:28:07Z',
u'ami',
u'4dfdcf14a20940799d89c7a5e7345978',
'False',
0,
0,
u'4eada48c2843d2a262c814ddc92ecf2c',
25165824,
u'/v2/images/6934941f-3eef-43f7-9198-9b3c188e4aab/file',
u'15ed89b8-588d-47ad-8ee0-207ed8010569',
u'c244d5c7-1c83-414c-a90d-af7cea1dd3b5',
u'/v2/schemas/image',
u'public'),
...
At this point in the test, update_from_datasource() has already been run, so all it does is check that the driver's self.state has the expected value.
.. code-block:: python
self.assertEqual(self.driver.state, expected)
4.3.3 Glance test code in full
*******************************
.. code-block:: python
import mock
from congress.datasources import glancev2_driver
from congress.tests import base
from congress.tests import helper
class TestGlanceV2Driver(base.TestCase):
def setUp(self):
super(TestGlanceV2Driver, self).setUp()
self.keystone_client_p = mock.patch(
"keystoneclient.v2_0.client.Client")
self.keystone_client_p.start()
self.glance_client_p = mock.patch("glanceclient.v2.client.Client")
self.glance_client_p.start()
args = helper.datasource_openstack_args()
args['poll_time'] = 0
args['client'] = mock.MagicMock()
self.driver = glancev2_driver.GlanceV2Driver(args=args)
self.mock_images = {'images': [
{u'checksum': u'9e486c3bf76219a6a37add392e425b36',
u'container_format': u'bare',
u'created_at': u'2014-10-01T20:28:08Z',
u'disk_format': u'qcow2',
u'file': u'/v2/images/c42736e7-8b09-4906-abd2-d6dc8673c297/file',
u'id': u'c42736e7-8b09-4906-abd2-d6dc8673c297',
u'min_disk': 0,
u'min_ram': 0,
u'name': u'Fedora-x86_64-20-20140618-sda',
u'owner': u'4dfdcf14a20940799d89c7a5e7345978',
u'protected': False,
u'schema': u'/v2/schemas/image',
u'size': 209649664,
u'status': u'active',
u'tags': ['type=xen2', 'type=xen'],
u'updated_at': u'2014-10-01T20:28:09Z',
u'visibility': u'public'},
{u'checksum': u'4eada48c2843d2a262c814ddc92ecf2c',
u'container_format': u'ami',
u'created_at': u'2014-10-01T20:28:06Z',
u'disk_format': u'ami',
u'file': u'/v2/images/6934941f-3eef-43f7-9198-9b3c188e4aab/file',
u'id': u'6934941f-3eef-43f7-9198-9b3c188e4aab',
u'kernel_id': u'15ed89b8-588d-47ad-8ee0-207ed8010569',
u'min_disk': 0,
u'min_ram': 0,
u'name': u'cirros-0.3.2-x86_64-uec',
u'owner': u'4dfdcf14a20940799d89c7a5e7345978',
u'protected': False,
u'ramdisk_id': u'c244d5c7-1c83-414c-a90d-af7cea1dd3b5',
u'schema': u'/v2/schemas/image',
u'size': 25165824,
u'status': u'active',
u'tags': [],
u'updated_at': u'2014-10-01T20:28:07Z',
u'visibility': u'public'}]}
def test_update_from_datasource(self):
with mock.patch.object(self.driver.glance.images, "list") as img_list:
img_list.return_value = self.mock_images['images']
self.driver.update_from_datasource()
expected = {'images': set([
(u'6934941f-3eef-43f7-9198-9b3c188e4aab',
u'active',
u'cirros-0.3.2-x86_64-uec',
u'ami',
u'2014-10-01T20:28:06Z',
u'2014-10-01T20:28:07Z',
u'ami',
u'4dfdcf14a20940799d89c7a5e7345978',
'False',
0,
0,
u'4eada48c2843d2a262c814ddc92ecf2c',
25165824,
u'/v2/images/6934941f-3eef-43f7-9198-9b3c188e4aab/file',
u'15ed89b8-588d-47ad-8ee0-207ed8010569',
u'c244d5c7-1c83-414c-a90d-af7cea1dd3b5',
u'/v2/schemas/image',
u'public'),
(u'c42736e7-8b09-4906-abd2-d6dc8673c297',
u'active',
u'Fedora-x86_64-20-20140618-sda',
u'bare',
u'2014-10-01T20:28:08Z',
u'2014-10-01T20:28:09Z',
u'qcow2',
u'4dfdcf14a20940799d89c7a5e7345978',
'False',
0,
0,
u'9e486c3bf76219a6a37add392e425b36',
209649664,
u'/v2/images/c42736e7-8b09-4906-abd2-d6dc8673c297/file',
'None',
'None',
u'/v2/schemas/image',
u'public')]),
'tags': set([
(u'c42736e7-8b09-4906-abd2-d6dc8673c297', 'type=xen'),
(u'c42736e7-8b09-4906-abd2-d6dc8673c297', 'type=xen2')])}
self.assertEqual(self.driver.state, expected)