Initial EC2-API service commit.

This code introduces standalone service which proxies its calls to
existing nova EC2-API.
All the code here except for the ec2api/api/proxy.py,
ec2api/api/ec2client.py and some util functions is taken from current
nova and unused functionality is cut of it.
The proxy.py and ec2client.py files implement the new code which
proxies incoming request (on port 8788) to original EC2 API in nova
on port 8773.
The result is transparently translated back to user.

Change-Id: I4cb84f833d7d4f0e379672710ed39562811d43e0

.testr.conf

@@ -0,0 +1,4 @@
+[DEFAULT]
+test_command=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ec2api/tests $LISTOPT $IDOPTION
+test_id_option=--load-list $IDFILE
+test_list_option=--list

HACKING.rst

@@ -0,0 +1,43 @@
+Ec2api Style Commandments
+=========================
+
+- Step 1: Read the OpenStack Style Commandments
+  https://github.com/openstack-dev/hacking/blob/master/doc/source/index.rst
+- Step 2: Read on
+
+Ec2api Specific Commandments
+----------------------------
+
+General
+-------
+- Do not use locals(). Example::
+
+    LOG.debug(_("volume %(vol_name)s: creating size %(vol_size)sG") %
+              locals()) # BAD
+
+    LOG.debug(_("volume %(vol_name)s: creating size %(vol_size)sG") %
+              {'vol_name': vol_name,
+               'vol_size': vol_size}) # OKAY
+
+- Use 'raise' instead of 'raise e' to preserve original traceback or exception being reraised::
+
+    except Exception as e:
+        ...
+        raise e  # BAD
+
+    except Exception:
+        ...
+        raise  # OKAY
+
+
+
+Creating Unit Tests
+-------------------
+For every new feature, unit tests should be created that both test and
+(implicitly) document the usage of said feature. If submitting a patch for a
+bug that had no unit test, a new passing unit test should be added. If a
+submitted bug fix does have a unit test, be sure to add a new one that fails
+without the patch and passes with the patch.
+
+For more information on creating unit tests and utilizing the testing
+infrastructure in OpenStack Ec2api, please read ec2api/testing/README.rst.

LICENSE

@@ -0,0 +1,176 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+

MANIFEST.in

@@ -0,0 +1,20 @@
+include run_tests.sh ChangeLog
+include README.rst builddeb.sh
+include MANIFEST.in pylintrc
+include AUTHORS
+include run_tests.py
+include HACKING.rst
+include LICENSE
+include ChangeLog
+include babel.cfg tox.ini
+include openstack-common.conf
+include ec2api/openstack/common/README
+include ec2api/db/sqlalchemy/migrate_repo/README
+include ec2api/db/sqlalchemy/migrate_repo/migrate.cfg
+include ec2api/db/sqlalchemy/migrate_repo/versions/*.sql
+graft doc
+graft etc
+graft ec2api/locale
+graft ec2api/tests
+graft tools
+global-exclude *.pyc

README.rst

@@ -0,0 +1,68 @@
+OpenStack EC2 API README
+-----------------------------
+
+Support of EC2 API for OpenStack.
+This project provides a standalone EC2 API service which pursues two goals:
+1. Implement VPC API which now absent in nova's EC2 API
+2. Create a standalone service for EC2 API support which later can accommodate
+not only the VPC API but the rest of the EC2 API currently present in nova as 
+well.
+
+This service implements VPC API related commands only. For the rest of the 
+EC2 API functionality it redirects request to original EC2 API in nova.
+
+It doesn't replace existing nova EC2 API service in deployment it gets 
+installed to a different port (8788 by default).
+
+Installation
+=====
+
+Run install.sh
+
+#TODO: The following should be automated later.
+
+Change /etc/ec2api/ec2api.conf:
+[database]
+connection_nova = <connection to nova> #should be taken from nova.conf
+[DEFAULT]
+external_network = <public network name> #obtained by neutron net-external-list
+
+The service gets installed on port 8788 by default. It can be changed before the
+installation in install.sh script.
+
+Usage
+=====
+
+Download aws cli from Amazon.
+Create configuration file for aws cli in your home directory ~/.aws/config:
+
+[default]
+aws_access_key_id = 1b013f18d5ed47ae8ed0fbb8debc036b
+aws_secret_access_key = 9bbc6f270ffd4dfdbe0e896947f41df3
+region = us-east-1
+
+Change the aws_access_key_id and aws_secret_acces_key above to the values
+appropriate for your cloud (can be obtained by "keystone ec2-credentials-list"
+command).
+
+Run aws cli commands using new EC2 API endpoint URL (can be obtained from
+keystone with the new port 8788) like this:
+
+aws --endpoint-url http://10.0.2.15:8788/services/Cloud ec2 describe-instances 
+
+
+Limitations
+===========
+
+This is an alpha-version, Tempest tests are not run yet.  
+VPN-related functionality is not supported yet. 
+Route-tables functionality is limited. 
+Filtering in describe functions can be done by IDs only.
+Security groups are attached to network interfaces only, not to instances yet.
+Rollbacks in case of failure during object creation are not supported yet.
+Some other not-listed here limitations exist also.
+
+Supported Features
+==================
+
+VPC API except for the Limitations above is supported.

babel.cfg

@@ -0,0 +1 @@
+[python: **.py]

bin/ec2api-db-setup

@@ -0,0 +1,292 @@
+#!/bin/bash -e
+#
+#    Copyright 2014 Cloudscaling Group, Inc
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+#
+
+#
+# Print --help output and exit.
+#
+usage() {
+
+cat << EOF
+Set up a local MySQL database for use with ec2api.
+This script will create a 'ec2api' database that is accessible
+only on localhost by user 'ec2api' with password 'ec2api'.
+
+Usage: ec2api-db-setup <rpm|deb> [options]
+Options:
+        select a distro type (rpm or debian)
+
+        --help | -h
+                Print usage information.
+        --password <pw> | -p <pw>
+                Specify the password for the 'ec2api' MySQL user that will
+                use to connect to the 'ec2api' MySQL database. By default,
+                the password 'ec2api' will be used.
+        --rootpw <pw> | -r <pw>
+                Specify the root MySQL password. If the script installs
+                the MySQL server, it will set the root password to this value
+                instead of prompting for a password. If the MySQL server is
+                already installed, this password will be used to connect to the
+                database instead of having to prompt for it.
+        --yes | -y
+                In cases where the script would normally ask for confirmation
+                before doing something, such as installing mysql-server,
+                just assume yes. This is useful if you want to run the script
+                non-interactively.
+EOF
+
+        exit 0
+}
+
+install_mysql_server() {
+        if [ -z "${ASSUME_YES}" ] ; then
+                $PACKAGE_INSTALL mysql-server
+        else
+                $PACKAGE_INSTALL -y mysql-server
+        fi
+}
+
+start_mysql_server() {
+        $SERVICE_START
+}
+
+MYSQL_EC2API_PW_DEFAULT="ec2api"
+MYSQL_EC2API_PW=${MYSQL_EC2API_PW_DEFAULT}
+EC2API_CONFIG="/etc/ec2api/ec2api.conf"
+ASSUME_YES=""
+ELEVATE=""
+
+# Check for root privileges
+if [[ $EUID -ne 0 ]] ; then
+        echo "This operation requires superuser privileges, using sudo:"
+        if sudo -l > /dev/null ; then
+                ELEVATE="sudo"
+        else
+                exit 1
+        fi
+fi
+
+case "$1" in
+        rpm)
+                echo "Installing on an RPM system."
+                PACKAGE_INSTALL="$ELEVATE yum install"
+                PACKAGE_STATUS="rpm -q"
+                SERVICE_MYSQLD="mysqld"
+                SERVICE_START="$ELEVATE service $SERVICE_MYSQLD start"
+                SERVICE_STATUS="service $SERVICE_MYSQLD status"
+                SERVICE_ENABLE="$ELEVATE chkconfig"
+                ;;
+        deb)
+                echo "Installing on a Debian system."
+                PACKAGE_INSTALL="$ELEVATE apt-get install"
+                PACKAGE_STATUS="dpkg-query -s"
+                SERVICE_MYSQLD="mysql"
+                SERVICE_START="$ELEVATE service $SERVICE_MYSQLD start"
+                SERVICE_STATUS="$ELEVATE service $SERVICE_MYSQLD status"
+                SERVICE_ENABLE=""
+                ;;
+        *)
+                usage
+                ;;
+esac
+
+while [ $# -gt 0 ]
+do
+        case "$1" in
+                -h|--help)
+                        usage
+                        ;;
+                -p|--password)
+                        shift
+                        MYSQL_EC2API_PW=${1}
+                        ;;
+                -r|--rootpw)
+                        shift
+                        MYSQL_ROOT_PW=${1}
+                        ;;
+                -y|--yes)
+                        ASSUME_YES="yes"
+                        ;;
+                *)
+                        # ignore
+                        ;;
+        esac
+        shift
+done
+
+
+# Make sure MySQL is installed.
+
+NEW_MYSQL_INSTALL=0
+if ! $PACKAGE_STATUS mysql-server && ! $PACKAGE_STATUS mariadb-server && ! $PACKAGE_STATUS mariadb-galera-server > /dev/null
+then
+        if [ -z "${ASSUME_YES}" ] ; then
+                printf "mysql-server is not installed. Would you like to install it now? (y/n): "
+                read response
+                case "$response" in
+                        y|Y)
+                                ;;
+                        n|N)
+                                echo "mysql-server must be installed. Please install it before proceeding."
+                                exit 0
+                                ;;
+                        *)
+                                echo "Invalid response."
+                                exit 1
+                esac
+        fi
+
+        NEW_MYSQL_INSTALL=1
+        install_mysql_server
+fi
+
+
+# Make sure mysqld is running.
+
+if ! $SERVICE_STATUS > /dev/null
+then
+        if [ -z "${ASSUME_YES}" ] ; then
+                printf "$SERVICE_MYSQLD is not running. Would you like to start it now? (y/n): "
+                read response
+                case "$response" in
+                        y|Y)
+                                ;;
+                        n|N)
+                                echo "$SERVICE_MYSQLD must be running. Please start it before proceeding."
+                                exit 0
+                                ;;
+                        *)
+                                echo "Invalid response."
+                                exit 1
+                esac
+        fi
+
+        start_mysql_server
+
+        # If we both installed and started, ensure it starts at boot
+        [ $NEW_MYSQL_INSTALL -eq 1 ] && $SERVICE_ENABLE $SERVICE_MYSQLD on
+fi
+
+
+# Get MySQL root access.
+
+if [ $NEW_MYSQL_INSTALL -eq 1 ]
+then
+        if [ ! "${MYSQL_ROOT_PW+defined}" ] ; then
+                echo "Since this is a fresh installation of MySQL, please set a password for the 'root' mysql user."
+
+                PW_MATCH=0
+                while [ $PW_MATCH -eq 0 ]
+                do
+                        printf "Enter new password for 'root' mysql user: "
+                        read -s MYSQL_ROOT_PW
+                        echo
+                        printf "Enter new password again: "
+                        read -s PW2
+                        echo
+                        if [ "${MYSQL_ROOT_PW}" = "${PW2}" ] ; then
+                                PW_MATCH=1
+                        else
+                                echo "Passwords did not match."
+                        fi
+                done
+        fi
+
+        echo "UPDATE mysql.user SET password = password('${MYSQL_ROOT_PW}') WHERE user = 'root'; DELETE FROM mysql.user WHERE user = ''; flush privileges;" | mysql -u root
+        if ! [ $? -eq 0 ] ; then
+                echo "Failed to set password for 'root' MySQL user."
+                exit 1
+        fi
+elif [ ! "${MYSQL_ROOT_PW+defined}" ] ; then
+        printf "Please enter the password for the 'root' MySQL user: "
+        read -s MYSQL_ROOT_PW
+        echo
+fi
+
+
+# Sanity check MySQL credentials.
+
+MYSQL_ROOT_PW_ARG=""
+if [ "${MYSQL_ROOT_PW+defined}" ]
+then
+        MYSQL_ROOT_PW_ARG="--password=${MYSQL_ROOT_PW}"
+fi
+echo "SELECT 1;" | mysql -u root ${MYSQL_ROOT_PW_ARG} > /dev/null
+if ! [ $? -eq 0 ]
+then
+        echo "Failed to connect to the MySQL server. Please check your root user credentials."
+        exit 1
+fi
+echo "Verified connectivity to MySQL."
+
+
+# Now create the db.
+
+echo "Creating 'ec2api' database."
+cat << EOF | mysql -u root ${MYSQL_ROOT_PW_ARG}
+DROP DATABASE IF EXISTS ec2api;
+CREATE DATABASE IF NOT EXISTS ec2api DEFAULT CHARACTER SET utf8;
+GRANT ALL ON ec2api.* TO 'ec2api'@'localhost' IDENTIFIED BY '${MYSQL_EC2API_PW}';
+GRANT ALL ON ec2api.* TO 'ec2api'@'%' IDENTIFIED BY '${MYSQL_EC2API_PW}';
+flush privileges;
+EOF
+
+
+# Make sure ec2api configuration has the right MySQL password.
+
+if [ "${MYSQL_EC2API_PW}" != "${MYSQL_EC2API_PW_DEFAULT}" ] ; then
+        echo "Updating 'ec2api' database password in ${EC2API_CONFIG}"
+        sed -i -e "s/mysql:\/\/ec2api:\(.*\)@/mysql:\/\/ec2api:${MYSQL_EC2API_PW}@/" ${EC2API_CONFIG}
+fi
+
+# override the logging config in ec2api.conf
+log_conf=$(mktemp /tmp/ec2api-logging.XXXXXXXXXX.conf)
+cat <<EOF > $log_conf
+[loggers]
+keys=root
+
+[handlers]
+keys=consoleHandler
+
+[formatters]
+keys=simpleFormatter
+
+[logger_root]
+level=INFO
+handlers=consoleHandler
+
+[handler_consoleHandler]
+class=StreamHandler
+formatter=simpleFormatter
+args=(sys.stdout,)
+
+[formatter_simpleFormatter]
+format=%(name)s - %(levelname)s - %(message)s
+EOF
+
+ec2-api-manage --log-config=$log_conf db_sync
+rm $log_conf
+
+# Do a final sanity check on the database.
+
+echo "SELECT * FROM migrate_version;" | mysql -u ec2api --password=${MYSQL_EC2API_PW} ec2api > /dev/null
+if ! [ $? -eq 0 ]
+then
+        echo "Final sanity check failed."
+        exit 1
+fi
+
+echo "Complete!"

ec2api/.project

@@ -0,0 +1,18 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<projectDescription>
+	<name>ec2api</name>
+	<comment></comment>
+	<projects>
+	</projects>
+	<buildSpec>
+		<buildCommand>
+			<name>org.python.pydev.PyDevBuilder</name>
+			<arguments>
+			</arguments>
+		</buildCommand>
+	</buildSpec>
+	<natures>
+		<nature>com.aptana.projects.webnature</nature>
+		<nature>org.python.pydev.pythonNature</nature>
+	</natures>
+</projectDescription>

ec2api/.pydevproject

@@ -0,0 +1,5 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<?eclipse-pydev version="1.0"?><pydev_project>
+<pydev_property name="org.python.pydev.PYTHON_PROJECT_INTERPRETER">Default</pydev_property>
+<pydev_property name="org.python.pydev.PYTHON_PROJECT_VERSION">python 2.7</pydev_property>
+</pydev_project>

ec2api/__init__.py

@@ -0,0 +1,27 @@
+#    Copyright 2014 Cloudscaling Group, Inc
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+"""
+:mod:`ec2api` -- Cloud IaaS Platform
+===================================
+
+.. automodule:: ec2api
+   :platform: Unix
+   :synopsis: Infrastructure-as-a-Service Cloud platform.
+"""
+
+import gettext
+
+
+gettext.install('ec2api', unicode=1)

ec2api/api/__init__.py

@@ -0,0 +1,400 @@
+#    Copyright 2014 Cloudscaling Group, Inc
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+"""
+Starting point for routing EC2 requests.
+"""
+
+from eventlet.green import httplib
+import netaddr
+from oslo.config import cfg
+import six
+import six.moves.urllib.parse as urlparse
+import webob
+import webob.dec
+import webob.exc
+
+from ec2api.api import apirequest
+from ec2api.api import ec2utils
+from ec2api.api import faults
+from ec2api.api import validator
+from ec2api import context
+from ec2api import exception
+from ec2api.openstack.common.gettextutils import _
+from ec2api.openstack.common import jsonutils
+from ec2api.openstack.common import log as logging
+from ec2api.openstack.common import timeutils
+from ec2api import wsgi
+
+
+LOG = logging.getLogger(__name__)
+
+ec2_opts = [
+    cfg.StrOpt('keystone_url',
+               default='http://localhost:5000/v2.0',
+               help='URL to get token from ec2 request.'),
+    cfg.IntOpt('ec2_timestamp_expiry',
+               default=300,
+               help='Time in seconds before ec2 timestamp expires'),
+]
+
+CONF = cfg.CONF
+CONF.register_opts(ec2_opts)
+CONF.import_opt('use_forwarded_for', 'ec2api.api.auth')
+
+
+# Fault Wrapper around all EC2 requests #
+class FaultWrapper(wsgi.Middleware):
+    """Calls the middleware stack, captures any exceptions into faults."""
+
+    @webob.dec.wsgify(RequestClass=wsgi.Request)
+    def __call__(self, req):
+        try:
+            return req.get_response(self.application)
+        except Exception as ex:
+            LOG.exception(_("FaultWrapper: %s"), unicode(ex))
+            return faults.Fault(webob.exc.HTTPInternalServerError())
+
+
+class RequestLogging(wsgi.Middleware):
+    """Access-Log akin logging for all EC2 API requests."""
+
+    @webob.dec.wsgify(RequestClass=wsgi.Request)
+    def __call__(self, req):
+        start = timeutils.utcnow()
+        rv = req.get_response(self.application)
+        self.log_request_completion(rv, req, start)
+        return rv
+
+    def log_request_completion(self, response, request, start):
+        apireq = request.environ.get('ec2.request', None)
+        if apireq:
+            action = apireq.action
+        else:
+            action = None
+        ctxt = request.environ.get('ec2api.context', None)
+        delta = timeutils.utcnow() - start
+        seconds = delta.seconds
+        microseconds = delta.microseconds
+        LOG.info(
+            "%s.%ss %s %s %s %s %s [%s] %s %s",
+            seconds,
+            microseconds,
+            request.remote_addr,
+            request.method,
+            "%s%s" % (request.script_name, request.path_info),
+            action,
+            response.status_int,
+            request.user_agent,
+            request.content_type,
+            response.content_type,
+            context=ctxt)
+
+
+class EC2KeystoneAuth(wsgi.Middleware):
+    """Authenticate an EC2 request with keystone and convert to context."""
+
+    @webob.dec.wsgify(RequestClass=wsgi.Request)
+    def __call__(self, req):
+        request_id = context.generate_request_id()
+        signature = req.params.get('Signature')
+        if not signature:
+            msg = _("Signature not provided")
+            return faults.ec2_error_response(request_id, "AuthFailure", msg,
+                                             status=400)
+        access = req.params.get('AWSAccessKeyId')
+        if not access:
+            msg = _("Access key not provided")
+            return faults.ec2_error_response(request_id, "AuthFailure", msg,
+                                             status=400)
+
+        # Make a copy of args for authentication and signature verification.
+        auth_params = dict(req.params)
+        # Not part of authentication args
+        auth_params.pop('Signature')
+
+        cred_dict = {
+            'access': access,
+            'signature': signature,
+            'host': req.host,
+            'verb': req.method,
+            'path': req.path,
+            'params': auth_params,
+        }
+        token_url = CONF.keystone_url + "/ec2tokens"
+        if "ec2" in token_url:
+            creds = {'ec2Credentials': cred_dict}
+        else:
+            creds = {'auth': {'OS-KSEC2:ec2Credentials': cred_dict}}
+        creds_json = jsonutils.dumps(creds)
+        headers = {'Content-Type': 'application/json'}
+
+        o = urlparse.urlparse(token_url)
+        if o.scheme == "http":
+            conn = httplib.HTTPConnection(o.netloc)
+        else:
+            conn = httplib.HTTPSConnection(o.netloc)
+        conn.request('POST', o.path, body=creds_json, headers=headers)
+        response = conn.getresponse()
+        data = response.read()
+        if response.status != 200:
+            if response.status == 401:
+                msg = response.reason
+            else:
+                msg = _("Failure communicating with keystone")
+            return faults.ec2_error_response(request_id, "AuthFailure", msg,
+                                             status=response.status)
+        result = jsonutils.loads(data)
+        conn.close()
+
+        try:
+            token_id = result['access']['token']['id']
+            user_id = result['access']['user']['id']
+            project_id = result['access']['token']['tenant']['id']
+            user_name = result['access']['user'].get('name')
+            project_name = result['access']['token']['tenant'].get('name')
+            roles = [role['name'] for role
+                     in result['access']['user']['roles']]
+        except (AttributeError, KeyError) as e:
+            LOG.exception(_("Keystone failure: %s") % e)
+            msg = _("Failure communicating with keystone")
+            return faults.ec2_error_response(request_id, "AuthFailure", msg,
+                                             status=400)
+
+        remote_address = req.remote_addr
+        if CONF.use_forwarded_for:
+            remote_address = req.headers.get('X-Forwarded-For',
+                                             remote_address)
+
+        headers["X-Auth-Token"] = token_id
+        o = urlparse.urlparse(CONF.keystone_url
+            + ("/users/%s/credentials/OS-EC2/%s" % (user_id, access)))
+        if o.scheme == "http":
+            conn = httplib.HTTPConnection(o.netloc)
+        else:
+            conn = httplib.HTTPSConnection(o.netloc)
+        conn.request('GET', o.path, headers=headers)
+        response = conn.getresponse()
+        data = response.read()
+        if response.status != 200:
+            if response.status == 401:
+                msg = response.reason
+            else:
+                msg = _("Failure communicating with keystone")
+            return faults.ec2_error_response(request_id, "AuthFailure", msg,
+                                             status=response.status)
+        ec2_creds = jsonutils.loads(data)
+        conn.close()
+
+        catalog = result['access']['serviceCatalog']
+        ctxt = context.RequestContext(user_id,
+                                      project_id,
+                                      ec2_creds["credential"]["access"],
+                                      ec2_creds["credential"]["secret"],
+                                      user_name=user_name,
+                                      project_name=project_name,
+                                      roles=roles,
+                                      auth_token=token_id,
+                                      remote_address=remote_address,
+                                      service_catalog=catalog,
+                                      api_version=req.params.get('Version'))
+
+        req.environ['ec2api.context'] = ctxt
+
+        return self.application
+
+
+class Requestify(wsgi.Middleware):
+
+    def __init__(self, app):
+        super(Requestify, self).__init__(app)
+
+    @webob.dec.wsgify(RequestClass=wsgi.Request)
+    def __call__(self, req):
+        non_args = ['Action', 'Signature', 'AWSAccessKeyId', 'SignatureMethod',
+                    'SignatureVersion', 'Version', 'Timestamp']
+        args = dict(req.params)
+        try:
+            expired = ec2utils.is_ec2_timestamp_expired(req.params,
+                            expires=CONF.ec2_timestamp_expiry)
+            if expired:
+                msg = _("Timestamp failed validation.")
+                LOG.exception(msg)
+                raise webob.exc.HTTPForbidden(explanation=msg)
+
+            # Raise KeyError if omitted
+            action = req.params['Action']
+            # Fix bug lp:720157 for older (version 1) clients
+            version = req.params['SignatureVersion']
+            if int(version) == 1:
+                non_args.remove('SignatureMethod')
+                if 'SignatureMethod' in args:
+                    args.pop('SignatureMethod')
+            for non_arg in non_args:
+                # Remove, but raise KeyError if omitted
+                args.pop(non_arg)
+        except KeyError:
+            raise webob.exc.HTTPBadRequest()
+        except exception.InvalidRequest as err:
+            raise webob.exc.HTTPBadRequest(explanation=unicode(err))
+
+        LOG.debug('action: %s', action)
+        for key, value in args.items():
+            LOG.debug('arg: %(key)s\t\tval: %(value)s',
+                      {'key': key, 'value': value})
+
+        # Success!
+        api_request = apirequest.APIRequest(
+            action, req.params['Version'], args)
+        req.environ['ec2.request'] = api_request
+        return self.application
+
+
+def validate_ec2_id(val):
+    if not validator.validate_str()(val):
+        return False
+    try:
+        ec2utils.ec2_id_to_id(val)
+    except exception.InvalidEc2Id:
+        return False
+    return True
+
+
+def is_valid_ipv4(address):
+    """Verify that address represents a valid IPv4 address."""
+    try:
+        return netaddr.valid_ipv4(address)
+    except Exception:
+        return False
+
+
+class Validator(wsgi.Middleware):
+
+    validator.validate_ec2_id = validate_ec2_id
+
+    validator.DEFAULT_VALIDATOR = {
+        'instance_id': validate_ec2_id,
+        'volume_id': validate_ec2_id,
+        'image_id': validate_ec2_id,
+        'attribute': validator.validate_str(),
+        'image_location': validator.validate_image_path,
+        'public_ip': is_valid_ipv4,
+        'region_name': validator.validate_str(),
+        'group_name': validator.validate_str(max_length=255),
+        'group_description': validator.validate_str(max_length=255),
+        'size': validator.validate_int(),
+        'user_data': validator.validate_user_data
+    }
+
+    def __init__(self, application):
+        super(Validator, self).__init__(application)
+
+    @webob.dec.wsgify(RequestClass=wsgi.Request)
+    def __call__(self, req):
+        if validator.validate(req.environ['ec2.request'].args,
+                              validator.DEFAULT_VALIDATOR):
+            return self.application
+        else:
+            raise webob.exc.HTTPBadRequest()
+
+
+def exception_to_ec2code(ex):
+    """Helper to extract EC2 error code from exception.
+
+    For other than EC2 exceptions (those without ec2_code attribute),
+    use exception name.
+    """
+    if hasattr(ex, 'ec2_code'):
+        code = ex.ec2_code
+    else:
+        code = type(ex).__name__
+    return code
+
+
+def ec2_error_ex(ex, req, code=None, message=None):
+    """Return an EC2 error response based on passed exception and log it."""
+    if not code:
+        code = exception_to_ec2code(ex)
+    status = getattr(ex, 'code', None)
+    if not status:
+        status = 500
+
+    log_fun = LOG.error
+    log_msg = _("Unexpected %(ex_name)s raised: %(ex_str)s")
+
+    context = req.environ['ec2api.context']
+    request_id = context.request_id
+    log_msg_args = {
+        'ex_name': type(ex).__name__,
+        'ex_str': unicode(ex)
+    }
+    log_fun(log_msg % log_msg_args, context=context)
+
+    if ex.args and not message and status < 500:
+        message = unicode(ex.args[0])
+    # Log filtered environment for unexpected errors.
+    env = req.environ.copy()
+    for k in env.keys():
+        if not isinstance(env[k], six.string_types):
+            env.pop(k)
+    log_fun(_('Environment: %s') % jsonutils.dumps(env))
+    if not message:
+        message = _('Unknown error occurred.')
+    return faults.ec2_error_response(request_id, code, message, status=status)
+
+
+class Executor(wsgi.Application):
+
+    """Execute an EC2 API request.
+
+    Executes 'ec2.action', passing 'ec2api.context' and
+    'ec2.action_args' (all variables in WSGI environ.)  Returns an XML
+    response, or a 400 upon failure.
+    """
+
+    @webob.dec.wsgify(RequestClass=wsgi.Request)
+    def __call__(self, req):
+        context = req.environ['ec2api.context']
+        api_request = req.environ['ec2.request']
+        try:
+            result = api_request.invoke(context)
+        except exception.InstanceNotFound as ex:
+            ec2_id = ec2utils.id_to_ec2_inst_id(ex.kwargs['instance_id'])
+            message = ex.msg_fmt % {'instance_id': ec2_id}
+            return ec2_error_ex(ex, req, message=message)
+        except exception.MethodNotFound:
+            try:
+                http, response = api_request.proxy(req)
+                resp = webob.Response()
+                resp.status = http["status"]
+                resp.headers["content-type"] = http["content-type"]
+                resp.body = str(response)
+                return resp
+            except Exception as ex:
+                return ec2_error_ex(ex, req)
+        except exception.EC2ServerError as ex:
+            resp = webob.Response()
+            resp.status = ex.response['status']
+            resp.headers['Content-Type'] = ex.response['content-type']
+            resp.body = ex.content
+            return resp
+        except Exception as ex:
+            return ec2_error_ex(ex, req)
+        else:
+            resp = webob.Response()
+            resp.status = 200
+            resp.headers['Content-Type'] = 'text/xml'
+            resp.body = str(result)
+
+            return resp

ec2api/api/apirequest.py

@@ -0,0 +1,143 @@
+#    Copyright 2014 Cloudscaling Group, Inc
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+"""
+APIRequest class
+"""
+
+import datetime
+# TODO(termie): replace minidom with etree
+from xml.dom import minidom
+
+from lxml import etree
+
+from ec2api.api import cloud
+from ec2api.api import ec2utils
+from ec2api.api import proxy
+from ec2api import exception
+from ec2api.openstack.common import log as logging
+
+
+LOG = logging.getLogger(__name__)
+
+
+def _underscore_to_camelcase(st):
+    return ''.join([x[:1].upper() + x[1:] for x in st.split('_')])
+
+
+def _underscore_to_xmlcase(st):
+    res = _underscore_to_camelcase(st)
+    return res[:1].lower() + res[1:]
+
+
+def _database_to_isoformat(datetimeobj):
+    """Return a xs:dateTime parsable string from datatime."""
+    return datetimeobj.strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + 'Z'
+
+
+class APIRequest(object):
+    def __init__(self, action, version, args):
+        self.action = action
+        self.version = version
+        self.args = args
+        self.controller = cloud.CloudController()
+        self.proxyController = proxy.ProxyController()
+
+    def invoke(self, context):
+        try:
+            method = getattr(self.controller,
+                             ec2utils.camelcase_to_underscore(self.action))
+        except AttributeError:
+            raise exception.MethodNotFound(name=self.action)
+
+        args = ec2utils.dict_from_dotted_str(self.args.items())
+
+        for key in args.keys():
+            # NOTE(vish): Turn numeric dict keys into lists
+            # NOTE(Alex): Turn "value"-only dict keys into values
+            if isinstance(args[key], dict):
+                if args[key] == {}:
+                    continue
+                if args[key].keys()[0].isdigit():
+                    s = args[key].items()
+                    s.sort()
+                    args[key] = [v for k, v in s]
+                elif args[key].keys()[0] == 'value' and len(args[key]) == 1:
+                    args[key] = args[key]['value']
+
+        result = method(context, **args)
+        return self._render_response(result, context.request_id)
+
+    def proxy(self, req):
+        return self.proxyController.proxy(req, self.args)
+
+    def _render_response(self, response_data, request_id):
+        xml = minidom.Document()
+
+        response_el = xml.createElement(self.action + 'Response')
+        response_el.setAttribute('xmlns',
+                             'http://ec2.amazonaws.com/doc/%s/' % self.version)
+        request_id_el = xml.createElement('requestId')
+        request_id_el.appendChild(xml.createTextNode(request_id))
+        response_el.appendChild(request_id_el)
+        if response_data is True:
+            self._render_dict(xml, response_el, {'return': 'true'})
+        else:
+            self._render_dict(xml, response_el, response_data)
+
+        xml.appendChild(response_el)
+
+        response = xml.toxml()
+        root = etree.fromstring(response)
+        response = etree.tostring(root, pretty_print=True)
+
+        xml.unlink()
+
+        # Don't write private key to log
+        if self.action != "CreateKeyPair":
+            LOG.debug(response)
+        else:
+            LOG.debug("CreateKeyPair: Return Private Key")
+
+        return response
+
+    def _render_dict(self, xml, el, data):
+        try:
+            for key in data.keys():
+                val = data[key]
+                el.appendChild(self._render_data(xml, key, val))
+        except Exception:
+            LOG.debug(data)
+            raise
+
+    def _render_data(self, xml, el_name, data):
+        el_name = _underscore_to_xmlcase(el_name)
+        data_el = xml.createElement(el_name)
+
+        if isinstance(data, list):
+            for item in data:
+                data_el.appendChild(self._render_data(xml, 'item', item))
+        elif isinstance(data, dict):
+            self._render_dict(xml, data_el, data)
+        elif hasattr(data, '__dict__'):
+            self._render_dict(xml, data_el, data.__dict__)
+        elif isinstance(data, bool):
+            data_el.appendChild(xml.createTextNode(str(data).lower()))
+        elif isinstance(data, datetime.datetime):
+            data_el.appendChild(
+                  xml.createTextNode(_database_to_isoformat(data)))
+        elif data is not None:
+            data_el.appendChild(xml.createTextNode(str(data)))
+
+        return data_el

ec2api/api/auth.py

@@ -0,0 +1,54 @@
+#    Copyright 2014 Cloudscaling Group, Inc
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+"""
+Common Auth Middleware.
+
+"""
+
+from oslo.config import cfg
+
+from ec2api.openstack.common import log as logging
+
+
+auth_opts = [
+    cfg.BoolOpt('api_rate_limit',
+                default=False,
+                help='whether to use per-user rate limiting for the api.'),
+    cfg.BoolOpt('use_forwarded_for',
+                default=False,
+                help='Treat X-Forwarded-For as the canonical remote address. '
+                     'Only enable this if you have a sanitizing proxy.'),
+]
+
+CONF = cfg.CONF
+CONF.register_opts(auth_opts)
+
+LOG = logging.getLogger(__name__)
+
+
+def pipeline_factory(loader, global_conf, **local_conf):
+    """A paste pipeline replica that keys off of auth_strategy."""
+    auth_strategy = "keystone"
+    pipeline = local_conf[auth_strategy]
+    if not CONF.api_rate_limit:
+        limit_name = auth_strategy + '_nolimit'
+        pipeline = local_conf.get(limit_name, pipeline)
+    pipeline = pipeline.split()
+    filters = [loader.get_filter(n) for n in pipeline[:-1]]
+    app = loader.get_app(pipeline[-1])
+    filters.reverse()
+    for fltr in filters:
+        app = fltr(app)
+    return app

ec2api/api/clients.py

@@ -0,0 +1,141 @@
+#    Copyright 2014 Cloudscaling Group, Inc
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+
+from keystoneclient.v2_0 import client as kc
+from novaclient import client as novaclient
+from novaclient import shell as novashell
+from oslo.config import cfg
+
+from ec2api.openstack.common.gettextutils import _
+from ec2api.openstack.common import log as logging
+
+logger = logging.getLogger(__name__)
+
+CONF = cfg.CONF
+
+
+try:
+    from neutronclient.v2_0 import client as neutronclient
+except ImportError:
+    neutronclient = None
+    logger.info(_('neutronclient not available'))
+try:
+    from cinderclient import client as cinderclient
+except ImportError:
+    cinderclient = None
+    logger.info(_('cinderclient not available'))
+try:
+    from glanceclient import client as glanceclient
+except ImportError:
+    glanceclient = None
+    logger.info(_('glanceclient not available'))
+
+
+def nova(context, service_type='compute'):
+    computeshell = novashell.OpenStackComputeShell()
+    extensions = computeshell._discover_extensions("1.1")
+
+    args = {
+        'project_id': context.project_id,
+        'auth_url': CONF.keystone_url,
+        'service_type': service_type,
+        'username': None,
+        'api_key': None,
+        'extensions': extensions,
+    }
+
+    client = novaclient.Client(1.1, **args)
+
+    management_url = _url_for(context, service_type=service_type)
+    client.client.auth_token = context.auth_token
+    client.client.management_url = management_url
+
+    return client
+
+
+def neutron(context):
+    if neutronclient is None:
+        return None
+
+    args = {
+        'auth_url': CONF.keystone_url,
+        'service_type': 'network',
+        'token': context.auth_token,
+        'endpoint_url': _url_for(context, service_type='network'),
+    }
+
+    return neutronclient.Client(**args)
+
+
+def glance(context):
+    if glanceclient is None:
+        return None
+
+    args = {
+        'auth_url': CONF.keystone_url,
+        'service_type': 'image',
+        'token': context.auth_token,
+    }
+
+    return glanceclient.Client(
+        "1", endpoint=_url_for(context, service_type='image'), **args)
+
+
+def cinder(context):
+    if cinderclient is None:
+        return nova(context, 'volume')
+
+    args = {
+        'service_type': 'volume',
+        'auth_url': CONF.keystone_url,
+        'username': None,
+        'api_key': None,
+    }
+
+    _cinder = cinderclient.Client('1', **args)
+    management_url = _url_for(context, service_type='volume')
+    _cinder.client.auth_token = context.auth_token
+    _cinder.client.management_url = management_url
+
+    return _cinder
+
+
+def keystone(context):
+    _keystone = kc.Client(
+        token=context.auth_token,
+        tenant_id=context.project_id,
+        auth_url=CONF.keystone_url)
+
+    return _keystone
+
+
+def _url_for(context, **kwargs):
+    service_catalog = context.service_catalog
+    if not service_catalog:
+        catalog = keystone(context).service_catalog.catalog
+        service_catalog = catalog["serviceCatalog"]
+        context.service_catalog = service_catalog
+
+    service_type = kwargs["service_type"]
+    for service in service_catalog:
+        if service["type"] != service_type:
+            continue
+        for endpoint in service["endpoints"]:
+            if "publicURL" in endpoint:
+                return endpoint["publicURL"]
+        else:
+            return None
+
+    return None

ec2api/api/cloud.py

@@ -0,0 +1,41 @@
+#    Copyright 2014 Cloudscaling Group, Inc
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+
+"""
+Cloud Controller: Implementation of EC2 REST API calls, which are
+dispatched to other nodes via AMQP RPC. State is via distributed
+datastore.
+"""
+
+from oslo.config import cfg
+
+from ec2api.openstack.common import log as logging
+
+CONF = cfg.CONF
+LOG = logging.getLogger(__name__)
+
+
+class CloudController(object):
+    """Cloud Controller
+
+        Provides the critical dispatch between
+        inbound API calls through the endpoint and messages
+        sent to the other nodes.
+    """
+    def __init__(self):
+        pass
+
+    def __str__(self):
+        return 'CloudController'

ec2api/api/ec2client.py

@@ -0,0 +1,222 @@
+#    Copyright 2014 Cloudscaling Group, Inc
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+import base64
+import hashlib
+import hmac
+import re
+import time
+import types
+import urllib
+import urlparse
+
+import httplib2
+from lxml import etree
+from oslo.config import cfg
+
+from ec2api.api import ec2utils
+from ec2api import exception
+from ec2api.openstack.common import log as logging
+
+
+ec2_opts = [
+    cfg.StrOpt('base_ec2_host',
+               default="localhost",
+               help='The IP address of the EC2 API server'),
+    cfg.IntOpt('base_ec2_port',
+               default=8773,
+               help='The port of the EC2 API server'),
+    cfg.StrOpt('base_ec2_scheme',
+               default='http',
+               help='The protocol to use when connecting to the EC2 API '
+                    'server (http, https)'),
+    cfg.StrOpt('base_ec2_path',
+               default='/services/Cloud',
+               help='The path prefix used to call the ec2 API server'),
+]
+
+CONF = cfg.CONF
+CONF.register_opts(ec2_opts)
+LOG = logging.getLogger(__name__)
+
+ISO8601 = '%Y-%m-%dT%H:%M:%SZ'
+
+
+def ec2client(context):
+    return EC2Client(context)
+
+
+class EC2Requester(object):
+
+    def __init__(self, version, http_method):
+        self.http_obj = httplib2.Http(
+            disable_ssl_certificate_validation=True)
+        self.version = version
+        self.method = http_method
+
+    def request(self, context, action, args):
+        headers = {
+            'content-type': 'application/x-www-form-urlencoded',
+            'connection': 'close',
+        }
+        params = args
+        params['Action'] = action
+        params['Version'] = self.version
+        self._add_auth(context, params)
+        params = self._get_query_string(params)
+
+        if self.method == 'POST':
+            url = self._ec2_url
+            body = params
+        else:
+            url = '?'.join((self._ec2_url, params,))
+            body = None
+
+        response, content = self.http_obj.request(url, self.method,
+                                                  body=body, headers=headers)
+        return response, content
+
+    _ec2_url = '%s://%s:%s%s' % (CONF.base_ec2_scheme,
+                                 CONF.base_ec2_host,
+                                 CONF.base_ec2_port,
+                                 CONF.base_ec2_path)
+
+    @staticmethod
+    def _get_query_string(params):
+        pairs = []
+        for key in sorted(params):
+            value = params[key]
+            pairs.append(urllib.quote(key.encode('utf-8'), safe='') + '=' +
+                         urllib.quote(value.encode('utf-8'), safe='-_~'))
+        return '&'.join(pairs)
+
+    def _calc_signature(self, context, params):
+        LOG.debug('Calculating signature using v2 auth.')
+        split = urlparse.urlsplit(self._ec2_url)
+        path = split.path
+        if len(path) == 0:
+            path = '/'
+        string_to_sign = '%s\n%s\n%s\n' % (self.method,
+                                           split.netloc,
+                                           path)
+        secret = context.secret_key
+        lhmac = hmac.new(secret.encode('utf-8'), digestmod=hashlib.sha256)
+        string_to_sign += self._get_query_string(params)
+        LOG.debug('String to sign: %s', string_to_sign)
+        lhmac.update(string_to_sign.encode('utf-8'))
+        b64 = base64.b64encode(lhmac.digest()).strip().decode('utf-8')
+        return b64
+
+    def _add_auth(self, context, params):
+        params['AWSAccessKeyId'] = context.access_key
+        params['SignatureVersion'] = '2'
+        params['SignatureMethod'] = 'HmacSHA256'
+        params['Timestamp'] = time.strftime(ISO8601, time.gmtime())
+        signature = self._calc_signature(context, params)
+        params['Signature'] = signature
+
+
+class EC2Client(object):
+
+    def __init__(self, context):
+        self.context = context
+        self.requester = EC2Requester(context.api_version, 'POST')
+
+    def __getattr__(self, name):
+        ec2_name = self._underscore_to_camelcase(name)
+
+        def func(self, **kwargs):
+            params = self._build_params(**kwargs)
+            response, content = self.requester.request(self.context, ec2_name,
+                                                       params)
+            return self._process_response(response, content)
+
+        func.__name__ = name
+        setattr(self, name, types.MethodType(func, self, self.__class__))
+        setattr(self.__class__, name,
+                types.MethodType(func, None, self.__class__))
+        return getattr(self, name)
+
+    @staticmethod
+    def _process_response(response, content):
+        if response.status > 200:
+            raise exception.EC2ServerError(response, content)
+
+        res = EC2Client._parse_xml(content)
+
+        res = next(res.itervalues())
+        if 'return' in res:
+            return res['return']
+        else:
+            res.pop('requestId')
+            return res
+
+    @staticmethod
+    def _build_params(**kwargs):
+        def add_list_param(params, items, label):
+            for i in range(1, len(items) + 1):
+                item = items[i - 1]
+                item_label = '%s.%d' % (label, i)
+                if isinstance(item, dict):
+                    add_dict_param(params, item, item_label)
+                else:
+                    params[item_label] = str(item)
+
+        def add_dict_param(params, items, label=None):
+            for key, value in items.iteritems():
+                ec2_key = EC2Client._underscore_to_camelcase(key)
+                item_label = '%s.%s' % (label, ec2_key) if label else ec2_key
+                if isinstance(value, dict):
+                    add_dict_param(params, value, item_label)
+                elif isinstance(value, list):
+                    add_list_param(params, value, item_label)
+                else:
+                    params[item_label] = str(value)
+
+        params = {}
+        add_dict_param(params, kwargs)
+        return params
+
+    _xml_scheme = re.compile('\sxmlns=".*"')
+
+    @staticmethod
+    # NOTE(ft): this function is used in unit tests until it be moved to one
+    # of utils module
+    def _parse_xml(xml_string):
+        xml_string = EC2Client._xml_scheme.sub('', xml_string)
+        xml = etree.fromstring(xml_string)
+
+        def convert_node(node):
+            children = list(node)
+            if len(children):
+                if children[0].tag == 'item':
+                    val = list(convert_node(child)[1] for child in children)
+                else:
+                    val = dict(convert_node(child) for child in children)
+            elif node.tag.endswith('Set'):
+                val = []
+            else:
+                # TODO(ft): do not use private function
+                val = (ec2utils._try_convert(node.text)
+                       if node.text
+                       else node.text)
+            return node.tag, val
+
+        return dict([convert_node(xml)])
+
+    @staticmethod
+    # NOTE(ft): this function is copied from apirequest to avoid circular
+    # module reference. It should be moved to one of utils module
+    def _underscore_to_camelcase(st):
+        return ''.join([x[:1].upper() + x[1:] for x in st.split('_')])

ec2api/api/ec2utils.py

@@ -0,0 +1,186 @@
+#    Copyright 2014 Cloudscaling Group, Inc
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+import re
+
+from ec2api import exception
+from ec2api.openstack.common.gettextutils import _
+from ec2api.openstack.common import log as logging
+from ec2api.openstack.common import timeutils
+
+LOG = logging.getLogger(__name__)
+
+
+def resource_type_from_id(context, resource_id):
+    """Get resource type by ID
+
+    Returns a string representation of the Amazon resource type, if known.
+    Returns None on failure.
+
+    :param context: context under which the method is called
+    :param resource_id: resource_id to evaluate
+    """
+
+    known_types = {
+        'i': 'instance',
+        'r': 'reservation',
+        'vol': 'volume',
+        'snap': 'snapshot',
+        'ami': 'image',
+        'aki': 'image',
+        'ari': 'image'
+    }
+
+    type_marker = resource_id.split('-')[0]
+
+    return known_types.get(type_marker)
+
+
+_c2u = re.compile('(((?<=[a-z])[A-Z])|([A-Z](?![A-Z]|$)))')
+
+
+def camelcase_to_underscore(str):
+    return _c2u.sub(r'_\1', str).lower().strip('_')
+
+
+def _try_convert(value):
+    """Return a non-string from a string or unicode, if possible.
+
+    ============= =====================================================
+    When value is returns
+    ============= =====================================================
+    zero-length   ''
+    'None'        None
+    'True'        True case insensitive
+    'False'       False case insensitive
+    '0', '-0'     0
+    0xN, -0xN     int from hex (positive) (N is any number)
+    0bN, -0bN     int from binary (positive) (N is any number)
+    *             try conversion to int, float, complex, fallback value
+
+    """
+    def _negative_zero(value):
+        epsilon = 1e-7
+        return 0 if abs(value) < epsilon else value
+
+    if len(value) == 0:
+        return ''
+    if value == 'None':
+        return None
+    lowered_value = value.lower()
+    if lowered_value == 'true':
+        return True
+    if lowered_value == 'false':
+        return False
+    for prefix, base in [('0x', 16), ('0b', 2), ('0', 8), ('', 10)]:
+        try:
+            if lowered_value.startswith((prefix, "-" + prefix)):
+                return int(lowered_value, base)
+        except ValueError:
+            pass
+    try:
+        return _negative_zero(float(value))
+    except ValueError:
+        return value
+
+
+def dict_from_dotted_str(items):
+    """parse multi dot-separated argument into dict.
+
+    EBS boot uses multi dot-separated arguments like
+    BlockDeviceMapping.1.DeviceName=snap-id
+    Convert the above into
+    {'block_device_mapping': {'1': {'device_name': snap-id}}}
+    """
+    args = {}
+    for key, value in items:
+        parts = key.split(".")
+        key = str(camelcase_to_underscore(parts[0]))
+        if isinstance(value, str) or isinstance(value, unicode):
+            # NOTE(vish): Automatically convert strings back
+            #             into their respective values
+            value = _try_convert(value)
+
+            if len(parts) > 1:
+                d = args.get(key, {})
+                args[key] = d
+                for k in parts[1:-1]:
+                    k = camelcase_to_underscore(k)
+                    v = d.get(k, {})
+                    d[k] = v
+                    d = v
+                d[camelcase_to_underscore(parts[-1])] = value
+            else:
+                args[key] = value
+
+    return args
+
+
+_ms_time_regex = re.compile('^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3,6}Z$')
+
+
+def is_ec2_timestamp_expired(request, expires=None):
+    """Checks the timestamp or expiry time included in an EC2 request
+
+    and returns true if the request is expired
+    """
+    query_time = None
+    timestamp = request.get('Timestamp')
+    expiry_time = request.get('Expires')
+
+    def parse_strtime(strtime):
+        if _ms_time_regex.match(strtime):
+            # NOTE(MotoKen): time format for aws-sdk-java contains millisecond
+            time_format = "%Y-%m-%dT%H:%M:%S.%fZ"
+        else:
+            time_format = "%Y-%m-%dT%H:%M:%SZ"
+        return timeutils.parse_strtime(strtime, time_format)
+
+    try:
+        if timestamp and expiry_time:
+            msg = _("Request must include either Timestamp or Expires,"
+                    " but cannot contain both")
+            LOG.error(msg)
+            raise exception.InvalidRequest(msg)
+        elif expiry_time:
+            query_time = parse_strtime(expiry_time)
+            return timeutils.is_older_than(query_time, -1)
+        elif timestamp:
+            query_time = parse_strtime(timestamp)
+
+            # Check if the difference between the timestamp in the request
+            # and the time on our servers is larger than 5 minutes, the
+            # request is too old (or too new).
+            if query_time and expires:
+                return (timeutils.is_older_than(query_time, expires) or
+                        timeutils.is_newer_than(query_time, expires))
+        return False
+    except ValueError:
+        LOG.audit(_("Timestamp is invalid."))
+        return True
+
+
+# TODO(Alex) This function is copied as is from original cloud.py. It doesn't
+# check for the prefix which allows any prefix used for any object.
+def ec2_id_to_id(ec2_id):
+    """Convert an ec2 ID (i-[base 16 number]) to an instance id (int)."""
+    try:
+        return int(ec2_id.split('-')[-1], 16)
+    except ValueError:
+        raise exception.InvalidEc2Id(ec2_id=ec2_id)
+
+
+def id_to_ec2_id(instance_id, template='i-%08x'):
+    """Convert an instance ID (int) to an ec2 ID (i-[base 16 number])."""
+    return template % int(instance_id)

ec2api/api/faults.py

@@ -0,0 +1,96 @@
+#    Copyright 2014 Cloudscaling Group, Inc
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from xml.sax import saxutils
+
+from oslo.config import cfg
+import webob.dec
+import webob.exc
+
+import ec2api.api
+from ec2api import context
+from ec2api.openstack.common import gettextutils
+from ec2api.openstack.common import log as logging
+
+CONF = cfg.CONF
+LOG = logging.getLogger(__name__)
+
+
+def xhtml_escape(value):
+    """Escapes a string so it is valid within XML or XHTML.
+
+    """
+    return saxutils.escape(value, {'"': '"', "'": '''})
+
+
+def utf8(value):
+    """Try to turn a string into utf-8 if possible.
+
+    Code is directly from the utf8 function in
+    http://github.com/facebook/tornado/blob/master/tornado/escape.py
+
+    """
+    if isinstance(value, unicode):
+        return value.encode('utf-8')
+    elif isinstance(value, gettextutils.Message):
+        return unicode(value).encode('utf-8')
+    assert isinstance(value, str)
+    return value
+
+
+def ec2_error_response(request_id, code, message, status=500):
+    """Helper to construct an EC2 compatible error response."""
+    LOG.debug('EC2 error response: %(code)s: %(message)s',
+              {'code': code, 'message': message})
+    resp = webob.Response()
+    resp.status = status
+    resp.headers['Content-Type'] = 'text/xml'
+    resp.body = str('<?xml version="1.0"?>\n'
+                    '<Response><Errors><Error><Code>%s</Code>'
+                    '<Message>%s</Message></Error></Errors>'
+                    '<RequestID>%s</RequestID></Response>' %
+                    (xhtml_escape(utf8(code)),
+                     xhtml_escape(utf8(message)),
+                     xhtml_escape(utf8(request_id))))
+    return resp
+
+
+class Fault(webob.exc.HTTPException):
+    """Captures exception and return REST Response."""
+
+    def __init__(self, exception):
+        """Create a response for the given webob.exc.exception."""
+        self.wrapped_exc = exception
+
+    @webob.dec.wsgify
+    def __call__(self, req):
+        """Generate a WSGI response based on the exception passed to ctor."""
+        code = ec2api.api.exception_to_ec2code(self.wrapped_exc)
+        status = self.wrapped_exc.status_int
+        message = self.wrapped_exc.explanation
+
+        if status == 501:
+            message = "The requested function is not supported"
+
+        if 'AWSAccessKeyId' not in req.params:
+            raise webob.exc.HTTPBadRequest()
+        user_id, _sep, project_id = req.params['AWSAccessKeyId'].partition(':')
+        project_id = project_id or user_id
+        remote_address = getattr(req, 'remote_address', '127.0.0.1')
+        if CONF.use_forwarded_for:
+            remote_address = req.headers.get('X-Forwarded-For', remote_address)
+
+        resp = ec2_error_response(context.generate_request_id(), code,
+                                  message=message, status=status)
+        return resp

ec2api/api/proxy.py

@@ -0,0 +1,27 @@
+#    Copyright 2014 Cloudscaling Group, Inc
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from ec2api.api import ec2client
+
+
+class ProxyController(object):
+
+    def __str__(self):
+        return 'ProxyController'
+
+    def proxy(self, req, args):
+        requester = ec2client.EC2Requester(req.params["Version"],
+                                           req.environ["REQUEST_METHOD"])
+        return requester.request(req.environ['ec2api.context'],
+                                 req.params["Action"], args)

ec2api/api/validator.py

@@ -0,0 +1,132 @@
+#    Copyright 2014 Cloudscaling Group, Inc
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+import base64
+import re
+
+from ec2api.openstack.common.gettextutils import _
+from ec2api.openstack.common import log as logging
+
+
+LOG = logging.getLogger(__name__)
+
+
+def _get_path_validator_regex():
+    # rfc3986 path validator regex from
+    # http://jmrware.com/articles/2009/uri_regexp/URI_regex.html
+    pchar = "([A-Za-z0-9\-._~!$&'()*+,;=:@]|%[0-9A-Fa-f]{2})"
+    path = "((/{pchar}*)*|"
+    path += "/({pchar}+(/{pchar}*)*)?|"
+    path += "{pchar}+(/{pchar}*)*|"
+    path += "{pchar}+(/{pchar}*)*|)"
+    path = path.format(pchar=pchar)
+    return re.compile(path)
+
+
+VALIDATE_PATH_RE = _get_path_validator_regex()
+
+
+def validate_str(max_length=None):
+
+    def _do(val):
+        if not isinstance(val, basestring):
+            return False
+        if max_length and len(val) > max_length:
+            return False
+        return True
+
+    return _do
+
+
+def validate_int(max_value=None):
+
+    def _do(val):
+        if not isinstance(val, int):
+            return False
+        if max_value and val > max_value:
+            return False
+        return True
+
+    return _do
+
+
+def validate_url_path(val):
+    """True if val is matched by the path component grammar in rfc3986."""
+
+    if not validate_str()(val):
+        return False
+
+    return VALIDATE_PATH_RE.match(val).end() == len(val)
+
+
+def validate_image_path(val):
+    if not validate_str()(val):
+        return False
+
+    bucket_name = val.split('/')[0]
+    manifest_path = val[len(bucket_name) + 1:]
+    if not len(bucket_name) or not len(manifest_path):
+        return False
+
+    if val[0] == '/':
+        return False
+
+    # make sure the image path if rfc3986 compliant
+    # prepend '/' to make input validate
+    if not validate_url_path('/' + val):
+        return False
+
+    return True
+
+
+def validate_user_data(user_data):
+    """Check if the user_data is encoded properly."""
+    try:
+        user_data = base64.b64decode(user_data)
+    except TypeError:
+        return False
+    return True
+
+
+def validate(args, validator):
+    """Validate values of args against validators in validator.
+
+    :param args:      Dict of values to be validated.
+    :param validator: A dict where the keys map to keys in args
+                      and the values are validators.
+                      Applies each validator to ``args[key]``
+    :returns: True if validation succeeds. Otherwise False.
+
+    A validator should be a callable which accepts 1 argument and which
+    returns True if the argument passes validation. False otherwise.
+    A validator should not raise an exception to indicate validity of the
+    argument.
+
+    Only validates keys which show up in both args and validator.
+
+    """
+
+    for key in validator:
+        if key not in args:
+            continue
+
+        f = validator[key]
+        assert callable(f)
+
+        if not f(args[key]):
+            LOG.debug(_("%(key)s with value %(value)s failed"
+                        " validator %(name)s"),
+                      {'key': key, 'value': args[key], 'name': f.__name__})
+            return False
+    return True

ec2api/cmd/__init__.py

@@ -0,0 +1,16 @@
+#    Copyright 2014 Cloudscaling Group, Inc
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from ec2api.openstack.common import gettextutils
+gettextutils.install('ec2api')

ec2api/cmd/api.py

@@ -0,0 +1,42 @@
+#    Copyright 2014 Cloudscaling Group, Inc
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+"""
+EC2api API Server
+"""
+
+import sys
+
+from oslo.config import cfg
+
+from ec2api import config
+from ec2api.openstack.common import log as logging
+from ec2api import service
+
+CONF = cfg.CONF
+CONF.import_opt('use_ssl', 'ec2api.service')
+
+
+def main():
+    config.parse_args(sys.argv)
+    logging.setup('ec2api')
+
+    server = service.WSGIService(
+        'ec2api', use_ssl=CONF.use_ssl, max_url_len=16384)
+    service.serve(server)
+    service.wait()
+
+
+if __name__ == '__main__':
+    main()

ec2api/cmd/manage.py

@@ -0,0 +1,75 @@
+#    Copyright 2013 Cloudscaling Group, Inc
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+
+"""
+  CLI interface for EC2 API management.
+"""
+
+import sys
+
+from oslo.config import cfg
+
+from ec2api.db import migration
+from ec2api.openstack.common import log
+from ec2api import version
+
+
+CONF = cfg.CONF
+
+
+def do_db_version():
+    """Print database's current migration level."""
+    print(migration.db_version())
+
+
+def do_db_sync():
+    """Place a database under migration control and upgrade,
+
+    creating if necessary.
+    """
+    migration.db_sync(CONF.command.version)
+
+
+def add_command_parsers(subparsers):
+    parser = subparsers.add_parser('db_version')
+    parser.set_defaults(func=do_db_version)
+
+    parser = subparsers.add_parser('db_sync')
+    parser.set_defaults(func=do_db_sync)
+    parser.add_argument('version', nargs='?')
+    parser.add_argument('current_version', nargs='?')
+
+
+command_opt = cfg.SubCommandOpt('command',
+                                title='Commands',
+                                help='Available commands',
+                                handler=add_command_parsers)
+
+
+def main():
+    CONF.register_cli_opt(command_opt)
+    try:
+        default_config_files = cfg.find_config_files('ec2api')
+        CONF(sys.argv[1:], project='ec2api', prog='ec2-api-manage',
+             version=version.version_info.version_string(),
+             default_config_files=default_config_files)
+        log.setup("ec2api")
+    except RuntimeError as e:
+        sys.exit("ERROR: %s" % e)
+
+    try:
+        CONF.command.func()
+    except Exception as e:
+        sys.exit("ERROR: %s" % e)

ec2api/config.py

@@ -0,0 +1,30 @@
+#    Copyright 2014 Cloudscaling Group, Inc
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from oslo.config import cfg
+
+from ec2api.openstack.common.db import options
+from ec2api import paths
+from ec2api import version
+
+_DEFAULT_SQL_CONNECTION = 'sqlite:///' + paths.state_path_def('nova.sqlite')
+
+
+def parse_args(argv, default_config_files=None):
+    options.set_defaults(sql_connection=_DEFAULT_SQL_CONNECTION,
+                         sqlite_db='nova.sqlite')
+    cfg.CONF(argv[1:],
+             project='ec2api',
+             version=version.version_info.version_string(),
+             default_config_files=default_config_files)

ec2api/context.py

@@ -0,0 +1,150 @@
+#    Copyright 2014 Cloudscaling Group, Inc
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+"""RequestContext: context for requests that persist through all of ec2."""
+
+import uuid
+
+import six
+
+from ec2api import exception
+from ec2api.openstack.common.gettextutils import _
+from ec2api.openstack.common import local
+from ec2api.openstack.common import log as logging
+from ec2api.openstack.common import timeutils
+
+
+LOG = logging.getLogger(__name__)
+
+
+def generate_request_id():
+    return 'req-' + str(uuid.uuid4())
+
+
+class RequestContext(object):
+    """Security context and request information.
+
+    Represents the user taking a given action within the system.
+
+    """
+
+    def __init__(self, user_id, project_id, access_key, secret_key,
+                 is_admin=None, roles=None, remote_address=None,
+                 auth_token=None, user_name=None, project_name=None,
+                 overwrite=True, service_catalog=None, api_version=None,
+                 **kwargs):
+        """Parameters
+
+            :param overwrite: Set to False to ensure that the greenthread local
+                copy of the index is not overwritten.
+
+
+            :param kwargs: Extra arguments that might be present, but we ignore
+                because they possibly came in from older rpc messages.
+        """
+        if kwargs:
+            LOG.warn(_('Arguments dropped when creating context: %s') %
+                    str(kwargs))
+
+        self.user_id = user_id
+        self.project_id = project_id
+        self.access_key = access_key
+        self.secret_key = secret_key
+        self.roles = roles or []
+        self.remote_address = remote_address
+        timestamp = timeutils.utcnow()
+        if isinstance(timestamp, six.string_types):
+            timestamp = timeutils.parse_strtime(timestamp)
+        self.timestamp = timestamp
+        self.request_id = generate_request_id()
+        self.auth_token = auth_token
+
+        self.service_catalog = service_catalog
+        if self.service_catalog is None:
+            # if list is empty or none
+            self.service_catalog = []
+
+        self.user_name = user_name
+        self.project_name = project_name
+        self.is_admin = is_admin
+        self.api_version = api_version
+        if overwrite or not hasattr(local.store, 'context'):
+            self.update_store()
+
+    def update_store(self):
+        local.store.context = self
+
+    def to_dict(self):
+        return {'user_id': self.user_id,
+                'project_id': self.project_id,
+                'is_admin': self.is_admin,
+                'roles': self.roles,
+                'remote_address': self.remote_address,
+                'timestamp': timeutils.strtime(self.timestamp),
+                'request_id': self.request_id,
+                'auth_token': self.auth_token,
+                'user_name': self.user_name,
+                'service_catalog': self.service_catalog,
+                'project_name': self.project_name,
+                'tenant': self.tenant,
+                'user': self.user}
+
+    @classmethod
+    def from_dict(cls, values):
+        values.pop('user', None)
+        values.pop('tenant', None)
+        return cls(**values)
+
+    # NOTE(sirp): the openstack/common version of RequestContext uses
+    # tenant/user whereas the ec2 version uses project_id/user_id. We need
+    # this shim in order to use context-aware code from openstack/common, like
+    # logging, until we make the switch to using openstack/common's version of
+    # RequestContext.
+    @property
+    def tenant(self):
+        return self.project_id
+
+    @property
+    def user(self):
+        return self.user_id
+
+
+def get_admin_context(read_deleted="no"):
+    return RequestContext(user_id=None,
+                          project_id=None,
+                          access_key=None,
+                          secret_key=None,
+                          is_admin=True,
+                          read_deleted=read_deleted,
+                          overwrite=False)
+
+
+def is_user_context(context):
+    """Indicates if the request context is a normal user."""
+    if not context:
+        return False
+    if context.is_admin:
+        return False
+    if not context.user_id or not context.project_id:
+        return False
+    return True
+
+
+def require_context(ctxt):
+    """Raise exception.Forbidden()
+
+    if context is not a user or an admin context.
+    """
+    if not ctxt.is_admin and not is_user_context(ctxt):
+        raise exception.Forbidden()

ec2api/exception.py

@@ -0,0 +1,279 @@
+#    Copyright 2014 Cloudscaling Group, Inc
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+"""ec2api base exception handling.
+
+Includes decorator for re-raising ec2api-type exceptions.
+
+SHOULD include dedicated exception logging.
+
+"""
+
+import sys
+
+from oslo.config import cfg
+
+from ec2api.openstack.common.gettextutils import _
+from ec2api.openstack.common import log as logging
+
+LOG = logging.getLogger(__name__)
+
+exc_log_opts = [
+    cfg.BoolOpt('fatal_exception_format_errors',
+                default=False,
+                help='Make exception message format errors fatal'),
+]
+
+CONF = cfg.CONF
+CONF.register_opts(exc_log_opts)
+
+
+class EC2ServerError(Exception):
+
+    def __init__(self, response, content):
+        self.response = response
+        self.content = content
+
+
+class EC2Exception(Exception):
+
+    """Base EC2 Exception
+
+    To correctly use this class, inherit from it and define
+    a 'msg_fmt' property. That msg_fmt will get printf'd
+    with the keyword arguments provided to the constructor.
+
+    """
+    msg_fmt = _("An unknown exception occurred.")