Initial EC2-API service commit.

This code introduces standalone service which proxies its calls to
existing nova EC2-API.
All the code here except for the ec2api/api/proxy.py,
ec2api/api/ec2client.py and some util functions is taken from current
nova and unused functionality is cut of it.
The proxy.py and ec2client.py files implement the new code which
proxies incoming request (on port 8788) to original EC2 API in nova
on port 8773.
The result is transparently translated back to user.

Change-Id: I4cb84f833d7d4f0e379672710ed39562811d43e0
changes/09/106209/32
Alexandre Levine 9 years ago committed by alevine
parent 1592a74ea3
commit 66826e9e5b

@ -0,0 +1,4 @@
[DEFAULT]
test_command=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_LOG_CAPTURE=1 ${PYTHON:-python} -m subunit.run discover -t ./ ec2api/tests $LISTOPT $IDOPTION
test_id_option=--load-list $IDFILE
test_list_option=--list

@ -0,0 +1,43 @@
Ec2api Style Commandments
=========================
- Step 1: Read the OpenStack Style Commandments
https://github.com/openstack-dev/hacking/blob/master/doc/source/index.rst
- Step 2: Read on
Ec2api Specific Commandments
----------------------------
General
-------
- Do not use locals(). Example::
LOG.debug(_("volume %(vol_name)s: creating size %(vol_size)sG") %
locals()) # BAD
LOG.debug(_("volume %(vol_name)s: creating size %(vol_size)sG") %
{'vol_name': vol_name,
'vol_size': vol_size}) # OKAY
- Use 'raise' instead of 'raise e' to preserve original traceback or exception being reraised::
except Exception as e:
...
raise e # BAD
except Exception:
...
raise # OKAY
Creating Unit Tests
-------------------
For every new feature, unit tests should be created that both test and
(implicitly) document the usage of said feature. If submitting a patch for a
bug that had no unit test, a new passing unit test should be added. If a
submitted bug fix does have a unit test, be sure to add a new one that fails
without the patch and passes with the patch.
For more information on creating unit tests and utilizing the testing
infrastructure in OpenStack Ec2api, please read ec2api/testing/README.rst.

@ -0,0 +1,176 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.

@ -0,0 +1,20 @@
include run_tests.sh ChangeLog
include README.rst builddeb.sh
include MANIFEST.in pylintrc
include AUTHORS
include run_tests.py
include HACKING.rst
include LICENSE
include ChangeLog
include babel.cfg tox.ini
include openstack-common.conf
include ec2api/openstack/common/README
include ec2api/db/sqlalchemy/migrate_repo/README
include ec2api/db/sqlalchemy/migrate_repo/migrate.cfg
include ec2api/db/sqlalchemy/migrate_repo/versions/*.sql
graft doc
graft etc
graft ec2api/locale
graft ec2api/tests
graft tools
global-exclude *.pyc

@ -0,0 +1,68 @@
OpenStack EC2 API README
-----------------------------
Support of EC2 API for OpenStack.
This project provides a standalone EC2 API service which pursues two goals:
1. Implement VPC API which now absent in nova's EC2 API
2. Create a standalone service for EC2 API support which later can accommodate
not only the VPC API but the rest of the EC2 API currently present in nova as
well.
This service implements VPC API related commands only. For the rest of the
EC2 API functionality it redirects request to original EC2 API in nova.
It doesn't replace existing nova EC2 API service in deployment it gets
installed to a different port (8788 by default).
Installation
=====
Run install.sh
#TODO: The following should be automated later.
Change /etc/ec2api/ec2api.conf:
[database]
connection_nova = <connection to nova> #should be taken from nova.conf
[DEFAULT]
external_network = <public network name> #obtained by neutron net-external-list
The service gets installed on port 8788 by default. It can be changed before the
installation in install.sh script.
Usage
=====
Download aws cli from Amazon.
Create configuration file for aws cli in your home directory ~/.aws/config:
[default]
aws_access_key_id = 1b013f18d5ed47ae8ed0fbb8debc036b
aws_secret_access_key = 9bbc6f270ffd4dfdbe0e896947f41df3
region = us-east-1
Change the aws_access_key_id and aws_secret_acces_key above to the values
appropriate for your cloud (can be obtained by "keystone ec2-credentials-list"
command).
Run aws cli commands using new EC2 API endpoint URL (can be obtained from
keystone with the new port 8788) like this:
aws --endpoint-url http://10.0.2.15:8788/services/Cloud ec2 describe-instances
Limitations
===========
This is an alpha-version, Tempest tests are not run yet.
VPN-related functionality is not supported yet.
Route-tables functionality is limited.
Filtering in describe functions can be done by IDs only.
Security groups are attached to network interfaces only, not to instances yet.
Rollbacks in case of failure during object creation are not supported yet.
Some other not-listed here limitations exist also.
Supported Features
==================
VPC API except for the Limitations above is supported.

@ -0,0 +1 @@
[python: **.py]

@ -0,0 +1,292 @@
#!/bin/bash -e
#
# Copyright 2014 Cloudscaling Group, Inc
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#
# Print --help output and exit.
#
usage() {
cat << EOF
Set up a local MySQL database for use with ec2api.
This script will create a 'ec2api' database that is accessible
only on localhost by user 'ec2api' with password 'ec2api'.
Usage: ec2api-db-setup <rpm|deb> [options]
Options:
select a distro type (rpm or debian)
--help | -h
Print usage information.
--password <pw> | -p <pw>
Specify the password for the 'ec2api' MySQL user that will
use to connect to the 'ec2api' MySQL database. By default,
the password 'ec2api' will be used.
--rootpw <pw> | -r <pw>
Specify the root MySQL password. If the script installs
the MySQL server, it will set the root password to this value
instead of prompting for a password. If the MySQL server is
already installed, this password will be used to connect to the
database instead of having to prompt for it.
--yes | -y
In cases where the script would normally ask for confirmation
before doing something, such as installing mysql-server,
just assume yes. This is useful if you want to run the script
non-interactively.
EOF
exit 0
}
install_mysql_server() {
if [ -z "${ASSUME_YES}" ] ; then
$PACKAGE_INSTALL mysql-server
else
$PACKAGE_INSTALL -y mysql-server
fi
}
start_mysql_server() {
$SERVICE_START
}
MYSQL_EC2API_PW_DEFAULT="ec2api"
MYSQL_EC2API_PW=${MYSQL_EC2API_PW_DEFAULT}
EC2API_CONFIG="/etc/ec2api/ec2api.conf"
ASSUME_YES=""
ELEVATE=""
# Check for root privileges
if [[ $EUID -ne 0 ]] ; then
echo "This operation requires superuser privileges, using sudo:"
if sudo -l > /dev/null ; then
ELEVATE="sudo"
else
exit 1
fi
fi
case "$1" in
rpm)
echo "Installing on an RPM system."
PACKAGE_INSTALL="$ELEVATE yum install"
PACKAGE_STATUS="rpm -q"
SERVICE_MYSQLD="mysqld"
SERVICE_START="$ELEVATE service $SERVICE_MYSQLD start"
SERVICE_STATUS="service $SERVICE_MYSQLD status"
SERVICE_ENABLE="$ELEVATE chkconfig"
;;
deb)
echo "Installing on a Debian system."
PACKAGE_INSTALL="$ELEVATE apt-get install"
PACKAGE_STATUS="dpkg-query -s"
SERVICE_MYSQLD="mysql"
SERVICE_START="$ELEVATE service $SERVICE_MYSQLD start"
SERVICE_STATUS="$ELEVATE service $SERVICE_MYSQLD status"
SERVICE_ENABLE=""
;;
*)
usage
;;
esac
while [ $# -gt 0 ]
do
case "$1" in
-h|--help)
usage
;;
-p|--password)
shift
MYSQL_EC2API_PW=${1}
;;
-r|--rootpw)
shift
MYSQL_ROOT_PW=${1}
;;
-y|--yes)
ASSUME_YES="yes"
;;
*)
# ignore
;;
esac
shift
done
# Make sure MySQL is installed.
NEW_MYSQL_INSTALL=0
if ! $PACKAGE_STATUS mysql-server && ! $PACKAGE_STATUS mariadb-server && ! $PACKAGE_STATUS mariadb-galera-server > /dev/null
then
if [ -z "${ASSUME_YES}" ] ; then
printf "mysql-server is not installed. Would you like to install it now? (y/n): "
read response
case "$response" in
y|Y)
;;
n|N)
echo "mysql-server must be installed. Please install it before proceeding."
exit 0
;;
*)
echo "Invalid response."
exit 1
esac
fi
NEW_MYSQL_INSTALL=1
install_mysql_server
fi
# Make sure mysqld is running.
if ! $SERVICE_STATUS > /dev/null
then
if [ -z "${ASSUME_YES}" ] ; then
printf "$SERVICE_MYSQLD is not running. Would you like to start it now? (y/n): "
read response
case "$response" in
y|Y)
;;
n|N)
echo "$SERVICE_MYSQLD must be running. Please start it before proceeding."
exit 0
;;
*)
echo "Invalid response."
exit 1
esac
fi
start_mysql_server
# If we both installed and started, ensure it starts at boot
[ $NEW_MYSQL_INSTALL -eq 1 ] && $SERVICE_ENABLE $SERVICE_MYSQLD on
fi
# Get MySQL root access.
if [ $NEW_MYSQL_INSTALL -eq 1 ]
then
if [ ! "${MYSQL_ROOT_PW+defined}" ] ; then
echo "Since this is a fresh installation of MySQL, please set a password for the 'root' mysql user."
PW_MATCH=0
while [ $PW_MATCH -eq 0 ]
do
printf "Enter new password for 'root' mysql user: "
read -s MYSQL_ROOT_PW
echo
printf "Enter new password again: "
read -s PW2
echo
if [ "${MYSQL_ROOT_PW}" = "${PW2}" ] ; then
PW_MATCH=1
else
echo "Passwords did not match."
fi
done
fi
echo "UPDATE mysql.user SET password = password('${MYSQL_ROOT_PW}') WHERE user = 'root'; DELETE FROM mysql.user WHERE user = ''; flush privileges;" | mysql -u root
if ! [ $? -eq 0 ] ; then
echo "Failed to set password for 'root' MySQL user."
exit 1
fi
elif [ ! "${MYSQL_ROOT_PW+defined}" ] ; then
printf "Please enter the password for the 'root' MySQL user: "
read -s MYSQL_ROOT_PW
echo
fi
# Sanity check MySQL credentials.
MYSQL_ROOT_PW_ARG=""
if [ "${MYSQL_ROOT_PW+defined}" ]
then
MYSQL_ROOT_PW_ARG="--password=${MYSQL_ROOT_PW}"
fi
echo "SELECT 1;" | mysql -u root ${MYSQL_ROOT_PW_ARG} > /dev/null
if ! [ $? -eq 0 ]
then
echo "Failed to connect to the MySQL server. Please check your root user credentials."
exit 1
fi
echo "Verified connectivity to MySQL."
# Now create the db.
echo "Creating 'ec2api' database."
cat << EOF | mysql -u root ${MYSQL_ROOT_PW_ARG}
DROP DATABASE IF EXISTS ec2api;
CREATE DATABASE IF NOT EXISTS ec2api DEFAULT CHARACTER SET utf8;
GRANT ALL ON ec2api.* TO 'ec2api'@'localhost' IDENTIFIED BY '${MYSQL_EC2API_PW}';
GRANT ALL ON ec2api.* TO 'ec2api'@'%' IDENTIFIED BY '${MYSQL_EC2API_PW}';
flush privileges;
EOF
# Make sure ec2api configuration has the right MySQL password.
if [ "${MYSQL_EC2API_PW}" != "${MYSQL_EC2API_PW_DEFAULT}" ] ; then
echo "Updating 'ec2api' database password in ${EC2API_CONFIG}"
sed -i -e "s/mysql:\/\/ec2api:\(.*\)@/mysql:\/\/ec2api:${MYSQL_EC2API_PW}@/" ${EC2API_CONFIG}
fi
# override the logging config in ec2api.conf
log_conf=$(mktemp /tmp/ec2api-logging.XXXXXXXXXX.conf)
cat <<EOF > $log_conf
[loggers]
keys=root
[handlers]
keys=consoleHandler
[formatters]
keys=simpleFormatter
[logger_root]
level=INFO
handlers=consoleHandler
[handler_consoleHandler]
class=StreamHandler
formatter=simpleFormatter
args=(sys.stdout,)
[formatter_simpleFormatter]
format=%(name)s - %(levelname)s - %(message)s
EOF
ec2-api-manage --log-config=$log_conf db_sync
rm $log_conf
# Do a final sanity check on the database.
echo "SELECT * FROM migrate_version;" | mysql -u ec2api --password=${MYSQL_EC2API_PW} ec2api > /dev/null
if ! [ $? -eq 0 ]
then
echo "Final sanity check failed."
exit 1
fi
echo "Complete!"

@ -0,0 +1,18 @@
<?xml version="1.0" encoding="UTF-8"?>
<projectDescription>
<name>ec2api</name>
<comment></comment>
<projects>
</projects>
<buildSpec>
<buildCommand>
<name>org.python.pydev.PyDevBuilder</name>
<arguments>
</arguments>
</buildCommand>
</buildSpec>
<natures>
<nature>com.aptana.projects.webnature</nature>
<nature>org.python.pydev.pythonNature</nature>
</natures>
</projectDescription>

@ -0,0 +1,5 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?eclipse-pydev version="1.0"?><pydev_project>
<pydev_property name="org.python.pydev.PYTHON_PROJECT_INTERPRETER">Default</pydev_property>
<pydev_property name="org.python.pydev.PYTHON_PROJECT_VERSION">python 2.7</pydev_property>
</pydev_project>

@ -0,0 +1,27 @@
# Copyright 2014 Cloudscaling Group, Inc
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""
:mod:`ec2api` -- Cloud IaaS Platform
===================================
.. automodule:: ec2api
:platform: Unix
:synopsis: Infrastructure-as-a-Service Cloud platform.
"""
import gettext
gettext.install('ec2api', unicode=1)

@ -0,0 +1,400 @@
# Copyright 2014 Cloudscaling Group, Inc
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""
Starting point for routing EC2 requests.
"""
from eventlet.green import httplib
import netaddr
from oslo.config import cfg
import six
import six.moves.urllib.parse as urlparse
import webob
import webob.dec
import webob.exc
from ec2api.api import apirequest
from ec2api.api import ec2utils
from ec2api.api import faults
from ec2api.api import validator
from ec2api import context
from ec2api import exception
from ec2api.openstack.common.gettextutils import _
from ec2api.openstack.common import jsonutils
from ec2api.openstack.common import log as logging
from ec2api.openstack.common import timeutils
from ec2api import wsgi
LOG = logging.getLogger(__name__)
ec2_opts = [
cfg.StrOpt('keystone_url',
default='http://localhost:5000/v2.0',
help='URL to get token from ec2 request.'),
cfg.IntOpt('ec2_timestamp_expiry',
default=300,
help='Time in seconds before ec2 timestamp expires'),
]
CONF = cfg.CONF
CONF.register_opts(ec2_opts)
CONF.import_opt('use_forwarded_for', 'ec2api.api.auth')
# Fault Wrapper around all EC2 requests #
class FaultWrapper(wsgi.Middleware):
"""Calls the middleware stack, captures any exceptions into faults."""
@webob.dec.wsgify(RequestClass=wsgi.Request)
def __call__(self, req):
try:
return req.get_response(self.application)
except Exception as ex:
LOG.exception(_("FaultWrapper: %s"), unicode(ex))
return faults.Fault(webob.exc.HTTPInternalServerError())
class RequestLogging(wsgi.Middleware):
"""Access-Log akin logging for all EC2 API requests."""
@webob.dec.wsgify(RequestClass=wsgi.Request)
def __call__(self, req):
start = timeutils.utcnow()
rv = req.get_response(self.application)
self.log_request_completion(rv, req, start)
return rv
def log_request_completion(self, response, request, start):
apireq = request.environ.get('ec2.request', None)
if apireq:
action = apireq.action
else:
action = None
ctxt = request.environ.get('ec2api.context', None)
delta = timeutils.utcnow() - start
seconds = delta.seconds
microseconds = delta.microseconds
LOG.info(
"%s.%ss %s %s %s %s %s [%s] %s %s",
seconds,
microseconds,
request.remote_addr,
request.method,
"%s%s" % (request.script_name, request.path_info),
action,
response.status_int,
request.user_agent,
request.content_type,
response.content_type,
context=ctxt)
class EC2KeystoneAuth(wsgi.Middleware):
"""Authenticate an EC2 request with keystone and convert to context."""
@webob.dec.wsgify(RequestClass=wsgi.Request)
def __call__(self, req):
request_id = context.generate_request_id()
signature = req.params.get('Signature')
if not signature:
msg = _("Signature not provided")
return faults.ec2_error_response(request_id, "AuthFailure", msg,
status=400)
access = req.params.get('AWSAccessKeyId')
if not access:
msg = _("Access key not provided")
return faults.ec2_error_response(request_id, "AuthFailure", msg,
status=400)
# Make a copy of args for authentication and signature verification.
auth_params = dict(req.params)
# Not part of authentication args
auth_params.pop('Signature')
cred_dict = {
'access': access,
'signature': signature,
'host': req.host,
'verb': req.method,
'path': req.path,
'params': auth_params,
}
token_url = CONF.keystone_url + "/ec2tokens"
if "ec2" in token_url:
creds = {'ec2Credentials': cred_dict}
else:
creds = {'auth': {'OS-KSEC2:ec2Credentials': cred_dict}}
creds_json = jsonutils.dumps(creds)
headers = {'Content-Type': 'application/json'}
o = urlparse.urlparse(token_url)
if o.scheme == "http":
conn = httplib.HTTPConnection(o.netloc)
else:
conn = httplib.HTTPSConnection(o.netloc)
conn.request('POST', o.path, body=creds_json, headers=headers)
response = conn.getresponse()
data = response.read()
if response.status != 200:
if response.status == 401:
msg = response.reason
else:
msg = _("Failure communicating with keystone")
return faults.ec2_error_response(request_id, "AuthFailure", msg,
status=response.status)
result = jsonutils.loads(data)
conn.close()
try:
token_id = result['access']['token']['id']
user_id = result['access']['user']['id']
project_id = result['access']['token']['tenant']['id']
user_name = result['access']['user'].get('name')
project_name = result['access']['token']['tenant'].get('name')
roles = [role['name'] for role
in result['access']['user']['roles']]
except (AttributeError, KeyError) as e:
LOG.exception(_("Keystone failure: %s") % e)
msg = _("Failure communicating with keystone")
return faults.ec2_error_response(request_id, "AuthFailure", msg,
status=400)
remote_address = req.remote_addr
if CONF.use_forwarded_for:
remote_address = req.headers.get('X-Forwarded-For',
remote_address)
headers["X-Auth-Token"] = token_id
o = urlparse.urlparse(CONF.keystone_url
+ ("/users/%s/credentials/OS-EC2/%s" % (user_id, access)))
if o.scheme == "http":
conn = httplib.HTTPConnection(o.netloc)
else:
conn = httplib.HTTPSConnection(o.netloc)
conn.request('GET', o.path, headers=headers)
response = conn.getresponse()
data = response.read()
if response.status != 200:
if response.status == 401:
msg = response.reason
else:
msg = _("Failure communicating with keystone")
return faults.ec2_error_response(request_id, "AuthFailure", msg,
status=response.status)
ec2_creds = jsonutils.loads(data)
conn.close()
catalog = result['access']['serviceCatalog']
ctxt = context.RequestContext(user_id,
project_id,
ec2_creds["credential"]["access"],
ec2_creds["credential"]["secret"],
user_name=user_name,
project_name=project_name,
roles=roles,
auth_token=token_id,
remote_address=remote_address,
service_catalog=catalog,
api_version=req.params.get('Version'))
req.environ['ec2api.context'] = ctxt
return self.application
class Requestify(wsgi.Middleware):
def __init__(self, app):
super(Requestify, self).__init__(app)
@webob.dec.wsgify(RequestClass=wsgi.Request)
def __call__(self, req):
non_args = ['Action', 'Signature', 'AWSAccessKeyId', 'SignatureMethod',
'SignatureVersion', 'Version', 'Timestamp']
args = dict(req.params)
try:
expired = ec2utils.is_ec2_timestamp_expired(req.params,
expires=CONF.ec2_timestamp_expiry)
if expired:
msg = _("Timestamp failed validation.")
LOG.exception(msg)
raise webob.exc.HTTPForbidden(explanation=msg)
# Raise KeyError if omitted
action = req.params['Action']
# Fix bug lp:720157 for older (version 1) clients
version = req.params['SignatureVersion']
if int(version) == 1:
non_args.remove('SignatureMethod')
if 'SignatureMethod' in args:
args.pop('SignatureMethod')
for non_arg in non_args:
# Remove, but raise KeyError if omitted
args.pop(non_arg)
except KeyError:
raise webob.exc.HTTPBadRequest()
except exception.InvalidRequest as err:
raise webob.exc.HTTPBadRequest(explanation=unicode(err))
LOG.debug('action: %s', action)
for key, value in args.items():
LOG.debug('arg: %(key)s\t\tval: %(value)s',
{'key': key, 'value': value})
# Success!
api_request = apirequest.APIRequest(
action, req.params['Version'], args)
req.environ['ec2.request'] = api_request
return self.application
def validate_ec2_id(val):
if not validator.validate_str()(val):
return False
try:
ec2utils.ec2_id_to_id(val)
except exception.InvalidEc2Id:
return False
return True
def is_valid_ipv4(address):
"""Verify that address represents a valid IPv4 address."""
try:
return netaddr.valid_ipv4(address)
except Exception:
return False
class Validator(wsgi.Middleware):
validator.validate_ec2_id = validate_ec2_id
validator.DEFAULT_VALIDATOR = {
'instance_id': validate_ec2_id,
'volume_id': validate_ec2_id,
'image_id': validate_ec2_id,
'attribute': validator.validate_str(),
'image_location': validator.validate_image_path,
'public_ip': is_valid_ipv4,
'region_name': validator.validate_str(),
'group_name': validator.validate_str(max_length=255),
'group_description': validator.validate_str(max_length=255),
'size': validator.validate_int(),
'user_data': validator.validate_user_data
}
def __init__(self, application):
super(Validator, self).__init__(application)
@webob.dec.wsgify(RequestClass=wsgi.Request)
def __call__(self, req):
if validator.validate(req.environ['ec2.request'].args,
validator.DEFAULT_VALIDATOR):
return self.application
else:
raise webob.exc.HTTPBadRequest()
def exception_to_ec2code(ex):
"""Helper to extract EC2 error code from exception.
For other than EC2 exceptions (those without ec2_code attribute),
use exception name.
"""
if hasattr(ex, 'ec2_code'):
code = ex.ec2_code
else:
code = type(ex).__name__
return code
def ec2_error_ex(ex, req, code=None, message=None):
"""Return an EC2 error response based on passed exception and log it."""
if not code:
code = exception_to_ec2code(ex)
status = getattr(ex, 'code', None)
if not status:
status = 500
log_fun = LOG.error
log_msg = _("Unexpected %(ex_name)s raised: %(ex_str)s")
context = req.environ['ec2api.context']
request_id = context.request_id
log_msg_args = {
'ex_name': type(ex).__name__,
'ex_str': unicode(ex)
}
log_fun(log_msg % log_msg_args, context=context)
if ex.args and not message and status < 500:
message = unicode(ex.args[0])
# Log filtered environment for unexpected errors.
env = req.environ.copy()
for k in env.keys():
if not isinstance(env[k], six.string_types):
env.pop(k)
log_fun(_('Environment: %s') % jsonutils.dumps(env))
if not message:
message = _('Unknown error occurred.')
return faults.ec2_error_response(request_id, code, message, status=status)
class Executor(wsgi.Application):
"""Execute an EC2 API request.
Executes 'ec2.action', passing 'ec2api.context' and
'ec2.action_args' (all variables in WSGI environ.) Returns an XML
response, or a 400 upon failure.
"""
@webob.dec.wsgify(RequestClass=wsgi.Request)
def __call__(self, req):
context = req.environ['ec2api.context']
api_request = req.environ['ec2.request']
try:
result = api_request.invoke(context)
except exception.InstanceNotFound as ex:
ec2_id = ec2utils.id_to_ec2_inst_id(ex.kwargs['instance_id'])
message = ex.msg_fmt % {'instance_id': ec2_id}
return ec2_error_ex(ex, req, message=message)
except exception.MethodNotFound:
try:
http, response = api_request.proxy(req)
resp = webob.Response()
resp.status = http["status"]
resp.headers["content-type"] = http["content-type"]
resp.body = str(response)
return resp
except Exception as ex:
return ec2_error_ex(ex, req)
except exception.EC2ServerError as ex:
resp = webob.Response()
resp.status = ex.response['status']
resp.headers['Content-Type'] = ex.response['content-type']
resp.body = ex.content
return resp
except Exception as ex:
return ec2_error_ex(ex, req)
else:
resp = webob.Response()
resp.status = 200
resp.headers['Content-Type'] = 'text/xml'
resp.body = str(result)
return resp

@ -0,0 +1,143 @@
# Copyright 2014 Cloudscaling Group, Inc
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""
APIRequest class
"""
import datetime
# TODO(termie): replace minidom with etree
from xml.dom import minidom
from lxml import etree
from ec2api.api import cloud
from ec2api.api import ec2utils
from ec2api.api import proxy
from ec2api import exception
from ec2api.openstack.common import log as logging
LOG = logging.getLogger(__name__)
def _underscore_to_camelcase(st):
return ''.join([x[:1].upper() + x[1:] for x in st.split('_')])
def _underscore_to_xmlcase(st):
res = _underscore_to_camelcase(st)
return res[:1].lower() + res[1:]
def _database_to_isoformat(datetimeobj):
"""Return a xs:dateTime parsable string from datatime."""
return datetimeobj.strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + 'Z'
class APIRequest(object):
def __init__(self, action, version, args):
self.action = action
self.version = version
self.args = args
self.controller = cloud.CloudController()
self.proxyController = proxy.ProxyController()
def invoke(self, context):
try:
method = getattr(self.controller,
ec2utils.camelcase_to_underscore(self.action))
except AttributeError:
raise exception.MethodNotFound(name=self.action)
args = ec2utils.dict_from_dotted_str(self.args.items())
for key in args.keys():
# NOTE(vish): Turn numeric dict keys into lists
# NOTE(Alex): Turn "value"-only dict keys into values
if isinstance(args[key], dict):
if args[key] == {}:
continue
if args[key].keys()[0].isdigit():
s = args[key].items()
s.sort()
args[key] = [v for k, v in s]
elif args[key].keys()[0] == 'value' and len(args[key]) == 1:
args[key] = args[key]['value']
result = method(context, **args)
return self._render_response(result, context.request_id)
def proxy(self, req):
return self.proxyController.proxy(req, self.args)
def _render_response(self, response_data, request_id):
xml = minidom.Document()
response_el = xml.createElement(self.action + 'Response')
response_el.setAttribute('xmlns',
'http://ec2.amazonaws.com/doc/%s/' % self.version)
request_id_el = xml.createElement('requestId')
request_id_el.appendChild(xml.createTextNode(request_id))
response_el.appendChild(request_id_el)
if response_data is True:
self._render_dict(xml, response_el, {'return': 'true'})
else:
self._render_dict(xml, response_el, response_data)
xml.appendChild(response_el)
response = xml.toxml()
root = etree.fromstring(response)
response = etree.tostring(root, pretty_print=True)
xml.unlink()
# Don't write private key to log
if self.action != "CreateKeyPair":
LOG.debug(response)
else:
LOG.debug("CreateKeyPair: Return Private Key")
return response
def _render_dict(self, xml, el, data):
try:
for key in data.keys():
val = data[key]
el.appendChild(self._render_data(xml, key, val))
except Exception:
LOG.debug(data)
raise
def _render_data(self, xml, el_name, data):
el_name = _underscore_to_xmlcase(el_name)
data_el = xml.createElement(el_name)
if isinstance(data, list):
for item in data:
data_el.appendChild(self._render_data(xml, 'item', item))
elif isinstance(data, dict):
self._render_dict(xml, data_el, data)
elif hasattr(data, '__dict__'):
self._render_dict(xml, data_el, data.__dict__)
elif isinstance(data, bool):
data_el.appendChild(xml.createTextNode(str(data).lower()))
elif isinstance(data, datetime.datetime):
data_el.appendChild(
xml.createTextNode(_database_to_isoformat(data)))
elif data is not None:
data_el.appendChild(xml.createTextNode(str(data)))
return data_el

@ -0,0 +1,54 @@
# Copyright 2014 Cloudscaling Group, Inc
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""
Common Auth Middleware.
"""
from oslo.config import cfg
from ec2api.openstack.common import log as logging
auth_opts = [
cfg.BoolOpt('api_rate_limit',
default=False,
help='whether to use per-user rate limiting for the api.'),
cfg.BoolOpt('use_forwarded_for',
default=False,
help='Treat X-Forwarded-For as the canonical remote address. '
'Only enable this if you have a sanitizing proxy.'),
]
CONF = cfg.CONF
CONF.register_opts(auth_opts)
LOG = logging.getLogger(__name__)
def pipeline_factory(loader, global_conf, **local_conf):
"""A paste pipeline replica that keys off of auth_strategy."""
auth_strategy = "keystone"
pipeline = local_conf[auth_strategy]
if not CONF.api_rate_limit:
limit_name = auth_strategy + '_nolimit'
pipeline = local_conf.get(limit_name, pipeline)
pipeline = pipeline.split()
filters = [loader.get_filter(n) for n in pipeline[:-1]]
app = loader.get_app(pipeline[-1])
filters.reverse()
for fltr in filters:
app = fltr(app)
return app

@ -0,0 +1,141 @@
# Copyright 2014 Cloudscaling Group, Inc
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from keystoneclient.v2_0 import client as kc
from novaclient import client as novaclient
from novaclient import shell as novashell
from oslo.config import cfg
from ec2api.openstack.common.gettextutils import _
from ec2api.openstack.common import log as logging
logger = logging.getLogger(__name__)
CONF = cfg.CONF
try:
from neutronclient.v2_0 import client as neutronclient
except ImportError:
neutronclient = None
logger.info(_('neutronclient not available'))
try:
from cinderclient import client as cinderclient
except ImportError:
cinderclient = None
logger.info(_('cinderclient not available'))
try:
from glanceclient import client as glanceclient
except ImportError:
glanceclient = None
logger.info(_('glanceclient not available'))
def nova(context, service_type='compute'):
computeshell = novashell.OpenStackComputeShell()
extensions = computeshell._discover_extensions("1.1")
args = {
'project_id': context.project_id,
'auth_url': CONF.keystone_url,
'service_type': service_type,
'username': None,
'api_key': None,
'extensions': extensions,
}
client = novaclient.Client(1.1, **args)
management_url = _url_for(context, service_type=service_type)
client.client.auth_token = context.auth_token
client.client.management_url = management_url
return client
def neutron(context):
if neutronclient is None:
return None
args = {
'auth_url': CONF.keystone_url,
'service_type': 'network',
'token': context.auth_token,
'endpoint_url': _url_for(context, service_type='network'),
}
return neutronclient.Client(**args)
def glance(context):
if glanceclient is None:
return None
args = {
'auth_url': CONF.keystone_url,
'service_type': 'image',
'token': context.auth_token,
}
return glanceclient.Client(
"1", endpoint=_url_for(context, service_type='image'), **args)
def cinder(context):
if cinderclient is None:
return nova(context, 'volume')
args = {
'service_type': 'volume',
'auth_url': CONF.keystone_url,
'username': None,
'api_key': None,
}
_cinder = cinderclient.Client('1', **args)
management_url = _url_for(context, service_type='volume')
_cinder.client.auth_token = context.auth_token
_cinder.client.management_url = management_url
return _cinder
def keystone(context):
_keystone = kc.Client(
token=context.auth_token,
tenant_id=context.project_id,
auth_url=CONF.keystone_url)
return _keystone
def _url_for(context, **kwargs):
service_catalog = context.service_catalog
if not service_catalog:
catalog = keystone(context).service_catalog.catalog
service_catalog = catalog["serviceCatalog"]
context.service_catalog = service_catalog
service_type = kwargs["service_type"]
for service in service_catalog:
if service["type"] != service_type:
continue
for endpoint in service["endpoints"]:
if "publicURL" in endpoint:
return endpoint["publicURL"]
else:
return None
return None

@ -0,0 +1,41 @@
# Copyright 2014 Cloudscaling Group, Inc
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""
Cloud Controller: Implementation of EC2 REST API calls, which are
dispatched to other nodes via AMQP RPC. State is via distributed
datastore.
"""
from oslo.config import cfg
from ec2api.openstack.common import log as logging
CONF = cfg.CONF
LOG = logging.getLogger(__name__)
class CloudController(object):
"""Cloud Controller
Provides the critical dispatch between
inbound API calls through the endpoint and messages
sent to the other nodes.
"""
def __init__(self):
pass
def __str__(self):
return 'CloudController'

@ -0,0 +1,222 @@
# Copyright 2014 Cloudscaling Group, Inc
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import base64
import hashlib
import hmac
import re
import time
import types
import urllib
import urlparse
import httplib2
from lxml import etree
from oslo.config import cfg
from ec2api.api import ec2utils
from ec2api import exception
from ec2api.openstack.common import log as logging
ec2_opts = [
cfg.StrOpt('base_ec2_host',
default="localhost",
help='The IP address of the EC2 API server'),
cfg.IntOpt('base_ec2_port',
default=8773,
help='The port of the EC2 API server'),
cfg.StrOpt('base_ec2_scheme',
default='http',
help='The protocol to use when connecting to the EC2 API '
'server (http, https)'),
cfg.StrOpt('base_ec2_path',
default='/services/Cloud',
help='The path prefix used to call the ec2 API server'),
]
CONF = cfg.CONF
CONF.register_opts(ec2_opts)
LOG = logging.getLogger(__name__)
ISO8601 = '%Y-%m-%dT%H:%M:%SZ'
def ec2client(context):
return EC2Client(context)
class EC2Requester(object):
def __init__(self, version, http_method):
self.http_obj = httplib2.Http(
disable_ssl_certificate_validation=True)
self.version = version
self.method = http_method
def request(self, context, action, args):
headers = {
'content-type': 'application/x-www-form-urlencoded',
'connection': 'close',
}
params = args
params['Action'] = action
params['Version'] = self.version
self._add_auth(context, params)
params = self._get_query_string(params)
if self.method == 'POST':
url = self._ec2_url
body = params
else:
url = '?'.join((self._ec2_url, params,))
body = None
response, content = self.http_obj.request(url, self.method,
body=body, headers=headers)
return response, content
_ec2_url = '%s://%s:%s%s' % (CONF.base_ec2_scheme,
CONF.base_ec2_host,
CONF.base_ec2_port,
CONF.base_ec2_path)
@staticmethod
def _get_query_string(params):
pairs = []
for key in sorted(params):
value = params[key]
pairs.append(urllib.quote(key.encode('utf-8'), safe='') + '=' +
urllib.quote(value.encode('utf-8'), safe='-_~'))
return '&'.join(pairs)
def _calc_signature(self, context, params):
LOG.debug('Calculating signature using v2 auth.')
split = urlparse.urlsplit(self._ec2_url)
path = split.path
if len(path) == 0:
path = '/'
string_to_sign = '%s\n%s\n%s\n' % (self.method,
split.netloc,
path)
secret = context.secret_key
lhmac = hmac.new(secret.encode('utf-8'), digestmod=hashlib.sha256)
string_to_sign += self._get_query_string