Merge "Drop support for diablo to essex migrations"

This commit is contained in:
Jenkins 2013-08-29 04:17:14 +00:00 committed by Gerrit Code Review
commit e446e0ac52
6 changed files with 0 additions and 651 deletions

View File

@ -435,9 +435,6 @@ service catalog will not change very much over time.
The value of ``template_file`` is expected to be an absolute path to your
service catalog configuration. An example ``template_file`` is included in
Keystone, however you should create your own to reflect your deployment.
If you are migrating from a legacy deployment, a tool is available to help with
this task (see `Migrating your Service Catalog from legacy versions of
Keystone`_).
Another such example is `available in devstack
(files/default_catalog.templates)
@ -701,124 +698,6 @@ empty list from your new database)::
values, or deployed Keystone to a different endpoint, you will need to
change the provided command accordingly.
Migrating from legacy versions of Keystone
==========================================
Migration support is provided for the following legacy Keystone versions:
* diablo-5
* stable/diablo
* essex-2
* essex-3
.. NOTE::
Before you can import your legacy data, you must first
`prepare your deployment`_.
Step 1: Ensure your deployment can access your legacy database
--------------------------------------------------------------------
Your legacy ``keystone.conf`` contains a SQL configuration section called
``[keystone.backends.sqlalchemy]`` connection string which, by default,
looks like::
sql_connection = sqlite:///keystone.db
This connection string needs to be accessible from your deployment (e.g.
you may need to copy your SQLite ``*.db`` file to a new server, adjust the
relative path as appropriate, or open a firewall for MySQL, etc).
Step 2: Import your legacy data
-------------------------------
Use the following command to import your old data using the value of
``sql_connection`` from step 3::
$ keystone-manage import_legacy <sql_connection>
You should now be able to run the same command you used to test your new
database above, but now you'll see your legacy Keystone data::
$ keystone --token ADMIN --endpoint http://127.0.0.1:35357/v2.0/ tenant-list
+----------------------------------+----------------+---------+
| id | name | enabled |
+----------------------------------+----------------+---------+
| 12edde26a6224199a66ece67b762a065 | project-y | True |
| 593715ed4359404999915ea7005a7da1 | ANOTHER:TENANT | True |
| be57fed798b049bc9637d2be30bfa857 | coffee-tea | True |
| e3c382f4757a4385b502056431763cca | customer-x | True |
+----------------------------------+----------------+---------+
Migrating your Service Catalog from legacy versions of Keystone
===============================================================
While legacy Keystone deployments stored the service catalog in the database,
the service catalog is stored in a flat ``template_file``. An example
service catalog template file may be found in
``etc/default_catalog.templates``. You can change the path to your service
catalog template in ``keystone.conf`` by changing the value of
``[catalog] template_file``.
Import your legacy catalog and redirect the output to your ``template_file``::
$ keystone-manage export_legacy_catalog <sql_connection> > <template_file>
.. NOTE::
After executing this command, you will need to restart the Keystone
service to see your changes.
Migrating from Nova Auth
========================
Migration of users, projects (aka tenants), roles and EC2 credentials
is supported for the Essex and later releases of Nova. To migrate your auth
data from Nova, use the following steps:
.. NOTE::
Before you can migrate from nova auth, you must first
`prepare your deployment`_.
Step 1: Export your data from Nova
----------------------------------
Use the following command to export your data from Nova to a ``dump_file``::
$ nova-manage export auth > /path/to/dump
It is important to redirect the output to a file so it can be imported in the
next step.
Step 2: Import your data to Keystone
------------------------------------
Import your Nova auth data from a ``dump_file`` created with ``nova-manage``::
$ keystone-manage import_nova_auth <dump_file>
.. NOTE::
Users are added to Keystone with the user ID from Nova as the user name.
Nova's projects are imported with the project ID as the tenant name. The
password used to authenticate a user in Keystone will be the API key
(also EC2 access key) used in Nova. Users also lose any administrative
privileges they had in Nova. The necessary admin role must be explicitly
re-assigned to each user.
.. NOTE::
Users in Nova's auth system have a single set of EC2 credentials that
works with all projects (tenants) that user can access. In Keystone, these
credentials are scoped to a single user/tenant pair. In order to use the
same secret keys from Nova, you must prefix each corresponding access key
with the ID of the project used in Nova. For example, if you had access
to the 'Beta' project in your Nova installation with the access/secret
keys 'ACCESS'/'SECRET', you should use 'Beta:ACCESS'/'SECRET' in Keystone.
These credentials are active once your migration is complete.
Initializing Keystone
=====================
@ -826,9 +705,6 @@ Initializing Keystone
through the normal REST API. At the moment, the following calls are supported:
* ``db_sync``: Sync the database schema.
* ``import_legacy``: Import data from a legacy (pre-Essex) database.
* ``export_legacy_catalog``: Export service catalog from a legacy (pre-Essex) database.
* ``import_nova_auth``: Load auth data from a dump created with ``nova-manage``.
* ``pki_setup``: Initialize the certificates for PKI based tokens.
* ``ssl_setup``: Generate certificates for HTTPS.

View File

@ -45,9 +45,6 @@ Available commands:
* ``db_sync``: Sync the database.
* ``db_version``: Print the current migration version of the database.
* ``export_legacy_catalog``: Export the service catalog from a legacy database.
* ``import_legacy``: Import a legacy database.
* ``import_nova_auth``: Import a dump of nova auth data into keystone.
* ``pki_setup``: Initialize the certificates used to sign tokens.
* ``ssl_setup``: Generate certificates for SSL.
* ``token_flush``: Purge expired tokens.

View File

@ -30,7 +30,6 @@ from keystone.common.sql import migration
from keystone import config
from keystone import contrib
from keystone.openstack.common import importutils
from keystone.openstack.common import jsonutils
from keystone import token
CONF = config.CONF
@ -190,67 +189,9 @@ class TokenFlush(BaseApp):
token_manager.driver.flush_expired_tokens()
class ImportLegacy(BaseApp):
"""Import a legacy database."""
name = 'import_legacy'
@classmethod
def add_argument_parser(cls, subparsers):
parser = super(ImportLegacy, cls).add_argument_parser(subparsers)
parser.add_argument('old_db')
return parser
@staticmethod
def main():
from keystone.common.sql import legacy
migration = legacy.LegacyMigration(CONF.command.old_db)
migration.migrate_all()
class ExportLegacyCatalog(BaseApp):
"""Export the service catalog from a legacy database."""
name = 'export_legacy_catalog'
@classmethod
def add_argument_parser(cls, subparsers):
parser = super(ExportLegacyCatalog,
cls).add_argument_parser(subparsers)
parser.add_argument('old_db')
return parser
@staticmethod
def main():
from keystone.common.sql import legacy
migration = legacy.LegacyMigration(CONF.command.old_db)
print('\n'.join(migration.dump_catalog()))
class ImportNovaAuth(BaseApp):
"""Import a dump of nova auth data into keystone."""
name = 'import_nova_auth'
@classmethod
def add_argument_parser(cls, subparsers):
parser = super(ImportNovaAuth, cls).add_argument_parser(subparsers)
parser.add_argument('dump_file')
return parser
@staticmethod
def main():
from keystone.common.sql import nova
dump_data = jsonutils.loads(open(CONF.command.dump_file).read())
nova.import_auth(dump_data)
CMDS = [
DbSync,
DbVersion,
ExportLegacyCatalog,
ImportLegacy,
ImportNovaAuth,
PKISetup,
SSLSetup,
TokenFlush,

View File

@ -1,197 +0,0 @@
# vim: tabstop=4 shiftwidth=4 softtabstop=4
# Copyright 2012 OpenStack LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import json
import re
import sqlalchemy
from sqlalchemy import exc
from keystone.assignment.backends import sql as assignment_sql
from keystone.common import utils
from keystone import config
from keystone.credential.backends import sql as ec2_sql
from keystone.identity.backends import sql as identity_sql
from keystone.openstack.common import log as logging
LOG = logging.getLogger(__name__)
CONF = config.CONF
DEFAULT_DOMAIN_ID = CONF.identity.default_domain_id
def export_db(db):
table_names = db.table_names()
migration_data = {}
for table_name in table_names:
query = db.execute("select * from %s" % table_name)
table_data = []
for row in query.fetchall():
entry = {}
for k in row.keys():
entry[k] = row[k]
table_data.append(entry)
migration_data[table_name] = table_data
return migration_data
def _translate_replacements(s):
if '%' not in str(s):
return s
return re.sub(r'%([\w_]+)%', r'$(\1)s', s)
class LegacyMigration(object):
def __init__(self, db_string):
self.db = sqlalchemy.create_engine(db_string)
#TODO(ayoung): Replace with call via Manager
self.identity_driver = identity_sql.Identity()
self.assignment_driver = assignment_sql.Assignment()
self.identity_driver.assignment_api = self.assignment_driver
self.assignment_driver.identity_api = self.identity_driver
self.identity_driver.db_sync()
self.assignment_driver.db_sync()
self.ec2_driver = ec2_sql.Credential()
self._data = {}
self._user_map = {}
self._project_map = {}
self._role_map = {}
def migrate_all(self):
self._export_legacy_db()
self._migrate_projects()
self._migrate_users()
self._migrate_roles()
self._migrate_user_roles()
self._migrate_tokens()
self._migrate_ec2()
def dump_catalog(self):
"""Generate the contents of a catalog templates file."""
self._export_legacy_db()
services_by_id = dict((x['id'], x) for x in self._data['services'])
template = 'catalog.%(region)s.%(service_type)s.%(key)s = %(value)s'
o = []
for row in self._data['endpoint_templates']:
service = services_by_id[row['service_id']]
d = {'service_type': service['type'],
'region': row['region']}
for x in ['internal_url', 'public_url', 'admin_url', 'enabled']:
d['key'] = x.replace('_url', 'URL')
d['value'] = _translate_replacements(row[x])
o.append(template % d)
d['key'] = 'name'
d['value'] = service['desc']
o.append(template % d)
return o
def _export_legacy_db(self):
self._data = export_db(self.db)
def _migrate_projects(self):
for x in self._data['tenants']:
# map
new_dict = {'description': x.get('desc', ''),
'id': x.get('uid', x.get('id')),
'enabled': x.get('enabled', True),
'domain_id': x.get('domain_id', DEFAULT_DOMAIN_ID)}
new_dict['name'] = x.get('name', new_dict.get('id'))
# track internal ids
self._project_map[x.get('id')] = new_dict['id']
# create
#print 'create_project(%s, %s)' % (new_dict['id'], new_dict)
self.assignment_driver.create_project(new_dict['id'], new_dict)
def _migrate_users(self):
for x in self._data['users']:
# map
new_dict = {'email': x.get('email', ''),
'password': x.get('password', None),
'id': x.get('uid', x.get('id')),
'enabled': x.get('enabled', True),
'domain_id': x.get('domain_id', DEFAULT_DOMAIN_ID)}
if x.get('tenant_id'):
new_dict['tenant_id'] = self._project_map.get(x['tenant_id'])
new_dict['name'] = x.get('name', new_dict.get('id'))
# track internal ids
self._user_map[x.get('id')] = new_dict['id']
# create
#print 'create_user(%s, %s)' % (new_dict['id'], new_dict)
self.identity_driver.create_user(new_dict['id'], new_dict)
if new_dict.get('tenant_id'):
self.identity_driver.add_user_to_project(
new_dict['tenant_id'],
new_dict['id'])
def _migrate_roles(self):
for x in self._data['roles']:
# map
new_dict = {'id': x['id'],
'name': x.get('name', x['id'])}
# track internal ids
self._role_map[x.get('id')] = new_dict['id']
# create
self.assignment_driver.create_role(new_dict['id'], new_dict)
def _migrate_user_roles(self):
for x in self._data['user_roles']:
# map
if (not x.get('user_id')
or not x.get('tenant_id')
or not x.get('role_id')):
continue
user_id = self._user_map[x['user_id']]
tenant_id = self._project_map[x['tenant_id']]
role_id = self._role_map[x['role_id']]
try:
self.identity_driver.add_user_to_project(tenant_id, user_id)
except Exception:
pass
self.assignment_driver.add_role_to_user_and_project(
user_id, tenant_id, role_id)
def _migrate_tokens(self):
pass
def _migrate_ec2(self):
for x in self._data['credentials']:
blob = {
'access': x['key'],
'secret': x['secret']
}
credential_id = utils.hash_access_key(blob['access'])
new_dict = {'user_id': x['user_id'],
'blob': json.dumps(blob),
'project_id': x['tenant_id'],
'id': credential_id,
'type': 'ec2'}
try:
self.ec2_driver.create_credential(credential_id, new_dict)
except exc.IntegrityError:
LOG.exception(_('Cannot migrate EC2 credential: %s') % x)

View File

@ -1,148 +0,0 @@
# vim: tabstop=4 shiftwidth=4 softtabstop=4
# Copyright 2012 OpenStack LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""Export data from Nova database and import through Identity Service."""
import json
import uuid
from keystone import assignment
from keystone.common import utils
from keystone import config
from keystone.credential.backends import sql as ec2_sql
from keystone import identity
from keystone.openstack.common import log as logging
LOG = logging.getLogger(__name__)
CONF = config.CONF
DEFAULT_DOMAIN_ID = CONF.identity.default_domain_id
def import_auth(data):
identity_api = identity.Manager()
assignment_api = assignment.Manager()
tenant_map = _create_projects(assignment_api, data['tenants'])
user_map = _create_users(identity_api, data['users'])
_create_memberships(assignment_api, data['user_tenant_list'],
user_map, tenant_map)
role_map = _create_roles(assignment_api, data['roles'])
_assign_roles(assignment_api, data['role_user_tenant_list'],
role_map, user_map, tenant_map)
ec2_api = ec2_sql.Credential()
ec2_creds = data['ec2_credentials']
_create_ec2_creds(ec2_api, assignment_api, ec2_creds, user_map)
def _generate_uuid():
return uuid.uuid4().hex
def _create_projects(api, tenants):
tenant_map = {}
for tenant in tenants:
tenant_dict = {
'id': _generate_uuid(),
'name': tenant['id'],
'domain_id': tenant.get('domain_id', DEFAULT_DOMAIN_ID),
'description': tenant['description'],
'enabled': True,
}
tenant_map[tenant['id']] = tenant_dict['id']
LOG.debug(_('Create tenant %s') % tenant_dict)
api.create_project(tenant_dict['id'], tenant_dict)
return tenant_map
def _create_users(api, users):
user_map = {}
for user in users:
user_dict = {
'id': _generate_uuid(),
'name': user['id'],
'domain_id': user.get('domain_id', DEFAULT_DOMAIN_ID),
'email': '',
'password': user['password'],
'enabled': True,
}
user_map[user['id']] = user_dict['id']
LOG.debug(_('Create user %s') % user_dict)
api.create_user(user_dict['id'], user_dict)
return user_map
def _create_memberships(api, memberships, user_map, tenant_map):
for membership in memberships:
user_id = user_map[membership['user_id']]
tenant_id = tenant_map[membership['tenant_id']]
LOG.debug(_('Add user %(user_id)s to tenant %(tenant_id)s') % {
'user_id': user_id, 'tenant_id': tenant_id})
api.add_user_to_project(tenant_id, user_id)
def _create_roles(api, roles):
role_map = dict((r['name'], r['id']) for r in api.list_roles())
for role in roles:
if role in role_map:
LOG.debug(_('Ignoring existing role %s') % role)
continue
role_dict = {
'id': _generate_uuid(),
'name': role,
}
role_map[role] = role_dict['id']
LOG.debug(_('Create role %s') % role_dict)
api.create_role(role_dict['id'], role_dict)
return role_map
def _assign_roles(api, assignments, role_map, user_map, tenant_map):
for assignment in assignments:
role_id = role_map[assignment['role']]
user_id = user_map[assignment['user_id']]
tenant_id = tenant_map[assignment['tenant_id']]
LOG.debug(_(
'Assign role %(role_id)s to user %(user_id)s on tenant '
'%(tenant_id)s') % {
'role_id': role_id,
'user_id': user_id,
'tenant_id': tenant_id})
api.add_role_to_user_and_project(user_id, tenant_id, role_id)
def _create_ec2_creds(ec2_api, assignment_api, ec2_creds, user_map):
for ec2_cred in ec2_creds:
user_id = user_map[ec2_cred['user_id']]
for tenant_id in assignment_api.get_projects_for_user(user_id):
blob = {
'access': '%s:%s' % (tenant_id, ec2_cred['access_key']),
'secret': ec2_cred['secret_key'],
}
credential_id = utils.hash_access_key(blob['access'])
cred_dict = {
'user_id': user_id,
'blob': json.dumps(blob),
'project_id': tenant_id,
'id': credential_id,
'type': 'ec2',
}
LOG.debug(_(
'Creating ec2 cred for user %(user_id)s and tenant '
'%(tenant_id)s') % {
'user_id': user_id, 'tenant_id': tenant_id})
ec2_api.create_credential(credential_id, cred_dict)

View File

@ -1,120 +0,0 @@
# vim: tabstop=4 shiftwidth=4 softtabstop=4
# Copyright 2012 OpenStack LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import os
try:
import sqlite3 as dbapi
except ImportError:
from pysqlite2 import dbapi2 as dbapi
from keystone.tests import core as test
from keystone.catalog.backends import templated as catalog_templated
from keystone.common.sql import legacy
from keystone import config
from keystone import identity
from keystone.identity.backends import sql as identity_sql
CONF = config.CONF
class ImportLegacy(test.TestCase):
def setUp(self):
super(ImportLegacy, self).setUp()
self.config([test.etcdir('keystone.conf.sample'),
test.testsdir('test_overrides.conf'),
test.testsdir('backend_sql.conf'),
test.testsdir('backend_sql_disk.conf')])
test.setup_test_database()
self.identity_man = identity.Manager()
self.identity_api = identity_sql.Identity()
def tearDown(self):
test.teardown_test_database()
super(ImportLegacy, self).tearDown()
def setup_old_database(self, sql_dump):
sql_path = test.testsdir(sql_dump)
db_path = test.tmpdir('%s.db' % sql_dump)
try:
os.unlink(db_path)
except OSError:
pass
script_str = open(sql_path).read().strip()
conn = dbapi.connect(db_path)
conn.executescript(script_str)
conn.commit()
return db_path
def test_import_d5(self):
db_path = self.setup_old_database('legacy_d5.sqlite')
migration = legacy.LegacyMigration('sqlite:///%s' % db_path)
migration.migrate_all()
admin_id = '1'
user_ref = self.identity_api.get_user(admin_id)
self.assertEquals(user_ref['name'], 'admin')
self.assertEquals(user_ref['enabled'], True)
# check password hashing
user_ref = self.identity_man.authenticate(
user_id=admin_id, password='secrete')
# check catalog
self._check_catalog(migration)
def test_import_diablo(self):
db_path = self.setup_old_database('legacy_diablo.sqlite')
migration = legacy.LegacyMigration('sqlite:///%s' % db_path)
migration.migrate_all()
admin_id = '1'
user_ref = self.identity_api.get_user(admin_id)
self.assertEquals(user_ref['name'], 'admin')
self.assertEquals(user_ref['enabled'], True)
# check password hashing
user_ref = self.identity_man.authenticate(
user_id=admin_id, password='secrete')
# check catalog
self._check_catalog(migration)
def test_import_essex(self):
db_path = self.setup_old_database('legacy_essex.sqlite')
migration = legacy.LegacyMigration('sqlite:///%s' % db_path)
migration.migrate_all()
admin_id = 'c93b19ea3fa94484824213db8ac0afce'
user_ref = self.identity_api.get_user(admin_id)
self.assertEquals(user_ref['name'], 'admin')
self.assertEquals(user_ref['enabled'], True)
# check password hashing
user_ref = self.identity_man.authenticate(
user_id=admin_id, password='secrete')
# check catalog
self._check_catalog(migration)
def _check_catalog(self, migration):
catalog_lines = migration.dump_catalog()
catalog = catalog_templated.parse_templates(catalog_lines)
self.assert_('RegionOne' in catalog)
self.assert_('compute' in catalog['RegionOne'])
self.assert_('adminURL' in catalog['RegionOne']['compute'])