diff --git a/doc/source/configuration.rst b/doc/source/configuration.rst index 3f5653ffbd..ff03f88db7 100644 --- a/doc/source/configuration.rst +++ b/doc/source/configuration.rst @@ -435,9 +435,6 @@ service catalog will not change very much over time. The value of ``template_file`` is expected to be an absolute path to your service catalog configuration. An example ``template_file`` is included in Keystone, however you should create your own to reflect your deployment. -If you are migrating from a legacy deployment, a tool is available to help with -this task (see `Migrating your Service Catalog from legacy versions of -Keystone`_). Another such example is `available in devstack (files/default_catalog.templates) @@ -701,124 +698,6 @@ empty list from your new database):: values, or deployed Keystone to a different endpoint, you will need to change the provided command accordingly. -Migrating from legacy versions of Keystone -========================================== - -Migration support is provided for the following legacy Keystone versions: - -* diablo-5 -* stable/diablo -* essex-2 -* essex-3 - -.. NOTE:: - - Before you can import your legacy data, you must first - `prepare your deployment`_. - -Step 1: Ensure your deployment can access your legacy database --------------------------------------------------------------------- - -Your legacy ``keystone.conf`` contains a SQL configuration section called -``[keystone.backends.sqlalchemy]`` connection string which, by default, -looks like:: - - sql_connection = sqlite:///keystone.db - -This connection string needs to be accessible from your deployment (e.g. -you may need to copy your SQLite ``*.db`` file to a new server, adjust the -relative path as appropriate, or open a firewall for MySQL, etc). - -Step 2: Import your legacy data -------------------------------- - -Use the following command to import your old data using the value of -``sql_connection`` from step 3:: - - $ keystone-manage import_legacy - -You should now be able to run the same command you used to test your new -database above, but now you'll see your legacy Keystone data:: - - $ keystone --token ADMIN --endpoint http://127.0.0.1:35357/v2.0/ tenant-list - +----------------------------------+----------------+---------+ - | id | name | enabled | - +----------------------------------+----------------+---------+ - | 12edde26a6224199a66ece67b762a065 | project-y | True | - | 593715ed4359404999915ea7005a7da1 | ANOTHER:TENANT | True | - | be57fed798b049bc9637d2be30bfa857 | coffee-tea | True | - | e3c382f4757a4385b502056431763cca | customer-x | True | - +----------------------------------+----------------+---------+ - - -Migrating your Service Catalog from legacy versions of Keystone -=============================================================== - -While legacy Keystone deployments stored the service catalog in the database, -the service catalog is stored in a flat ``template_file``. An example -service catalog template file may be found in -``etc/default_catalog.templates``. You can change the path to your service -catalog template in ``keystone.conf`` by changing the value of -``[catalog] template_file``. - -Import your legacy catalog and redirect the output to your ``template_file``:: - - $ keystone-manage export_legacy_catalog > - -.. NOTE:: - - After executing this command, you will need to restart the Keystone - service to see your changes. - -Migrating from Nova Auth -======================== - -Migration of users, projects (aka tenants), roles and EC2 credentials -is supported for the Essex and later releases of Nova. To migrate your auth -data from Nova, use the following steps: - -.. NOTE:: - - Before you can migrate from nova auth, you must first - `prepare your deployment`_. - -Step 1: Export your data from Nova ----------------------------------- - -Use the following command to export your data from Nova to a ``dump_file``:: - - $ nova-manage export auth > /path/to/dump - -It is important to redirect the output to a file so it can be imported in the -next step. - -Step 2: Import your data to Keystone ------------------------------------- - -Import your Nova auth data from a ``dump_file`` created with ``nova-manage``:: - - $ keystone-manage import_nova_auth - -.. NOTE:: - - Users are added to Keystone with the user ID from Nova as the user name. - Nova's projects are imported with the project ID as the tenant name. The - password used to authenticate a user in Keystone will be the API key - (also EC2 access key) used in Nova. Users also lose any administrative - privileges they had in Nova. The necessary admin role must be explicitly - re-assigned to each user. - -.. NOTE:: - - Users in Nova's auth system have a single set of EC2 credentials that - works with all projects (tenants) that user can access. In Keystone, these - credentials are scoped to a single user/tenant pair. In order to use the - same secret keys from Nova, you must prefix each corresponding access key - with the ID of the project used in Nova. For example, if you had access - to the 'Beta' project in your Nova installation with the access/secret - keys 'ACCESS'/'SECRET', you should use 'Beta:ACCESS'/'SECRET' in Keystone. - These credentials are active once your migration is complete. - Initializing Keystone ===================== @@ -826,9 +705,6 @@ Initializing Keystone through the normal REST API. At the moment, the following calls are supported: * ``db_sync``: Sync the database schema. -* ``import_legacy``: Import data from a legacy (pre-Essex) database. -* ``export_legacy_catalog``: Export service catalog from a legacy (pre-Essex) database. -* ``import_nova_auth``: Load auth data from a dump created with ``nova-manage``. * ``pki_setup``: Initialize the certificates for PKI based tokens. * ``ssl_setup``: Generate certificates for HTTPS. diff --git a/doc/source/man/keystone-manage.rst b/doc/source/man/keystone-manage.rst index 34cbd57af5..c8dbdb0693 100644 --- a/doc/source/man/keystone-manage.rst +++ b/doc/source/man/keystone-manage.rst @@ -45,9 +45,6 @@ Available commands: * ``db_sync``: Sync the database. * ``db_version``: Print the current migration version of the database. -* ``export_legacy_catalog``: Export the service catalog from a legacy database. -* ``import_legacy``: Import a legacy database. -* ``import_nova_auth``: Import a dump of nova auth data into keystone. * ``pki_setup``: Initialize the certificates used to sign tokens. * ``ssl_setup``: Generate certificates for SSL. * ``token_flush``: Purge expired tokens. diff --git a/keystone/cli.py b/keystone/cli.py index 6575f2e9af..6ee47fcd63 100644 --- a/keystone/cli.py +++ b/keystone/cli.py @@ -30,7 +30,6 @@ from keystone.common.sql import migration from keystone import config from keystone import contrib from keystone.openstack.common import importutils -from keystone.openstack.common import jsonutils from keystone import token CONF = config.CONF @@ -190,67 +189,9 @@ class TokenFlush(BaseApp): token_manager.driver.flush_expired_tokens() -class ImportLegacy(BaseApp): - """Import a legacy database.""" - - name = 'import_legacy' - - @classmethod - def add_argument_parser(cls, subparsers): - parser = super(ImportLegacy, cls).add_argument_parser(subparsers) - parser.add_argument('old_db') - return parser - - @staticmethod - def main(): - from keystone.common.sql import legacy - migration = legacy.LegacyMigration(CONF.command.old_db) - migration.migrate_all() - - -class ExportLegacyCatalog(BaseApp): - """Export the service catalog from a legacy database.""" - - name = 'export_legacy_catalog' - - @classmethod - def add_argument_parser(cls, subparsers): - parser = super(ExportLegacyCatalog, - cls).add_argument_parser(subparsers) - parser.add_argument('old_db') - return parser - - @staticmethod - def main(): - from keystone.common.sql import legacy - migration = legacy.LegacyMigration(CONF.command.old_db) - print('\n'.join(migration.dump_catalog())) - - -class ImportNovaAuth(BaseApp): - """Import a dump of nova auth data into keystone.""" - - name = 'import_nova_auth' - - @classmethod - def add_argument_parser(cls, subparsers): - parser = super(ImportNovaAuth, cls).add_argument_parser(subparsers) - parser.add_argument('dump_file') - return parser - - @staticmethod - def main(): - from keystone.common.sql import nova - dump_data = jsonutils.loads(open(CONF.command.dump_file).read()) - nova.import_auth(dump_data) - - CMDS = [ DbSync, DbVersion, - ExportLegacyCatalog, - ImportLegacy, - ImportNovaAuth, PKISetup, SSLSetup, TokenFlush, diff --git a/keystone/common/sql/legacy.py b/keystone/common/sql/legacy.py deleted file mode 100644 index c9a54ba6a0..0000000000 --- a/keystone/common/sql/legacy.py +++ /dev/null @@ -1,197 +0,0 @@ -# vim: tabstop=4 shiftwidth=4 softtabstop=4 - -# Copyright 2012 OpenStack LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); you may -# not use this file except in compliance with the License. You may obtain -# a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT -# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the -# License for the specific language governing permissions and limitations -# under the License. - -import json -import re - -import sqlalchemy -from sqlalchemy import exc - - -from keystone.assignment.backends import sql as assignment_sql -from keystone.common import utils -from keystone import config -from keystone.credential.backends import sql as ec2_sql -from keystone.identity.backends import sql as identity_sql -from keystone.openstack.common import log as logging - - -LOG = logging.getLogger(__name__) -CONF = config.CONF -DEFAULT_DOMAIN_ID = CONF.identity.default_domain_id - - -def export_db(db): - table_names = db.table_names() - - migration_data = {} - for table_name in table_names: - query = db.execute("select * from %s" % table_name) - table_data = [] - for row in query.fetchall(): - entry = {} - for k in row.keys(): - entry[k] = row[k] - table_data.append(entry) - - migration_data[table_name] = table_data - - return migration_data - - -def _translate_replacements(s): - if '%' not in str(s): - return s - return re.sub(r'%([\w_]+)%', r'$(\1)s', s) - - -class LegacyMigration(object): - def __init__(self, db_string): - self.db = sqlalchemy.create_engine(db_string) - #TODO(ayoung): Replace with call via Manager - self.identity_driver = identity_sql.Identity() - self.assignment_driver = assignment_sql.Assignment() - self.identity_driver.assignment_api = self.assignment_driver - self.assignment_driver.identity_api = self.identity_driver - self.identity_driver.db_sync() - self.assignment_driver.db_sync() - - self.ec2_driver = ec2_sql.Credential() - self._data = {} - self._user_map = {} - self._project_map = {} - self._role_map = {} - - def migrate_all(self): - self._export_legacy_db() - self._migrate_projects() - self._migrate_users() - self._migrate_roles() - self._migrate_user_roles() - self._migrate_tokens() - self._migrate_ec2() - - def dump_catalog(self): - """Generate the contents of a catalog templates file.""" - self._export_legacy_db() - - services_by_id = dict((x['id'], x) for x in self._data['services']) - template = 'catalog.%(region)s.%(service_type)s.%(key)s = %(value)s' - - o = [] - for row in self._data['endpoint_templates']: - service = services_by_id[row['service_id']] - d = {'service_type': service['type'], - 'region': row['region']} - - for x in ['internal_url', 'public_url', 'admin_url', 'enabled']: - d['key'] = x.replace('_url', 'URL') - d['value'] = _translate_replacements(row[x]) - o.append(template % d) - - d['key'] = 'name' - d['value'] = service['desc'] - o.append(template % d) - - return o - - def _export_legacy_db(self): - self._data = export_db(self.db) - - def _migrate_projects(self): - for x in self._data['tenants']: - # map - new_dict = {'description': x.get('desc', ''), - 'id': x.get('uid', x.get('id')), - 'enabled': x.get('enabled', True), - 'domain_id': x.get('domain_id', DEFAULT_DOMAIN_ID)} - new_dict['name'] = x.get('name', new_dict.get('id')) - # track internal ids - self._project_map[x.get('id')] = new_dict['id'] - # create - #print 'create_project(%s, %s)' % (new_dict['id'], new_dict) - self.assignment_driver.create_project(new_dict['id'], new_dict) - - def _migrate_users(self): - for x in self._data['users']: - # map - new_dict = {'email': x.get('email', ''), - 'password': x.get('password', None), - 'id': x.get('uid', x.get('id')), - 'enabled': x.get('enabled', True), - 'domain_id': x.get('domain_id', DEFAULT_DOMAIN_ID)} - if x.get('tenant_id'): - new_dict['tenant_id'] = self._project_map.get(x['tenant_id']) - new_dict['name'] = x.get('name', new_dict.get('id')) - # track internal ids - self._user_map[x.get('id')] = new_dict['id'] - # create - #print 'create_user(%s, %s)' % (new_dict['id'], new_dict) - self.identity_driver.create_user(new_dict['id'], new_dict) - if new_dict.get('tenant_id'): - self.identity_driver.add_user_to_project( - new_dict['tenant_id'], - new_dict['id']) - - def _migrate_roles(self): - for x in self._data['roles']: - # map - new_dict = {'id': x['id'], - 'name': x.get('name', x['id'])} - # track internal ids - self._role_map[x.get('id')] = new_dict['id'] - # create - self.assignment_driver.create_role(new_dict['id'], new_dict) - - def _migrate_user_roles(self): - for x in self._data['user_roles']: - # map - if (not x.get('user_id') - or not x.get('tenant_id') - or not x.get('role_id')): - continue - user_id = self._user_map[x['user_id']] - tenant_id = self._project_map[x['tenant_id']] - role_id = self._role_map[x['role_id']] - - try: - self.identity_driver.add_user_to_project(tenant_id, user_id) - except Exception: - pass - - self.assignment_driver.add_role_to_user_and_project( - user_id, tenant_id, role_id) - - def _migrate_tokens(self): - pass - - def _migrate_ec2(self): - for x in self._data['credentials']: - blob = { - 'access': x['key'], - 'secret': x['secret'] - } - credential_id = utils.hash_access_key(blob['access']) - new_dict = {'user_id': x['user_id'], - 'blob': json.dumps(blob), - 'project_id': x['tenant_id'], - 'id': credential_id, - 'type': 'ec2'} - - try: - self.ec2_driver.create_credential(credential_id, new_dict) - except exc.IntegrityError: - LOG.exception(_('Cannot migrate EC2 credential: %s') % x) diff --git a/keystone/common/sql/nova.py b/keystone/common/sql/nova.py deleted file mode 100644 index 5a502183de..0000000000 --- a/keystone/common/sql/nova.py +++ /dev/null @@ -1,148 +0,0 @@ -# vim: tabstop=4 shiftwidth=4 softtabstop=4 - -# Copyright 2012 OpenStack LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); you may -# not use this file except in compliance with the License. You may obtain -# a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT -# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the -# License for the specific language governing permissions and limitations -# under the License. - -"""Export data from Nova database and import through Identity Service.""" - -import json -import uuid - -from keystone import assignment -from keystone.common import utils -from keystone import config -from keystone.credential.backends import sql as ec2_sql -from keystone import identity -from keystone.openstack.common import log as logging - - -LOG = logging.getLogger(__name__) -CONF = config.CONF -DEFAULT_DOMAIN_ID = CONF.identity.default_domain_id - - -def import_auth(data): - identity_api = identity.Manager() - assignment_api = assignment.Manager() - - tenant_map = _create_projects(assignment_api, data['tenants']) - user_map = _create_users(identity_api, data['users']) - _create_memberships(assignment_api, data['user_tenant_list'], - user_map, tenant_map) - role_map = _create_roles(assignment_api, data['roles']) - _assign_roles(assignment_api, data['role_user_tenant_list'], - role_map, user_map, tenant_map) - - ec2_api = ec2_sql.Credential() - ec2_creds = data['ec2_credentials'] - _create_ec2_creds(ec2_api, assignment_api, ec2_creds, user_map) - - -def _generate_uuid(): - return uuid.uuid4().hex - - -def _create_projects(api, tenants): - tenant_map = {} - for tenant in tenants: - tenant_dict = { - 'id': _generate_uuid(), - 'name': tenant['id'], - 'domain_id': tenant.get('domain_id', DEFAULT_DOMAIN_ID), - 'description': tenant['description'], - 'enabled': True, - } - tenant_map[tenant['id']] = tenant_dict['id'] - LOG.debug(_('Create tenant %s') % tenant_dict) - api.create_project(tenant_dict['id'], tenant_dict) - return tenant_map - - -def _create_users(api, users): - user_map = {} - for user in users: - user_dict = { - 'id': _generate_uuid(), - 'name': user['id'], - 'domain_id': user.get('domain_id', DEFAULT_DOMAIN_ID), - 'email': '', - 'password': user['password'], - 'enabled': True, - } - user_map[user['id']] = user_dict['id'] - LOG.debug(_('Create user %s') % user_dict) - api.create_user(user_dict['id'], user_dict) - return user_map - - -def _create_memberships(api, memberships, user_map, tenant_map): - for membership in memberships: - user_id = user_map[membership['user_id']] - tenant_id = tenant_map[membership['tenant_id']] - LOG.debug(_('Add user %(user_id)s to tenant %(tenant_id)s') % { - 'user_id': user_id, 'tenant_id': tenant_id}) - api.add_user_to_project(tenant_id, user_id) - - -def _create_roles(api, roles): - role_map = dict((r['name'], r['id']) for r in api.list_roles()) - for role in roles: - if role in role_map: - LOG.debug(_('Ignoring existing role %s') % role) - continue - role_dict = { - 'id': _generate_uuid(), - 'name': role, - } - role_map[role] = role_dict['id'] - LOG.debug(_('Create role %s') % role_dict) - api.create_role(role_dict['id'], role_dict) - return role_map - - -def _assign_roles(api, assignments, role_map, user_map, tenant_map): - for assignment in assignments: - role_id = role_map[assignment['role']] - user_id = user_map[assignment['user_id']] - tenant_id = tenant_map[assignment['tenant_id']] - LOG.debug(_( - 'Assign role %(role_id)s to user %(user_id)s on tenant ' - '%(tenant_id)s') % { - 'role_id': role_id, - 'user_id': user_id, - 'tenant_id': tenant_id}) - api.add_role_to_user_and_project(user_id, tenant_id, role_id) - - -def _create_ec2_creds(ec2_api, assignment_api, ec2_creds, user_map): - for ec2_cred in ec2_creds: - user_id = user_map[ec2_cred['user_id']] - for tenant_id in assignment_api.get_projects_for_user(user_id): - blob = { - 'access': '%s:%s' % (tenant_id, ec2_cred['access_key']), - 'secret': ec2_cred['secret_key'], - } - credential_id = utils.hash_access_key(blob['access']) - cred_dict = { - 'user_id': user_id, - 'blob': json.dumps(blob), - 'project_id': tenant_id, - 'id': credential_id, - 'type': 'ec2', - } - LOG.debug(_( - 'Creating ec2 cred for user %(user_id)s and tenant ' - '%(tenant_id)s') % { - 'user_id': user_id, 'tenant_id': tenant_id}) - ec2_api.create_credential(credential_id, cred_dict) diff --git a/keystone/tests/test_import_legacy.py b/keystone/tests/test_import_legacy.py deleted file mode 100644 index b3b83c0f16..0000000000 --- a/keystone/tests/test_import_legacy.py +++ /dev/null @@ -1,120 +0,0 @@ -# vim: tabstop=4 shiftwidth=4 softtabstop=4 - -# Copyright 2012 OpenStack LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); you may -# not use this file except in compliance with the License. You may obtain -# a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT -# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the -# License for the specific language governing permissions and limitations -# under the License. - -import os - -try: - import sqlite3 as dbapi -except ImportError: - from pysqlite2 import dbapi2 as dbapi - -from keystone.tests import core as test - -from keystone.catalog.backends import templated as catalog_templated -from keystone.common.sql import legacy -from keystone import config -from keystone import identity -from keystone.identity.backends import sql as identity_sql - - -CONF = config.CONF - - -class ImportLegacy(test.TestCase): - def setUp(self): - super(ImportLegacy, self).setUp() - self.config([test.etcdir('keystone.conf.sample'), - test.testsdir('test_overrides.conf'), - test.testsdir('backend_sql.conf'), - test.testsdir('backend_sql_disk.conf')]) - test.setup_test_database() - self.identity_man = identity.Manager() - self.identity_api = identity_sql.Identity() - - def tearDown(self): - test.teardown_test_database() - super(ImportLegacy, self).tearDown() - - def setup_old_database(self, sql_dump): - sql_path = test.testsdir(sql_dump) - db_path = test.tmpdir('%s.db' % sql_dump) - try: - os.unlink(db_path) - except OSError: - pass - script_str = open(sql_path).read().strip() - conn = dbapi.connect(db_path) - conn.executescript(script_str) - conn.commit() - return db_path - - def test_import_d5(self): - db_path = self.setup_old_database('legacy_d5.sqlite') - migration = legacy.LegacyMigration('sqlite:///%s' % db_path) - migration.migrate_all() - - admin_id = '1' - user_ref = self.identity_api.get_user(admin_id) - self.assertEquals(user_ref['name'], 'admin') - self.assertEquals(user_ref['enabled'], True) - - # check password hashing - user_ref = self.identity_man.authenticate( - user_id=admin_id, password='secrete') - - # check catalog - self._check_catalog(migration) - - def test_import_diablo(self): - db_path = self.setup_old_database('legacy_diablo.sqlite') - migration = legacy.LegacyMigration('sqlite:///%s' % db_path) - migration.migrate_all() - - admin_id = '1' - user_ref = self.identity_api.get_user(admin_id) - self.assertEquals(user_ref['name'], 'admin') - self.assertEquals(user_ref['enabled'], True) - - # check password hashing - user_ref = self.identity_man.authenticate( - user_id=admin_id, password='secrete') - - # check catalog - self._check_catalog(migration) - - def test_import_essex(self): - db_path = self.setup_old_database('legacy_essex.sqlite') - migration = legacy.LegacyMigration('sqlite:///%s' % db_path) - migration.migrate_all() - - admin_id = 'c93b19ea3fa94484824213db8ac0afce' - user_ref = self.identity_api.get_user(admin_id) - self.assertEquals(user_ref['name'], 'admin') - self.assertEquals(user_ref['enabled'], True) - - # check password hashing - user_ref = self.identity_man.authenticate( - user_id=admin_id, password='secrete') - - # check catalog - self._check_catalog(migration) - - def _check_catalog(self, migration): - catalog_lines = migration.dump_catalog() - catalog = catalog_templated.parse_templates(catalog_lines) - self.assert_('RegionOne' in catalog) - self.assert_('compute' in catalog['RegionOne']) - self.assert_('adminURL' in catalog['RegionOne']['compute'])