diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst new file mode 100644 index 0000000..9ed5407 --- /dev/null +++ b/CONTRIBUTING.rst @@ -0,0 +1,21 @@ +If you would like to contribute to the development of OpenStack, +you must follow the steps in the "If you're a developer" +section of this page: + + http://wiki.openstack.org/HowToContribute + +You can find more Meteos-specific info in our How To Participate guide: + + http://docs.openstack.org/developer/python-meteosclient/devref/how_to_participate.html + +Once those steps have been completed, changes to OpenStack +should be submitted for review via the Gerrit tool, following +the workflow documented at: + + http://wiki.openstack.org/GerritWorkflow + +Pull requests submitted through GitHub will be ignored. + +Bugs should be filed on Launchpad, not GitHub: + + https://bugs.launchpad.net/python-meteosclient diff --git a/HACKING.rst b/HACKING.rst new file mode 100644 index 0000000..158bb0b --- /dev/null +++ b/HACKING.rst @@ -0,0 +1,12 @@ +Meteos Style Commandments +========================= + +- Step 1: Read the OpenStack Style Commandments + http://docs.openstack.org/developer/hacking/ +- Step 2: Read on + +Meteos Specific Commandments +---------------------------- + +None so far + diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..68c771a --- /dev/null +++ b/LICENSE @@ -0,0 +1,176 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + diff --git a/MANIFEST.in b/MANIFEST.in new file mode 100644 index 0000000..f1c38fb --- /dev/null +++ b/MANIFEST.in @@ -0,0 +1,9 @@ +include AUTHORS +include README.rst +include ChangeLog +include LICENSE + +exclude .gitignore +exclude .gitreview + +global-exclude *.pyc diff --git a/README.rst b/README.rst new file mode 100644 index 0000000..9778626 --- /dev/null +++ b/README.rst @@ -0,0 +1,33 @@ +Python bindings to the OpenStack Meteos API +=========================================== + +This is a client for the OpenStack Meteos API. There's a Python API (the +``meteosclient`` module), and a command-line script (``meteos``). Each +implements the OpenStack Meteos API. You can find documentation for both +Python bindings and CLI in `Docs`_. + +Development takes place via the usual OpenStack processes as outlined +in the `developer guide +`_. + +.. _Docs: http://docs.openstack.org/developer/python-meteosclient/ + +* License: Apache License, Version 2.0 +* `PyPi`_ - package installation +* `Online Documentation`_ +* `Launchpad project`_ - release management +* `Blueprints`_ - feature specifications +* `Bugs`_ - issue tracking +* `Source`_ +* `Specs`_ +* `How to Contribute`_ + +.. _PyPi: https://pypi.python.org/pypi/python-meteosclient +.. _Online Documentation: http://docs.openstack.org/developer/python-meteosclient +.. _Launchpad project: https://launchpad.net/python-meteosclient +.. _Blueprints: https://blueprints.launchpad.net/python-meteosclient +.. _Bugs: https://bugs.launchpad.net/python-meteosclient +.. _Source: https://git.openstack.org/cgit/openstack/python-meteosclient +.. _How to Contribute: http://docs.openstack.org/infra/manual/developers.html +.. _Specs: http://specs.openstack.org/openstack/meteos-specs/ + diff --git a/meteosclient/__init__.py b/meteosclient/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/meteosclient/api/__init__.py b/meteosclient/api/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/meteosclient/api/base.py b/meteosclient/api/base.py new file mode 100644 index 0000000..6bf3c6b --- /dev/null +++ b/meteosclient/api/base.py @@ -0,0 +1,272 @@ +# Copyright (c) 2013 Mirantis Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or +# implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import copy +import json +import logging + +import six +from six.moves.urllib import parse + +from meteosclient.openstack.common._i18n import _ + +LOG = logging.getLogger(__name__) + + +class Resource(object): + resource_name = 'Something' + defaults = {} + + def __init__(self, manager, info): + self.manager = manager + info = info.copy() + self._info = info + self._set_defaults(info) + self._add_details(info) + + def _set_defaults(self, info): + for name, value in six.iteritems(self.defaults): + if name not in info: + info[name] = value + + def _add_details(self, info): + for (k, v) in six.iteritems(info): + try: + setattr(self, k, v) + self._info[k] = v + except AttributeError: + # In this case we already defined the attribute on the class + pass + + def to_dict(self): + return copy.deepcopy(self._info) + + def __str__(self): + return '%s %s' % (self.resource_name, str(self._info)) + + +def _check_items(obj, searches): + try: + return all(getattr(obj, attr) == value for (attr, value) in searches) + except AttributeError: + return False + + +class NotUpdated(object): + """A sentinel class to signal that parameter should not be updated.""" + def __repr__(self): + return 'NotUpdated' + + +class ResourceManager(object): + resource_class = None + + def __init__(self, api): + self.api = api + + def find(self, **kwargs): + return [i for i in self.list() if _check_items(i, kwargs.items())] + + def find_unique(self, **kwargs): + found = self.find(**kwargs) + if not found: + raise APIException(error_code=404, + error_message=_("No matches found.")) + if len(found) > 1: + raise APIException(error_code=409, + error_message=_("Multiple matches found.")) + return found[0] + + def _create(self, url, data, response_key=None, dump_json=True): + if dump_json: + kwargs = {'json': data} + else: + kwargs = {'data': data} + + resp = self.api.post(url, **kwargs) + + if resp.status_code != 202 and resp.status_code != 200: + self._raise_api_exception(resp) + + if response_key is not None: + data = get_json(resp)[response_key] + else: + data = get_json(resp) + return self.resource_class(self, data) + + def _update(self, url, data, response_key=None, dump_json=True): + if dump_json: + kwargs = {'json': data} + else: + kwargs = {'data': data} + + resp = self.api.put(url, **kwargs) + + if resp.status_code != 202: + self._raise_api_exception(resp) + if response_key is not None: + data = get_json(resp)[response_key] + else: + data = get_json(resp) + + return self.resource_class(self, data) + + def _patch(self, url, data, response_key=None, dump_json=True): + if dump_json: + kwargs = {'json': data} + else: + kwargs = {'data': data} + + resp = self.api.patch(url, **kwargs) + + if resp.status_code != 202: + self._raise_api_exception(resp) + if response_key is not None: + data = get_json(resp)[response_key] + else: + data = get_json(resp) + + return self.resource_class(self, data) + + def _post(self, url, data, response_key=None, dump_json=True): + if dump_json: + kwargs = {'json': data} + else: + kwargs = {'data': data} + + resp = self.api.post(url, **kwargs) + + if resp.status_code != 202: + self._raise_api_exception(resp) + + if response_key is not None: + data = get_json(resp)[response_key] + else: + data = get_json(resp) + + return self.resource_class(self, data) + + def _list(self, url, response_key): + resp = self.api.get(url) + if resp.status_code == 200: + data = get_json(resp)[response_key] + + return [self.resource_class(self, res) + for res in data] + else: + self._raise_api_exception(resp) + + def _page(self, url, response_key, limit=None): + resp = self.api.get(url) + if resp.status_code == 200: + result = get_json(resp) + data = result[response_key] + meta = result.get('markers') + + next, prev = None, None + + if meta: + prev = meta.get('prev') + next = meta.get('next') + + l = [self.resource_class(self, res) + for res in data] + + return Page(l, prev, next, limit) + else: + self._raise_api_exception(resp) + + def _get(self, url, response_key=None): + resp = self.api.get(url) + + if resp.status_code == 200: + if response_key is not None: + data = get_json(resp)[response_key] + else: + data = get_json(resp) + return self.resource_class(self, data) + else: + self._raise_api_exception(resp) + + def _delete(self, url): + resp = self.api.delete(url) + + if resp.status_code != 202 and resp.status_code != 204: + self._raise_api_exception(resp) + + def _plurify_resource_name(self): + return self.resource_class.resource_name + 's' + + def _raise_api_exception(self, resp): + try: + error_data = get_json(resp) + error_msg = error_data.values()[0] + except Exception: + msg = _("Failed to parse response from Meteos: %s") % resp.reason + raise APIException( + error_code=resp.status_code, + error_message=msg) + + raise APIException(error_code=error_msg.get("code"), + error_name=error_data.keys()[0], + error_message=error_msg.get("message")) + + +def get_json(response): + """Provide backward compatibility with old versions of requests library.""" + + json_field_or_function = getattr(response, 'json', None) + if callable(json_field_or_function): + return response.json() + else: + return json.loads(response.content) + + +class APIException(Exception): + def __init__(self, error_code=None, error_name=None, error_message=None): + super(APIException, self).__init__(error_message) + self.error_code = error_code + self.error_name = error_name + self.error_message = error_message + + +def get_query_string(search_opts, limit=None, marker=None, sort_by=None, + reverse=None): + opts = {} + if marker is not None: + opts['marker'] = marker + if limit is not None: + opts['limit'] = limit + if sort_by is not None: + if reverse: + opts['sort_by'] = "-%s" % sort_by + else: + opts['sort_by'] = sort_by + if search_opts is not None: + opts.update(search_opts) + if opts: + qparams = sorted(opts.items(), key=lambda x: x[0]) + query_string = "?%s" % parse.urlencode(qparams, doseq=True) + else: + query_string = "" + return query_string + + +class Page(list): + def __init__(self, l, prev, next, limit): + super(Page, self).__init__(l) + self.prev = prev + self.next = next + self.limit = limit diff --git a/meteosclient/api/client.py b/meteosclient/api/client.py new file mode 100644 index 0000000..be5a6c0 --- /dev/null +++ b/meteosclient/api/client.py @@ -0,0 +1,169 @@ +# Copyright (c) 2013 Mirantis Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or +# implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import warnings + +from keystoneauth1 import adapter +from keystoneauth1 import exceptions +from keystoneauth1.identity import v2 +from keystoneauth1.identity import v3 +from keystoneauth1 import session as keystone_session +from keystoneauth1 import token_endpoint + +from meteosclient.api import templates +from meteosclient.api import experiments +from meteosclient.api import datasets +from meteosclient.api import models +from meteosclient.api import learnings + + +USER_AGENT = 'python-meteosclient' + + +class HTTPClient(adapter.Adapter): + + def request(self, *args, **kwargs): + kwargs.setdefault('raise_exc', False) + return super(HTTPClient, self).request(*args, **kwargs) + + +class Client(object): + """Client for the OpenStack Data Processing v1 API. + + :param str username: Username for Keystone authentication. + :param str api_key: Password for Keystone authentication. + :param str project_id: Keystone Tenant id. + :param str project_name: Keystone Tenant name. + :param str auth_url: Keystone URL that will be used for authentication. + :param str meteos_url: Meteos REST API URL to communicate with. + :param str endpoint_type: Desired Meteos endpoint type. + :param str service_type: Meteos service name in Keystone catalog. + :param str input_auth_token: Keystone authorization token. + :param session: Keystone Session object. + :param auth: Keystone Authentication Plugin object. + :param boolean insecure: Allow insecure. + :param string cacert: Path to the Privacy Enhanced Mail (PEM) file + which contains certificates needed to establish + SSL connection with the identity service. + :param string region_name: Name of a region to select when choosing an + endpoint from the service catalog. + """ + def __init__(self, username=None, api_key=None, project_id=None, + project_name=None, auth_url=None, meteos_url=None, + endpoint_type='publicURL', service_type='machine-learning', + input_auth_token=None, session=None, auth=None, + insecure=False, cacert=None, region_name=None, **kwargs): + + if not session: + warnings.simplefilter('once', category=DeprecationWarning) + warnings.warn('Passing authentication parameters to meteosclient ' + 'is deprecated. Please construct and pass an ' + 'authenticated session object directly.', + DeprecationWarning) + warnings.resetwarnings() + + if input_auth_token: + auth = token_endpoint.Token(meteos_url, input_auth_token) + + else: + auth = self._get_keystone_auth(auth_url=auth_url, + username=username, + api_key=api_key, + project_id=project_id, + project_name=project_name) + + verify = True + if insecure: + verify = False + elif cacert: + verify = cacert + + session = keystone_session.Session(verify=verify) + + if not auth: + auth = session.auth + + # NOTE(Toan): bug #1512801. If meteos_url is provided, it does not + # matter if service_type is orthographically correct or not. + # Only find Meteos service_type and endpoint in Keystone catalog + # if meteos_url is not provided. + if not meteos_url: + service_type = self._determine_service_type(session, + auth, + service_type, + endpoint_type) + + kwargs['user_agent'] = USER_AGENT + kwargs.setdefault('interface', endpoint_type) + kwargs.setdefault('endpoint_override', meteos_url) + + client = HTTPClient(session=session, + auth=auth, + service_type=service_type, + region_name=region_name, + **kwargs) + + self.templates = templates.TemplateManager(client) + self.experiments = experiments.ExperimentManager(client) + self.datasets = datasets.DatasetManager(client) + self.models = models.ModelManager(client) + self.learnings = learnings.LearningManager(client) + + def _get_keystone_auth(self, username=None, api_key=None, auth_url=None, + project_id=None, project_name=None): + if not auth_url: + raise RuntimeError("No auth url specified") + + if 'v2.0' in auth_url: + return v2.Password(auth_url=auth_url, + username=username, + password=api_key, + tenant_id=project_id, + tenant_name=project_name) + else: + # NOTE(jamielennox): Setting these to default is what + # keystoneclient does in the event they are not passed. + return v3.Password(auth_url=auth_url, + username=username, + password=api_key, + user_domain_id='default', + project_id=project_id, + project_name=project_name, + project_domain_id='default') + + @staticmethod + def _determine_service_type(session, auth, service_type, interface): + """Check a catalog for machine-learning or data_processing""" + + # NOTE(jamielennox): calling get_endpoint forces an auth on + # initialization which is required for backwards compatibility. It + # also allows us to reset the service type if not in the catalog. + for st in (service_type, service_type.replace('-', '_')): + try: + url = auth.get_endpoint(session, + service_type=st, + interface=interface) + except exceptions.Unauthorized: + raise RuntimeError("Not Authorized") + except exceptions.EndpointNotFound: + # NOTE(jamielennox): bug #1428447. This should not be + # raised, instead None should be returned. Handle in case + # it changes in the future + url = None + + if url: + return st + + raise RuntimeError("Could not find Meteos endpoint in catalog") diff --git a/meteosclient/api/datasets.py b/meteosclient/api/datasets.py new file mode 100644 index 0000000..b043185 --- /dev/null +++ b/meteosclient/api/datasets.py @@ -0,0 +1,68 @@ +# Copyright (c) 2013 Mirantis Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or +# implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import base64 +from six.moves.urllib import parse +from meteosclient.api import base + + +class Dataset(base.Resource): + resource_name = 'Dataset' + + +class DatasetManager(base.ResourceManager): + resource_class = Dataset + NotUpdated = base.NotUpdated() + + def create(self, method=None, source_dataset_url=None, display_name=None, + display_description=None, experiment_id=None, params=None, + swift_tenant=None, swift_username=None, swift_password=None): + """Create a Dataset.""" + + data = { + 'method': method, + 'source_dataset_url': source_dataset_url, + 'display_name': display_name, + 'display_description': display_description, + 'experiment_id': experiment_id, + 'params': base64.b64encode(str(params)), + 'swift_tenant': swift_tenant, + 'swift_username': swift_username, + 'swift_password': swift_password, + } + + body = {'dataset': data} + + return self._create('/datasets', body, 'dataset') + + def list(self, search_opts=None, limit=None, marker=None, + sort_by=None, reverse=None): + """Get a list of Dataset Datasets.""" + query = base.get_query_string(search_opts, limit=limit, marker=marker, + sort_by=sort_by, reverse=reverse) + url = "/datasets%s" % query + return self._page(url, 'datasets', limit) + + def get(self, dataset_id, show_progress=False): + """Get information about a Dataset.""" + url = ('/datasets/%(dataset_id)s?%(params)s' % + {"dataset_id": dataset_id, + "params": parse.urlencode({"show_progress": show_progress})}) + + return self._get(url, 'dataset') + + def delete(self, dataset_id): + """Delete a Dataset Dataset.""" + self._delete('/datasets/%s' % dataset_id) diff --git a/meteosclient/api/experiments.py b/meteosclient/api/experiments.py new file mode 100644 index 0000000..4168cd1 --- /dev/null +++ b/meteosclient/api/experiments.py @@ -0,0 +1,64 @@ +# Copyright (c) 2013 Mirantis Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or +# implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from six.moves.urllib import parse + +from meteosclient.api import base + + +class Experiment(base.Resource): + resource_name = 'Experiment' + + +class ExperimentManager(base.ResourceManager): + resource_class = Experiment + NotUpdated = base.NotUpdated() + + def create(self, display_name=None, display_description=None, + template_id=None, key_name=None, + neutron_management_network=None): + """Create a Experiment.""" + + data = { + 'display_name': display_name, + 'display_description': display_description, + 'template_id': template_id, + 'key_name': key_name, + 'neutron_management_network': neutron_management_network, + } + + body = {'experiment': data} + + return self._create('/experiments', body, 'experiment') + + def list(self, search_opts=None, limit=None, marker=None, + sort_by=None, reverse=None): + """Get a list of Experiment Experiments.""" + query = base.get_query_string(search_opts, limit=limit, marker=marker, + sort_by=sort_by, reverse=reverse) + url = "/experiments%s" % query + return self._page(url, 'experiments', limit) + + def get(self, experiment_id, show_progress=False): + """Get information about a Experiment.""" + url = ('/experiments/%(experiment_id)s?%(params)s' % + {"experiment_id": experiment_id, + "params": parse.urlencode({"show_progress": show_progress})}) + + return self._get(url, 'experiment') + + def delete(self, experiment_id): + """Delete a Experiment Experiment.""" + self._delete('/experiments/%s' % experiment_id) diff --git a/meteosclient/api/learnings.py b/meteosclient/api/learnings.py new file mode 100644 index 0000000..0941cba --- /dev/null +++ b/meteosclient/api/learnings.py @@ -0,0 +1,65 @@ +# Copyright (c) 2013 Mirantis Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or +# implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import base64 +from six.moves.urllib import parse + +from meteosclient.api import base + + +class Learning(base.Resource): + resource_name = 'Learning' + + +class LearningManager(base.ResourceManager): + resource_class = Learning + NotUpdated = base.NotUpdated() + + def create(self, display_name=None, display_description=None, + experiment_id=None, model_id=None, method=None, args=None): + """Create a Learning.""" + + data = { + 'display_name': display_name, + 'display_description': display_description, + 'experiment_id': experiment_id, + 'model_id': model_id, + 'method': method, + 'args': base64.b64encode(str(args)), + } + + body = {'learning': data} + + return self._create('/learnings', body, 'learning') + + def list(self, search_opts=None, limit=None, marker=None, + sort_by=None, reverse=None): + """Get a list of Learnings.""" + query = base.get_query_string(search_opts, limit=limit, marker=marker, + sort_by=sort_by, reverse=reverse) + url = "/learnings%s" % query + return self._page(url, 'learnings', limit) + + def get(self, learning_id, show_progress=False): + """Get information about a Learning.""" + url = ('/learnings/%(learning_id)s?%(params)s' % + {"learning_id": learning_id, + "params": parse.urlencode({"show_progress": show_progress})}) + + return self._get(url, 'learning') + + def delete(self, learning_id): + """Delete a Learning.""" + self._delete('/learnings/%s' % learning_id) diff --git a/meteosclient/api/models.py b/meteosclient/api/models.py new file mode 100644 index 0000000..5a8d4b9 --- /dev/null +++ b/meteosclient/api/models.py @@ -0,0 +1,72 @@ +# Copyright (c) 2013 Mirantis Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or +# implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import base64 +from six.moves.urllib import parse + +from meteosclient.api import base + + +class Model(base.Resource): + resource_name = 'Model' + + +class ModelManager(base.ResourceManager): + resource_class = Model + NotUpdated = base.NotUpdated() + + def create(self, display_name=None, display_description=None, + source_dataset_url=None, experiment_id=None, + model_type=None, model_params=None, dataset_format=None, + swift_tenant=None, swift_username=None, + swift_password=None): + """Create a Model.""" + + data = { + 'display_name': display_name, + 'display_description': display_description, + 'source_dataset_url': source_dataset_url, + 'experiment_id': experiment_id, + 'model_type': model_type, + 'model_params': base64.b64encode(model_params), + 'dataset_format': dataset_format, + 'swift_tenant': swift_tenant, + 'swift_username': swift_username, + 'swift_password': swift_password, + } + + body = {'model': data} + + return self._create('/models', body, 'model') + + def list(self, search_opts=None, limit=None, marker=None, + sort_by=None, reverse=None): + """Get a list of Model Models.""" + query = base.get_query_string(search_opts, limit=limit, marker=marker, + sort_by=sort_by, reverse=reverse) + url = "/models%s" % query + return self._page(url, 'models', limit) + + def get(self, model_id, show_progress=False): + """Get information about a Model.""" + url = ('/models/%(model_id)s?%(params)s' % + {"model_id": model_id, + "params": parse.urlencode({"show_progress": show_progress})}) + + return self._get(url, 'model') + + def delete(self, model_id): + """Delete a Model Model.""" + self._delete('/models/%s' % model_id) diff --git a/meteosclient/api/shell.py b/meteosclient/api/shell.py new file mode 100644 index 0000000..3a2a3f0 --- /dev/null +++ b/meteosclient/api/shell.py @@ -0,0 +1,330 @@ +# Copyright 2013 Red Hat, Inc. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +import argparse +import datetime +import inspect +import json +import os.path +import sys + +from meteosclient.openstack.common.apiclient import exceptions +from meteosclient.openstack.common import cliutils as utils + + +def _print_list_field(field): + return lambda obj: ', '.join(getattr(obj, field)) + + +def _filter_call_args(args, func, remap={}): + """Filter args according to func's parameter list. + + Take three arguments: + * args - a dictionary + * func - a function + * remap - a dictionary + Remove from dct all the keys which are not among the parameters + of func. Before filtering, remap the keys in the args dict + according to remap dict. + """ + + for name, new_name in remap.items(): + if name in args: + args[new_name] = args[name] + del args[name] + + valid_args = inspect.getargspec(func).args + for name in args.keys(): + if name not in valid_args: + print('WARNING: "%s" is not a valid parameter and will be ' + 'discarded from the request' % name) + del args[name] + + +def _show_dict(dict): + utils.print_dict(dict._info) + + +# +# Templates +# ~~~~~~~~ +# template-list +# +# template-show +# +# template-create [--json ] +# +# template-delete +# + +def do_template_list(cs, args): + """Print a list of available templates.""" + templates = cs.templates.list() + + columns = ('id', + 'name', + 'description', + 'status', + 'master_nodes', + 'worker_nodes', + 'spark_version') + utils.print_list(templates, columns) + + +@utils.arg('id', + metavar='', + help='ID of the template to show.') +def do_template_show(cs, args): + """Show details of a template.""" + template = cs.templates.get(args.id) + _show_dict(template) + + +@utils.arg('--json', + default=sys.stdin, + type=argparse.FileType('r'), + help='JSON representation of template.') +def do_template_create(cs, args): + """Create a template.""" + template = json.loads(args.json.read()) + + _filter_call_args(template, cs.templates.create) + _show_dict(cs.templates.create(**template)) + + +@utils.arg('id', + metavar='', + help='ID of the template to delete.') +def do_template_delete(cs, args): + """Delete a template.""" + cs.templates.delete(args.id) + + +# +# Experiments +# ~~~~~~~~ +# experiment-list +# +# experiment-show +# +# experiment-create [--json ] +# +# experiment-delete +# + +def do_experiment_list(cs, args): + """Print a list of available experiments.""" + experiments = cs.experiments.list() + + columns = ('id', + 'name', + 'description', + 'status', + 'created_at') + utils.print_list(experiments, columns) + + +@utils.arg('id', + metavar='', + help='ID of the experiment to show.') +def do_experiment_show(cs, args): + """Show details of a experiment.""" + experiment = cs.experiments.get(args.id) + _show_dict(experiment) + + +@utils.arg('--json', + default=sys.stdin, + type=argparse.FileType('r'), + help='JSON representation of experiment.') +def do_experiment_create(cs, args): + """Create a experiment.""" + experiment = json.loads(args.json.read()) + + _filter_call_args(experiment, cs.experiments.create) + _show_dict(cs.experiments.create(**experiment)) + + +@utils.arg('id', + metavar='', + help='ID of the experiment to delete.') +def do_experiment_delete(cs, args): + """Delete a experiment.""" + cs.experiments.delete(args.id) + +# +# Datasets +# ~~~~~~~~ +# dataset-list +# +# dataset-show +# +# dataset-create [--json ] +# +# dataset-delete +# + + +def do_dataset_list(cs, args): + """Print a list of available datasets.""" + datasets = cs.datasets.list() + + columns = ('id', + 'name', + 'description', + 'status', + 'source_dataset_url', + 'created_at') + utils.print_list(datasets, columns) + + +@utils.arg('id', + metavar='', + help='ID of the dataset to show.') +def do_dataset_show(cs, args): + """Show details of a dataset.""" + dataset = cs.datasets.get(args.id) + _show_dict(dataset) + + +@utils.arg('--json', + default=sys.stdin, + type=argparse.FileType('r'), + help='JSON representation of dataset.') +def do_dataset_create(cs, args): + """Create a dataset.""" + dataset = json.loads(args.json.read()) + + _filter_call_args(dataset, cs.datasets.create) + _show_dict(cs.datasets.create(**dataset)) + + +@utils.arg('id', + metavar='', + help='ID of the dataset to delete.') +def do_dataset_delete(cs, args): + """Delete a dataset.""" + cs.datasets.delete(args.id) + +# +# Models +# ~~~~~~~~ +# model-list +# +# model-show +# +# model-create [--json ] +# +# model-delete +# + + +def do_model_list(cs, args): + """Print a list of available models.""" + models = cs.models.list() + + columns = ('id', + 'name', + 'description', + 'status', + 'type', + 'source_dataset_url', + 'created_at') + utils.print_list(models, columns) + + +@utils.arg('id', + metavar='', + help='ID of the model to show.') +def do_model_show(cs, args): + """Show details of a model.""" + model = cs.models.get(args.id) + _show_dict(model) + + +@utils.arg('--json', + default=sys.stdin, + type=argparse.FileType('r'), + help='JSON representation of model.') +def do_model_create(cs, args): + """Create a model.""" + model = json.loads(args.json.read()) + + _filter_call_args(model, cs.models.create) + _show_dict(cs.models.create(**model)) + + +@utils.arg('id', + metavar='', + help='ID of the model to delete.') +def do_model_delete(cs, args): + """Delete a model.""" + cs.models.delete(args.id) + + +# +# Learnings +# ~~~~~~~~ +# learning-list +# +# learning-show +# +# learning-create [--json ] +# +# learning-delete +# + +def do_learning_list(cs, args): + """Print a list of available learnings.""" + learnings = cs.learnings.list() + + columns = ('id', + 'name', + 'description', + 'status', + 'args', + 'stdout', + 'created_at') + base64_params = ['args'] + utils.print_list(learnings, columns, base64_params=base64_params) + + +@utils.arg('id', + metavar='', + help='ID of the learning to show.') +def do_learning_show(cs, args): + """Show details of a learning.""" + learning = cs.learnings.get(args.id) + _show_dict(learning) + + +@utils.arg('--json', + default=sys.stdin, + type=argparse.FileType('r'), + help='JSON representation of learning.') +def do_learning_create(cs, args): + """Create a learning.""" + learning = json.loads(args.json.read()) + + _filter_call_args(learning, cs.learnings.create) + _show_dict(cs.learnings.create(**learning)) + + +@utils.arg('id', + metavar='', + help='ID of the learning to delete.') +def do_learning_delete(cs, args): + """Delete a learning.""" + cs.learnings.delete(args.id) diff --git a/meteosclient/api/templates.py b/meteosclient/api/templates.py new file mode 100644 index 0000000..546fd13 --- /dev/null +++ b/meteosclient/api/templates.py @@ -0,0 +1,69 @@ +# Copyright (c) 2013 Mirantis Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or +# implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from six.moves.urllib import parse + +from meteosclient.api import base + + +class Template(base.Resource): + resource_name = 'Template' + + +class TemplateManager(base.ResourceManager): + resource_class = Template + NotUpdated = base.NotUpdated() + + def create(self, display_name=None, display_description=None, + image_id=None, master_nodes_num=None, master_flavor_id=None, + worker_nodes_num=None, worker_flavor_id=None, + spark_version=None, floating_ip_pool=None): + """Create a Experiment Template.""" + + data = { + 'display_name': display_name, + 'display_description': display_description, + 'image_id': image_id, + 'master_nodes_num': master_nodes_num, + 'master_flavor_id': master_flavor_id, + 'worker_nodes_num': worker_nodes_num, + 'worker_flavor_id': worker_flavor_id, + 'spark_version': spark_version, + 'floating_ip_pool': floating_ip_pool, + } + + body = {'template': data} + + return self._create('/templates', body, 'template') + + def list(self, search_opts=None, limit=None, marker=None, + sort_by=None, reverse=None): + """Get a list of Experiment Templates.""" + query = base.get_query_string(search_opts, limit=limit, marker=marker, + sort_by=sort_by, reverse=reverse) + url = "/templates%s" % query + return self._page(url, 'templates', limit) + + def get(self, template_id, show_progress=False): + """Get information about a Template.""" + url = ('/templates/%(template_id)s?%(params)s' % + {"template_id": template_id, + "params": parse.urlencode({"show_progress": show_progress})}) + + return self._get(url, 'template') + + def delete(self, template_id): + """Delete a Experiment Template.""" + self._delete('/templates/%s' % template_id) diff --git a/meteosclient/client.py b/meteosclient/client.py new file mode 100644 index 0000000..f3ca4a1 --- /dev/null +++ b/meteosclient/client.py @@ -0,0 +1,47 @@ +# Copyright (c) 2014 Mirantis Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or +# implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from oslo_utils import importutils + + +class UnsupportedVersion(Exception): + """Indication for using an unsupported version of the API. + + Indicates that the user is trying to use an unsupported + version of the API. + """ + pass + + +def get_client_class(version): + version_map = { + '1.0': 'meteosclient.api.client.Client', + '1.1': 'meteosclient.api.client.Client', + } + try: + client_path = version_map[str(version)] + except (KeyError, ValueError): + supported_versions = ', '.join(version_map.keys()) + msg = ("Invalid client version '%(version)s'; must be one of: " + "%(versions)s") % {'version': version, + 'versions': supported_versions} + raise UnsupportedVersion(msg) + + return importutils.import_class(client_path) + + +def Client(version, *args, **kwargs): + client_class = get_client_class(version) + return client_class(*args, **kwargs) diff --git a/meteosclient/openstack/__init__.py b/meteosclient/openstack/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/meteosclient/openstack/common/__init__.py b/meteosclient/openstack/common/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/meteosclient/openstack/common/_i18n.py b/meteosclient/openstack/common/_i18n.py new file mode 100644 index 0000000..2a98b35 --- /dev/null +++ b/meteosclient/openstack/common/_i18n.py @@ -0,0 +1,45 @@ +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""oslo.i18n integration module. + +See http://docs.openstack.org/developer/oslo.i18n/usage.html + +""" + +try: + import oslo_i18n + + # NOTE(dhellmann): This reference to o-s-l-o will be replaced by the + # application name when this module is synced into the separate + # repository. It is OK to have more than one translation function + # using the same domain, since there will still only be one message + # catalog. + _translators = oslo_i18n.TranslatorFactory(domain='meteosclient') + + # The primary translation function using the well-known name "_" + _ = _translators.primary + + # Translators for log levels. + # + # The abbreviated names are meant to reflect the usual use of a short + # name like '_'. The "L" is for "log" and the other letter comes from + # the level. + _LI = _translators.log_info + _LW = _translators.log_warning + _LE = _translators.log_error + _LC = _translators.log_critical +except ImportError: + # NOTE(dims): Support for cases where a project wants to use + # code from oslo-incubator, but is not ready to be internationalized + # (like tempest) + _ = _LI = _LW = _LE = _LC = lambda x: x diff --git a/meteosclient/openstack/common/apiclient/__init__.py b/meteosclient/openstack/common/apiclient/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/meteosclient/openstack/common/apiclient/auth.py b/meteosclient/openstack/common/apiclient/auth.py new file mode 100644 index 0000000..0d16666 --- /dev/null +++ b/meteosclient/openstack/common/apiclient/auth.py @@ -0,0 +1,234 @@ +# Copyright 2013 OpenStack Foundation +# Copyright 2013 Spanish National Research Council. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +# E0202: An attribute inherited from %s hide this method +# pylint: disable=E0202 + +######################################################################## +# +# THIS MODULE IS DEPRECATED +# +# Please refer to +# https://etherpad.openstack.org/p/kilo-meteosclient-library-proposals for +# the discussion leading to this deprecation. +# +# We recommend checking out the python-openstacksdk project +# (https://launchpad.net/python-openstacksdk) instead. +# +######################################################################## + +import abc +import argparse +import os + +import six +from stevedore import extension + +from meteosclient.openstack.common.apiclient import exceptions + + +_discovered_plugins = {} + + +def discover_auth_systems(): + """Discover the available auth-systems. + + This won't take into account the old style auth-systems. + """ + global _discovered_plugins + _discovered_plugins = {} + + def add_plugin(ext): + _discovered_plugins[ext.name] = ext.plugin + + ep_namespace = "meteosclient.openstack.common.apiclient.auth" + mgr = extension.ExtensionManager(ep_namespace) + mgr.map(add_plugin) + + +def load_auth_system_opts(parser): + """Load options needed by the available auth-systems into a parser. + + This function will try to populate the parser with options from the + available plugins. + """ + group = parser.add_argument_group("Common auth options") + BaseAuthPlugin.add_common_opts(group) + for name, auth_plugin in six.iteritems(_discovered_plugins): + group = parser.add_argument_group( + "Auth-system '%s' options" % name, + conflict_handler="resolve") + auth_plugin.add_opts(group) + + +def load_plugin(auth_system): + try: + plugin_class = _discovered_plugins[auth_system] + except KeyError: + raise exceptions.AuthSystemNotFound(auth_system) + return plugin_class(auth_system=auth_system) + + +def load_plugin_from_args(args): + """Load required plugin and populate it with options. + + Try to guess auth system if it is not specified. Systems are tried in + alphabetical order. + + :type args: argparse.Namespace + :raises: AuthPluginOptionsMissing + """ + auth_system = args.os_auth_system + if auth_system: + plugin = load_plugin(auth_system) + plugin.parse_opts(args) + plugin.sufficient_options() + return plugin + + for plugin_auth_system in sorted(six.iterkeys(_discovered_plugins)): + plugin_class = _discovered_plugins[plugin_auth_system] + plugin = plugin_class() + plugin.parse_opts(args) + try: + plugin.sufficient_options() + except exceptions.AuthPluginOptionsMissing: + continue + return plugin + raise exceptions.AuthPluginOptionsMissing(["auth_system"]) + + +@six.add_metaclass(abc.ABCMeta) +class BaseAuthPlugin(object): + """Base class for authentication plugins. + + An authentication plugin needs to override at least the authenticate + method to be a valid plugin. + """ + + auth_system = None + opt_names = [] + common_opt_names = [ + "auth_system", + "username", + "password", + "tenant_name", + "token", + "auth_url", + ] + + def __init__(self, auth_system=None, **kwargs): + self.auth_system = auth_system or self.auth_system + self.opts = dict((name, kwargs.get(name)) + for name in self.opt_names) + + @staticmethod + def _parser_add_opt(parser, opt): + """Add an option to parser in two variants. + + :param opt: option name (with underscores) + """ + dashed_opt = opt.replace("_", "-") + env_var = "OS_%s" % opt.upper() + arg_default = os.environ.get(env_var, "") + arg_help = "Defaults to env[%s]." % env_var + parser.add_argument( + "--os-%s" % dashed_opt, + metavar="<%s>" % dashed_opt, + default=arg_default, + help=arg_help) + parser.add_argument( + "--os_%s" % opt, + metavar="<%s>" % dashed_opt, + help=argparse.SUPPRESS) + + @classmethod + def add_opts(cls, parser): + """Populate the parser with the options for this plugin. + """ + for opt in cls.opt_names: + # use `BaseAuthPlugin.common_opt_names` since it is never + # changed in child classes + if opt not in BaseAuthPlugin.common_opt_names: + cls._parser_add_opt(parser, opt) + + @classmethod + def add_common_opts(cls, parser): + """Add options that are common for several plugins. + """ + for opt in cls.common_opt_names: + cls._parser_add_opt(parser, opt) + + @staticmethod + def get_opt(opt_name, args): + """Return option name and value. + + :param opt_name: name of the option, e.g., "username" + :param args: parsed arguments + """ + return (opt_name, getattr(args, "os_%s" % opt_name, None)) + + def parse_opts(self, args): + """Parse the actual auth-system options if any. + + This method is expected to populate the attribute `self.opts` with a + dict containing the options and values needed to make authentication. + """ + self.opts.update(dict(self.get_opt(opt_name, args) + for opt_name in self.opt_names)) + + def authenticate(self, http_client): + """Authenticate using plugin defined method. + + The method usually analyses `self.opts` and performs + a request to authentication server. + + :param http_client: client object that needs authentication + :type http_client: HTTPClient + :raises: AuthorizationFailure + """ + self.sufficient_options() + self._do_authenticate(http_client) + + @abc.abstractmethod + def _do_authenticate(self, http_client): + """Protected method for authentication. + """ + + def sufficient_options(self): + """Check if all required options are present. + + :raises: AuthPluginOptionsMissing + """ + missing = [opt + for opt in self.opt_names + if not self.opts.get(opt)] + if missing: + raise exceptions.AuthPluginOptionsMissing(missing) + + @abc.abstractmethod + def token_and_endpoint(self, endpoint_type, service_type): + """Return token and endpoint. + + :param service_type: Service type of the endpoint + :type service_type: string + :param endpoint_type: Type of endpoint. + Possible values: public or publicURL, + internal or internalURL, + admin or adminURL + :type endpoint_type: string + :returns: tuple of token and endpoint strings + :raises: EndpointException + """ diff --git a/meteosclient/openstack/common/apiclient/exceptions.py b/meteosclient/openstack/common/apiclient/exceptions.py new file mode 100644 index 0000000..af524b8 --- /dev/null +++ b/meteosclient/openstack/common/apiclient/exceptions.py @@ -0,0 +1,479 @@ +# Copyright 2010 Jacob Kaplan-Moss +# Copyright 2011 Nebula, Inc. +# Copyright 2013 Alessio Ababilov +# Copyright 2013 OpenStack Foundation +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +""" +Exception definitions. +""" + +######################################################################## +# +# THIS MODULE IS DEPRECATED +# +# Please refer to +# https://etherpad.openstack.org/p/kilo-meteosclient-library-proposals for +# the discussion leading to this deprecation. +# +# We recommend checking out the python-openstacksdk project +# (https://launchpad.net/python-openstacksdk) instead. +# +######################################################################## + +import inspect +import sys + +import six + +from meteosclient.openstack.common._i18n import _ + + +class ClientException(Exception): + """The base exception class for all exceptions this library raises. + """ + pass + + +class ValidationError(ClientException): + """Error in validation on API client side.""" + pass + + +class UnsupportedVersion(ClientException): + """User is trying to use an unsupported version of the API.""" + pass + + +class CommandError(ClientException): + """Error in CLI tool.""" + pass + + +class AuthorizationFailure(ClientException): + """Cannot authorize API client.""" + pass + + +class ConnectionError(ClientException): + """Cannot connect to API service.""" + pass + + +class ConnectionRefused(ConnectionError): + """Connection refused while trying to connect to API service.""" + pass + + +class AuthPluginOptionsMissing(AuthorizationFailure): + """Auth plugin misses some options.""" + def __init__(self, opt_names): + super(AuthPluginOptionsMissing, self).__init__( + _("Authentication failed. Missing options: %s") % + ", ".join(opt_names)) + self.opt_names = opt_names + + +class AuthSystemNotFound(AuthorizationFailure): + """User has specified an AuthSystem that is not installed.""" + def __init__(self, auth_system): + super(AuthSystemNotFound, self).__init__( + _("AuthSystemNotFound: %r") % auth_system) + self.auth_system = auth_system + + +class NoUniqueMatch(ClientException): + """Multiple entities found instead of one.""" + pass + + +class EndpointException(ClientException): + """Something is rotten in Service Catalog.""" + pass + + +class EndpointNotFound(EndpointException): + """Could not find requested endpoint in Service Catalog.""" + pass + + +class AmbiguousEndpoints(EndpointException): + """Found more than one matching endpoint in Service Catalog.""" + def __init__(self, endpoints=None): + super(AmbiguousEndpoints, self).__init__( + _("AmbiguousEndpoints: %r") % endpoints) + self.endpoints = endpoints + + +class HttpError(ClientException): + """The base exception class for all HTTP exceptions. + """ + http_status = 0 + message = _("HTTP Error") + + def __init__(self, message=None, details=None, + response=None, request_id=None, + url=None, method=None, http_status=None): + self.http_status = http_status or self.http_status + self.message = message or self.message + self.details = details + self.request_id = request_id + self.response = response + self.url = url + self.method = method + formatted_string = "%s (HTTP %s)" % (self.message, self.http_status) + if request_id: + formatted_string += " (Request-ID: %s)" % request_id + super(HttpError, self).__init__(formatted_string) + + +class HTTPRedirection(HttpError): + """HTTP Redirection.""" + message = _("HTTP Redirection") + + +class HTTPClientError(HttpError): + """Client-side HTTP error. + + Exception for cases in which the client seems to have erred. + """ + message = _("HTTP Client Error") + + +class HttpServerError(HttpError): + """Server-side HTTP error. + + Exception for cases in which the server is aware that it has + erred or is incapable of performing the request. + """ + message = _("HTTP Server Error") + + +class MultipleChoices(HTTPRedirection): + """HTTP 300 - Multiple Choices. + + Indicates multiple options for the resource that the client may follow. + """ + + http_status = 300 + message = _("Multiple Choices") + + +class BadRequest(HTTPClientError): + """HTTP 400 - Bad Request. + + The request cannot be fulfilled due to bad syntax. + """ + http_status = 400 + message = _("Bad Request") + + +class Unauthorized(HTTPClientError): + """HTTP 401 - Unauthorized. + + Similar to 403 Forbidden, but specifically for use when authentication + is required and has failed or has not yet been provided. + """ + http_status = 401 + message = _("Unauthorized") + + +class PaymentRequired(HTTPClientError): + """HTTP 402 - Payment Required. + + Reserved for future use. + """ + http_status = 402 + message = _("Payment Required") + + +class Forbidden(HTTPClientError): + """HTTP 403 - Forbidden. + + The request was a valid request, but the server is refusing to respond + to it. + """ + http_status = 403 + message = _("Forbidden") + + +class NotFound(HTTPClientError): + """HTTP 404 - Not Found. + + The requested resource could not be found but may be available again + in the future. + """ + http_status = 404 + message = _("Not Found") + + +class MethodNotAllowed(HTTPClientError): + """HTTP 405 - Method Not Allowed. + + A request was made of a resource using a request method not supported + by that resource. + """ + http_status = 405 + message = _("Method Not Allowed") + + +class NotAcceptable(HTTPClientError): + """HTTP 406 - Not Acceptable. + + The requested resource is only capable of generating content not + acceptable according to the Accept headers sent in the request. + """ + http_status = 406 + message = _("Not Acceptable") + + +class ProxyAuthenticationRequired(HTTPClientError): + """HTTP 407 - Proxy Authentication Required. + + The client must first authenticate itself with the proxy. + """ + http_status = 407 + message = _("Proxy Authentication Required") + + +class RequestTimeout(HTTPClientError): + """HTTP 408 - Request Timeout. + + The server timed out waiting for the request. + """ + http_status = 408 + message = _("Request Timeout") + + +class Conflict(HTTPClientError): + """HTTP 409 - Conflict. + + Indicates that the request could not be processed because of conflict + in the request, such as an edit conflict. + """ + http_status = 409 + message = _("Conflict") + + +class Gone(HTTPClientError): + """HTTP 410 - Gone. + + Indicates that the resource requested is no longer available and will + not be available again. + """ + http_status = 410 + message = _("Gone") + + +class LengthRequired(HTTPClientError): + """HTTP 411 - Length Required. + + The request did not specify the length of its content, which is + required by the requested resource. + """ + http_status = 411 + message = _("Length Required") + + +class PreconditionFailed(HTTPClientError): + """HTTP 412 - Precondition Failed. + + The server does not meet one of the preconditions that the requester + put on the request. + """ + http_status = 412 + message = _("Precondition Failed") + + +class RequestEntityTooLarge(HTTPClientError): + """HTTP 413 - Request Entity Too Large. + + The request is larger than the server is willing or able to process. + """ + http_status = 413 + message = _("Request Entity Too Large") + + def __init__(self, *args, **kwargs): + try: + self.retry_after = int(kwargs.pop('retry_after')) + except (KeyError, ValueError): + self.retry_after = 0 + + super(RequestEntityTooLarge, self).__init__(*args, **kwargs) + + +class RequestUriTooLong(HTTPClientError): + """HTTP 414 - Request-URI Too Long. + + The URI provided was too long for the server to process. + """ + http_status = 414 + message = _("Request-URI Too Long") + + +class UnsupportedMediaType(HTTPClientError): + """HTTP 415 - Unsupported Media Type. + + The request entity has a media type which the server or resource does + not support. + """ + http_status = 415 + message = _("Unsupported Media Type") + + +class RequestedRangeNotSatisfiable(HTTPClientError): + """HTTP 416 - Requested Range Not Satisfiable. + + The client has asked for a portion of the file, but the server cannot + supply that portion. + """ + http_status = 416 + message = _("Requested Range Not Satisfiable") + + +class ExpectationFailed(HTTPClientError): + """HTTP 417 - Expectation Failed. + + The server cannot meet the requirements of the Expect request-header field. + """ + http_status = 417 + message = _("Expectation Failed") + + +class UnprocessableEntity(HTTPClientError): + """HTTP 422 - Unprocessable Entity. + + The request was well-formed but was unable to be followed due to semantic + errors. + """ + http_status = 422 + message = _("Unprocessable Entity") + + +class InternalServerError(HttpServerError): + """HTTP 500 - Internal Server Error. + + A generic error message, given when no more specific message is suitable. + """ + http_status = 500 + message = _("Internal Server Error") + + +# NotImplemented is a python keyword. +class HttpNotImplemented(HttpServerError): + """HTTP 501 - Not Implemented. + + The server either does not recognize the request method, or it lacks + the ability to fulfill the request. + """ + http_status = 501 + message = _("Not Implemented") + + +class BadGateway(HttpServerError): + """HTTP 502 - Bad Gateway. + + The server was acting as a gateway or proxy and received an invalid + response from the upstream server. + """ + http_status = 502 + message = _("Bad Gateway") + + +class ServiceUnavailable(HttpServerError): + """HTTP 503 - Service Unavailable. + + The server is currently unavailable. + """ + http_status = 503 + message = _("Service Unavailable") + + +class GatewayTimeout(HttpServerError): + """HTTP 504 - Gateway Timeout. + + The server was acting as a gateway or proxy and did not receive a timely + response from the upstream server. + """ + http_status = 504 + message = _("Gateway Timeout") + + +class HttpVersionNotSupported(HttpServerError): + """HTTP 505 - HttpVersion Not Supported. + + The server does not support the HTTP protocol version used in the request. + """ + http_status = 505 + message = _("HTTP Version Not Supported") + + +# _code_map contains all the classes that have http_status attribute. +_code_map = dict( + (getattr(obj, 'http_status', None), obj) + for name, obj in six.iteritems(vars(sys.modules[__name__])) + if inspect.isclass(obj) and getattr(obj, 'http_status', False) +) + + +def from_response(response, method, url): + """Returns an instance of :class:`HttpError` or subclass based on response. + + :param response: instance of `requests.Response` class + :param method: HTTP method used for request + :param url: URL used for request + """ + + req_id = response.headers.get("x-openstack-request-id") + # NOTE(hdd) true for older versions of nova and cinder + if not req_id: + req_id = response.headers.get("x-compute-request-id") + kwargs = { + "http_status": response.status_code, + "response": response, + "method": method, + "url": url, + "request_id": req_id, + } + if "retry-after" in response.headers: + kwargs["retry_after"] = response.headers["retry-after"] + + content_type = response.headers.get("Content-Type", "") + if content_type.startswith("application/json"): + try: + body = response.json() + except ValueError: + pass + else: + if isinstance(body, dict): + error = body.get(list(body)[0]) + if isinstance(error, dict): + kwargs["message"] = (error.get("message") or + error.get("faultstring")) + kwargs["details"] = (error.get("details") or + six.text_type(body)) + elif content_type.startswith("text/"): + kwargs["details"] = response.text + + try: + cls = _code_map[response.status_code] + except KeyError: + if 500 <= response.status_code < 600: + cls = HttpServerError + elif 400 <= response.status_code < 500: + cls = HTTPClientError + else: + cls = HttpError + return cls(**kwargs) diff --git a/meteosclient/openstack/common/cliutils.py b/meteosclient/openstack/common/cliutils.py new file mode 100644 index 0000000..f3ce667 --- /dev/null +++ b/meteosclient/openstack/common/cliutils.py @@ -0,0 +1,277 @@ +# Copyright 2012 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +# W0603: Using the global statement +# W0621: Redefining name %s from outer scope +# pylint: disable=W0603,W0621 + +from __future__ import print_function + +import base64 +import getpass +import inspect +import os +import sys +import textwrap + +from oslo_utils import encodeutils +from oslo_utils import strutils +import prettytable +import six +from six import moves + +from meteosclient.openstack.common._i18n import _ + + +class MissingArgs(Exception): + """Supplied arguments are not sufficient for calling a function.""" + def __init__(self, missing): + self.missing = missing + msg = _("Missing arguments: %s") % ", ".join(missing) + super(MissingArgs, self).__init__(msg) + + +def validate_args(fn, *args, **kwargs): + """Check that the supplied args are sufficient for calling a function. + + >>> validate_args(lambda a: None) + Traceback (most recent call last): + ... + MissingArgs: Missing argument(s): a + >>> validate_args(lambda a, b, c, d: None, 0, c=1) + Traceback (most recent call last): + ... + MissingArgs: Missing argument(s): b, d + + :param fn: the function to check + :param arg: the positional arguments supplied + :param kwargs: the keyword arguments supplied + """ + argspec = inspect.getargspec(fn) + + num_defaults = len(argspec.defaults or []) + required_args = argspec.args[:len(argspec.args) - num_defaults] + + def isbound(method): + return getattr(method, '__self__', None) is not None + + if isbound(fn): + required_args.pop(0) + + missing = [arg for arg in required_args if arg not in kwargs] + missing = missing[len(args):] + if missing: + raise MissingArgs(missing) + + +def arg(*args, **kwargs): + """Decorator for CLI args. + + Example: + + >>> @arg("name", help="Name of the new entity") + ... def entity_create(args): + ... pass + """ + def _decorator(func): + add_arg(func, *args, **kwargs) + return func + return _decorator + + +def env(*args, **kwargs): + """Returns the first environment variable set. + + If all are empty, defaults to '' or keyword arg `default`. + """ + for arg in args: + value = os.environ.get(arg) + if value: + return value + return kwargs.get('default', '') + + +def add_arg(func, *args, **kwargs): + """Bind CLI arguments to a shell.py `do_foo` function.""" + + if not hasattr(func, 'arguments'): + func.arguments = [] + + # NOTE(sirp): avoid dups that can occur when the module is shared across + # tests. + if (args, kwargs) not in func.arguments: + # Because of the semantics of decorator composition if we just append + # to the options list positional options will appear to be backwards. + func.arguments.insert(0, (args, kwargs)) + + +def unauthenticated(func): + """Adds 'unauthenticated' attribute to decorated function. + + Usage: + + >>> @unauthenticated + ... def mymethod(f): + ... pass + """ + func.unauthenticated = True + return func + + +def isunauthenticated(func): + """Checks if the function does not require authentication. + + Mark such functions with the `@unauthenticated` decorator. + + :returns: bool + """ + return getattr(func, 'unauthenticated', False) + + +def print_list(objs, fields, base64_params=[], formatters=None, sortby_index=0, + mixed_case_fields=None, field_labels=None): + """Print a list of objects as a table, one row per object. + + :param objs: iterable of :class:`Resource` + :param fields: attributes that correspond to columns, in order + :param base64_params: indicate a column which encoded by base64 + :param formatters: `dict` of callables for field formatting + :param sortby_index: index of the field for sorting table rows + :param mixed_case_fields: fields corresponding to object attributes that + have mixed case names (e.g., 'serverId') + :param field_labels: Labels to use in the heading of the table, default to + fields. + """ + formatters = formatters or {} + mixed_case_fields = mixed_case_fields or [] + field_labels = field_labels or fields + if len(field_labels) != len(fields): + raise ValueError(_("Field labels list %(labels)s has different number " + "of elements than fields list %(fields)s"), + {'labels': field_labels, 'fields': fields}) + + if sortby_index is None: + kwargs = {} + else: + kwargs = {'sortby': field_labels[sortby_index]} + pt = prettytable.PrettyTable(field_labels) + pt.align = 'l' + + for o in objs: + row = [] + for field in fields: + if field in formatters: + row.append(formatters[field](o)) + else: + if field in mixed_case_fields: + field_name = field.replace(' ', '_') + else: + field_name = field.lower().replace(' ', '_') + if field in base64_params: + data = base64.b64decode(getattr(o, field_name, '')) + else: + data = getattr(o, field_name, '') + row.append(data) + pt.add_row(row) + + if six.PY3: + print(encodeutils.safe_encode(pt.get_string(**kwargs)).decode()) + else: + print(encodeutils.safe_encode(pt.get_string(**kwargs))) + + +def print_dict(dct, dict_property="Property", wrap=0, dict_value='Value'): + """Print a `dict` as a table of two columns. + + :param dct: `dict` to print + :param dict_property: name of the first column + :param wrap: wrapping for the second column + :param dict_value: header label for the value (second) column + """ + pt = prettytable.PrettyTable([dict_property, dict_value]) + pt.align = 'l' + for k, v in sorted(dct.items()): + # convert dict to str to check length + if isinstance(v, dict): + v = six.text_type(v) + if wrap > 0: + v = textwrap.fill(six.text_type(v), wrap) + # if value has a newline, add in multiple rows + # e.g. fault with stacktrace + if v and isinstance(v, six.string_types) and r'\n' in v: + lines = v.strip().split(r'\n') + col1 = k + for line in lines: + pt.add_row([col1, line]) + col1 = '' + else: + pt.add_row([k, v]) + + if six.PY3: + print(encodeutils.safe_encode(pt.get_string()).decode()) + else: + print(encodeutils.safe_encode(pt.get_string())) + + +def get_password(max_password_prompts=3): + """Read password from TTY.""" + verify = strutils.bool_from_string(env("OS_VERIFY_PASSWORD")) + pw = None + if hasattr(sys.stdin, "isatty") and sys.stdin.isatty(): + # Check for Ctrl-D + try: + for __ in moves.range(max_password_prompts): + pw1 = getpass.getpass("OS Password: ") + if verify: + pw2 = getpass.getpass("Please verify: ") + else: + pw2 = pw1 + if pw1 == pw2 and pw1: + pw = pw1 + break + except EOFError: + pass + return pw + + +def service_type(stype): + """Adds 'service_type' attribute to decorated function. + + Usage: + + .. code-block:: python + + @service_type('volume') + def mymethod(f): + ... + """ + def inner(f): + f.service_type = stype + return f + return inner + + +def get_service_type(f): + """Retrieves service type from function.""" + return getattr(f, 'service_type', None) + + +def pretty_choice_list(l): + return ', '.join("'%s'" % i for i in l) + + +def exit(msg=''): + if msg: + print (msg, file=sys.stderr) + sys.exit(1) diff --git a/meteosclient/shell.py b/meteosclient/shell.py new file mode 100644 index 0000000..2fc9263 --- /dev/null +++ b/meteosclient/shell.py @@ -0,0 +1,726 @@ +# Copyright 2010 Jacob Kaplan-Moss +# Copyright 2011 OpenStack Foundation +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +### +# This code is taken from python-novaclient. Goal is minimal modification. +### + +""" +Command-line interface to the OpenStack Meteos API. +""" + +from __future__ import print_function +import argparse +import getpass +import logging +import sys +import warnings + +import six + + +HAS_KEYRING = False +all_errors = ValueError +try: + import keyring + HAS_KEYRING = True + try: + if isinstance(keyring.get_keyring(), keyring.backend.GnomeKeyring): + import gnomekeyring + all_errors = (ValueError, + gnomekeyring.IOError, + gnomekeyring.NoKeyringDaemonError) + except Exception: + pass +except ImportError: + pass + +from keystoneauth1.identity.generic import password +from keystoneauth1.identity.generic import token +from keystoneauth1.loading import session +from keystoneclient.auth.identity import v3 as identity +from oslo_utils import encodeutils +from oslo_utils import strutils + +from meteosclient.api import client +from meteosclient.api import shell as shell_api +from meteosclient.openstack.common.apiclient import auth +from meteosclient.openstack.common.apiclient import exceptions as exc +from meteosclient.openstack.common import cliutils +from meteosclient import version + +DEFAULT_API_VERSION = 'api' +DEFAULT_ENDPOINT_TYPE = 'publicURL' +DEFAULT_SERVICE_TYPE = 'machine-learning' + +logger = logging.getLogger(__name__) + + +def positive_non_zero_float(text): + if text is None: + return None + try: + value = float(text) + except ValueError: + msg = "%s must be a float" % text + raise argparse.ArgumentTypeError(msg) + if value <= 0: + msg = "%s must be greater than 0" % text + raise argparse.ArgumentTypeError(msg) + return value + + +class SecretsHelper(object): + def __init__(self, args, client): + self.args = args + self.client = client + self.key = None + + def _validate_string(self, text): + if text is None or len(text) == 0: + return False + return True + + def _make_key(self): + if self.key is not None: + return self.key + keys = [ + self.client.auth_url, + self.client.projectid, + self.client.user, + self.client.region_name, + self.client.endpoint_type, + self.client.service_type, + self.client.service_name, + self.client.volume_service_name, + ] + for (index, key) in enumerate(keys): + if key is None: + keys[index] = '?' + else: + keys[index] = str(keys[index]) + self.key = "/".join(keys) + return self.key + + def _prompt_password(self, verify=True): + pw = None + if hasattr(sys.stdin, 'isatty') and sys.stdin.isatty(): + # Check for Ctl-D + try: + while True: + pw1 = getpass.getpass('OS Password: ') + if verify: + pw2 = getpass.getpass('Please verify: ') + else: + pw2 = pw1 + if pw1 == pw2 and self._validate_string(pw1): + pw = pw1 + break + except EOFError: + pass + return pw + + def save(self, auth_token, management_url, tenant_id): + if not HAS_KEYRING or not self.args.os_cache: + return + if (auth_token == self.auth_token and + management_url == self.management_url): + # Nothing changed.... + return + if not all([management_url, auth_token, tenant_id]): + raise ValueError("Unable to save empty management url/auth token") + value = "|".join([str(auth_token), + str(management_url), + str(tenant_id)]) + keyring.set_password("meteosclient_auth", self._make_key(), value) + + @property + def password(self): + if self._validate_string(self.args.os_password): + return self.args.os_password + verify_pass = ( + strutils.bool_from_string(cliutils.env("OS_VERIFY_PASSWORD")) + ) + return self._prompt_password(verify_pass) + + @property + def management_url(self): + if not HAS_KEYRING or not self.args.os_cache: + return None + management_url = None + try: + block = keyring.get_password('meteosclient_auth', + self._make_key()) + if block: + _token, management_url, _tenant_id = block.split('|', 2) + except all_errors: + pass + return management_url + + @property + def auth_token(self): + # Now is where it gets complicated since we + # want to look into the keyring module, if it + # exists and see if anything was provided in that + # file that we can use. + if not HAS_KEYRING or not self.args.os_cache: + return None + token = None + try: + block = keyring.get_password('meteosclient_auth', + self._make_key()) + if block: + token, _management_url, _tenant_id = block.split('|', 2) + except all_errors: + pass + return token + + @property + def tenant_id(self): + if not HAS_KEYRING or not self.args.os_cache: + return None + tenant_id = None + try: + block = keyring.get_password('meteosclient_auth', + self._make_key()) + if block: + _token, _management_url, tenant_id = block.split('|', 2) + except all_errors: + pass + return tenant_id + + +class MeteosClientArgumentParser(argparse.ArgumentParser): + + def __init__(self, *args, **kwargs): + super(MeteosClientArgumentParser, self).__init__(*args, **kwargs) + + def error(self, message): + """error(message: string) + + Prints a usage message incorporating the message to stderr and + exits. + """ + self.print_usage(sys.stderr) + # FIXME(lzyeval): if changes occur in argparse.ArgParser._check_value + choose_from = ' (choose from' + progparts = self.prog.partition(' ') + self.exit(2, "error: %(errmsg)s\nTry '%(mainp)s help %(subp)s'" + " for more information.\n" % + {'errmsg': message.split(choose_from)[0], + 'mainp': progparts[0], + 'subp': progparts[2]}) + + +class OpenStackMeteosShell(object): + + def get_base_parser(self): + parser = MeteosClientArgumentParser( + prog='meteos', + description=__doc__.strip(), + epilog='See "meteos help COMMAND" ' + 'for help on a specific command.', + add_help=False, + formatter_class=OpenStackHelpFormatter, + ) + + # Global arguments + parser.add_argument('-h', '--help', + action='store_true', + help=argparse.SUPPRESS) + + parser.add_argument('--version', + action='version', + version=version.version_info.version_string()) + + parser.add_argument('--debug', + default=False, + action='store_true', + help="Print debugging output.") + + parser.add_argument('--os-cache', + default=strutils.bool_from_string( + cliutils.env('OS_CACHE', default=False)), + action='store_true', + help="Use the auth token cache. Defaults to False " + "if env[OS_CACHE] is not set.") + + +# TODO(mattf) - add get_timings support to Client +# parser.add_argument('--timings', +# default=False, +# action='store_true', +# help="Print call timing info") + +# TODO(mattf) - use timeout +# parser.add_argument('--timeout', +# default=600, +# metavar='', +# type=positive_non_zero_float, +# help="Set HTTP call timeout (in seconds)") + + parser.add_argument('--region-name', + metavar='', + default=cliutils.env('SAHARA_REGION_NAME', + 'OS_REGION_NAME'), + help='Defaults to env[OS_REGION_NAME].') + parser.add_argument('--region_name', + help=argparse.SUPPRESS) + + parser.add_argument('--service-type', + metavar='', + help='Defaults to machine-learning for all ' + 'actions.') + parser.add_argument('--service_type', + help=argparse.SUPPRESS) + +# NA +# parser.add_argument('--service-name', +# metavar='', +# default=utils.env('SAHARA_SERVICE_NAME'), +# help='Defaults to env[SAHARA_SERVICE_NAME]') +# parser.add_argument('--service_name', +# help=argparse.SUPPRESS) + +# NA +# parser.add_argument('--volume-service-name', +# metavar='', +# default=utils.env('NOVA_VOLUME_SERVICE_NAME'), +# help='Defaults to env[NOVA_VOLUME_SERVICE_NAME]') +# parser.add_argument('--volume_service_name', +# help=argparse.SUPPRESS) + + parser.add_argument('--endpoint-type', + metavar='', + default=cliutils.env( + 'SAHARA_ENDPOINT_TYPE', + 'OS_ENDPOINT_TYPE', + default=DEFAULT_ENDPOINT_TYPE), + help=('Defaults to env[SAHARA_ENDPOINT_TYPE] or' + ' env[OS_ENDPOINT_TYPE] or ') + + DEFAULT_ENDPOINT_TYPE + '.') + # NOTE(dtroyer): We can't add --endpoint_type here due to argparse + # thinking usage-list --end is ambiguous; but it + # works fine with only --endpoint-type present + # Go figure. I'm leaving this here for doc purposes. + # parser.add_argument('--endpoint_type', + # help=argparse.SUPPRESS) + + parser.add_argument('--meteos-api-version', + metavar='', + default=cliutils.env( + 'SAHARA_API_VERSION', + default=DEFAULT_API_VERSION), + help='Accepts "api", ' + 'defaults to env[SAHARA_API_VERSION].') + parser.add_argument('--meteos_api_version', + help=argparse.SUPPRESS) + + parser.add_argument('--bypass-url', + metavar='', + default=cliutils.env('BYPASS_URL', default=None), + dest='bypass_url', + help="Use this API endpoint instead of the " + "Service Catalog.") + parser.add_argument('--bypass_url', + help=argparse.SUPPRESS) + + parser.add_argument('--os-tenant-name', + default=cliutils.env('OS_TENANT_NAME'), + help='Defaults to env[OS_TENANT_NAME].') + + parser.add_argument('--os-tenant-id', + default=cliutils.env('OS_TENANT_ID'), + help='Defaults to env[OS_TENANT_ID].') + + parser.add_argument('--os-auth-system', + default=cliutils.env('OS_AUTH_SYSTEM'), + help='Defaults to env[OS_AUTH_SYSTEM].') + + parser.add_argument('--os-auth-token', + default=cliutils.env('OS_AUTH_TOKEN'), + help='Defaults to env[OS_AUTH_TOKEN].') + + # Use Keystoneclient/Keystoneauth API to parse authentication arguments + session.Session().register_argparse_arguments(parser) + identity.Password.register_argparse_arguments(parser) + + return parser + + def get_subcommand_parser(self, version): + parser = self.get_base_parser() + + self.subcommands = {} + subparsers = parser.add_subparsers(metavar='') + + try: + actions_module = { + 'api': shell_api, + }[version] + except KeyError: + actions_module = shell_api + actions_module = shell_api + + self._find_actions(subparsers, actions_module) + self._find_actions(subparsers, self) + + self._add_bash_completion_subparser(subparsers) + + return parser + + def _add_bash_completion_subparser(self, subparsers): + subparser = ( + subparsers.add_parser('bash_completion', + add_help=False, + formatter_class=OpenStackHelpFormatter) + ) + self.subcommands['bash_completion'] = subparser + subparser.set_defaults(func=self.do_bash_completion) + + def _find_actions(self, subparsers, actions_module): + for attr in (a for a in dir(actions_module) if a.startswith('do_')): + # I prefer to be hyphen-separated instead of underscores. + command = attr[3:].replace('_', '-') + callback = getattr(actions_module, attr) + desc = callback.__doc__ or '' + action_help = desc.strip() + arguments = getattr(callback, 'arguments', []) + + subparser = ( + subparsers.add_parser(command, + help=action_help, + description=desc, + add_help=False, + formatter_class=OpenStackHelpFormatter) + ) + subparser.add_argument('-h', '--help', + action='help', + help=argparse.SUPPRESS,) + self.subcommands[command] = subparser + for (args, kwargs) in arguments: + subparser.add_argument(*args, **kwargs) + subparser.set_defaults(func=callback) + + def setup_debugging(self, debug): + if not debug: + return + + streamformat = "%(levelname)s (%(module)s:%(lineno)d) %(message)s" + # Set up the root logger to debug so that the submodules can + # print debug messages + logging.basicConfig(level=logging.DEBUG, + format=streamformat) + + def _get_keystone_auth(self, session, auth_url, **kwargs): + auth_token = kwargs.pop('auth_token', None) + if auth_token: + return token.Token(auth_url, auth_token, **kwargs) + else: + return password.Password( + auth_url, + username=kwargs.pop('username'), + user_id=kwargs.pop('user_id'), + password=kwargs.pop('password'), + user_domain_id=kwargs.pop('user_domain_id'), + user_domain_name=kwargs.pop('user_domain_name'), + **kwargs) + + def main(self, argv): + + # Parse args once to find version and debug settings + parser = self.get_base_parser() + (options, args) = parser.parse_known_args(argv) + self.setup_debugging(options.debug) + self.options = options + + # NOTE(dtroyer): Hackery to handle --endpoint_type due to argparse + # thinking usage-list --end is ambiguous; but it + # works fine with only --endpoint-type present + # Go figure. + if '--endpoint_type' in argv: + spot = argv.index('--endpoint_type') + argv[spot] = '--endpoint-type' + + subcommand_parser = ( + self.get_subcommand_parser(options.meteos_api_version) + ) + self.parser = subcommand_parser + + if options.help or not argv: + subcommand_parser.print_help() + return 0 + + args = subcommand_parser.parse_args(argv) + + # Short-circuit and deal with help right away. + if args.func == self.do_help: + self.do_help(args) + return 0 + elif args.func == self.do_bash_completion: + self.do_bash_completion(args) + return 0 + +# (os_username, os_tenant_name, os_tenant_id, os_auth_url, +# os_region_name, os_auth_system, endpoint_type, insecure, +# service_type, service_name, volume_service_name, +# bypass_url, os_cache, cacert) = ( #, timeout) = ( +# args.os_username, +# args.os_tenant_name, args.os_tenant_id, +# args.os_auth_url, +# args.os_region_name, +# args.os_auth_system, +# args.endpoint_type, args.insecure, +# args.service_type, +# args.service_name, args.volume_service_name, +# args.bypass_url, args.os_cache, +# args.os_cacert, args.timeout) + (os_username, os_tenant_name, os_tenant_id, + os_auth_url, os_auth_system, endpoint_type, + service_type, bypass_url, os_cacert, insecure, region_name) = ( + (args.os_username, args.os_tenant_name, args.os_tenant_id, + args.os_auth_url, args.os_auth_system, args.endpoint_type, + args.service_type, args.bypass_url, args.os_cacert, args.insecure, + args.region_name) + ) + + if os_auth_system and os_auth_system != "keystone": + auth_plugin = auth.load_plugin(os_auth_system) + else: + auth_plugin = None + + # Fetched and set later as needed + os_password = None + + if not endpoint_type: + endpoint_type = DEFAULT_ENDPOINT_TYPE + + if not service_type: + service_type = DEFAULT_SERVICE_TYPE +# NA - there is only one service this CLI accesses +# service_type = utils.get_service_type(args.func) or service_type + + # FIXME(usrleon): Here should be restrict for project id same as + # for os_username or os_password but for compatibility it is not. + if not cliutils.isunauthenticated(args.func): + if auth_plugin: + auth_plugin.parse_opts(args) + + if not auth_plugin or not auth_plugin.opts: + if not os_username: + raise exc.CommandError("You must provide a username " + "via either --os-username or " + "env[OS_USERNAME]") + + if not os_auth_url: + if os_auth_system and os_auth_system != 'keystone': + os_auth_url = auth_plugin.get_auth_url() + + if not os_auth_url: + raise exc.CommandError("You must provide an auth url " + "via either --os-auth-url or " + "env[OS_AUTH_URL] or specify an " + "auth_system which defines a " + "default url with --os-auth-system " + "or env[OS_AUTH_SYSTEM]") + +# NA +# if (options.os_compute_api_version and +# options.os_compute_api_version != '1.0'): +# if not os_tenant_name and not os_tenant_id: +# raise exc.CommandError("You must provide a tenant name " +# "or tenant id via --os-tenant-name, " +# "--os-tenant-id, env[OS_TENANT_NAME] " +# "or env[OS_TENANT_ID]") +# +# if not os_auth_url: +# raise exc.CommandError("You must provide an auth url " +# "via either --os-auth-url or env[OS_AUTH_URL]") + +# NOTE: The Meteos client authenticates when you create it. So instead of +# creating here and authenticating later, which is what the novaclient +# does, we just create the client later. + + # Now check for the password/token of which pieces of the + # identifying keyring key can come from the underlying client + if not cliutils.isunauthenticated(args.func): + # NA - Client can't be used with SecretsHelper + # helper = SecretsHelper(args, self.cs.client) + if (auth_plugin and auth_plugin.opts and + "os_password" not in auth_plugin.opts): + use_pw = False + else: + use_pw = True + +# tenant_id, auth_token, management_url = (helper.tenant_id, +# helper.auth_token, +# helper.management_url) +# +# if tenant_id and auth_token and management_url: +# self.cs.client.tenant_id = tenant_id +# self.cs.client.auth_token = auth_token +# self.cs.client.management_url = management_url +# # authenticate just sets up some values in this case, no REST +# # calls +# self.cs.authenticate() + if use_pw: + # Auth using token must have failed or not happened + # at all, so now switch to password mode and save + # the token when its gotten... using our keyring + # saver + # os_password = helper.password + os_password = args.os_password + if not os_password: + raise exc.CommandError( + 'Expecting a password provided via either ' + '--os-password, env[OS_PASSWORD], or ' + 'prompted response') +# self.cs.client.password = os_password +# self.cs.client.keyring_saver = helper + + # V3 stuff + project_info_provided = (self.options.os_tenant_name or + self.options.os_tenant_id or + (self.options.os_project_name and + (self.options.os_project_domain_name or + self.options.os_project_domain_id)) or + self.options.os_project_id) + + if (not project_info_provided): + raise exc.CommandError( + ("You must provide a tenant_name, tenant_id, " + "project_id or project_name (with " + "project_domain_name or project_domain_id) via " + " --os-tenant-name (env[OS_TENANT_NAME])," + " --os-tenant-id (env[OS_TENANT_ID])," + " --os-project-id (env[OS_PROJECT_ID])" + " --os-project-name (env[OS_PROJECT_NAME])," + " --os-project-domain-id " + "(env[OS_PROJECT_DOMAIN_ID])" + " --os-project-domain-name " + "(env[OS_PROJECT_DOMAIN_NAME])")) + + if not os_auth_url: + raise exc.CommandError( + "You must provide an auth url " + "via either --os-auth-url or env[OS_AUTH_URL]") + + keystone_session = None + keystone_auth = None + if not auth_plugin: + project_id = args.os_project_id or args.os_tenant_id + project_name = args.os_project_name or args.os_tenant_name + + keystone_session = (session.Session(). + load_from_argparse_arguments(args)) + keystone_auth = self._get_keystone_auth( + keystone_session, + args.os_auth_url, + username=args.os_username, + user_id=args.os_user_id, + user_domain_id=args.os_user_domain_id, + user_domain_name=args.os_user_domain_name, + password=args.os_password, + auth_token=args.os_auth_token, + project_id=project_id, + project_name=project_name, + project_domain_id=args.os_project_domain_id, + project_domain_name=args.os_project_domain_name) + + self.cs = client.Client(username=os_username, + api_key=os_password, + project_id=os_tenant_id, + project_name=os_tenant_name, + auth_url=os_auth_url, + meteos_url=bypass_url, + endpoint_type=endpoint_type, + session=keystone_session, + auth=keystone_auth, + cacert=os_cacert, + insecure=insecure, + service_type=service_type, + region_name=region_name) + + args.func(self.cs, args) + +# TODO(mattf) - add get_timings support to Client +# if args.timings: +# self._dump_timings(self.cs.get_timings()) + + def _dump_timings(self, timings): + class Tyme(object): + def __init__(self, url, seconds): + self.url = url + self.seconds = seconds + results = [Tyme(url, end - start) for url, start, end in timings] + total = 0.0 + for tyme in results: + total += tyme.seconds + results.append(Tyme("Total", total)) + cliutils.print_list(results, ["url", "seconds"], sortby_index=None) + + def do_bash_completion(self, _args): + """Prints arguments for bash-completion. + + Prints all of the commands and options to stdout so that the + meteos.bash_completion script doesn't have to hard code them. + """ + commands = set() + options = set() + for sc_str, sc in self.subcommands.items(): + commands.add(sc_str) + for option in sc._optionals._option_string_actions.keys(): + options.add(option) + + commands.remove('bash-completion') + commands.remove('bash_completion') + print(' '.join(commands | options)) + + @cliutils.arg('command', metavar='', nargs='?', + help='Display help for .') + def do_help(self, args): + """Display help about this program or one of its subcommands.""" + if args.command: + if args.command in self.subcommands: + self.subcommands[args.command].print_help() + else: + raise exc.CommandError("'%s' is not a valid subcommand" % + args.command) + else: + self.parser.print_help() + + +# I'm picky about my shell help. +class OpenStackHelpFormatter(argparse.HelpFormatter): + def start_section(self, heading): + # Title-case the headings + heading = '%s%s' % (heading[0].upper(), heading[1:]) + super(OpenStackHelpFormatter, self).start_section(heading) + + +def main(): + try: + argv = [encodeutils.safe_decode(a) for a in sys.argv[1:]] + OpenStackMeteosShell().main(argv) + + except Exception as e: + logger.debug(e, exc_info=1) + print("ERROR: %s" % encodeutils.safe_encode(six.text_type(e)), + file=sys.stderr) + sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/meteosclient/tests/__init__.py b/meteosclient/tests/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/meteosclient/version.py b/meteosclient/version.py new file mode 100644 index 0000000..ae650b3 --- /dev/null +++ b/meteosclient/version.py @@ -0,0 +1,18 @@ +# Copyright (c) 2014 Mirantis Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or +# implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from pbr import version + +version_info = version.VersionInfo('python-meteosclient') diff --git a/openstack-common.conf b/openstack-common.conf new file mode 100644 index 0000000..1479ace --- /dev/null +++ b/openstack-common.conf @@ -0,0 +1,7 @@ +[DEFAULT] +base=meteosclient + +module=apiclient.auth +module=apiclient.exceptions +module=cliutils +module=_i18n diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..7445442 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,16 @@ +# The order of packages is significant, because pip processes them in the order +# of appearance. Changing the order has an impact on the overall integration +# process, which may cause wedges in the gate later. + +pbr>=1.6 # Apache-2.0 + +Babel>=2.3.4 # BSD +keystoneauth1>=2.10.0 # Apache-2.0 +oslo.log>=3.11.0 # Apache-2.0 +oslo.serialization>=1.10.0 # Apache-2.0 +oslo.i18n>=2.1.0 # Apache-2.0 +oslo.utils>=3.16.0 # Apache-2.0 +python-keystoneclient!=2.1.0,>=2.0.0 # Apache-2.0 +requests>=2.10.0 # Apache-2.0 +six>=1.9.0 # MIT +PrettyTable<0.8,>=0.7 # BSD diff --git a/run_tests.sh b/run_tests.sh new file mode 100755 index 0000000..73c3450 --- /dev/null +++ b/run_tests.sh @@ -0,0 +1,164 @@ +#!/bin/bash + +set -eu + +function usage { + echo "Usage: $0 [OPTION]..." + echo "Run python-meteosclient test suite" + echo "" + echo " -V, --virtual-env Always use virtualenv. Install automatically if not present" + echo " -N, --no-virtual-env Don't use virtualenv. Run tests in local environment" + echo " -s, --no-site-packages Isolate the virtualenv from the global Python environment" + echo " -x, --stop Stop running tests after the first error or failure." + echo " -f, --force Force a clean re-build of the virtual environment. Useful when dependencies have been added." + echo " -p, --pep8 Just run pep8" + echo " -P, --no-pep8 Don't run pep8" + echo " -c, --coverage Generate coverage report" + echo " -h, --help Print this usage message" + echo " --hide-elapsed Don't print the elapsed time for each test along with slow test list" + echo "" + echo "Note: with no options specified, the script will try to run the tests in a virtual environment," + echo " If no virtualenv is found, the script will ask if you would like to create one. If you " + echo " prefer to run tests NOT in a virtual environment, simply pass the -N option." + exit +} + +function process_option { + case "$1" in + -h|--help) usage;; + -V|--virtual-env) always_venv=1; never_venv=0;; + -N|--no-virtual-env) always_venv=0; never_venv=1;; + -s|--no-site-packages) no_site_packages=1;; + -f|--force) force=1;; + -p|--pep8) just_pep8=1;; + -P|--no-pep8) no_pep8=1;; + -c|--coverage) coverage=1;; + -*) testropts="$testropts $1";; + *) testrargs="$testrargs $1" + esac +} + +venv=.venv +with_venv=tools/with_venv.sh +always_venv=0 +never_venv=0 +force=0 +no_site_packages=0 +installvenvopts= +testrargs= +testropts= +wrapper="" +just_pep8=0 +no_pep8=0 +coverage=0 + +LANG=en_US.UTF-8 +LANGUAGE=en_US:en +LC_ALL=C + +for arg in "$@"; do + process_option $arg +done + +if [ $no_site_packages -eq 1 ]; then + installvenvopts="--no-site-packages" +fi + +function init_testr { + if [ ! -d .testrepository ]; then + ${wrapper} testr init + fi +} + +function run_tests { + # Cleanup *pyc + ${wrapper} find . -type f -name "*.pyc" -delete + + if [ $coverage -eq 1 ]; then + # Do not test test_coverage_ext when gathering coverage. + if [ "x$testrargs" = "x" ]; then + testrargs="^(?!.*test_coverage_ext).*$" + fi + export PYTHON="${wrapper} coverage run --source meteosclient --parallel-mode" + fi + # Just run the test suites in current environment + set +e + TESTRTESTS="$TESTRTESTS $testrargs" + echo "Running \`${wrapper} $TESTRTESTS\`" + ${wrapper} $TESTRTESTS + RESULT=$? + set -e + + copy_subunit_log + + return $RESULT +} + +function copy_subunit_log { + LOGNAME=`cat .testrepository/next-stream` + LOGNAME=$(($LOGNAME - 1)) + LOGNAME=".testrepository/${LOGNAME}" + cp $LOGNAME subunit.log +} + +function run_pep8 { + echo "Running flake8 ..." + ${wrapper} flake8 +} + +TESTRTESTS="testr run --parallel $testropts" + +if [ $never_venv -eq 0 ] +then + # Remove the virtual environment if --force used + if [ $force -eq 1 ]; then + echo "Cleaning virtualenv..." + rm -rf ${venv} + fi + if [ -e ${venv} ]; then + wrapper="${with_venv}" + else + if [ $always_venv -eq 1 ]; then + # Automatically install the virtualenv + python tools/install_venv.py $installvenvopts + wrapper="${with_venv}" + else + echo -e "No virtual environment found...create one? (Y/n) \c" + read use_ve + if [ "x$use_ve" = "xY" -o "x$use_ve" = "x" -o "x$use_ve" = "xy" ]; then + # Install the virtualenv and run the test suite in it + python tools/install_venv.py $installvenvopts + wrapper=${with_venv} + fi + fi + fi +fi + +# Delete old coverage data from previous runs +if [ $coverage -eq 1 ]; then + ${wrapper} coverage erase +fi + +if [ $just_pep8 -eq 1 ]; then + run_pep8 + exit +fi + +init_testr +run_tests + +# NOTE(sirp): we only want to run pep8 when we're running the full-test suite, +# not when we're running tests individually. To handle this, we need to +# distinguish between options (noseopts), which begin with a '-', and +# arguments (testrargs). +if [ -z "$testrargs" ]; then + if [ $no_pep8 -eq 0 ]; then + run_pep8 + fi +fi + +if [ $coverage -eq 1 ]; then + echo "Generating coverage report in covhtml/" + ${wrapper} coverage combine + ${wrapper} coverage html --include='meteosclient/*' --omit='meteosclient/openstack/common/*' -d covhtml -i +fi diff --git a/sample/data/decision_tree_data.txt b/sample/data/decision_tree_data.txt new file mode 100644 index 0000000..e8193d5 --- /dev/null +++ b/sample/data/decision_tree_data.txt @@ -0,0 +1,202 @@ +1 1:-5 2:3 3:1 4:-5 5:-4 6:3 +0 1:2 2:-5 3:0 4:0 5:1 6:-4 +1 1:-3 2:1 3:4 4:-5 5:-3 6:4 +0 1:4 2:-4 3:4 4:4 5:0 6:-1 +1 1:-4 2:4 3:0 4:-5 5:-4 6:0 +0 1:1 2:-4 3:3 4:3 5:0 6:-4 +1 1:-3 2:4 3:0 4:-4 5:-3 6:2 +0 1:2 2:-5 3:2 4:2 5:4 6:-3 +1 1:-1 2:0 3:1 4:-2 5:-5 6:4 +0 1:0 2:-4 3:1 4:1 5:4 6:-4 +1 1:-3 2:4 3:4 4:-5 5:-4 6:0 +0 1:2 2:-1 3:2 4:2 5:2 6:-4 +1 1:-1 2:4 3:2 4:-1 5:-3 6:0 +0 1:2 2:-1 3:1 4:1 5:1 6:-2 +1 1:-3 2:2 3:2 4:-4 5:-2 6:2 +0 1:3 2:-4 3:2 4:2 5:4 6:-5 +1 1:-5 2:0 3:1 4:-2 5:-4 6:2 +0 1:0 2:-1 3:1 4:1 5:3 6:-1 +1 1:-2 2:3 3:0 4:-5 5:-4 6:1 +0 1:0 2:-4 3:4 4:4 5:1 6:-2 +1 1:-4 2:1 3:3 4:-5 5:-3 6:1 +0 1:1 2:-4 3:0 4:0 5:4 6:-5 +1 1:-3 2:4 3:0 4:-2 5:-3 6:4 +0 1:0 2:-4 3:2 4:2 5:3 6:-4 +1 1:-5 2:1 3:3 4:-2 5:-1 6:4 +0 1:1 2:-1 3:0 4:0 5:0 6:-5 +1 1:-2 2:4 3:0 4:-2 5:-4 6:2 +0 1:2 2:-1 3:1 4:1 5:1 6:-5 +1 1:-1 2:4 3:3 4:-3 5:-4 6:1 +0 1:0 2:-4 3:1 4:1 5:2 6:-5 +1 1:-3 2:1 3:1 4:-4 5:-2 6:3 +0 1:3 2:-1 3:4 4:4 5:1 6:-3 +1 1:-4 2:1 3:1 4:-3 5:-5 6:4 +0 1:2 2:-2 3:3 4:3 5:0 6:-1 +1 1:-5 2:0 3:3 4:-2 5:-3 6:4 +0 1:2 2:-4 3:0 4:0 5:2 6:-5 +1 1:-5 2:3 3:1 4:-3 5:-3 6:0 +0 1:4 2:-5 3:2 4:2 5:0 6:-1 +1 1:-3 2:2 3:2 4:-3 5:-5 6:4 +0 1:4 2:-5 3:0 4:0 5:1 6:-4 +1 1:-5 2:1 3:4 4:-2 5:-1 6:4 +0 1:0 2:-3 3:2 4:2 5:3 6:-4 +1 1:-1 2:0 3:1 4:-5 5:-2 6:3 +0 1:4 2:-5 3:2 4:2 5:0 6:-5 +1 1:-4 2:2 3:3 4:-2 5:-5 6:3 +0 1:4 2:-1 3:1 4:1 5:1 6:-2 +1 1:-4 2:1 3:2 4:-1 5:-1 6:3 +0 1:3 2:-2 3:2 4:2 5:0 6:-4 +1 1:-5 2:3 3:1 4:-1 5:-2 6:1 +0 1:3 2:-5 3:0 4:0 5:3 6:-1 +1 1:-4 2:1 3:3 4:-2 5:-3 6:4 +0 1:4 2:-4 3:4 4:4 5:4 6:-2 +1 1:-1 2:4 3:3 4:-5 5:-2 6:2 +0 1:4 2:-5 3:4 4:4 5:4 6:-3 +1 1:-4 2:3 3:4 4:-2 5:-3 6:4 +0 1:4 2:-5 3:4 4:4 5:4 6:-3 +1 1:-1 2:3 3:0 4:-4 5:-5 6:1 +0 1:1 2:-1 3:0 4:0 5:4 6:-1 +1 1:-2 2:2 3:3 4:-5 5:-1 6:3 +0 1:2 2:-4 3:3 4:3 5:3 6:-1 +1 1:-1 2:4 3:4 4:-5 5:-5 6:2 +0 1:3 2:-3 3:1 4:1 5:1 6:-2 +1 1:-5 2:4 3:1 4:-3 5:-3 6:2 +0 1:0 2:-4 3:1 4:1 5:0 6:-5 +1 1:-4 2:1 3:4 4:-1 5:-3 6:1 +0 1:4 2:-3 3:0 4:0 5:2 6:-4 +1 1:-1 2:1 3:2 4:-4 5:-1 6:3 +0 1:3 2:-5 3:4 4:4 5:2 6:-4 +1 1:-4 2:4 3:4 4:-4 5:-4 6:3 +0 1:1 2:-5 3:3 4:3 5:3 6:-2 +1 1:-3 2:2 3:3 4:-3 5:-3 6:3 +0 1:1 2:-4 3:3 4:3 5:4 6:-5 +1 1:-3 2:1 3:4 4:-5 5:-3 6:3 +0 1:2 2:-5 3:4 4:4 5:0 6:-5 +1 1:-3 2:0 3:3 4:-5 5:-3 6:4 +0 1:1 2:-5 3:4 4:4 5:3 6:-2 +1 1:-1 2:2 3:3 4:-1 5:-3 6:4 +0 1:2 2:-2 3:2 4:2 5:1 6:-1 +1 1:-3 2:0 3:3 4:-1 5:-5 6:0 +0 1:1 2:-3 3:4 4:4 5:2 6:-5 +1 1:-5 2:3 3:1 4:-4 5:-2 6:4 +0 1:0 2:-5 3:1 4:1 5:0 6:-2 +1 1:-3 2:3 3:4 4:-5 5:-1 6:2 +0 1:0 2:-2 3:0 4:0 5:1 6:-2 +1 1:-2 2:2 3:4 4:-2 5:-3 6:4 +0 1:1 2:-5 3:4 4:4 5:1 6:-5 +1 1:-2 2:2 3:3 4:-1 5:-2 6:3 +0 1:3 2:-5 3:1 4:1 5:4 6:-2 +1 1:-4 2:1 3:2 4:-4 5:-2 6:2 +0 1:0 2:-2 3:3 4:3 5:2 6:-2 +1 1:-3 2:2 3:2 4:-4 5:-2 6:1 +0 1:0 2:-2 3:4 4:4 5:1 6:-5 +1 1:-4 2:0 3:3 4:-2 5:-4 6:0 +0 1:2 2:-4 3:3 4:3 5:0 6:-3 +1 1:-5 2:4 3:0 4:-3 5:-3 6:4 +0 1:3 2:-1 3:3 4:3 5:3 6:-5 +1 1:-1 2:2 3:0 4:-5 5:-2 6:2 +0 1:1 2:-2 3:1 4:1 5:1 6:-3 +1 1:-2 2:3 3:0 4:-5 5:-1 6:0 +0 1:1 2:-1 3:2 4:2 5:2 6:-1 +1 1:-3 2:2 3:0 4:-4 5:-5 6:3 +0 1:0 2:-2 3:3 4:3 5:1 6:-2 +1 1:-5 2:1 3:2 4:-1 5:-1 6:0 +0 1:2 2:-1 3:1 4:1 5:1 6:-1 +1 1:-2 2:2 3:1 4:-4 5:-2 6:0 +0 1:1 2:-1 3:0 4:0 5:3 6:-5 +1 1:-1 2:1 3:0 4:-3 5:-3 6:0 +0 1:1 2:-4 3:0 4:0 5:2 6:-3 +1 1:-1 2:0 3:0 4:-4 5:-4 6:0 +0 1:0 2:-4 3:3 4:3 5:4 6:-5 +1 1:-5 2:2 3:0 4:-2 5:-5 6:4 +0 1:4 2:-1 3:1 4:1 5:1 6:-5 +1 1:-5 2:4 3:3 4:-1 5:-5 6:4 +0 1:4 2:-3 3:2 4:2 5:2 6:-2 +1 1:-2 2:2 3:1 4:-5 5:-2 6:2 +0 1:1 2:-3 3:4 4:4 5:2 6:-1 +1 1:-3 2:2 3:2 4:-2 5:-2 6:2 +0 1:2 2:-2 3:3 4:3 5:2 6:-5 +1 1:-1 2:1 3:3 4:-4 5:-3 6:1 +0 1:2 2:-2 3:2 4:2 5:3 6:-3 +1 1:-2 2:0 3:4 4:-3 5:-4 6:1 +0 1:3 2:-2 3:3 4:3 5:2 6:-5 +1 1:-5 2:1 3:0 4:-2 5:-5 6:0 +0 1:1 2:-3 3:4 4:4 5:3 6:-5 +1 1:-5 2:0 3:4 4:-1 5:-2 6:3 +0 1:2 2:-2 3:1 4:1 5:1 6:-2 +1 1:-1 2:4 3:0 4:-3 5:-2 6:1 +0 1:2 2:-4 3:1 4:1 5:0 6:-2 +1 1:-4 2:1 3:1 4:-5 5:-5 6:3 +0 1:2 2:-5 3:0 4:0 5:0 6:-1 +1 1:-3 2:0 3:4 4:-1 5:-4 6:1 +0 1:3 2:-3 3:2 4:2 5:0 6:-4 +1 1:-4 2:2 3:4 4:-3 5:-5 6:2 +0 1:3 2:-4 3:1 4:1 5:1 6:-4 +1 1:-1 2:0 3:3 4:-3 5:-3 6:2 +0 1:0 2:-5 3:2 4:2 5:4 6:-1 +1 1:-5 2:2 3:0 4:-3 5:-4 6:1 +0 1:4 2:-4 3:0 4:0 5:2 6:-2 +1 1:-4 2:2 3:3 4:-4 5:-4 6:1 +0 1:1 2:-5 3:4 4:4 5:4 6:-1 +1 1:-3 2:2 3:3 4:-2 5:-4 6:0 +0 1:0 2:-5 3:0 4:0 5:2 6:-5 +1 1:-2 2:1 3:2 4:-5 5:-1 6:1 +0 1:1 2:-3 3:1 4:1 5:0 6:-3 +1 1:-3 2:1 3:3 4:-4 5:-5 6:0 +0 1:4 2:-3 3:1 4:1 5:4 6:-4 +1 1:-1 2:1 3:1 4:-1 5:-4 6:4 +0 1:4 2:-2 3:3 4:3 5:4 6:-4 +1 1:-3 2:1 3:0 4:-5 5:-1 6:2 +0 1:3 2:-2 3:2 4:2 5:4 6:-5 +1 1:-2 2:3 3:1 4:-5 5:-3 6:0 +0 1:3 2:-1 3:1 4:1 5:1 6:-2 +1 1:-2 2:4 3:0 4:-1 5:-5 6:2 +0 1:0 2:-2 3:4 4:4 5:0 6:-2 +1 1:-1 2:1 3:3 4:-4 5:-1 6:4 +0 1:0 2:-5 3:0 4:0 5:3 6:-2 +1 1:-4 2:4 3:0 4:-3 5:-1 6:4 +0 1:1 2:-2 3:2 4:2 5:4 6:-2 +1 1:-2 2:3 3:0 4:-4 5:-4 6:2 +0 1:3 2:-5 3:4 4:4 5:0 6:-1 +1 1:-1 2:1 3:4 4:-2 5:-2 6:4 +0 1:4 2:-5 3:3 4:3 5:0 6:-4 +1 1:-2 2:4 3:4 4:-1 5:-3 6:0 +0 1:3 2:-2 3:1 4:1 5:4 6:-1 +1 1:-1 2:2 3:1 4:-2 5:-3 6:4 +0 1:4 2:-5 3:3 4:3 5:3 6:-5 +1 1:-3 2:2 3:1 4:-3 5:-1 6:2 +0 1:1 2:-3 3:0 4:0 5:2 6:-2 +1 1:-1 2:4 3:0 4:-1 5:-3 6:3 +0 1:3 2:-5 3:4 4:4 5:4 6:-3 +1 1:-2 2:1 3:0 4:-5 5:-1 6:2 +0 1:0 2:-1 3:0 4:0 5:1 6:-3 +1 1:-5 2:3 3:2 4:-5 5:-5 6:0 +0 1:0 2:-2 3:4 4:4 5:3 6:-3 +1 1:-4 2:4 3:4 4:-3 5:-1 6:3 +0 1:1 2:-4 3:1 4:1 5:2 6:-1 +1 1:-1 2:0 3:1 4:-3 5:-5 6:1 +0 1:4 2:-5 3:3 4:3 5:2 6:-2 +1 1:-1 2:0 3:1 4:-2 5:-3 6:1 +0 1:1 2:-3 3:3 4:3 5:1 6:-3 +1 1:-2 2:3 3:0 4:-1 5:-3 6:3 +0 1:0 2:-4 3:3 4:3 5:3 6:-4 +1 1:-2 2:1 3:3 4:-1 5:-3 6:0 +0 1:0 2:-5 3:3 4:3 5:1 6:-2 +1 1:-4 2:0 3:4 4:-4 5:-4 6:3 +0 1:4 2:-3 3:4 4:4 5:2 6:-5 +1 1:-3 2:4 3:1 4:-2 5:-2 6:1 +0 1:1 2:-1 3:2 4:2 5:4 6:-1 +1 1:-5 2:3 3:2 4:-5 5:-4 6:3 +0 1:0 2:-3 3:2 4:2 5:3 6:-5 +1 1:-5 2:1 3:0 4:-1 5:-5 6:0 +0 1:2 2:-4 3:2 4:2 5:1 6:-1 +1 1:-3 2:0 3:2 4:-4 5:-4 6:4 +0 1:0 2:-5 3:3 4:3 5:3 6:-4 +1 1:-5 2:2 3:4 4:-1 5:-4 6:1 +0 1:1 2:-5 3:1 4:1 5:2 6:-2 +1 1:-1 2:4 3:1 4:-5 5:-1 6:3 +0 1:0 2:-5 3:2 4:2 5:3 6:-3 +1 1:-5 2:1 3:3 4:-3 5:-5 6:1 +0 1:3 2:-1 3:4 4:4 5:0 6:-2 +1 1:-5 2:2 3:3 4:-1 5:-1 6:0 +0 1:1 2:-2 3:1 4:1 5:3 6:-3 diff --git a/sample/data/kmeans_data.txt b/sample/data/kmeans_data.txt new file mode 100644 index 0000000..850786a --- /dev/null +++ b/sample/data/kmeans_data.txt @@ -0,0 +1,50 @@ +1,1.0,1.0,0.0,0.0,0.0 +2,3.0,3.0,2.0,3.0,3.0 +3,5.0,5.0,5.0,4.0,5.0 +4,7.0,7.0,6.0,6.0,7.0 +5,7.0,8.0,7.0,8.0,8.0 +6,0.0,1.0,1.0,0.0,1.0 +7,3.0,2.0,2.0,2.0,3.0 +8,4.0,4.0,4.0,5.0,5.0 +9,6.0,6.0,6.0,7.0,6.0 +10,8.0,8.0,8.0,7.0,7.0 +11,0.0,1.0,0.0,0.0,0.0 +12,3.0,3.0,2.0,2.0,3.0 +13,5.0,5.0,5.0,4.0,5.0 +14,7.0,6.0,7.0,7.0,7.0 +15,8.0,7.0,8.0,8.0,7.0 +16,1.0,1.0,1.0,0.0,1.0 +17,2.0,2.0,2.0,2.0,3.0 +18,5.0,5.0,5.0,5.0,5.0 +19,6.0,6.0,6.0,7.0,6.0 +20,7.0,7.0,8.0,7.0,7.0 +21,1.0,1.0,1.0,1.0,1.0 +22,3.0,2.0,3.0,2.0,3.0 +23,5.0,5.0,5.0,5.0,5.0 +24,6.0,7.0,7.0,7.0,7.0 +25,7.0,8.0,7.0,8.0,7.0 +26,1.0,0.0,0.0,1.0,1.0 +27,2.0,2.0,3.0,2.0,3.0 +28,4.0,5.0,4.0,4.0,4.0 +29,6.0,7.0,7.0,6.0,6.0 +30,7.0,7.0,8.0,7.0,8.0 +31,1.0,1.0,1.0,0.0,1.0 +32,2.0,2.0,3.0,3.0,2.0 +33,5.0,5.0,5.0,4.0,5.0 +34,7.0,7.0,6.0,6.0,7.0 +35,8.0,7.0,8.0,7.0,8.0 +36,1.0,0.0,0.0,1.0,0.0 +37,3.0,2.0,3.0,2.0,2.0 +38,5.0,4.0,4.0,4.0,5.0 +39,6.0,7.0,6.0,7.0,6.0 +40,8.0,7.0,7.0,7.0,7.0 +41,0.0,0.0,0.0,0.0,0.0 +42,3.0,2.0,3.0,2.0,2.0 +43,5.0,5.0,5.0,4.0,5.0 +44,6.0,7.0,7.0,6.0,6.0 +45,7.0,8.0,7.0,8.0,7.0 +46,1.0,0.0,0.0,1.0,1.0 +47,2.0,3.0,2.0,3.0,2.0 +48,4.0,4.0,4.0,4.0,4.0 +49,7.0,7.0,7.0,7.0,7.0 +50,7.0,7.0,7.0,7.0,8.0 diff --git a/sample/data/linear_data.txt b/sample/data/linear_data.txt new file mode 100644 index 0000000..1bef49f --- /dev/null +++ b/sample/data/linear_data.txt @@ -0,0 +1,182 @@ +4500,1,1,2016,5,0,40,50 +8000,2,1,2016,6,1,60,80 +9500,3,1,2016,0,2,88,92 +5000,4,1,2016,1,3,90,90 +0,5,1,2016,2,2,90,80 +4500,6,1,2016,3,3,80,90 +4000,7,1,2016,4,1,60,80 +4500,8,1,2016,5,0,40,50 +8000,9,1,2016,6,0,30,50 +9500,10,1,2016,0,0,40,50 +5000,11,1,2016,1,1,60,80 +0,12,1,2016,2,2,88,92 +5000,13,1,2016,3,3,90,90 +4500,14,1,2016,4,2,90,80 +5500,15,1,2016,5,3,80,90 +10000,16,1,2016,6,1,60,80 +9500,17,1,2016,0,0,40,50 +5000,18,1,2016,1,0,30,50 +0,19,1,2016,2,0,40,50 +4500,20,1,2016,3,1,60,80 +4000,21,1,2016,4,2,88,92 +4500,22,1,2016,5,3,90,90 +8000,23,1,2016,6,2,90,80 +9500,24,1,2016,0,3,80,90 +5000,25,1,2016,1,1,60,80 +0,26,1,2016,2,0,40,50 +4500,27,1,2016,3,0,30,50 +4000,28,1,2016,4,0,40,50 +4500,29,1,2016,5,1,60,80 +8000,30,1,2016,6,2,88,92 +9500,31,1,2016,0,3,90,90 +5000,1,2,2016,1,2,90,80 +0,2,2,2016,2,3,80,90 +4500,3,2,2016,3,1,60,80 +4000,4,2,2016,4,0,40,50 +5000,5,2,2016,5,0,30,50 +8500,6,2,2016,6,0,40,50 +9000,7,2,2016,0,1,60,80 +4500,8,2,2016,1,2,88,92 +0,9,2,2016,2,3,90,90 +4500,10,2,2016,3,2,90,80 +4000,11,2,2016,4,3,80,90 +4500,12,2,2016,5,1,60,80 +8000,13,2,2016,6,0,40,50 +9500,14,2,2016,0,0,30,50 +5000,15,2,2016,1,0,40,50 +0,16,2,2016,2,1,60,80 +4500,17,2,2016,3,2,88,92 +4000,18,2,2016,4,3,90,90 +5000,19,2,2016,5,2,90,80 +8500,20,2,2016,6,3,80,90 +9000,21,2,2016,0,1,60,80 +4500,22,2,2016,1,0,40,50 +0,23,2,2016,2,0,30,50 +4500,24,2,2016,3,0,40,50 +4000,25,2,2016,4,1,60,80 +4500,26,2,2016,5,2,88,92 +8000,27,2,2016,6,3,90,90 +9500,28,2,2016,0,2,90,80 +5000,29,2,2016,1,3,80,90 +0,1,3,2016,2,1,60,80 +5000,2,3,2016,3,0,40,50 +4500,3,3,2016,4,0,30,50 +5500,4,3,2016,5,0,40,50 +10000,5,3,2016,6,1,60,80 +9500,6,3,2016,0,2,88,92 +5000,7,3,2016,1,3,90,90 +0,8,3,2016,2,2,90,80 +4500,9,3,2016,3,3,80,90 +4000,10,3,2016,4,1,60,80 +4500,11,3,2016,5,0,40,50 +8000,12,3,2016,6,0,30,50 +9500,13,3,2016,0,0,40,50 +5000,14,3,2016,1,1,60,80 +0,15,3,2016,2,2,88,92 +4500,16,3,2016,3,3,90,90 +4000,17,3,2016,4,2,90,80 +4500,18,3,2016,5,3,80,90 +8000,19,3,2016,6,1,60,80 +9500,20,3,2016,0,0,40,50 +5000,21,3,2016,1,0,30,50 +0,22,3,2016,2,0,40,50 +4500,23,3,2016,3,1,60,80 +4000,24,3,2016,4,2,88,92 +4500,25,3,2016,5,3,90,90 +8000,26,3,2016,6,2,90,80 +9500,27,3,2016,0,3,80,90 +5000,28,3,2016,1,1,60,80 +0,29,3,2016,2,0,40,50 +5000,30,3,2016,3,0,30,50 +4500,31,3,2016,4,0,40,50 +5500,1,4,2016,5,1,60,80 +10000,2,4,2016,6,2,88,92 +9500,3,4,2016,0,3,90,90 +5000,4,4,2016,1,2,90,80 +0,5,4,2016,2,3,80,90 +4500,6,4,2016,3,1,60,80 +4000,7,4,2016,4,0,40,50 +4500,8,4,2016,5,0,30,50 +8000,9,4,2016,6,0,40,50 +9500,10,4,2016,0,1,60,80 +5000,11,4,2016,1,2,88,92 +0,12,4,2016,2,3,90,90 +4500,13,4,2016,3,2,90,80 +4000,14,4,2016,4,3,80,90 +4500,15,4,2016,5,1,60,80 +8000,16,4,2016,6,0,40,50 +9500,17,4,2016,0,0,30,50 +5000,18,4,2016,1,0,40,50 +0,19,4,2016,2,1,60,80 +4500,20,4,2016,3,2,88,92 +4000,21,4,2016,4,3,90,90 +4500,22,4,2016,5,2,90,80 +8000,23,4,2016,6,3,80,90 +9500,24,4,2016,0,1,60,80 +5000,25,4,2016,1,0,40,50 +0,26,4,2016,2,0,30,50 +5000,27,4,2016,3,0,40,50 +4500,28,4,2016,4,1,60,80 +5500,29,4,2016,5,2,88,92 +10000,30,4,2016,6,3,90,90 +9500,1,5,2016,0,2,90,80 +5000,2,5,2016,1,3,80,90 +0,3,5,2016,2,1,60,80 +4500,4,5,2016,3,0,40,50 +4000,5,5,2016,4,0,30,50 +4500,6,5,2016,5,0,40,50 +8000,7,5,2016,6,1,60,80 +9500,8,5,2016,0,2,88,92 +5000,9,5,2016,1,3,90,90 +0,10,5,2016,2,2,90,80 +4500,11,5,2016,3,3,80,90 +4000,12,5,2016,4,1,60,80 +4500,13,5,2016,5,0,40,50 +8000,14,5,2016,6,0,30,50 +9500,15,5,2016,0,0,40,50 +5000,16,5,2016,1,1,60,80 +0,17,5,2016,2,2,88,92 +4500,18,5,2016,3,3,90,90 +4000,19,5,2016,4,2,90,80 +4500,20,5,2016,5,3,80,90 +8000,21,5,2016,6,1,60,80 +9500,22,5,2016,0,0,40,50 +5000,23,5,2016,1,0,30,50 +0,24,5,2016,2,0,40,50 +5000,25,5,2016,3,1,60,80 +4500,26,5,2016,4,2,88,92 +5500,27,5,2016,5,3,90,90 +10000,28,5,2016,6,2,90,80 +9500,29,5,2016,0,3,80,90 +5000,30,5,2016,1,1,60,80 +0,31,5,2016,2,0,40,50 +4500,1,6,2016,3,0,30,50 +4000,2,6,2016,4,0,40,50 +4500,3,6,2016,5,1,60,80 +8000,4,6,2016,6,2,88,92 +9500,5,6,2016,0,3,90,90 +5000,6,6,2016,1,2,90,80 +0,7,6,2016,2,3,80,90 +4500,8,6,2016,3,1,60,80 +4000,9,6,2016,4,0,40,50 +4500,10,6,2016,5,0,30,50 +8000,11,6,2016,6,0,40,50 +9500,12,6,2016,0,1,60,80 +5000,13,6,2016,1,2,88,92 +0,14,6,2016,2,3,90,90 +4500,15,6,2016,3,2,90,80 +4000,16,6,2016,4,3,80,90 +4500,17,6,2016,5,1,60,80 +8000,18,6,2016,6,0,40,50 +9500,19,6,2016,0,0,30,50 +5000,20,6,2016,1,0,40,50 +0,21,6,2016,2,1,60,80 +5000,22,6,2016,3,2,88,92 +4500,23,6,2016,4,3,90,90 +5500,24,6,2016,5,2,90,80 +10000,25,6,2016,6,3,80,90 +9500,26,6,2016,0,1,60,80 +5000,27,6,2016,1,0,40,50 +0,28,6,2016,2,0,30,50 +4500,29,6,2016,3,0,40,50 +4000,30,6,2016,4,1,60,80 diff --git a/sample/data/logistic_data.txt b/sample/data/logistic_data.txt new file mode 100644 index 0000000..27d5278 --- /dev/null +++ b/sample/data/logistic_data.txt @@ -0,0 +1,11 @@ +1,500000,1,10,2016,6,0,68,50 +1,550000,2,10,2016,0,1,68,90 +1,300000,3,10,2016,1,0,60,55 +1,350000,4,10,2016,2,2,58,87 +0,0,5,10,2016,3,3,58,60 +1,400000,6,10,2016,4,3,60,60 +1,330000,7,10,2016,5,2,62,87 +1,550000,8,10,2016,6,1,66,92 +1,600000,9,10,2016,0,1,55,93 +1,330000,10,10,2016,1,0,57,55 +0,0,11,10,2016,3,3,90,90 diff --git a/sample/data/recommendation_data.txt b/sample/data/recommendation_data.txt new file mode 100644 index 0000000..cccbf0f --- /dev/null +++ b/sample/data/recommendation_data.txt @@ -0,0 +1,19 @@ +1,1,4.5 +1,2,1.5 +1,3,5.0 +1,4,2.0 +2,1,5.0 +2,2,1.0 +2,3,4.0 +2,4,1.0 +3,1,1.5 +3,2,4.0 +3,3,2.0 +3,4,5.0 +4,1,2.0 +4,2,5.0 +4,3,2.0 +4,4,4.0 +5,1,1.5 +5,2,4.5 +5,4,4.5 diff --git a/sample/json/dataset_download.json b/sample/json/dataset_download.json new file mode 100644 index 0000000..0ce8278 --- /dev/null +++ b/sample/json/dataset_download.json @@ -0,0 +1,10 @@ +{ + "source_dataset_url": "swift://meteos/linear_data.txt", + "display_name": "sample-data", + "display_description": "This is a sample dataset", + "method": "download", + "experiment_id": "", + "swift_tenant": "demo", + "swift_username": "demo", + "swift_password": "nova" +} diff --git a/sample/json/dataset_parse.json b/sample/json/dataset_parse.json new file mode 100644 index 0000000..b3f9aff --- /dev/null +++ b/sample/json/dataset_parse.json @@ -0,0 +1,11 @@ +{ + "source_dataset_url": "swift://meteos/linear_data.txt", + "display_name": "sample-data", + "display_description": "This is a sample dataset", + "method": "parse", + "params": [{"method": "filter", "args": "lambda l: l.split(',')[0] != '0'"}], + "experiment_id": "", + "swift_tenant": "demo", + "swift_username": "demo", + "swift_password": "nova" +} diff --git a/sample/json/experiment.json b/sample/json/experiment.json new file mode 100644 index 0000000..976c95d --- /dev/null +++ b/sample/json/experiment.json @@ -0,0 +1,7 @@ +{ + "display_name": "example-experiment", + "display_description": "This is a sample experiment", + "key_name": "", + "neutron_management_network": "", + "template_id": "" +} diff --git a/sample/json/learning.json b/sample/json/learning.json new file mode 100644 index 0000000..4870b8f --- /dev/null +++ b/sample/json/learning.json @@ -0,0 +1,8 @@ +{ + "display_name": "example-learning-job", + "display_description": "This is a sample job", + "experiment_id": "", + "model_id": "", + "method": "predict", + "args": "" +} diff --git a/sample/json/model_decision_tree.json b/sample/json/model_decision_tree.json new file mode 100644 index 0000000..6615a1d --- /dev/null +++ b/sample/json/model_decision_tree.json @@ -0,0 +1,12 @@ +{ + "display_name": "sample-tree-model", + "display_description": "Sample Decision Tree Model", + "source_dataset_url": "swift://meteos/decision_tree_data.txt", + "model_type": "DecisionTreeRegression", + "model_params": "{'numIterations': 100}", + "dataset_format": "libsvm", + "experiment_id": "", + "swift_tenant": "demo", + "swift_username": "demo", + "swift_password": "nova" +} diff --git a/sample/json/model_kmeans.json b/sample/json/model_kmeans.json new file mode 100644 index 0000000..4db427f --- /dev/null +++ b/sample/json/model_kmeans.json @@ -0,0 +1,11 @@ +{ + "display_name": "sample-kmeans-model", + "display_description": "Sample KMeans Model", + "source_dataset_url": "swift://meteos/kmeans_data.txt", + "model_type": "KMeans", + "model_params": "{'numIterations': 5, 'numClasses':5}", + "experiment_id": "", + "swift_tenant": "demo", + "swift_username": "demo", + "swift_password": "nova" +} diff --git a/sample/json/model_linear.json b/sample/json/model_linear.json new file mode 100644 index 0000000..5a6d16a --- /dev/null +++ b/sample/json/model_linear.json @@ -0,0 +1,8 @@ +{ + "display_name": "sample-linear-model", + "display_description": "Sample LinearRegression Model", + "source_dataset_url": "internal://", + "model_type": "LinearRegression", + "model_params": "{'numIterations': 100}", + "experiment_id": "" +} diff --git a/sample/json/model_logistic.json b/sample/json/model_logistic.json new file mode 100644 index 0000000..1108edb --- /dev/null +++ b/sample/json/model_logistic.json @@ -0,0 +1,11 @@ +{ + "display_name": "sample-logistic-model", + "display_description": "Sample LogisticRegression Model", + "source_dataset_url": "swift://meteos/logistic_data.txt", + "model_type": "LogisticRegression", + "model_params": "{'numIterations': 100}", + "experiment_id": "", + "swift_tenant": "demo", + "swift_username": "demo", + "swift_password": "nova" +} diff --git a/sample/json/model_recommendation.json b/sample/json/model_recommendation.json new file mode 100644 index 0000000..94fbefb --- /dev/null +++ b/sample/json/model_recommendation.json @@ -0,0 +1,11 @@ +{ + "display_name": "sample-recommendation-model", + "display_description": "Sample Recommendation Model", + "source_dataset_url": "swift://meteos/recommendation_data.txt", + "model_type": "Recommendation", + "model_params": "{'numIterations': 5}", + "experiment_id": "", + "swift_tenant": "demo", + "swift_username": "demo", + "swift_password": "nova" +} diff --git a/sample/json/template.json b/sample/json/template.json new file mode 100644 index 0000000..2a053dc --- /dev/null +++ b/sample/json/template.json @@ -0,0 +1,11 @@ +{ + "display_name": "example-template", + "display_description": "This is a sample template of experiment", + "image_id" : "", + "master_nodes_num": 1, + "master_flavor_id": "4", + "worker_nodes_num": 2, + "worker_flavor_id": "2", + "spark_version": "1.6.0", + "floating_ip_pool": "" +} diff --git a/setup.cfg b/setup.cfg new file mode 100644 index 0000000..db794e8 --- /dev/null +++ b/setup.cfg @@ -0,0 +1,39 @@ +[metadata] +name = python-meteosclient +summary = Client library for Meteos API +description-file = + README.rst +license = Apache License, Version 2.0 +author = OpenStack +author-email = openstack-dev@lists.openstack.org +home-page = https://github.com/openstack/python-meteosclient +classifier = + Environment :: OpenStack + Intended Audience :: Information Technology + Intended Audience :: System Administrators + License :: OSI Approved :: Apache Software License + Operating System :: POSIX :: Linux + Programming Language :: Python + Programming Language :: Python :: 2 + Programming Language :: Python :: 2.7 + Programming Language :: Python :: 3 + Programming Language :: Python :: 3.4 + +[global] +setup-hooks = pbr.hooks.setup_hook + +[files] +packages = + meteosclient + +[entry_points] +console_scripts = + meteos = meteosclient.shell:main + +[build_sphinx] +all_files = 1 +build-dir = doc/build +source-dir = doc/source + +[wheel] +universal = 1 diff --git a/setup.py b/setup.py new file mode 100644 index 0000000..782bb21 --- /dev/null +++ b/setup.py @@ -0,0 +1,29 @@ +# Copyright (c) 2013 Hewlett-Packard Development Company, L.P. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or +# implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# THIS FILE IS MANAGED BY THE GLOBAL REQUIREMENTS REPO - DO NOT EDIT +import setuptools + +# In python < 2.7.4, a lazy loading of package `pbr` will break +# setuptools if some other modules registered functions in `atexit`. +# solution from: http://bugs.python.org/issue15881#msg170215 +try: + import multiprocessing # noqa +except ImportError: + pass + +setuptools.setup( + setup_requires=['pbr>=1.8'], + pbr=True) diff --git a/test-requirements.txt b/test-requirements.txt new file mode 100644 index 0000000..afe12c7 --- /dev/null +++ b/test-requirements.txt @@ -0,0 +1,15 @@ +# The order of packages is significant, because pip processes them in the order +# of appearance. Changing the order has an impact on the overall integration +# process, which may cause wedges in the gate later. + +hacking<0.11,>=0.10.0 + +coverage>=3.6 # Apache-2.0 +mock>=2.0 # BSD +oslosphinx>=4.7.0 # Apache-2.0 +oslotest>=1.10.0 # Apache-2.0 +os-testr>=0.7.0 # Apache-2.0 +reno>=1.8.0 # Apache2 +requests-mock>=1.1 # Apache-2.0 +sphinx!=1.3b1,<1.4,>=1.2.1 # BSD +testrepository>=0.0.18 # Apache-2.0/BSD diff --git a/tools/install_venv.py b/tools/install_venv.py new file mode 100644 index 0000000..397cbe5 --- /dev/null +++ b/tools/install_venv.py @@ -0,0 +1,75 @@ +# Copyright 2010 United States Government as represented by the +# Administrator of the National Aeronautics and Space Administration. +# All Rights Reserved. +# +# Copyright 2010 OpenStack Foundation +# Copyright 2013 IBM Corp. +# Copyright (c) 2013 Hewlett-Packard Development Company, L.P. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +import ConfigParser +import os +import sys + +import install_venv_common as install_venv + + +def print_help(project, venv, root): + help = """ + %(project)s development environment setup is complete. + + %(project)s development uses virtualenv to track and manage Python + dependencies while in development and testing. + + To activate the %(project)s virtualenv for the extent of your current + shell session you can run: + + $ source %(venv)s/bin/activate + + Or, if you prefer, you can run commands in the virtualenv on a case by + case basis by running: + + $ %(root)s/tools/with_venv.sh + """ + print(help % dict(project=project, venv=venv, root=root)) + + +def main(argv): + root = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) + + if os.environ.get('tools_path'): + root = os.environ['tools_path'] + venv = os.path.join(root, '.venv') + if os.environ.get('venv'): + venv = os.environ['venv'] + + pip_requires = os.path.join(root, 'requirements.txt') + test_requires = os.path.join(root, 'test-requirements.txt') + py_version = "python%s.%s" % (sys.version_info[0], sys.version_info[1]) + setup_cfg = ConfigParser.ConfigParser() + setup_cfg.read('setup.cfg') + project = setup_cfg.get('metadata', 'name') + + install = install_venv.InstallVenv( + root, venv, pip_requires, test_requires, py_version, project) + options = install.parse_args(argv) + install.check_python_version() + install.check_dependencies() + install.create_virtualenv(no_site_packages=options.no_site_packages) + install.install_dependencies() + install.post_process() + print_help(project, venv, root) + +if __name__ == '__main__': + main(sys.argv) diff --git a/tools/install_venv_common.py b/tools/install_venv_common.py new file mode 100644 index 0000000..d9e4de0 --- /dev/null +++ b/tools/install_venv_common.py @@ -0,0 +1,212 @@ +# Copyright 2013 OpenStack Foundation +# Copyright 2013 IBM Corp. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""Provides methods needed by installation script for OpenStack development +virtual environments. + +Since this script is used to bootstrap a virtualenv from the system's Python +environment, it should be kept strictly compatible with Python 2.6. + +Synced in from openstack-common +""" + +from __future__ import print_function + +import optparse +import os +import subprocess +import sys + + +class InstallVenv(object): + + def __init__(self, root, venv, requirements, + test_requirements, py_version, + project): + self.root = root + self.venv = venv + self.requirements = requirements + self.test_requirements = test_requirements + self.py_version = py_version + self.project = project + + def die(self, message, *args): + print(message % args, file=sys.stderr) + sys.exit(1) + + def check_python_version(self): + if sys.version_info < (2, 6): + self.die("Need Python Version >= 2.6") + + def run_command_with_code(self, cmd, redirect_output=True, + check_exit_code=True): + """Runs a command in an out-of-process shell. + + Returns the output of that command. Working directory is self.root. + """ + if redirect_output: + stdout = subprocess.PIPE + else: + stdout = None + + proc = subprocess.Popen(cmd, cwd=self.root, stdout=stdout) + output = proc.communicate()[0] + if check_exit_code and proc.returncode != 0: + self.die('Command "%s" failed.\n%s', ' '.join(cmd), output) + return (output, proc.returncode) + + def run_command(self, cmd, redirect_output=True, check_exit_code=True): + return self.run_command_with_code(cmd, redirect_output, + check_exit_code)[0] + + def get_distro(self): + if (os.path.exists('/etc/fedora-release') or + os.path.exists('/etc/redhat-release')): + return Fedora( + self.root, self.venv, self.requirements, + self.test_requirements, self.py_version, self.project) + else: + return Distro( + self.root, self.venv, self.requirements, + self.test_requirements, self.py_version, self.project) + + def check_dependencies(self): + self.get_distro().install_virtualenv() + + def create_virtualenv(self, no_site_packages=True): + """Creates the virtual environment and installs PIP. + + Creates the virtual environment and installs PIP only into the + virtual environment. + """ + if not os.path.isdir(self.venv): + print('Creating venv...', end=' ') + if no_site_packages: + self.run_command(['virtualenv', '-q', '--no-site-packages', + self.venv]) + else: + self.run_command(['virtualenv', '-q', self.venv]) + print('done.') + else: + print("venv already exists...") + pass + + def pip_install(self, *args): + self.run_command(['tools/with_venv.sh', + 'pip', 'install', '--upgrade'] + list(args), + redirect_output=False) + + def install_dependencies(self): + print('Installing dependencies with pip (this can take a while)...') + + # First things first, make sure our venv has the latest pip and + # setuptools and pbr + self.pip_install('pip>=1.4') + self.pip_install('setuptools') + self.pip_install('pbr') + + self.pip_install('-r', self.requirements) + self.pip_install('-r', self.test_requirements) + + def post_process(self): + self.get_distro().post_process() + + def parse_args(self, argv): + """Parses command-line arguments.""" + parser = optparse.OptionParser() + parser.add_option('-n', '--no-site-packages', + action='store_true', + help="Do not inherit packages from global Python " + "install") + return parser.parse_args(argv[1:])[0] + + +class Distro(InstallVenv): + + def check_cmd(self, cmd): + return bool(self.run_command(['which', cmd], + check_exit_code=False).strip()) + + def install_virtualenv(self): + if self.check_cmd('virtualenv'): + return + + if self.check_cmd('easy_install'): + print('Installing virtualenv via easy_install...', end=' ') + if self.run_command(['easy_install', 'virtualenv']): + print('Succeeded') + return + else: + print('Failed') + + self.die('ERROR: virtualenv not found.\n\n%s development' + ' requires virtualenv, please install it using your' + ' favorite package management tool' % self.project) + + def post_process(self): + """Any distribution-specific post-processing gets done here. + + In particular, this is useful for applying patches to code inside + the venv. + """ + pass + + +class Fedora(Distro): + """This covers all Fedora-based distributions. + + Includes: Fedora, RHEL, CentOS, Scientific Linux + """ + + def check_pkg(self, pkg): + return self.run_command_with_code(['rpm', '-q', pkg], + check_exit_code=False)[1] == 0 + + def apply_patch(self, originalfile, patchfile): + self.run_command(['patch', '-N', originalfile, patchfile], + check_exit_code=False) + + def install_virtualenv(self): + if self.check_cmd('virtualenv'): + return + + if not self.check_pkg('python-virtualenv'): + self.die("Please install 'python-virtualenv'.") + + super(Fedora, self).install_virtualenv() + + def post_process(self): + """Workaround for a bug in eventlet. + + This currently affects RHEL6.1, but the fix can safely be + applied to all RHEL and Fedora distributions. + + This can be removed when the fix is applied upstream. + + Nova: https://bugs.launchpad.net/nova/+bug/884915 + Upstream: https://bitbucket.org/eventlet/eventlet/issue/89 + RHEL: https://bugzilla.redhat.com/958868 + """ + + if os.path.exists('contrib/redhat-eventlet.patch'): + # Install "patch" program if it's not there + if not self.check_pkg('patch'): + self.die("Please install 'patch'.") + + # Apply the eventlet patch + self.apply_patch(os.path.join(self.venv, 'lib', self.py_version, + 'site-packages', + 'eventlet/green/subprocess.py'), + 'contrib/redhat-eventlet.patch') diff --git a/tools/with_venv.sh b/tools/with_venv.sh new file mode 100755 index 0000000..c8d2940 --- /dev/null +++ b/tools/with_venv.sh @@ -0,0 +1,4 @@ +#!/bin/bash +TOOLS=`dirname $0` +VENV=$TOOLS/../.venv +source $VENV/bin/activate && $@ diff --git a/tox.ini b/tox.ini new file mode 100644 index 0000000..860a702 --- /dev/null +++ b/tox.ini @@ -0,0 +1,68 @@ +[tox] +envlist = py27,py34,pypy,pep8,releasenotes +minversion = 1.6 +skipsdist = True + +[testenv] +usedevelop = True +install_command = pip install -U {opts} {packages} +setenv = + VIRTUAL_ENV={envdir} + DISCOVER_DIRECTORY=meteosclient/tests/unit +deps = + -r{toxinidir}/requirements.txt + -r{toxinidir}/test-requirements.txt +commands = find . -type f -name "*.pyc" -delete + ostestr {posargs} +whitelist_externals = find +passenv = http_proxy HTTP_PROXY https_proxy HTTPS_PROXY no_proxy NO_PROXY + +[testenv:debug] +commands = oslo_debug_helper -t meteosclient/tests/unit {posargs} + +[testenv:debug-py27] +basepython = python2.7 +commands = oslo_debug_helper -t meteosclient/tests/unit {posargs} + +[testenv:debug-py34] +basepython = python3.4 +commands = oslo_debug_helper -t meteosclient/tests/unit {posargs} + +[testenv:cover] +commands = python setup.py test --coverage --testr-args='{posargs}' + +[tox:jenkins] +sitepackages = False + +[testenv:pep8] +sitepackages = False +commands = flake8 + +[testenv:doc8] +deps = + -r{toxinidir}/requirements.txt + -r{toxinidir}/test-requirements.txt + doc8 +commands = doc8 doc/source + +[testenv:venv] +commands = {posargs} + +[testenv:docs] +commands = + rm -rf doc/html doc/build + rm -rf doc/source/apidoc doc/source/api + python setup.py build_sphinx +whitelist_externals = + rm + +[testenv:releasenotes] +commands = sphinx-build -a -E -W -d releasenotes/build/doctrees -b html releasenotes/source releasenotes/build/html + +[flake8] +show-source = true +builtins = _ +exclude=.venv,.git,.tox,dist,doc,*openstack/common*,*lib/python*,*egg,tools + +[hacking] +import_exceptions = meteosclient.openstack.common._i18n._