diff --git a/README.rst b/README.rst index 5989a8aae..7079da180 100644 --- a/README.rst +++ b/README.rst @@ -6,45 +6,95 @@ with OpenStack clouds. The project aims to provide a consistent and complete set of interactions with OpenStack's many services, along with complete documentation, examples, and tools. -It also contains a simple interface layer. Clouds can do many things, but +It also contains an abstraction interface layer. Clouds can do many things, but there are probably only about 10 of them that most people care about with any regularity. If you want to do complicated things, the per-service oriented -portions of the SDK are for you. However, if what you want is to be able to +portions of the SDK are for you. However, if what you want to be able to write an application that talks to clouds no matter what crazy choices the deployer has made in an attempt to be more hipster than their self-entitled -narcissist peers, then the ``openstack.cloud`` layer is for you. +narcissist peers, then the Cloud Abstraction layer is for you. A Brief History --------------- +.. TODO(shade) This history section should move to the docs. We can put a + link to the published URL here in the README, but it's too long. + openstacksdk started its life as three different libraries: shade, os-client-config and python-openstacksdk. -``shade`` started its life as some code inside of OpenStack Infra's nodepool -project, and as some code inside of Ansible. Ansible had a bunch of different -OpenStack related modules, and there was a ton of duplicated code. Eventually, -between refactoring that duplication into an internal library, and adding logic -and features that the OpenStack Infra team had developed to run client -applications at scale, it turned out that we'd written nine-tenths of what we'd -need to have a standalone library. +``shade`` started its life as some code inside of OpenStack Infra's `nodepool`_ +project, and as some code inside of the `Ansible OpenStack Modules`_. +Ansible had a bunch of different OpenStack related modules, and there was a +ton of duplicated code. Eventually, between refactoring that duplication into +an internal library, and adding the logic and features that the OpenStack Infra +team had developed to run client applications at scale, it turned out that we'd +written nine-tenths of what we'd need to have a standalone library. + +Because of its background from nodepool, shade contained abstractions to +work around deployment differences and is resource oriented rather than service +oriented. This allows a user to think about Security Groups without having to +know whether Security Groups are provided by Nova or Neutron on a given cloud. +On the other hand, as an interface that provides an abstraction, it deviates +from the published OpenStack REST API and adds its own opinions, which may not +get in the way of more advanced users with specific needs. ``os-client-config`` was a library for collecting client configuration for -using an OpenStack cloud in a consistent and comprehensive manner. -In parallel, the python-openstacksdk team was working on a library to expose -the OpenStack APIs to developers in a consistent and predictable manner. After -a while it became clear that there was value in both a high-level layer that -contains business logic, a lower-level SDK that exposes services and their -resources as Python objects, and also to be able to make direct REST calls -when needed with a properly configured Session or Adapter from python-requests. -This led to the merger of the three projects. +using an OpenStack cloud in a consistent and comprehensive manner, which +introduced the ``clouds.yaml`` file for expressing named cloud configurations. -The contents of the shade library have been moved into ``openstack.cloud`` -and os-client-config has been moved in to ``openstack.config``. The next -release of shade will be a thin compatibility layer that subclasses the objects -from ``openstack.cloud`` and provides different argument defaults where needed -for compat. Similarly the next release of os-client-config will be a compat +``python-openstacksdk`` was a library that exposed the OpenStack APIs to +developers in a consistent and predictable manner. + +After a while it became clear that there was value in both the high-level +layer that contains additional business logic and the lower-level SDK that +exposes services and their resources faithfully and consistently as Python +objects. + +Even with both of those layers, it is still beneficial at times to be able to +make direct REST calls and to do so with the same properly configured +`Session`_ from `python-requests`_. + +This led to the merge of the three projects. + +The original contents of the shade library have been moved into +``openstack.cloud`` and os-client-config has been moved in to +``openstack.config``. The next release of shade will be a thin compatibility +layer that subclasses the objects from ``openstack.cloud`` and provides +different argument defaults where needed for compat. +Similarly the next release of os-client-config will be a compat layer shim around ``openstack.config``. +.. note:: + + The ``openstack.cloud.OpenStackCloud`` object and the + ``openstack.connection.Connection`` object are going to be merged. It is + recommended to not write any new code which consumes objects from the + ``openstack.cloud`` namespace until that merge is complete. + +.. _nodepool: https://docs.openstack.org/infra/nodepool/ +.. _Ansible OpenStack Modules: http://docs.ansible.com/ansible/latest/list_of_cloud_modules.html#openstack +.. _Session: http://docs.python-requests.org/en/master/user/advanced/#session-objects +.. _python-requests: http://docs.python-requests.org/en/master/ + +openstack +========= + +List servers using objects configured with the ``clouds.yaml`` file: + +.. code-block:: python + + import openstack + + # Initialize and turn on debug logging + openstack.enable_logging(debug=True) + + # Initialize cloud + conn = openstack.connect(cloud='mordred') + + for server in conn.compute.servers(): + print(server.to_dict()) + openstack.config ================ @@ -88,10 +138,10 @@ Create a server using objects configured with the ``clouds.yaml`` file: .. code-block:: python - import openstack.cloud + import openstack # Initialize and turn on debug logging - openstack.cloud.simple_logging(debug=True) + openstack.enable_logging(debug=True) # Initialize cloud # Cloud configs are read with openstack.config diff --git a/SHADE-MERGE-TODO.rst b/SHADE-MERGE-TODO.rst index 6155dc628..32baa0925 100644 --- a/SHADE-MERGE-TODO.rst +++ b/SHADE-MERGE-TODO.rst @@ -72,7 +72,6 @@ shade integration * Investigate auto-generating the bulk of shade's API based on introspection of SDK objects, leaving only the code with extra special logic in the shade layer. -* Rationalize openstack.util.enable_logging and shade.simple_logging. Service Proxies --------------- diff --git a/doc/source/user/guides/logging.rst b/doc/source/user/guides/logging.rst index 00014e6b7..a1bc944cd 100644 --- a/doc/source/user/guides/logging.rst +++ b/doc/source/user/guides/logging.rst @@ -1,56 +1,44 @@ +======= Logging ======= -Logging can save you time and effort when developing your code or looking -for help. If your code is not behaving how you expect it to, enabling and -configuring logging can quickly give you valuable insight into the root -cause of the issue. If you need help from the OpenStack community, the -logs can help the people there assist you. +.. note:: TODO(shade) This document is written from a shade POV. It needs to + be combined with the existing logging guide, but also the logging + systems need to be rationalized. -.. note:: By default, no logging is done. +`openstacksdk` uses `Python Logging`_. As `openstacksdk` is a library, it does +not configure logging handlers automatically, expecting instead for that to be +the purview of the consuming application. -Enable SDK Logging ------------------- +Simple Usage +------------ -To enable logging you use :func:`~openstack.utils.enable_logging`. +For consumers who just want to get a basic logging setup without thinking +about it too deeply, there is a helper method. If used, it should be called +before any other openstacksdk functionality. -The ``debug`` parameter controls the logging level. Set ``debug=True`` to -log debug and higher messages. Set ``debug=False`` to log warning and higher -messages. +.. autofunction:: openstack.enable_logging -To log debug and higher messages:: +.. code-block:: python - import sys - from openstack import utils - - utils.enable_logging(debug=True, stream=sys.stdout) - -The ``path`` parameter controls the location of a log file. If set, this -parameter will send log messages to a file using a -:py:class:`~logging.FileHandler`. - -To log messages to a file called ``openstack.log``:: - - from openstack import utils - - utils.enable_logging(debug=True, path='openstack.log') + import openstack + openstack.enable_logging() The ``stream`` parameter controls the stream where log message are written to. -If set to ``sys.stdout`` or ``sys.stderr``, this parameter will send log -messages to that stream using a :py:class:`~logging.StreamHandler` +It defaults to `sys.stdout` which will result in log messages being written +to STDOUT. It can be set to another output stream, or to ``None`` to disable +logging to the console. -To log messages to the console on ``stdout``:: - - import sys - from openstack import utils - - utils.enable_logging(debug=True, stream=sys.stdout) +The ``path`` parameter sets up logging to log to a file. By default, if +``path`` is given and ``stream`` is not, logging will only go to ``path``. You can combine the ``path`` and ``stream`` parameters to log to both places simultaneously. To log messages to a file called ``openstack.log`` and the console on -``stdout``:: +``stdout``: + +.. code-block:: python import sys from openstack import utils @@ -58,23 +46,68 @@ To log messages to a file called ``openstack.log`` and the console on utils.enable_logging(debug=True, path='openstack.log', stream=sys.stdout) -Enable requests Logging ------------------------ +`openstack.enable_logging` also sets up a few other loggers and +squelches some warnings or log messages that are otherwise uninteresting or +unactionable by an openstacksdk user. -The SDK depends on a small number other libraries. Notably, it uses -`requests `_ for its transport layer. -To get even more information about the request/response cycle, you enable -logging of requests the same as you would any other library. +Advanced Usage +-------------- -To log messages to the console on ``stdout``:: +`openstacksdk` logs to a set of different named loggers. - import logging - import sys +Most of the logging is set up to log to the root ``openstack`` logger. +There are additional sub-loggers that are used at times, primarily so that a +user can decide to turn on or off a specific type of logging. They are listed +below. - logger = logging.getLogger('requests') - formatter = logging.Formatter( - '%(asctime)s %(levelname)s: %(name)s %(message)s') - console = logging.StreamHandler(sys.stdout) - console.setFormatter(formatter) - logger.setLevel(logging.DEBUG) - logger.addHandler(console) +openstack.config + Issues pertaining to configuration are logged to the ``openstack.config`` + logger. + +openstack.task_manager + `openstacksdk` uses a Task Manager to perform remote calls. The + ``openstack.task_manager`` logger emits messages at the start and end + of each Task announcing what it is going to run and then what it ran and + how long it took. Logging ``openstack.task_manager`` is a good way to + get a trace of external actions `openstacksdk` is taking without full + `HTTP Tracing`_. + +openstack.iterate_timeout + When `openstacksdk` needs to poll a resource, it does so in a loop that waits + between iterations and ultimately times out. The + ``openstack.iterate_timeout`` logger emits messages for each iteration + indicating it is waiting and for how long. These can be useful to see for + long running tasks so that one can know things are not stuck, but can also + be noisy. + +openstack.fnmatch + `openstacksdk` will try to use `fnmatch`_ on given `name_or_id` arguments. + It's a best effort attempt, so pattern misses are logged to + ``openstack.fnmatch``. A user may not be intending to use an fnmatch + pattern - such as if they are trying to find an image named + ``Fedora 24 [official]``, so these messages are logged separately. + +.. _fnmatch: https://pymotw.com/2/fnmatch/ + +HTTP Tracing +------------ + +HTTP Interactions are handled by `keystoneauth`_. If you want to enable HTTP +tracing while using openstacksdk and are not using `openstack.enable_logging`, +set the log level of the ``keystoneauth`` logger to ``DEBUG``. + +For more information see https://docs.openstack.org/keystoneauth/latest/using-sessions.html#logging + +.. _keystoneauth: https://docs.openstack.org/keystoneauth/latest/ + +Python Logging +-------------- + +Python logging is a standard feature of Python and is documented fully in the +Python Documentation, which varies by version of Python. + +For more information on Python Logging for Python v2, see +https://docs.python.org/2/library/logging.html. + +For more information on Python Logging for Python v3, see +https://docs.python.org/3/library/logging.html. diff --git a/doc/source/user/index.rst b/doc/source/user/index.rst index d28c2abb6..778148176 100644 --- a/doc/source/user/index.rst +++ b/doc/source/user/index.rst @@ -22,8 +22,6 @@ These guides walk you through how to make use of the libraries we provide to work with each OpenStack service. If you're looking for a cookbook approach, this is where you'll want to begin. -.. TODO(shade) Merge guides/logging and logging - .. toctree:: :maxdepth: 1 @@ -32,7 +30,6 @@ approach, this is where you'll want to begin. Connect to an OpenStack Cloud Using a Config File Using Cloud Abstration Layer Logging - Shade Logging Microversions Baremetal Block Storage diff --git a/doc/source/user/logging.rst b/doc/source/user/logging.rst deleted file mode 100644 index e3239d3ff..000000000 --- a/doc/source/user/logging.rst +++ /dev/null @@ -1,99 +0,0 @@ -======= -Logging -======= - -.. note:: TODO(shade) This document is written from a shade POV. It needs to - be combined with the existing logging guide, but also the logging - systems need to be rationalized. - -`openstacksdk` uses `Python Logging`_. As `openstacksdk` is a library, it does -not configure logging handlers automatically, expecting instead for that to be -the purview of the consuming application. - -Simple Usage ------------- - -For consumers who just want to get a basic logging setup without thinking -about it too deeply, there is a helper method. If used, it should be called -before any other `shade` functionality. - -.. code-block:: python - - import openstack.cloud - openstack.cloud.simple_logging() - -`openstack.cloud.simple_logging` takes two optional boolean arguments: - -debug - Turns on debug logging. - -http_debug - Turns on debug logging as well as debug logging of the underlying HTTP calls. - -`openstack.cloud.simple_logging` also sets up a few other loggers and -squelches some warnings or log messages that are otherwise uninteresting or -unactionable by a `openstack.cloud` user. - -Advanced Usage --------------- - -`openstack.cloud` logs to a set of different named loggers. - -Most of the logging is set up to log to the root `openstack.cloud` logger. -There are additional sub-loggers that are used at times, primarily so that a -user can decide to turn on or off a specific type of logging. They are listed -below. - -openstack.task_manager - `openstack.cloud` uses a Task Manager to perform remote calls. The - `openstack.cloud.task_manager` logger emits messages at the start and end - of each Task announcing what it is going to run and then what it ran and - how long it took. Logging `openstack.cloud.task_manager` is a good way to - get a trace of external actions `openstack.cloud` is taking without full - `HTTP Tracing`_. - -openstack.cloud.exc - If `log_inner_exceptions` is set to True, `shade` will emit any wrapped - exception to the `openstack.cloud.exc` logger. Wrapped exceptions are usually - considered implementation details, but can be useful for debugging problems. - -openstack.iterate_timeout - When `shade` needs to poll a resource, it does so in a loop that waits - between iterations and ultimately timesout. The - `openstack.iterate_timeout` logger emits messages for each iteration - indicating it is waiting and for how long. These can be useful to see for - long running tasks so that one can know things are not stuck, but can also - be noisy. - -openstack.cloud.http - `shade` will sometimes log additional information about HTTP interactions - to the `openstack.cloud.http` logger. This can be verbose, as it sometimes - logs entire response bodies. - -openstack.cloud.fnmatch - `shade` will try to use `fnmatch`_ on given `name_or_id` arguments. It's a - best effort attempt, so pattern misses are logged to - `openstack.cloud.fnmatch`. A user may not be intending to use an fnmatch - pattern - such as if they are trying to find an image named - ``Fedora 24 [official]``, so these messages are logged separately. - -.. _fnmatch: https://pymotw.com/2/fnmatch/ - -HTTP Tracing ------------- - -HTTP Interactions are handled by `keystoneauth`. If you want to enable HTTP -tracing while using `shade` and are not using `openstack.cloud.simple_logging`, -set the log level of the `keystoneauth` logger to `DEBUG`. - -Python Logging --------------- - -Python logging is a standard feature of Python and is documented fully in the -Python Documentation, which varies by version of Python. - -For more information on Python Logging for Python v2, see -https://docs.python.org/2/library/logging.html. - -For more information on Python Logging for Python v3, see -https://docs.python.org/3/library/logging.html. diff --git a/doc/source/user/multi-cloud-demo.rst b/doc/source/user/multi-cloud-demo.rst index 6d6914231..aafd3a164 100644 --- a/doc/source/user/multi-cloud-demo.rst +++ b/doc/source/user/multi-cloud-demo.rst @@ -62,10 +62,10 @@ Complete Example .. code:: python - import openstack.cloud + import openstack # Initialize and turn on debug logging - openstack.cloud.simple_logging(debug=True) + openstack.enable_logging(debug=True) for cloud_name, region_name in [ ('my-vexxhost', 'ca-ymq-1'), @@ -314,10 +314,10 @@ Complete Example Again .. code:: python - import openstack.cloud + import openstack # Initialize and turn on debug logging - openstack.cloud.simple_logging(debug=True) + openstack.enable_logging(debug=True) for cloud_name, region_name in [ ('my-vexxhost', 'ca-ymq-1'), @@ -346,27 +346,25 @@ Import the library .. code:: python - import openstack.cloud + import openstack Logging ======= -* `shade` uses standard python logging -* Special `openstack.cloud.request_ids` logger for API request IDs -* `simple_logging` does easy defaults +* `openstacksdk` uses standard python logging +* ``openstack.enable_logging`` does easy defaults * Squelches some meaningless warnings * `debug` * Logs shade loggers at debug level - * Includes `openstack.cloud.request_ids` debug logging * `http_debug` Implies `debug`, turns on HTTP tracing .. code:: python # Initialize and turn on debug logging - openstack.cloud.simple_logging(debug=True) + openstack.enable_logging(debug=True) Example with Debug Logging ========================== @@ -375,8 +373,8 @@ Example with Debug Logging .. code:: python - import openstack.cloud - openstack.cloud.simple_logging(debug=True) + import openstack + openstack.enable_logging(debug=True) cloud = openstack.openstack_cloud( cloud='my-vexxhost', region_name='ca-ymq-1') @@ -389,8 +387,8 @@ Example with HTTP Debug Logging .. code:: python - import openstack.cloud - openstack.cloud.simple_logging(http_debug=True) + import openstack + openstack.enable_logging(http_debug=True) cloud = openstack.openstack_cloud( cloud='my-vexxhost', region_name='ca-ymq-1') @@ -486,10 +484,10 @@ Image and Flavor by Name or ID .. code:: python - import openstack.cloud + import openstack # Initialize and turn on debug logging - openstack.cloud.simple_logging(debug=True) + openstack.enable_logging(debug=True) for cloud_name, region_name, image, flavor in [ ('my-vexxhost', 'ca-ymq-1', @@ -533,10 +531,10 @@ Image and Flavor by Dict .. code:: python - import openstack.cloud + import openstack # Initialize and turn on debug logging - openstack.cloud.simple_logging(debug=True) + openstack.enable_logging(debug=True) for cloud_name, region_name, image, flavor_id in [ ('my-vexxhost', 'ca-ymq-1', 'Ubuntu 16.04.1 LTS [2017-03-03]', @@ -564,8 +562,8 @@ Munch Objects .. code:: python - import openstack.cloud - openstack.cloud.simple_logging(debug=True) + import openstack + openstack.enable_logging(debug=True) cloud = openstack.openstack_cloud(cloud='zetta', region_name='no-osl1') image = cloud.get_image('Ubuntu 14.04 (AMD64) [Local Storage]') @@ -596,10 +594,10 @@ Cleanup Script .. code:: python - import openstack.cloud + import openstack # Initialize and turn on debug logging - openstack.cloud.simple_logging(debug=True) + openstack.enable_logging(debug=True) for cloud_name, region_name in [ ('my-vexxhost', 'ca-ymq-1'), @@ -618,8 +616,8 @@ Normalization .. code:: python - import openstack.cloud - openstack.cloud.simple_logging() + import openstack + openstack.enable_logging() cloud = openstack.openstack_cloud(cloud='fuga', region_name='cystack') image = cloud.get_image( @@ -634,8 +632,8 @@ Strict Normalized Results .. code:: python - import openstack.cloud - openstack.cloud.simple_logging() + import openstack + openstack.enable_logging() cloud = openstack.openstack_cloud( cloud='fuga', region_name='cystack', strict=True) @@ -651,8 +649,8 @@ How Did I Find the Image Name for the Last Example? .. code:: python - import openstack.cloud - openstack.cloud.simple_logging() + import openstack + openstack.enable_logging() cloud = openstack.openstack_cloud(cloud='fuga', region_name='cystack') cloud.pprint([ @@ -672,8 +670,8 @@ Added / Modified Information .. code:: python - import openstack.cloud - openstack.cloud.simple_logging(debug=True) + import openstack + openstack.enable_logging(debug=True) cloud = openstack.openstack_cloud(cloud='my-citycloud', region_name='Buf1') try: @@ -714,8 +712,8 @@ User Agent Info .. code:: python - import openstack.cloud - openstack.cloud.simple_logging(http_debug=True) + import openstack + openstack.enable_logging(http_debug=True) cloud = openstack.openstack_cloud( cloud='datacentred', app_name='AmazingApp', app_version='1.0') @@ -732,8 +730,8 @@ Uploading Large Objects .. code:: python - import openstack.cloud - openstack.cloud.simple_logging(debug=True) + import openstack + openstack.enable_logging(debug=True) cloud = openstack.openstack_cloud(cloud='ovh', region_name='SBG1') cloud.create_object( @@ -753,8 +751,8 @@ Uploading Large Objects .. code:: python - import openstack.cloud - openstack.cloud.simple_logging(debug=True) + import openstack + openstack.enable_logging(debug=True) cloud = openstack.openstack_cloud(cloud='ovh', region_name='SBG1') cloud.create_object( @@ -769,8 +767,8 @@ Service Conditionals .. code:: python - import openstack.cloud - openstack.cloud.simple_logging(debug=True) + import openstack + openstack.enable_logging(debug=True) cloud = openstack.openstack_cloud(cloud='kiss', region_name='region1') print(cloud.has_service('network')) @@ -783,8 +781,8 @@ Service Conditional Overrides .. code:: python - import openstack.cloud - openstack.cloud.simple_logging(debug=True) + import openstack + openstack.enable_logging(debug=True) cloud = openstack.openstack_cloud(cloud='rax', region_name='DFW') print(cloud.has_service('network')) diff --git a/examples/cloud/cleanup-servers.py b/examples/cloud/cleanup-servers.py index 7f9257b58..7620f4c0d 100644 --- a/examples/cloud/cleanup-servers.py +++ b/examples/cloud/cleanup-servers.py @@ -13,7 +13,7 @@ import openstack # Initialize and turn on debug logging -openstack.simple_logging(debug=True) +openstack.enable_logging(debug=True) for cloud_name, region_name in [ ('my-vexxhost', 'ca-ymq-1'), diff --git a/examples/cloud/create-server-dict.py b/examples/cloud/create-server-dict.py index 4557ee188..7b7505010 100644 --- a/examples/cloud/create-server-dict.py +++ b/examples/cloud/create-server-dict.py @@ -13,7 +13,7 @@ import openstack # Initialize and turn on debug logging -openstack.simple_logging(debug=True) +openstack.enable_logging(debug=True) for cloud_name, region_name, image, flavor_id in [ ('my-vexxhost', 'ca-ymq-1', 'Ubuntu 16.04.1 LTS [2017-03-03]', diff --git a/examples/cloud/create-server-name-or-id.py b/examples/cloud/create-server-name-or-id.py index a269f2fe6..170904a9d 100644 --- a/examples/cloud/create-server-name-or-id.py +++ b/examples/cloud/create-server-name-or-id.py @@ -13,7 +13,7 @@ import openstack # Initialize and turn on debug logging -openstack.simple_logging(debug=True) +openstack.enable_logging(debug=True) for cloud_name, region_name, image, flavor in [ ('my-vexxhost', 'ca-ymq-1', diff --git a/examples/cloud/debug-logging.py b/examples/cloud/debug-logging.py index a9cc31c2b..0874e7b6c 100644 --- a/examples/cloud/debug-logging.py +++ b/examples/cloud/debug-logging.py @@ -11,7 +11,7 @@ # under the License. import openstack -openstack.simple_logging(debug=True) +openstack.enable_logging(debug=True) cloud = openstack.openstack_cloud( cloud='my-vexxhost', region_name='ca-ymq-1') diff --git a/examples/cloud/find-an-image.py b/examples/cloud/find-an-image.py index 089b08640..9be76817e 100644 --- a/examples/cloud/find-an-image.py +++ b/examples/cloud/find-an-image.py @@ -11,7 +11,7 @@ # under the License. import openstack -openstack.simple_logging() +openstack.enable_logging() cloud = openstack.openstack_cloud(cloud='fuga', region_name='cystack') cloud.pprint([ diff --git a/examples/cloud/http-debug-logging.py b/examples/cloud/http-debug-logging.py index e5b1a7e9d..62b942532 100644 --- a/examples/cloud/http-debug-logging.py +++ b/examples/cloud/http-debug-logging.py @@ -11,7 +11,7 @@ # under the License. import openstack -openstack.simple_logging(http_debug=True) +openstack.enable_logging(http_debug=True) cloud = openstack.openstack_cloud( cloud='my-vexxhost', region_name='ca-ymq-1') diff --git a/examples/cloud/munch-dict-object.py b/examples/cloud/munch-dict-object.py index df3a0e76c..5ba6e63ea 100644 --- a/examples/cloud/munch-dict-object.py +++ b/examples/cloud/munch-dict-object.py @@ -11,7 +11,7 @@ # under the License. import openstack -openstack.simple_logging(debug=True) +openstack.enable_logging(debug=True) cloud = openstack.openstack_cloud(cloud='ovh', region_name='SBG1') image = cloud.get_image('Ubuntu 16.10') diff --git a/examples/cloud/normalization.py b/examples/cloud/normalization.py index 80f1ff4c9..40a3748d3 100644 --- a/examples/cloud/normalization.py +++ b/examples/cloud/normalization.py @@ -11,7 +11,7 @@ # under the License. import openstack -openstack.simple_logging() +openstack.enable_logging() cloud = openstack.openstack_cloud(cloud='fuga', region_name='cystack') image = cloud.get_image( diff --git a/examples/cloud/server-information.py b/examples/cloud/server-information.py index 8ae710631..b851e96ad 100644 --- a/examples/cloud/server-information.py +++ b/examples/cloud/server-information.py @@ -11,7 +11,7 @@ # under the License. import openstack -openstack.simple_logging(debug=True) +openstack.enable_logging(debug=True) cloud = openstack.openstack_cloud(cloud='my-citycloud', region_name='Buf1') try: diff --git a/examples/cloud/service-conditional-overrides.py b/examples/cloud/service-conditional-overrides.py index 7991a1ec3..d3a2a88de 100644 --- a/examples/cloud/service-conditional-overrides.py +++ b/examples/cloud/service-conditional-overrides.py @@ -11,7 +11,7 @@ # under the License. import openstack -openstack.simple_logging(debug=True) +openstack.enable_logging(debug=True) cloud = openstack.openstack_cloud(cloud='rax', region_name='DFW') print(cloud.has_service('network')) diff --git a/examples/cloud/service-conditionals.py b/examples/cloud/service-conditionals.py index 5e81d1dad..46b3d2e91 100644 --- a/examples/cloud/service-conditionals.py +++ b/examples/cloud/service-conditionals.py @@ -11,7 +11,7 @@ # under the License. import openstack -openstack.simple_logging(debug=True) +openstack.enable_logging(debug=True) cloud = openstack.openstack_cloud(cloud='kiss', region_name='region1') print(cloud.has_service('network')) diff --git a/examples/cloud/strict-mode.py b/examples/cloud/strict-mode.py index 74cfd05ff..96eaa13b9 100644 --- a/examples/cloud/strict-mode.py +++ b/examples/cloud/strict-mode.py @@ -11,7 +11,7 @@ # under the License. import openstack -openstack.simple_logging() +openstack.enable_logging() cloud = openstack.openstack_cloud( cloud='fuga', region_name='cystack', strict=True) diff --git a/examples/cloud/upload-large-object.py b/examples/cloud/upload-large-object.py index f274c2e7e..d89ae557d 100644 --- a/examples/cloud/upload-large-object.py +++ b/examples/cloud/upload-large-object.py @@ -11,7 +11,7 @@ # under the License. import openstack -openstack.simple_logging(debug=True) +openstack.enable_logging(debug=True) cloud = openstack.openstack_cloud(cloud='ovh', region_name='SBG1') cloud.create_object( diff --git a/examples/cloud/upload-object.py b/examples/cloud/upload-object.py index f274c2e7e..d89ae557d 100644 --- a/examples/cloud/upload-object.py +++ b/examples/cloud/upload-object.py @@ -11,7 +11,7 @@ # under the License. import openstack -openstack.simple_logging(debug=True) +openstack.enable_logging(debug=True) cloud = openstack.openstack_cloud(cloud='ovh', region_name='SBG1') cloud.create_object( diff --git a/examples/cloud/user-agent.py b/examples/cloud/user-agent.py index 309355b5e..578c3c2e5 100644 --- a/examples/cloud/user-agent.py +++ b/examples/cloud/user-agent.py @@ -11,7 +11,7 @@ # under the License. import openstack -openstack.simple_logging(http_debug=True) +openstack.enable_logging(http_debug=True) cloud = openstack.openstack_cloud( cloud='datacentred', app_name='AmazingApp', app_version='1.0') diff --git a/openstack/__init__.py b/openstack/__init__.py index 9d67f9083..3fd98a2a5 100644 --- a/openstack/__init__.py +++ b/openstack/__init__.py @@ -12,15 +12,21 @@ # See the License for the specific language governing permissions and # limitations under the License. -import logging +__all__ = [ + '__version__', + 'connect', + 'enable_logging', +] + import warnings import keystoneauth1.exceptions import pbr.version import requestsexceptions -from openstack import _log +from openstack._log import enable_logging # noqa from openstack.cloud.exc import * # noqa +# TODO(shade) These two want to be removed before we make a release from openstack.cloud.openstackcloud import OpenStackCloud from openstack.cloud.operatorcloud import OperatorCloud import openstack.connection @@ -34,43 +40,11 @@ if requestsexceptions.SubjectAltNameWarning: def _get_openstack_config(app_name=None, app_version=None): import openstack.config - # Protect against older versions of os-client-config that don't expose this - try: - return openstack.config.OpenStackConfig( - app_name=app_name, app_version=app_version) - except Exception: - return openstack.config.OpenStackConfig() + return openstack.config.OpenStackConfig( + app_name=app_name, app_version=app_version) -def simple_logging(debug=False, http_debug=False): - if http_debug: - debug = True - if debug: - log_level = logging.DEBUG - else: - log_level = logging.INFO - if http_debug: - # Enable HTTP level tracing - log = _log.setup_logging('keystoneauth') - log.addHandler(logging.StreamHandler()) - log.setLevel(log_level) - # We only want extra shade HTTP tracing in http debug mode - log = _log.setup_logging('openstack.cloud.http') - log.setLevel(log_level) - else: - # We only want extra shade HTTP tracing in http debug mode - log = _log.setup_logging('openstack.cloud.http') - log.setLevel(logging.WARNING) - log = _log.setup_logging('openstack.cloud') - log.addHandler(logging.StreamHandler()) - log.setLevel(log_level) - # Suppress warning about keystoneauth loggers - log = _log.setup_logging('keystoneauth.identity.base') - log = _log.setup_logging('keystoneauth.identity.generic.base') - - -# TODO(shade) Document this and add some examples -# TODO(shade) This wants to be renamed before we make a release. +# TODO(shade) This wants to be remove before we make a release. def openstack_clouds( config=None, debug=False, cloud=None, strict=False, app_name=None, app_version=None): @@ -99,10 +73,7 @@ def openstack_clouds( "Invalid cloud configuration: {exc}".format(exc=str(e))) -# TODO(shade) This wants to be renamed before we make a release - there is -# ultimately no reason to have an openstack_cloud and a connect -# factory function - but we have a few steps to go first and this is used -# in the imported tests from shade. +# TODO(shade) This wants to be removed before we make a release. def openstack_cloud( config=None, strict=False, app_name=None, app_version=None, **kwargs): if not config: @@ -115,10 +86,7 @@ def openstack_cloud( return OpenStackCloud(cloud_config=cloud_region, strict=strict) -# TODO(shade) This wants to be renamed before we make a release - there is -# ultimately no reason to have an operator_cloud and a connect -# factory function - but we have a few steps to go first and this is used -# in the imported tests from shade. +# TODO(shade) This wants to be removed before we make a release. def operator_cloud( config=None, strict=False, app_name=None, app_version=None, **kwargs): if not config: @@ -134,3 +102,16 @@ def operator_cloud( def connect(*args, **kwargs): """Create a `openstack.connection.Connection`.""" return openstack.connection.Connection(*args, **kwargs) + + +def connect_all(config=None, app_name=None, app_version=None): + if not config: + config = _get_openstack_config(app_name, app_version) + try: + return [ + openstack.connection.Connection(config=cloud_region) + for cloud_region in config.get_all() + ] + except keystoneauth1.exceptions.auth_plugins.NoMatchingPlugin as e: + raise OpenStackCloudException( + "Invalid cloud configuration: {exc}".format(exc=str(e))) diff --git a/openstack/_log.py b/openstack/_log.py index ff2f2eac7..4637dfc8b 100644 --- a/openstack/_log.py +++ b/openstack/_log.py @@ -13,16 +13,102 @@ # limitations under the License. import logging +import sys -class NullHandler(logging.Handler): - def emit(self, record): - pass +def setup_logging(name, handlers=None, level=None): + """Set up logging for a named logger. + Gets and initializes a named logger, ensuring it at least has a + `logging.NullHandler` attached. -def setup_logging(name): + :param str name: + Name of the logger. + :param list handlers: + A list of `logging.Handler` objects to attach to the logger. + :param int level: + Log level to set the logger at. + + :returns: A `logging.Logger` object that can be used to emit log messages. + """ + handlers = handlers or [] log = logging.getLogger(name) - if len(log.handlers) == 0: - h = NullHandler() + if len(log.handlers) == 0 and not handlers: + h = logging.NullHandler() log.addHandler(h) + for h in handlers: + log.addHandler(h) + if level: + log.setLevel(level) return log + + +def enable_logging( + debug=False, http_debug=False, path=None, stream=None, + format_stream=False, + format_template='%(asctime)s %(levelname)s: %(name)s %(message)s'): + """Enable logging output. + + Helper function to enable logging. This function is available for + debugging purposes and for folks doing simple applications who want an + easy 'just make it work for me'. For more complex applications or for + those who want more flexibility, the standard library ``logging`` package + will receive these messages in any handlers you create. + + :param bool debug: + Set this to ``True`` to receive debug messages. + :param bool http_debug: + Set this to ``True`` to receive debug messages including + HTTP requests and responses. This implies ``debug=True``. + :param str path: + If a *path* is specified, logging output will written to that file + in addition to sys.stderr. + The path is passed to logging.FileHandler, which will append messages + the file (and create it if needed). + :param stream: + One of ``None `` or ``sys.stdout`` or ``sys.stderr``. + If it is ``None``, nothing is logged to a stream. + If it isn't ``None``, console output is logged to this stream. + :param bool format_stream: + If format_stream is False, the default, apply ``format_template`` to + ``path`` but not to ``stream`` outputs. If True, apply + ``format_template`` to ``stream`` outputs as well. + :param str format_template: + Template to pass to :class:`logging.Formatter`. + + :rtype: None + """ + if not stream and not path: + stream = sys.stdout + + if http_debug: + debug = True + if debug: + level = logging.DEBUG + else: + level = logging.INFO + + formatter = logging.Formatter(format_template) + + handlers = [] + + if stream is not None: + console = logging.StreamHandler(stream) + if format_stream: + console.setFormatter(formatter) + handlers.append(console) + + if path is not None: + file_handler = logging.FileHandler(path) + file_handler.setFormatter(formatter) + handlers.append(file_handler) + + if http_debug: + # Enable HTTP level tracing + setup_logging('keystoneauth', handlers=handlers, level=level) + + setup_logging('openstack', handlers=handlers, level=level) + # Suppress warning about keystoneauth loggers + setup_logging('keystoneauth.discovery') + setup_logging('keystoneauth.identity.base') + setup_logging('keystoneauth.identity.generic.base') diff --git a/openstack/cloud/_utils.py b/openstack/cloud/_utils.py index b5eb032b4..270dd95dd 100644 --- a/openstack/cloud/_utils.py +++ b/openstack/cloud/_utils.py @@ -94,7 +94,7 @@ def _filter_list(data, name_or_id, filters): # The logger is openstack.cloud.fmmatch to allow a user/operator to # configure logging not to communicate about fnmatch misses # (they shouldn't be too spammy, but one never knows) - log = _log.setup_logging('openstack.cloud.fnmatch') + log = _log.setup_logging('openstack.fnmatch') if name_or_id: # name_or_id might already be unicode name_or_id = _make_unicode(name_or_id) diff --git a/openstack/cloud/meta.py b/openstack/cloud/meta.py index 35bc242d2..3c57a1a04 100644 --- a/openstack/cloud/meta.py +++ b/openstack/cloud/meta.py @@ -232,7 +232,7 @@ def find_best_address(addresses, family, public=False, cloud_public=True): pass # Give up and return the first - none work as far as we can tell if do_check: - log = _log.setup_logging('shade') + log = _log.setup_logging('openstack') log.debug( 'The cloud returned multiple addresses, and none of them seem' ' to work. That might be what you wanted, but we have no clue' @@ -381,7 +381,7 @@ def _get_supplemental_addresses(cloud, server): # This SHOULD return one and only one FIP - but doing # it as a search/list lets the logic work regardless if fip['fixed_ip_address'] not in fixed_ip_mapping: - log = _log.setup_logging('shade') + log = _log.setup_logging('openstack') log.debug( "The cloud returned floating ip %(fip)s attached" " to server %(server)s but the fixed ip associated" diff --git a/openstack/cloud/openstackcloud.py b/openstack/cloud/openstackcloud.py index 9a3103f66..4cb9380d7 100644 --- a/openstack/cloud/openstackcloud.py +++ b/openstack/cloud/openstackcloud.py @@ -152,7 +152,7 @@ class OpenStackCloud(_normalize.Normalizer): if log_inner_exceptions: OpenStackCloudException.log_inner_exceptions = True - self.log = _log.setup_logging('openstack.cloud') + self.log = _log.setup_logging('openstack') if not cloud_config: config = openstack.config.OpenStackConfig( diff --git a/openstack/config/cloud_region.py b/openstack/config/cloud_region.py index ffef7f60e..7c2541caf 100644 --- a/openstack/config/cloud_region.py +++ b/openstack/config/cloud_region.py @@ -69,7 +69,7 @@ class CloudRegion(object): self.name = name self.region_name = region_name self.config = config - self.log = _log.setup_logging(__name__) + self.log = _log.setup_logging('openstack.config') self._force_ipv4 = force_ipv4 self._auth = auth_plugin self._openstack_config = openstack_config diff --git a/openstack/config/loader.py b/openstack/config/loader.py index dcc22c14b..f5d853bfa 100644 --- a/openstack/config/loader.py +++ b/openstack/config/loader.py @@ -182,7 +182,7 @@ class OpenStackConfig(object): pw_func=None, session_constructor=None, app_name=None, app_version=None, load_yaml_config=True): - self.log = _log.setup_logging(__name__) + self.log = _log.setup_logging('openstack.config') self._session_constructor = session_constructor self._app_name = app_name self._app_version = app_version diff --git a/openstack/connection.py b/openstack/connection.py index f5797b7df..e59d8cc67 100644 --- a/openstack/connection.py +++ b/openstack/connection.py @@ -75,22 +75,20 @@ try to find it and if that fails, you would create it:: """ import importlib -import logging -import sys import keystoneauth1.exceptions import os_service_types from six.moves import urllib +from openstack import _log import openstack.config from openstack.config import cloud_region from openstack import exceptions from openstack import proxy from openstack import proxy2 from openstack import task_manager -from openstack import utils -_logger = logging.getLogger(__name__) +_logger = _log.setup_logging('openstack') def from_config(cloud=None, config=None, options=None, **kwargs): @@ -119,9 +117,6 @@ def from_config(cloud=None, config=None, options=None, **kwargs): config = openstack.config.OpenStackConfig().get_one( cloud=cloud, argparse=options) - if config.debug: - utils.enable_logging(True, stream=sys.stdout) - return Connection(config=config) diff --git a/openstack/image/v2/image.py b/openstack/image/v2/image.py index f1b8a8f57..4b397a443 100644 --- a/openstack/image/v2/image.py +++ b/openstack/image/v2/image.py @@ -11,16 +11,16 @@ # under the License. import hashlib -import logging import jsonpatch +from openstack import _log from openstack import exceptions from openstack.image import image_service from openstack import resource2 from openstack import utils -_logger = logging.getLogger(__name__) +_logger = _log.setup_logging('openstack') class Image(resource2.Resource): diff --git a/openstack/profile.py b/openstack/profile.py index d70c05afa..588fd60c7 100644 --- a/openstack/profile.py +++ b/openstack/profile.py @@ -52,8 +52,8 @@ The resulting preference print out would look something like:: """ import copy -import logging +from openstack import _log from openstack.baremetal import baremetal_service from openstack.block_storage import block_storage_service from openstack.clustering import clustering_service @@ -72,7 +72,7 @@ from openstack.object_store import object_store_service from openstack.orchestration import orchestration_service from openstack.workflow import workflow_service -_logger = logging.getLogger(__name__) +_logger = _log.setup_logging('openstack') class Profile(object): diff --git a/openstack/task_manager.py b/openstack/task_manager.py index b015ac85f..8a49ba4b2 100644 --- a/openstack/task_manager.py +++ b/openstack/task_manager.py @@ -22,10 +22,10 @@ import time import keystoneauth1.exceptions import six +import openstack._log from openstack import exceptions -from openstack import utils -_log = utils.setup_logging(__name__) +_log = openstack._log.setup_logging('openstack.task_manager') class Task(object): diff --git a/openstack/tests/unit/image/v2/test_image.py b/openstack/tests/unit/image/v2/test_image.py index cf4a96355..14e2cf1ed 100644 --- a/openstack/tests/unit/image/v2/test_image.py +++ b/openstack/tests/unit/image/v2/test_image.py @@ -272,7 +272,7 @@ class TestImage(testtools.TestCase): self.sess.get.side_effect = [resp1, resp2] - with self.assertLogs(logger=image.__name__, level="WARNING") as log: + with self.assertLogs(logger='openstack', level="WARNING") as log: rv = sot.download(self.sess) self.assertEqual(len(log.records), 1, diff --git a/openstack/tests/unit/test_utils.py b/openstack/tests/unit/test_utils.py index f0ea12d60..1110330ff 100644 --- a/openstack/tests/unit/test_utils.py +++ b/openstack/tests/unit/test_utils.py @@ -10,67 +10,83 @@ # License for the specific language governing permissions and limitations # under the License. +import logging import mock import sys import testtools +import fixtures + from openstack import utils class Test_enable_logging(testtools.TestCase): - def _console_tests(self, fake_logging, level, debug, stream): - the_logger = mock.Mock() - fake_logging.getLogger.return_value = the_logger + def setUp(self): + super(Test_enable_logging, self).setUp() + self.openstack_logger = mock.Mock() + self.openstack_logger.handlers = [] + self.ksa_logger_1 = mock.Mock() + self.ksa_logger_1.handlers = [] + self.ksa_logger_2 = mock.Mock() + self.ksa_logger_2.handlers = [] + self.ksa_logger_3 = mock.Mock() + self.ksa_logger_3.handlers = [] + self.fake_get_logger = mock.Mock() + self.fake_get_logger.side_effect = [ + self.openstack_logger, + self.ksa_logger_1, + self.ksa_logger_2, + self.ksa_logger_3 + ] + self.useFixture( + fixtures.MonkeyPatch('logging.getLogger', self.fake_get_logger)) + + def _console_tests(self, level, debug, stream): utils.enable_logging(debug=debug, stream=stream) - self.assertEqual(the_logger.addHandler.call_count, 2) - the_logger.setLevel.assert_called_with(level) + self.assertEqual(self.openstack_logger.addHandler.call_count, 1) + self.openstack_logger.setLevel.assert_called_with(level) - def _file_tests(self, fake_logging, level, debug): - the_logger = mock.Mock() - fake_logging.getLogger.return_value = the_logger + def _file_tests(self, level, debug): + file_handler = mock.Mock() + self.useFixture( + fixtures.MonkeyPatch('logging.FileHandler', file_handler)) fake_path = "fake/path.log" utils.enable_logging(debug=debug, path=fake_path) - fake_logging.FileHandler.assert_called_with(fake_path) - self.assertEqual(the_logger.addHandler.call_count, 2) - the_logger.setLevel.assert_called_with(level) + file_handler.assert_called_with(fake_path) + self.assertEqual(self.openstack_logger.addHandler.call_count, 1) + self.openstack_logger.setLevel.assert_called_with(level) def test_none(self): - self.assertRaises( - ValueError, utils.enable_logging, - debug=True, path=None, stream=None) + utils.enable_logging(debug=True) + self.fake_get_logger.assert_has_calls([]) + self.openstack_logger.setLevel.assert_called_with(logging.DEBUG) + self.assertEqual(self.openstack_logger.addHandler.call_count, 1) + self.assertIsInstance( + self.openstack_logger.addHandler.call_args_list[0][0][0], + logging.StreamHandler) - @mock.patch("openstack.utils.logging") - def test_debug_console_stderr(self, fake_logging): - self._console_tests(fake_logging, - fake_logging.DEBUG, True, sys.stderr) + def test_debug_console_stderr(self): + self._console_tests(logging.DEBUG, True, sys.stderr) - @mock.patch("openstack.utils.logging") - def test_warning_console_stderr(self, fake_logging): - self._console_tests(fake_logging, - fake_logging.WARNING, False, sys.stderr) + def test_warning_console_stderr(self): + self._console_tests(logging.INFO, False, sys.stderr) - @mock.patch("openstack.utils.logging") - def test_debug_console_stdout(self, fake_logging): - self._console_tests(fake_logging, - fake_logging.DEBUG, True, sys.stdout) + def test_debug_console_stdout(self): + self._console_tests(logging.DEBUG, True, sys.stdout) - @mock.patch("openstack.utils.logging") - def test_warning_console_stdout(self, fake_logging): - self._console_tests(fake_logging, - fake_logging.WARNING, False, sys.stdout) + def test_warning_console_stdout(self): + self._console_tests(logging.INFO, False, sys.stdout) - @mock.patch("openstack.utils.logging") - def test_debug_file(self, fake_logging): - self._file_tests(fake_logging, fake_logging.DEBUG, True) + def test_debug_file(self): + self._file_tests(logging.DEBUG, True) - @mock.patch("openstack.utils.logging") - def test_warning_file(self, fake_logging): - self._file_tests(fake_logging, fake_logging.WARNING, False) + def test_warning_file(self): + self._file_tests(logging.INFO, False) class Test_urljoin(testtools.TestCase): diff --git a/openstack/utils.py b/openstack/utils.py index 606705531..462bb8cc4 100644 --- a/openstack/utils.py +++ b/openstack/utils.py @@ -11,12 +11,13 @@ # under the License. import functools -import logging import time import deprecation from openstack import _log +# SDK has had enable_logging in utils. Import the symbol here to not break them +from openstack._log import enable_logging # noqa from openstack import exceptions from openstack import version @@ -47,67 +48,6 @@ def deprecated(deprecated_in=None, removed_in=None, details=details) -class NullHandler(logging.Handler): - def emit(self, record): - pass - - -def setup_logging(name): - '''Get a logging.Logger and make sure there is at least a NullHandler.''' - log = logging.getLogger(name) - if len(log.handlers) == 0: - h = NullHandler() - log.addHandler(h) - return log - - -def enable_logging(debug=False, path=None, stream=None): - """Enable logging to a file at path and/or a console stream. - - This function is available for debugging purposes. If you wish to - log this package's message in your application, the standard library - ``logging`` package will receive these messages in any handlers you - create. - - :param bool debug: Set this to ``True`` to receive debug messages, - which includes HTTP requests and responses, - or ``False`` for warning messages. - :param str path: If a *path* is specified, logging output will - written to that file in addition to sys.stderr. - The path is passed to logging.FileHandler, - which will append messages the file (and create - it if needed). - :param stream: One of ``None `` or ``sys.stdout`` or ``sys.stderr``. - If it is ``None``, nothing is logged to a stream. - If it isn't ``None``, console output is logged - to this stream. - - :rtype: None - """ - if path is None and stream is None: - raise ValueError("path and/or stream must be set") - - logger = logging.getLogger('openstack') - ksalog = logging.getLogger('keystoneauth') - formatter = logging.Formatter( - '%(asctime)s %(levelname)s: %(name)s %(message)s') - - if stream is not None: - console = logging.StreamHandler(stream) - console.setFormatter(formatter) - logger.addHandler(console) - ksalog.addHandler(console) - - if path is not None: - file_handler = logging.FileHandler(path) - file_handler.setFormatter(formatter) - logger.addHandler(file_handler) - ksalog.addHandler(file_handler) - - logger.setLevel(logging.DEBUG if debug else logging.WARNING) - ksalog.setLevel(logging.DEBUG if debug else logging.WARNING) - - def urljoin(*args): """A custom version of urljoin that simply joins strings into a path.