Add profiling for Ansible tasks

This should help find the slowest Ansible tasks that need to be optimized.

It is only enabled by default for gate jobs.

The profiling callback code is MIT licensed.  Per legal-discuss, I've added
the MIT license to the LICENSE.txt at the root of the repository along with
a note about which code is MIT licensed.

Thread on legal-discuss:

http://lists.openstack.org/pipermail/legal-discuss/2015-September/000398.html

Closes-Bug: #1488639

Change-Id: I46a2c495d9f8f1131ec08be99696efc39133f291
(cherry picked from commit 059cae3d76)

LICENSE.txt

@@ -199,3 +199,31 @@ Apache License
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
+
+-------------------------------------------------------------------------------
+
+The MIT License (MIT)
+
+Copyright (c) [year] [fullname]
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+Files in this project licensed under the MIT license:
+
+  - playbooks/plugins/callbacks/profile_tasks.py

README.rst

@@ -46,6 +46,7 @@ Notes
 * The tool ``lxc-system-manage`` is available on all lxc hosts and can assist in recreating parts of the LXC system whenever its needed.
 * Inventory is generated by executing the ``playbooks/inventory/dynamic_inventory.py`` script. This is configured in the ``playbooks/ansible.cfg`` file.
 * If you don't use the pw-token-gen.py script you will want to ensure the permissions on /etc/openstack_deploy/user_secrets.yml are more secure. ``chmod 0600 /etc/openstack_deploy/user_secrets.yml``
+* Finding slow Ansible tasks is easier with the ``profile_tasks`` callback plugin.  Review the notes in ``playbooks/plugins/callbacks/profile_tasks.py`` to enable it.
 
 
 Bugs and Blueprints

playbooks/plugins/callbacks/profile_tasks.py

@@ -0,0 +1,77 @@
+# The MIT License (MIT)
+#
+# Copyright (c) 2015, Red Hat, Inc. and others
+# Copyright (c) 2015, Rackspace US, Inc.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+#
+# ----------------------------------------------------------------------------
+#
+# Note that this callback plugin isn't enabled by default.  If you'd like to
+# enable it, add the following line to ansible.cfg in the 'playbooks'
+# directory in this repository:
+#
+#    callback_plugins = plugins/callbacks
+#
+# Add that line prior to running the playbooks and you will have detailed
+# timing information for Ansible tasks right after each playbook finishes
+# running.
+#
+import time
+
+
+class CallbackModule(object):
+    """
+    A plugin for timing tasks
+    """
+    def __init__(self):
+        self.stats = {}
+        self.current = None
+
+    def playbook_on_task_start(self, name, is_conditional):
+        """
+        Logs the start of each task
+        """
+        if self.current is not None:
+            # Record the running time of the last executed task
+            self.stats[self.current] = time.time() - self.stats[self.current]
+
+        # Record the start time of the current task
+        self.current = name
+        self.stats[self.current] = time.time()
+
+    def playbook_on_stats(self, stats):
+        """
+        Prints the timings
+        """
+        # Record the timing of the very last task
+        if self.current is not None:
+            self.stats[self.current] = time.time() - self.stats[self.current]
+
+        # Sort the tasks by their running time
+        results = sorted(self.stats.items(), key=lambda value: value[1],
+                         reverse=True)
+
+        # Just keep the top 10
+        results = results[:10]
+
+        # Print the timings
+        for name, elapsed in results:
+            print "{0:-<70}{1:->9}".format('{0} '.format(name),
+                                           ' {0:.02f}s'.format(elapsed))

scripts/gate-check-commit.sh

@@ -67,6 +67,9 @@ if [ -fs /etc/nodepool/provider ]; then
   fi
 fi
 
+# Enable detailed task profiling
+sed -i '/\[defaults\]/a callback_plugins = plugins/callbacks' $(dirname ${0})/../playbooks/ansible.cfg
+
 # Bootstrap an AIO setup if required
 if [ "${BOOTSTRAP_AIO}" == "yes" ]; then
   source $(dirname ${0})/bootstrap-aio.sh