Add CI Watch, a third-party CI monitoring dashboard

Original Change-Id: I8611f25a6700c6e0c64c3fadf820dbc9adcd5ea5

Change-Id: I847016f87ebd6da559ecd6298c5ad007bc935cb8

LICENSE

@@ -0,0 +1,176 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+

README.md

@@ -0,0 +1,53 @@
+# CI Watch
+
+## Installation
+
+From this folder, run the following commands.
+
+```
+pip install -r requirements.txt
+pip install -e .
+```
+
+These instructions are for development and testing installations.
+
+## Usage
+
+At the moment, this package provides three commands.
+
+`ci-watch-server`.
+Launch a development server.
+
+`ci-watch-stream-events`.
+Stream events from Gerrit and append valid events to `third-party-ci.log`.
+
+`ci-watch-populate-database`.
+Add all entries from `third-party-ci.log` to the database.
+
+
+## Configuration
+
+Configuration is stored in the `ci-watch.conf` file. Importantly, you can
+specify a directory to store the `third-party-ci.log` file (data\_dir) as well
+as the database to connect to. Look at `ci-watch.conf.sample` for an example.
+
+Other settings should be self explanatory based on the provided configuration
+file.
+
+## State of the project
+
+This project is a work in progress and the code is pretty rough in some places.
+
+## TODO
+
+* Add tests.
+* Use a different cache other than SimpleCache. It is not threadsafe. We
+  should use something like redis instead.
+
+These items are far from the only work needed for this project.
+
+
+## Acknowledgements
+
+This code was originally forked from John Griffith's sos-ci project. Some of it
+can still be found in the code and configuration file.

ci-watch.conf.sample

@@ -0,0 +1,18 @@
+[AccountInfo]
+gerrit_ssh_key = /path/to/private/key
+gerrit_username = your_username
+gerrit_host = review.openstack.org
+gerrit_port = 29418
+
+[Data]
+debug = True
+data_dir = /var/data
+
+[database]
+connection = sqlite:///:memory
+
+[misc]
+; This is a more complete list that is not missing any CI's from the wiki
+; It may include projects that don't actually do the third party CI thing
+; projects = cinder,nova,swift,rally,murano,keystone,ironic,octavia,os-brick,neutron,tempest,neutron-lbaas,devstack,designate,manila
+projects = cinder,nova,swift,rally,murano,ironic,octavia,os-brick,neutron,neutron-lbaas,devstack,manila

ciwatch.wsgi

@@ -0,0 +1,17 @@
+# Copyright (c) 2015 Tintri. All rights reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+import sys
+sys.path.insert(0, "/var/www/ciwatch")
+from ciwatch import app as application

ciwatch/__init__.py

@@ -0,0 +1,31 @@
+# Copyright (c) 2015 Tintri. All rights reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from flask import Flask
+
+app = Flask(__name__)
+
+from ciwatch import views  # noqa
+from ciwatch import filters  # noqa
+
+
+__version__ = "0.0.1"
+
+
+def main():
+    app.run(debug=True)
+
+
+if __name__ == '__main__':
+    main()

ciwatch/api.py

@@ -0,0 +1,87 @@
+# Copyright (c) 2015 Tintri. All rights reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from collections import OrderedDict
+from datetime import datetime, timedelta
+
+from flask import request
+from sqlalchemy import and_
+from sqlalchemy import desc
+
+from ciwatch import db
+from ciwatch.models import CiServer, Project, PatchSet
+
+
+TIME_OPTIONS = OrderedDict([  # Map time options to hours
+    ("24 hours", 24),
+    ("48 hours", 48),
+    ("7 days", 7 * 24),
+])
+
+DEFAULT_TIME_OPTION = "24 hours"
+DEFAULT_PROJECT = "cinder"
+
+
+def _get_ci_info_for_patch_sets(ci, patch_sets):
+    ci_info = {"name": ci.name, "trusted": ci.trusted, "results": []}
+    for patch_set in patch_sets:
+        for comment in patch_set.comments:
+            if comment.ci_server_id == ci.id:
+                ci_info["results"].append(comment)
+                break
+        else:  # nobreak
+            ci_info["results"].append(None)
+    return ci_info
+
+
+def get_projects():
+    return db.session.query(Project).order_by(Project.name).all()
+
+
+def get_ci_servers():
+    return db.session.query(CiServer).order_by(
+        desc(CiServer.trusted), CiServer.name).all()
+
+
+def get_patch_sets(project, since):
+    return db.session.query(PatchSet).filter(
+        and_(PatchSet.project == project, PatchSet.created >= since)
+        ).order_by(PatchSet.created.desc()).all()
+
+
+def get_time_options():
+    return TIME_OPTIONS.keys()
+
+
+def get_context():
+    project = request.args.get('project', DEFAULT_PROJECT)
+    time = request.args.get('time', DEFAULT_TIME_OPTION)
+    since = datetime.now() - timedelta(hours=TIME_OPTIONS[time])
+    project = db.session.query(Project).filter(
+        Project.name == project).one()
+    patch_sets = get_patch_sets(project=project, since=since)
+    results = OrderedDict()
+    for ci in get_ci_servers():
+        ci_info = _get_ci_info_for_patch_sets(ci, patch_sets)
+        if any(result for result in ci_info["results"]):
+            results[ci.ci_owner] = results.get(ci.ci_owner, [])
+            results[ci.ci_owner].append(
+                _get_ci_info_for_patch_sets(ci, patch_sets))
+
+    return {"time_options": get_time_options(),
+            "time_option": time,
+            "patch_sets": patch_sets,
+            "project": project,
+            "projects": get_projects(),
+            "user_results": results}

ciwatch/cache.py

@@ -0,0 +1,38 @@
+# Copyright (c) 2015 Tintri. All rights reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from functools import wraps
+
+from flask import request
+from werkzeug.contrib.cache import SimpleCache
+
+
+cache = SimpleCache()
+
+
+def cached(func):
+    @wraps(func)
+    def wrapper(*args, **kwargs):
+        key = _get_cache_key()
+        result = cache.get(key)
+        if result is None:
+            result = func(*args, **kwargs)
+            cache.set(key, result, timeout=60)
+        return result
+    return wrapper
+
+
+def _get_cache_key():
+    args = request.args
+    return request.path + str([(key, args[key]) for key in sorted(args)])

ciwatch/config.py

@@ -0,0 +1,28 @@
+# Copyright (c) 2015 Tintri. All rights reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+import os
+
+from iniparse import INIConfig
+
+_fdir = os.path.dirname(os.path.realpath(__file__))
+_conf_dir = os.path.dirname(_fdir)
+cfg = INIConfig(open(_conf_dir + '/ci-watch.conf'))
+
+
+def get_projects():
+    projects = []
+    for name in cfg.misc.projects.split(','):
+        projects.append(name)
+    return projects

ciwatch/db.py

@@ -0,0 +1,57 @@
+# Copyright (c) 2015 Tintri. All rights reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from sqlalchemy import create_engine
+from sqlalchemy.orm import sessionmaker
+
+from ciwatch import models
+from ciwatch.config import cfg, get_projects
+
+
+engine = create_engine(cfg.database.connection)
+Session = sessionmaker()
+Session.configure(bind=engine)
+models.Base.metadata.create_all(engine)
+session = Session()
+
+
+def create_projects():
+    for name in get_projects():
+        get_or_create(models.Project,
+                      commit_=False,
+                      name=name)
+    session.commit()
+
+
+def update_or_create_comment(commit_=True, **kwargs):
+    comment = session.query(models.Comment).filter_by(
+        ci_server_id=kwargs['ci_server_id'],
+        patch_set_id=kwargs['patch_set_id']).scalar()
+    if comment is not None:
+        for key, value in kwargs.iteritems():
+            setattr(comment, key, value)
+    else:
+        session.add(models.Comment(**kwargs))
+    if commit_:
+        session.commit()
+
+
+def get_or_create(model, commit_=True, **kwargs):
+    result = session.query(model).filter_by(**kwargs).first()
+    if not result:
+        result = model(**kwargs)
+        session.add(result)
+        if commit_:
+            session.commit()
+    return result

ciwatch/events.py

@@ -0,0 +1,174 @@
+# Copyright (c) 2015 Tintri. All rights reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+import json
+import paramiko
+import time
+from datetime import datetime
+import re
+
+from ciwatch import db, models
+from ciwatch.log import logger, DATA_DIR
+from ciwatch.config import cfg, get_projects
+
+
+def _process_project_name(project_name):
+    return project_name.split('/')[-1]
+
+
+def _process_event(event):
+    comment = event['comment']
+    # Find all the CIs voting in this comment
+    lines = comment.splitlines()
+    event['ci-status'] = {}
+    for line in lines:
+        possible_results = "FAILURE|SUCCESS|NOT_REGISTERED|UNSTABLE"
+        pattern = re.compile("[-*]\s+([^\s*]+)\s+(http[^\s*]+) : (%s)" %
+                             possible_results)
+        match = pattern.search(line)
+        if match is not None:
+            ci_name = match.group(1)
+            log_url = match.group(2)
+            result = match.group(3)
+            event['ci-status'][ci_name] = {
+                "result": result,
+                "log_url": log_url}
+
+
+def _is_ci_user(name):
+    return 'CI' in name or 'Jenkins' in name
+
+
+# Check if this is a third party CI event
+def _is_valid(event):
+    if (event.get('type', 'nill') == 'comment-added' and
+            _is_ci_user(event['author'].get('name', '')) and
+            _process_project_name(event['change']['project']) in get_projects() and
+            event['change']['branch'] == 'master'):
+        return True
+    return False
+
+
+def _store_event(event):
+    with open(DATA_DIR + '/third-party-ci.log', 'a') as f:
+        json.dump(event, f)
+        f.write('\n')
+    add_event_to_db(event)
+    return event
+
+
+class GerritEventStream(object):
+    def __init__(self):
+
+        logger.debug('Connecting to %(host)s:%(port)d as '
+                     '%(user)s using %(key)s',
+                     {'user': cfg.AccountInfo.gerrit_username,
+                      'key': cfg.AccountInfo.gerrit_ssh_key,
+                      'host': cfg.AccountInfo.gerrit_host,
+                      'port': int(cfg.AccountInfo.gerrit_port)})
+
+        self.ssh = paramiko.SSHClient()
+        self.ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+        connected = False
+        while not connected:
+            try:
+                self.ssh.connect(cfg.AccountInfo.gerrit_host,
+                                 int(cfg.AccountInfo.gerrit_port),
+                                 cfg.AccountInfo.gerrit_username,
+                                 key_filename=cfg.AccountInfo.gerrit_ssh_key)
+                connected = True
+            except paramiko.SSHException as e:
+                logger.error('%s', e)
+                logger.warn('Gerrit may be down, will pause and retry...')
+                time.sleep(10)
+
+        self.stdin, self.stdout, self.stderr =\
+            self.ssh.exec_command("gerrit stream-events")
+
+    def __iter__(self):
+        return self
+
+    def next(self):
+        return self.stdout.readline()
+
+
+def parse_json_event(event):
+    try:
+        event = json.loads(event)
+    except Exception as ex:
+        logger.error('Failed json.loads on event: %s', event)
+        logger.exception(ex)
+        return None
+    if _is_valid(event):
+        _process_event(event)
+        logger.info('Parsed valid event: %s', event)
+        return event
+    return None
+
+
+def add_event_to_db(event, commit_=True):
+    project = db.session.query(models.Project).filter(
+        models.Project.name == _process_project_name(
+            event["change"]["project"])).one()
+    patch_set = db.get_or_create(
+        models.PatchSet,
+        commit_=False,
+        project_id=project.id,
+        ref=event['patchSet']['ref'],
+        commit_message=event['change']['commitMessage'],
+        created=datetime.fromtimestamp(
+            int(event['patchSet']['createdOn'])))
+
+    owner_name = event["author"]["name"]
+    owner = db.get_or_create(models.CiOwner, name=owner_name)
+    trusted = (event["author"]["username"] == "jenkins")
+
+    if trusted and "approvals" in event:
+        if event["approvals"][0]["value"] in ("+1", "+2"):
+            patch_set.verified = True
+        elif event["approvals"][0]["value"] in ("-1", "-2"):
+            patch_set.verified = False
+
+    for ci, data in event['ci-status'].iteritems():
+        ci_server = db.get_or_create(models.CiServer,
+                                     commit_=False,
+                                     name=ci,
+                                     trusted=trusted,
+                                     ci_owner_id=owner.id)
+        db.update_or_create_comment(commit_=False,
+                                    result=data["result"],
+                                    log_url=data["log_url"],
+                                    ci_server_id=ci_server.id,
+                                    patch_set_id=patch_set.id)
+    if commit_:
+        db.session.commit()
+
+
+def main():
+    db.create_projects()  # This will make sure the database has projects in it
+    while True:
+        try:
+            events = GerritEventStream()
+        except paramiko.SSHException as ex:
+            logger.exception('Error connecting to Gerrit: %s', ex)
+            time.sleep(60)
+        for event in events:
+            event = parse_json_event(event)
+            if event is not None:
+                _store_event(event)
+
+
+if __name__ == '__main__':
+    main()

ciwatch/filters.py

@@ -0,0 +1,32 @@
+# Copyright (c) 2015 Tintri. All rights reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+import re
+
+from jinja2 import evalcontextfilter, Markup, escape
+
+from ciwatch import app
+
+
+_paragraph_re = re.compile(r'(?:\r\n|\r|\n){2,}')
+
+
+@app.template_filter()
+@evalcontextfilter
+def nl2br(eval_ctx, value):
+    result = u'\n\n'.join(u'<p>%s</p>' % p.replace('\n', '<br>\n')
+                          for p in _paragraph_re.split(escape(value)))
+    if eval_ctx.autoescape:
+        result = Markup(result)
+    return result

ciwatch/log.py

@@ -0,0 +1,46 @@
+# Copyright (c) 2015 Tintri. All rights reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+import logging
+from logging import handlers
+import os
+
+from ciwatch.config import cfg
+
+
+def setup_logger(name):
+
+    logger = logging.getLogger(name)
+    logger.setLevel(logging.DEBUG)
+    log_formatter = logging.Formatter("%(asctime)s [%(levelname)-5.5s]"
+                                      "  %(message)s")
+
+    file_handler =\
+        handlers.RotatingFileHandler(name,
+                                     maxBytes=1048576,
+                                     backupCount=2,)
+    logger.addHandler(file_handler)
+
+    console_handler = logging.StreamHandler()
+    console_handler.setFormatter(log_formatter)
+    logger.addHandler(console_handler)
+    return logger
+
+
+DATA_DIR =\
+    os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + '/data'
+if cfg.Data.data_dir:
+    DATA_DIR = cfg.Data.data_dir
+
+logger = setup_logger(DATA_DIR + '/ci-watch.log')

ciwatch/models.py

@@ -0,0 +1,98 @@
+# Copyright (c) 2015 Tintri. All rights reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from sqlalchemy import Boolean, Column, DateTime, Integer, String, ForeignKey
+from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.orm import relationship, backref
+
+
+Base = declarative_base()
+
+
+class Project(Base):
+    __tablename__ = "projects"
+
+    id = Column(Integer, primary_key=True)
+    name = Column(String, unique=True)
+
+    def __repr__(self):
+        return "<Project(name='%s')>" % self.name
+
+
+class PatchSet(Base):
+    __tablename__ = "patch_sets"
+
+    id = Column(Integer, primary_key=True)
+    created = Column(DateTime)
+    # ref = Column(String, unique=True)
+    ref = Column(String)  # Why are there duplicate refs?
+    # Verified only represents Jenkin's vote
+    verified = Column(Boolean, nullable=True, default=None)
+
+    commit_message = Column(String)
+
+    project_id = Column(Integer, ForeignKey('projects.id'))
+    project = relationship("Project", backref=backref('patch_sets',
+                                                      order_by=id))
+
+    def __repr__(self):
+        return "<PatchSet(created='%s', ref='%s')>" % (
+            self.created, self.ref)
+
+
+class Comment(Base):
+    __tablename__ = "comments"
+
+    id = Column(Integer, primary_key=True)
+    result = Column(String)
+    log_url = Column(String, nullable=True, default=None)
+
+    ci_server_id = Column(Integer, ForeignKey('ci_servers.id'))
+    ci_server = relationship("CiServer", backref=backref('comments',
+                                                         order_by=id))
+
+    patch_set_id = Column(Integer, ForeignKey('patch_sets.id'))
+    patch_set = relationship("PatchSet", backref=backref('comments',
+                                                         order_by=id))
+
+    def __repr__(self):
+        return "<Comment(log_url='%s', result='%s')>" % (
+            self.log_url, self.result)
+
+
+class CiServer(Base):
+    __tablename__ = "ci_servers"
+
+    id = Column(Integer, primary_key=True)
+    name = Column(String)
+
+    # Official OpenStack CIs are trusted (e.g., Jenkins)
+    trusted = Column(Boolean, default=False)
+
+    ci_owner_id = Column(Integer, ForeignKey('ci_owners.id'))
+    ci_owner = relationship('CiOwner', backref=backref('ci_servers',
+                                                       order_by=id))
+
+    def __repr__(self):
+        return "<CiServer(name='%s')>" % self.name
+
+
+class CiOwner(Base):
+    __tablename__ = "ci_owners"
+
+    id = Column(Integer, primary_key=True)
+    name = Column(String, unique=True)
+
+    def __repr__(self):
+        return "<CiOwner(name='%s')>" % self.name

ciwatch/populate.py

@@ -0,0 +1,42 @@
+# Copyright (c) 2015 Tintri. All rights reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from ciwatch import db
+from ciwatch.events import parse_json_event, add_event_to_db
+
+
+def get_data():
+    data = []
+    with open('/var/data/third-party-ci.log') as file_:
+        for line in file_:
+            event = parse_json_event(line)
+            if event is not None:
+                data.append(event)
+    return data
+
+
+def load_data():
+    data = get_data()
+    for event in data:
+        add_event_to_db(event, commit_=False)
+    db.session.commit()
+
+
+def main():
+    db.create_projects()
+    load_data()
+
+
+if __name__ == '__main__':
+    main()

ciwatch/static/hover.js

@@ -0,0 +1,23 @@
+// Code adapted from https://css-tricks.com/row-and-column-highlighting/
+$(document).ready(function () {
+  $("table").delegate("td.result","mouseover mouseleave", function(e) {
+      if (e.type == "mouseover") {
+        $(this).parent().addClass("hover");
+        $("colgroup").eq($(this).index()).addClass("hover");
+      }
+      else {
+        $(this).parent().removeClass("hover");
+        $("colgroup").eq($(this).index()).removeClass("hover");
+      }
+  });
+  $("table").delegate("td.ci-name","mouseover mouseleave", function(e) {
+      if (e.type == "mouseover") {
+        $(this).parent().addClass("hover");
+      }
+      else {
+        $(this).parent().removeClass("hover");
+      }
+  });
+
+  $('[data-toggle="popover"]').popover({trigger: 'hover'});
+});

ciwatch/static/style.css

@@ -0,0 +1,65 @@
+table.table {
+  white-space: nowrap;
+}
+
+.hover {
+  background-color: #EEE;
+}
+
+.failure {
+  color: #C00;
+  font-weight: bold;
+}
+
+.success {
+  color: #0C0;
+  font-weight: bold;
+}
+
+.unstable {
+  color: #BB0;
+  font-weight: bold;
+}
+
+.unregistered {
+  color: #333;
+  font-weight: bold;
+}
+
+.ci-user {
+  background-color: #DDD !important;
+  font-style: italic;
+}
+
+.blank-row {
+  height: 20px;
+}
+
+.popover {
+  max-width: 100%;
+}
+
+.no-style-link:link {
+  text-decoration: none;
+  color: #333;
+}
+
+.no-style-link:visited {
+  text-decoration: none;
+  color: #333;
+}
+
+.no-style-link:hover {
+  text-decoration: none;
+  color: #333;
+}
+
+.no-style-link:active {
+  text-decoration: none;
+  color: #333;
+}
+
+.glyphicon-none:before {
+  content: "\2122";
+  color: transparent !important;
+}

ciwatch/static/verified.js

@@ -0,0 +1,33 @@
+/* Copyright (c) 2015 Tintri. All rights reserved.
+ *
+ *    Licensed under the Apache License, Version 2.0 (the "License"); you may
+ *    not use this file except in compliance with the License. You may obtain
+ *    a copy of the License at
+ *
+ *         http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *    Unless required by applicable law or agreed to in writing, software
+ *    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ *    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ *    License for the specific language governing permissions and limitations
+ *    under the License.
+ */
+
+var toggle1 = function () {
+  $(".verified-1").css("background-color", "#FAA");
+  $(this).one("click", toggle2);
+}
+
+var toggle2 = function () {
+  $(".verified-1").css("background-color", "");
+  $(this).one("click", toggle1);
+}
+
+$(document).ready(function () {
+  $("colgroup").each(function (i, elem) {
+    if ($(elem).hasClass("verified-1")) {
+      $("#results").find("td").filter(":nth-child(" + (i + 1) + ")").addClass("verified-1");
+    }
+  });
+  $("#verified-1-button").one("click", toggle1);
+});

ciwatch/templates/_contact.html.jinja

@@ -0,0 +1 @@
+<p>Send feedback to <a href="mailto:openstack-dev@tintri.com">openstack-dev.tintri.com</a></p>

ciwatch/templates/_header.html.jinja

@@ -0,0 +1,3 @@
+<div class="page-header">
+  <h1><a class="no-style-link" href="{{ url_for("home") }}">CI Watch</a><small> &mdash; an OpenStack third-party CI monitoring tool</small></h1>
+</div>

ciwatch/templates/_usage.html.jinja

@@ -0,0 +1,7 @@
+      <p>
+        Each project has a table of recent CI results.
+        Columns plot patch sets and rows plot CI servers.
+        Each cell shows results for the row's CI server and the column's patch set.
+        Click the icons to view relevant logs.
+        Newest results are shown furthest to the left.
+      </p>

ciwatch/templates/index.html.jinja

@@ -0,0 +1,111 @@
+<html>
+  <head>
+
+    <link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">
+    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/css/font-awesome.min.css">
+
+    <link rel=stylesheet type=text/css href="{{ url_for('static', filename='style.css') }}">
+  </head>
+
+  <body>
+
+    <div class="container">
+
+      {% include '_header.html.jinja' %}
+
+      <div class="jumbotron">
+        <h2>Select a project to view CI results</h2>
+        <br>
+        <div class="row">
+          {% for project in projects %}
+          <div class="col-xs-6 col-md-3">
+            <a href="{{ url_for('project', project=project.name) }}" class="thumbnail">
+              <p style="text-align:center">{{ project.name }}</p>
+            </a>
+          </div>
+          {% endfor %}
+        </div>
+      </div>
+
+      <h2>Usage</h2>
+      {% include '_usage.html.jinja' %}
+
+      <div class="panel panel-default">
+        <div class="panel-heading">Legend</div>
+        <table class="table table-bordered">
+          <thead>
+            <tr>
+              <th>Icon</th>
+              <th>Meaning</th>
+            </tr>
+          </thead>
+
+          <tbody>
+            <tr>
+              <td>
+                <a href="#">
+                  <span class="fa fa-external-link"></span>
+                </a>
+              </td>
+              <td>Link to patch set on Gerrit</td>
+            </tr>
+
+            <tr>
+              <td>
+                <p class="success">
+                  <span class="glyphicon glyphicon-ok">
+                  </span>
+                </p>
+              </td>
+              <td>CI voted <span class="success">SUCCESS</span> for patch set</p></td>
+            </tr>
+
+            <tr>
+              <td>
+                <p class="failure">
+                  <span class="glyphicon glyphicon-remove">
+                  </span>
+                </p>
+              </td>
+              <td>CI voted <span class="failure">FAILURE</span> for patch set</td>
+            </tr>
+
+            <tr>
+              <td>
+                <p class="unstable">
+                  <span class="fa fa-exclamation-triangle">
+                  </span>
+                </p>
+              </td>
+              <td>CI voted <span class="unstable">UNSTABLE</span> for patch set</td>
+            </tr>
+
+            <tr>
+              <td>
+                <p class="unregistered">N</p>
+              </td>
+              <td>CI voted <span class="unregistered">NOT_REGISTERED</span> patch set</td>
+            </tr>
+
+            <tr>
+              <td>
+                <span class="glyphicon glyphicon-none">
+                </span>
+              </td>
+              <td>CI has not yet voted on this patch set</td>
+            </tr>
+
+          </tbody>
+        </table>
+      </div>
+
+      <h2>Contact Us</h2>
+      {% include '_contact.html.jinja' %}
+
+    </div>
+
+    <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
+    <script src="http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/js/bootstrap.min.js"></script>
+
+  </body>
+</html>

ciwatch/templates/project.html.jinja

@@ -0,0 +1,109 @@
+<html>
+  <head>
+
+    <link rel="stylesheet" href="http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">
+    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/css/font-awesome.min.css">
+
+    <link rel=stylesheet type=text/css href="{{ url_for('static', filename='style.css') }}">
+  </head>
+  <body>
+
+    <div class="container">
+      {% include '_header.html.jinja' %}
+      {% include '_usage.html.jinja' %}
+      {% include '_contact.html.jinja' %}
+      <div class="alert alert-info">Viewing results for {{ project.name }} from the past {{ request.args.get('time', time_option) }}.</div>
+    </div>
+
+    <div class="btn-group">
+      <div class="btn-group">
+        <button class="btn btn-default dropdown-toggle" type="button" data-toggle="dropdown">Switch project
+          <span class="caret"></span>
+        </button>
+        <ul class="dropdown-menu">
+          {% for proj in projects %}
+          <li><a href="{{ url_for('project', project=proj.name, time=request.args.get('time', time_option)) }}">{{ proj.name }}</a></li>
+          {% endfor %}
+        </ul>
+      </div>
+
+      <div class="btn-group">
+        <button class="btn btn-default dropdown-toggle" type="button" data-toggle="dropdown">Past {{ time_option }}
+          <span class="caret"></span>
+        </button>
+        <ul class="dropdown-menu">
+          {% for option in time_options %}
+          <li><a href="{{ url_for('project', project=project.name, time=option) }}">{{ option }}</a></li>
+          {% endfor %}
+        </ul>
+      </div>
+
+      <button id="verified-1-button" class="btn btn-default" type="button" data-toggle="button">Highlight Jenkins -1 votes</button>
+    </div>
+
+    <br>
+
+    <div class="table">
+      <table id="results" class="table table-bordered table-condensed">
+
+        <colgroup></colgroup>
+        {% for patch_set in patch_sets %}
+        {% if patch_set.verified is not none and not patch_set.verfied %}
+        <colgroup class="verified-1"></colgroup>
+        {% else %}
+        <colgroup></colgroup>
+        {% endif %}
+        {% endfor %}
+
+        <tbody>
+
+          <tr>
+            <td>Patch Set</td>
+            {% for patch_set in patch_sets %}
+            <td style="text-align:center;">
+              <a href="https://review.openstack.org/#/c/{{ "/".join(patch_set.ref.split("/")[-2:]) }}"
+                 data-toggle="popover" data-html="true" data-placement="bottom" title="Commit Message" 
+                 data-content="{{ patch_set.commit_message|nl2br }}">
+                <span class="fa fa-external-link"></span>
+              </a>
+            </td>
+            {% endfor %}
+          </tr>
+
+          {% for owner, results in user_results.iteritems() %}
+          <tr class="ci-user"><td colspan="{{ patch_sets|length + 1}}" class="ci-user">{{ owner.name }}</td></tr>
+          {% for ci in results %}
+          <tr>
+            <td class="ci-name">{{ ci["name"] }}</td>
+            {% for comment in ci["results"] %}
+              <td class="result">
+              {% if comment is not none %}
+                <a href="{{ comment.log_url }}">
+                {% if comment.result == "SUCCESS" %}
+                  <p class="success"><span class="glyphicon glyphicon-ok"></span></p>
+                {% elif comment.result == "FAILURE" %}
+                  <p class="failure"><span class="glyphicon glyphicon-remove"></span></p>
+                {% elif comment.result == "UNSTABLE" %}
+                  <p class="unstable"><span class="fa fa-exclamation-triangle"></span></p>
+                {% elif comment.result == "NOT_REGISTERED" %}
+                  <p class="unregistered">N</p>
+                {% endif %}
+              </a>
+              {% endif %}
+            </td>
+            {% endfor %}
+          </tr>
+          {% endfor %}
+          <tr class="blank-row"></tr>
+          {% endfor %}
+        </tbody>
+      </table>
+    </div>
+
+    <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
+    <script src="http://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/js/bootstrap.min.js"></script>
+    <script src="{{ url_for('static', filename='hover.js') }}"></script>
+    <script src="{{ url_for('static', filename='verified.js') }}"></script>
+
+  </body>
+</html>

ciwatch/views.py

@@ -0,0 +1,38 @@
+# Copyright (c) 2015 Tintri. All rights reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from flask import render_template
+from sqlalchemy.orm.exc import NoResultFound
+from werkzeug.exceptions import abort
+
+from ciwatch import app
+from ciwatch.api import get_context
+from ciwatch.api import get_projects
+from ciwatch.cache import cached
+
+
+@app.route("/")
+@app.route("/index")
+@app.route("/home")
+def home():
+    return render_template('index.html.jinja', projects=get_projects())
+
+
+@app.route("/project")
+@cached
+def project():
+    try:
+        return render_template("project.html.jinja", **get_context())
+    except NoResultFound:
+        abort(404)

requirements.txt

@@ -0,0 +1,4 @@
+flask
+sqlalchemy
+iniparse
+paramiko

run.py

@@ -0,0 +1,18 @@
+# Copyright (c) 2015 Tintri. All rights reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+import ciwatch
+
+if __name__ == "__main__":
+    ciwatch.app.run(debug=True)

setup.py

@@ -0,0 +1,41 @@
+#!/usr/bin/env python
+
+# Copyright (c) 2015 Tintri. All rights reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from os.path import join, dirname
+
+from setuptools import setup
+
+import ciwatch
+
+
+setup(
+    name='ci-watch',
+    version=ciwatch.__version__,
+    long_description=open(join(dirname(__file__), 'README.md')).read(),
+    entry_points={
+        'console_scripts': [
+            'ci-watch-server = ciwatch:main',
+            'ci-watch-populate-database = ciwatch.populate:main',
+            'ci-watch-stream-events = ciwatch.events:main',
+            ],
+        },
+    install_requires=[
+        "flask",
+        "sqlalchemy",
+        "iniparse",
+        "paramiko",
+    ]
+)