Send mail tool

This is the script that will send an email to a list of
user stored in the config file with information regarding the
results of a tempest run, together with the ansible task that
will execute it when tempest finishes run.

Change-Id: I57f5f0928e5efec106c793b71e335af1ac77fa91
Implements: blueprint send-mail-tool
This commit is contained in:
Arx Cruz 2017-01-20 16:39:24 +01:00 committed by Ronelle Landy
parent c16d247666
commit a0ded5ef54
7 changed files with 570 additions and 1 deletions

View File

@ -26,6 +26,7 @@ Role Variables
* `tempest_until_failure`: false/true - default is false, repeat the run again and again until failure occurs
* `tempest_failing`: false/true - default is false, run only tests known to be failing
* `tempest_exit_on_failure`: true/false - whether to exit from role with tempest exit code (default: true)
* `tempestmail_config`: config.yaml - name of config file for tempestmail script
Skip tests file
---------------

View File

@ -34,3 +34,4 @@ tempest_exit_on_failure: true
tempest_version_dict: { 'mitaka': 'mitaka', 'newton': 'newton',
'ocata': '16.0.0', 'master': 'master'}
tempest_version: "{{ tempest_version_dict[release] }}"
tempestmail_config: config.yaml

View File

@ -0,0 +1,109 @@
Tempest mail tool
=================
Description
-----------
This is a tool to send mails to people interested in TripleO periodic jobs
status.
Usage
-----
```bash
tempestmail.py -c config.yaml --jobs periodic-tripleo-ci-centos-7-ovb-ha-tempest
```
Config file example
-------------------
```yaml
mail_username: username
mail_password: password
smtp_server: smtp.gmail.com:587
mail_from: username@gmail.com
template_path: template/
log_url: 'http://logs.openstack.org/periodic'
emails:
- mail: 'arxcruz@gmail.com'
name: 'Arx Cruz'
template: template.html
known_failures:
- test: 'tempest.scenario.test_volume_boot_pattern.*'
reason: 'http://bugzilla.redhat.com/1272289'
- test: 'tempest.api.identity.*v3.*'
reason: 'https://bugzilla.redhat.com/1266947'
- test: '.*test_external_network_visibility'
reason: 'https://bugs.launchpad.net/tripleo/+bug/1577769'
```
HTML template example
---------------------
Tempest mail uses Jinja2 to create templates in html, and it parses the
following data to HTML (stored in the data dictionary)
* run - Bool - Whether the job runs or not
* date - String - In the format %Y-%m-%d %H:%M
* link - String - Contain the log url
* job - String - The job name
* failed - List - List of tests that fails in string format
* covered - List - List of tests covered in dictionary format, containing:
* failure - String - Test name
* reason - String - Reason of the failure
* new - List - List of new failures
* errors - List - Errors found in the log
An example of output of the data is showed below:
```python
[
{
'errors': [],
'run': True,
'failed': [
u'tempest.api.object_storage.test_container_quotas.ContainerQuotasTest.test_upload_too_many_objects',
u'tempest.api.object_storage.test_container_quotas.ContainerQuotasTest.test_upload_valid_object'
],
'job': 'periodic-tripleo-ci-centos-7-ovb-ha-tempest',
'link': u'http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha-tempest/1ce5e95/console.html',
'covered': [],
'date': datetime.datetime(2017, 1, 19, 8, 27),
'new': [
u'tempest.api.object_storage.test_container_quotas.ContainerQuotasTest.test_upload_too_many_objects',
u'tempest.api.object_storage.test_container_quotas.ContainerQuotasTest.test_upload_valid_object'
]
}
]
```
And here's an example you can use as email template:
```html
<html>
<head></head>
<body>
<p>Hello,</p>
<p>Here's the result of the latest tempest run for job {{ data.get('job') }}.</p>
<p>The job ran on {{ data.get('date') }}.</p>
<p>For more details, you can check the URL: {{ data.get('link') }}
{% if data.get('new') %}</p>
<h2>New failures</h2>
<ul>
{% for fail in data.get('new') %}
<li>{{ fail }}</li>
{% endfor %}
</ul>
{% endif %}
{% if data.get('covered') %}
<h2>Known failures</h2>
<ul>
{% for fail in data.get('covered') %}
<li>{{ fail.get('failure') }} - {{ fail.get('reason') }}</li>
{% endfor %}
</ul>
{% endif %}
</body>
</html>
```

View File

@ -0,0 +1,18 @@
require_auth: True
mail_from: tripleoresults@gmail.com
templates_path: template/
log_url: 'http://logs.openstack.org/periodic/'
api_server: 'http://tempest-tripleoci.rhcloud.com/api/v1.0/sendmail'
use_api_server: True
default_log_url: 'http://logs.openstack.org'
emails:
- mail: 'arxcruz@redhat.com'
name: 'Arx Cruz'
- mail: 'whayutin@redhat.com'
name: 'Wes'
- mail: 'gcerami@redhat.com'
name: 'Gabriele'
- mail: 'sshnaidm@redhat.com'
name: 'Sagi Shnaidman'
template: template.html

View File

@ -0,0 +1,380 @@
#! /usr/bin/env python
# Copyright Red Hat, Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import argparse
import datetime
import logging
import os
import re
import requests
import smtplib
import sys
import yaml
from email.mime.text import MIMEText
from jinja2 import Environment
from jinja2 import FileSystemLoader
from six.moves.urllib.parse import urljoin
HREF = re.compile('href="([^"]+)"')
JOBRE = re.compile('[a-z0-9]{7}/')
TESTRE = re.compile('(tempest\.[^ \(\)]+|\w+\.tests\.[^ \(\)]+)')
TIMEST = re.compile('(\d{4}-\d{2}-\d{2} \d{2}:\d{2}):\d{2}\.\d+ \|')
TITLE = re.compile('<title>(.*?)</title>')
FAILED = "... FAILED"
OK = "... ok"
ERROR = "... ERROR"
SKIPPED = "... SKIPPED"
NLINKS = 1
def compare_tests(failures, config):
'''Detect fails covered by bugs and new'''
covered, new = [], []
for fail in failures:
for test in config.known_failures:
if re.search(test.get('test'), fail):
covered.append({'failure': fail, 'reason': test.get('reason')})
new = [fail for fail in failures if not any(
c['failure'] == fail for c in covered)]
return covered, new
def get_html(url):
try:
resp = requests.get(url)
if resp is None:
raise TypeError
except TypeError as e:
print("Exception %s" % str(e))
return
return resp
def get_tests_results(console):
'''Get results of tests from console'''
failed = [TESTRE.search(l).group(1)
for l in console.splitlines() if FAILED in l]
ok = [TESTRE.search(l).group(1)
for l in console.splitlines() if OK in l]
errors = [TESTRE.search(l).group(1)
for l in console.splitlines() if ERROR in l]
# all_skipped = [TESTRE.search(l).group(1)
# for l in console.splitlines() if SKIPPED in l]
return failed, ok, errors
class Config(object):
pass
class Mail(object):
def __init__(self, config):
self.config = config
self.log = logging.getLogger('Mail')
self.mail_from = config.mail_from
self.username = config.username
self.password = config.password
self.smtp = config.smtp
self.require_auth = config.require_auth
self.templates_path = os.path.join(os.path.dirname(__file__),
config.templates_path)
self.template = config.template
def render_template(self, data):
self.log.debug('Rendering template')
env = Environment(loader=FileSystemLoader(self.templates_path))
template = env.get_template(self.template)
return template.render(data=data)
def filter_emails(self, job, data):
has_errors = False
addresses = []
for error in [data.get(x, []) for x in ('new', 'failed', 'errors')]:
if error:
self.log.debug('There are tests with failed result')
has_errors = True
break
if has_errors:
# Check if the user is assigned for the job
# If there's no job assigned, we add the user anyway
emails = [m for m in self.config.emails if job in
m.get('jobs') or not
m.get('jobs')]
# Now we filter for regex if doesn't exists
addresses = [m.get('mail') for m in emails if not m.get('regex')]
# And finally, if regex exists
for email in emails:
for r in email.get('regex'):
if len(filter(r.search, data.get('new'))):
addresses.append(email.get('mail'))
break
else:
self.log.debug('No failures send email to everybody')
addresses = [m.get('mail') for m in self.config.emails]
data['has_errors'] = has_errors
return addresses
def _send_mail_local(self, addresses, message, subject, output):
msg = MIMEText(message, 'html')
msg['Subject'] = subject
msg['From'] = self.mail_from
msg['To'] = ",".join(addresses)
s = smtplib.SMTP(self.smtp)
if self.require_auth:
s.ehlo()
s.starttls()
s.login(self.username, self.password)
s.sendmail(self.mail_from, addresses, msg.as_string())
self.log.debug('Sending mail')
s.quit()
if output:
self.log.debug('Writing email in {}'.format(output))
with open(output, 'w') as f:
f.write(msg.as_string())
def _send_mail_api(self, addresses, message, subject):
data = {'addresses': addresses, 'message': message, 'subject': subject,
'mime_type': 'html'}
requests.post(self.config.api_server, data=data)
def send_mail(self, job, data, output):
addresses = self.filter_emails(job, data)
message = self.render_template(data)
subject = 'Job {} results'.format(job)
if self.config.use_api_server:
self._send_mail_api(addresses, message, subject)
else:
self._send_mail_local(addresses, message, subject, output)
class TempestMailCmd(object):
def parse_arguments(self):
parser = argparse.ArgumentParser(description='tempest-mail')
parser.add_argument('-c', dest='config',
default='/etc/tempest-mail/tempest-mail.yaml',
help='Path to config file')
parser.add_argument('-l', dest='logconfig',
help='Path to log config file')
parser.add_argument('--version', dest='version',
help='Show version')
parser.add_argument('--job', dest='job',
help='Job name', required=True)
parser.add_argument('--file', dest='file',
help='File containing tempest output')
parser.add_argument('--skip-file', dest='skip_file',
help='List of skip files')
parser.add_argument('--output', dest='output',
help='Save the email content in a file')
self.args = parser.parse_args()
def setup_logging(self):
self.log = logging.getLogger('tempestmail.TempestMail')
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s %(levelname)s %(name)s: '
'%(message)s')
def get_index(self):
'''Get index page of periodic job and returns all links to jobs'''
url = urljoin(self.config.log_url, self.args.job)
res = get_html(url)
if res is None or not res.ok:
return []
body = res.content.decode() if res.content else ''
hrefs = [HREF.search(l).group(1)
for l in body.splitlines() if HREF.search(l)]
links = ["/".join((url, link))
for link in hrefs if JOBRE.match(link)]
if links:
# Number of links to return
return links[:NLINKS]
else:
return []
def get_console(self, job_url=None):
'''Get console page of job'''
if self.args.file and not job_url:
try:
with open(self.args.file) as f:
console = f.read()
log_path = os.environ.get('LOG_PATH', None)
if log_path:
log_path = urljoin(self.config.default_log_url,
log_path)
else:
log_path = ('Not available yet. Check '
'https://logs.openstack.org')
return (console, datetime.datetime.now(), log_path)
except IOError:
return (None, None, None)
def _good_result(res):
if res is None or int(res.status_code) not in (200, 404):
return False
else:
return True
def _get_date(c):
text = c.splitlines()
# find last line with timestamp
for l in text[::-1]:
if TIMEST.match(l):
return datetime.datetime.strptime(
TIMEST.search(l).group(1),
"%Y-%m-%d %H:%M")
return None
url = urljoin(job_url, "console.html.gz")
res = get_html(url)
if not _good_result(res):
print("Error getting console %s" % url)
# Try again
res = get_html(url)
if not _good_result(res):
return (None, None, None)
elif int(res.status_code) == 404:
url = urljoin(job_url, "console.html")
res = get_html(url)
if not _good_result(res):
# Try again
res = get_html(url)
if not _good_result(res):
print("Error getting console %s" % url)
return (None, None, None)
console = res.content.decode('utf-8')
date = _get_date(console)
return console, date, url
def get_data(self, console, date, link):
fails, ok, errors = get_tests_results(console)
d = {
'run': True,
'date': date,
'link': link,
'job': self.args.job
}
if fails or errors:
covered, new = compare_tests(fails, self.config)
d.update({
'failed': fails,
'covered': covered,
'new': new,
'errors': errors,
})
elif ok:
d['ok'] = ok
elif not fails and not ok and not errors:
d['run'] = False
return d
def load_skip_file(self, skipfile):
known_failures = []
try:
skip = yaml.load(open(self.args.skip_file))
for t in skip.get('known_failures'):
known_failures.append({'test': t.get('test'),
'reason': t.get('reason')})
except Exception:
pass
finally:
return known_failures
def checkJobs(self):
data = []
if self.args.file:
console, date, link = self.get_console()
d = self.get_data(console, date, link)
data.append(d)
else:
index = self.get_index()
for run in index:
console, date, link = self.get_console(run)
if not console or not date:
continue
d = self.get_data(console, date, link)
data.append(d)
data = sorted(data, key=lambda x: x['date'])
last = data[-1]
send_mail = Mail(self.config)
send_mail.send_mail(self.args.job, last, self.args.output)
def setupConfig(self):
self.log.debug("Loading configuration")
config = yaml.load(open(self.args.config))
newconfig = Config()
known_failures = []
newconfig.emails = []
newconfig.username = config.get('mail_username', '')
newconfig.password = config.get('mail_password', '')
newconfig.mail_from = config.get('mail_from', '')
newconfig.smtp = config.get('smtp_server', '')
newconfig.templates_path = config.get('templates_path')
newconfig.template = config.get('template')
newconfig.log_url = config.get('log_url')
newconfig.require_auth = config.get('require_auth', False)
newconfig.default_log_url = config.get('default_log_url',
'http://logs.openstack.org')
for e in config.get('emails'):
regex = [re.compile(r) for r in e.get('regex', [])]
newconfig.emails.append({'name': e.get('name'),
'mail': e.get('mail'),
'jobs': e.get('jobs', []),
'regex': regex})
for t in config.get('known_failures', []):
known_failures.append({'test': t.get('test'),
'reason': t.get('reason')})
if self.args.skip_file:
known_failures = (known_failures +
self.load_skip_file(self.args.skip_file))
newconfig.known_failures = known_failures
newconfig.api_server = config.get('api_server')
newconfig.use_api_server = config.get('use_api_server', False)
self.config = newconfig
def main():
tmc = TempestMailCmd()
tmc.parse_arguments()
tmc.setup_logging()
tmc.setupConfig()
tmc.checkJobs()
if __name__ == '__main__':
sys.exit(main())

View File

@ -0,0 +1,38 @@
<html>
<head></head>
<body>
<p>Hello,</p>
<p>Here's the result of the latest tempest run for job {{ data.get('job') }}.</p>
<p>The job ran on {{ data.get('date') }}.</p>
<p>For more details, you can check the URL: {{ data.get('link') }} (It might take a few minutes to upload the logs).</p>
{% if 'new' in data and data.new %}
<h2>New failures</h2>
<ul>
{% for fail in data.new %}
<li>{{ fail }}</li>
{% endfor %}
</ul>
{% endif %}
{% if 'covered' in data and data.covered %}
<h2>Known failures</h2>
<ul>
{% for fail in data.covered %}
<li>{{ fail.failure }} - {{ fail.reason }}</li>
{% endfor %}
</ul>
{% endif %}
{% if ('has_errors' in data and 'run' in data) and not data.has_errors and data.run %}
<h2>Job ran successfully!</h2>
<p>We consider a successfull run even if it has Known failures, since these are covered.</p>
{% endif %}
{% if not data.run %}
<h2>There's no tempest results!</h2>
<p>This means that the TripleO installation might have finished sucessfully, however, tempest either fail before tests started, or didn't ran at all.</p>
{% endif %}
<p></p>
<p>You are receiving this email because someone from TripleO team though you were interested in these results.</p>
<p>
</body>
</html>

View File

@ -19,7 +19,6 @@
- ignore_errors: true
block:
- name: Generate testrepository.subunit results file
shell: >
{% if tempest_format == 'venv' %}source {{ working_dir }}/tempest_git/.venv/bin/activate; {% endif %}
@ -39,6 +38,29 @@
dest: "{{ lookup('env', 'PWD') }}/tempest.html"
flat: yes
- name: Copying tempestmail files
synchronize:
src: tempestmail/
dest: "{{ working_dir }}/tempestmail/"
use_ssh_args: true
- name: Copying skip file
synchronize:
src: "vars/tempest_skip_{{ release }}.yml"
dest: "{{ working_dir }}/tempestmail/"
use_ssh_args: true
- name: Send tempest results by mail
shell: >
{% if lookup('env', 'LOG_PATH') %}LOG_PATH='{{ lookup('env', 'LOG_PATH') }}' {% endif %}
./tempestmail.py -c {{ tempestmail_config }} --job
"{{ lookup('env', 'JOB_NAME')|default('Periodic job', true) }}"
--file "{{ working_dir }}/{{ tempest_log_file }}"
--skip-file "{{ working_dir }}/tempestmail/tempest_skip_{{ release }}.yml"
args:
chdir: "{{ working_dir }}/tempestmail"
ignore_errors: yes
- name: Exit with tempest result code if configured
shell: tail -10 {{ tempest_log_file }}; exit {{ tempest_result.rc }}
when: tempest_result.rc != 0 and tempest_exit_on_failure|bool