Browse Source

Grafana container deployment

This uses the Grafana container created with
Iddfafe852166fe95b3e433420e2e2a4a6380fc64 to run the
grafana.opendev.org service.

We retain the old model of an Apache reverse-proxy; it's well tested
and understood, it's much easier than trying to map all the SSL
termination/renewal/etc. into the Grafana container and we don't have
to convince ourselves the container is safe to be directly web-facing.

Otherwise this is a fairly straight forward deployment of the
container.  As before, it uses the graph configuration kept in
project-config which is loaded in with grafyaml, which is included in
the container.

Once nice advantage is that it makes it quite easy to develop graphs
locally, using the container which can talk to the public graphite
instance.  The documentation has been updated with a reference on how
to do this.

Change-Id: I0cc76d29b6911aecfebc71e5fdfe7cf4fcd071a4
changes/06/737406/19
Ian Wienand 1 month ago
parent
commit
b146181174
17 changed files with 314 additions and 20 deletions
  1. +50
    -18
      doc/source/grafana.rst
  2. +4
    -1
      inventory/service/groups.yaml
  3. +5
    -0
      inventory/service/host_vars/grafana01.opendev.org
  4. +1
    -0
      playbooks/roles/grafana/README.rst
  5. +4
    -0
      playbooks/roles/grafana/handlers/main.yaml
  6. +87
    -0
      playbooks/roles/grafana/tasks/main.yaml
  7. +18
    -0
      playbooks/roles/grafana/templates/docker-compose.yaml.j2
  8. +41
    -0
      playbooks/roles/grafana/templates/grafana.vhost.j2
  9. +3
    -0
      playbooks/roles/letsencrypt-create-certs/handlers/main.yaml
  10. +6
    -0
      playbooks/service-grafana.yaml
  11. +1
    -0
      playbooks/zuul/run-base.yaml
  12. +3
    -0
      playbooks/zuul/templates/host_vars/grafana01.opendev.org.yaml.j2
  13. +27
    -0
      testinfra/test_grafana.py
  14. +1
    -1
      zuul.d/docker-images/grafana.yaml
  15. +17
    -0
      zuul.d/infra-prod.yaml
  16. +16
    -0
      zuul.d/project.yaml
  17. +30
    -0
      zuul.d/system-config-run.yaml

+ 50
- 18
doc/source/grafana.rst View File

@@ -13,34 +13,66 @@ At a Glance
===========

:Hosts:
* http://grafana.openstack.org
:Puppet:
* https://github.com/bfraser/puppet-grafana
* :git_file:`modules/openstack_project/manifests/grafana.pp`
* https://grafana.opendev.org
:Projects:
* http://grafana.org
* https://grafana.org
:Bugs:
* https://storyboard.openstack.org/#!/project/748

Overview
========

Apache is configured as a reverse proxy and there is a MySQL database
backend.
Apache is configured as a reverse proxy to Grafana running in a
container, listening on port 3000.


Sysadmin
========
Local Development
=================

To develop dashboards, you can run the OpenDev Grafana container.
Firstly, get setup:

.. code-block:: shell-session

$ cd <work dir>
$ mkdir secrets
$ echo "password" > secrets/admin_password
$ echo "admin" > secrets/admin_user
$ echo "key" > secrets/secret_key

$ git clone https://opendev.org/openstack/project-config

Then run the container with the following options:

.. code-block:: shell-session

After bringing up a Grafana node with puppet, log in and configure Grafana by
hand:
$ cd <work dir>
$ sudo podman run \
-p 3000:3000 \
-v ./secrets:/etc/grafana/secrets \
-v ./project-config:/opt/project-config \
-e GF_AUTH_ANONYMOUS_ENABLED=true \
-e GF_USERS_ALLOW_SIGN_UP=false \
-e GF_SECURITY_ADMIN_PASSWORD__FILE=/etc/grafana/secrets/admin_password \
-e GF_SECURITY_ADMIN_USER__FILE=/etc/grafana/secrets/admin_user \
-e GF_SECURITY_SECRET_KEY__FILE=/etc/grafana/secrets/secret_key \
docker.io/opendevorg/grafana

#. Log in as the admin user.
At this point, Grafana will be running and listening on port 3000.
You can log into as ``admin`` with ``password`` (or using your secrets
above).

#. Under 'Data Sources', add a new entry with following:
This is unconfigured and does not yet talk to the OpenDev Graphite
instance. The dashboard definitions are kept in
``project-config/grafana``. To load them ``exec`` the
``update-grafana`` script in the container (i.e. ``podman exec
<running-container> update-grafana``). That will run ``grafyaml`` and
load in the updated dashboards (note it relies on things being mapped
as above). To work on dashboards, update the ``yaml`` files in
``project-config`` and re-run ``update-grafana``, then reload them in
the Grafana UI.

- name: OpenStack
- type: Graphite
- default: checked
- url: http://graphite.opendev.org
- access: direct
Alternatively, you can use the Grafana editor to make the dashboards,
and then under "Dashboard Settings" (gear icon) select "JSON Model"
and commit that (it seems you have to cut-and-paste, there isn't
currently a way to export the JSON as such).

+ 4
- 1
inventory/service/groups.yaml View File

@@ -60,7 +60,9 @@ groups:
gitea-lb:
- gitea-lb[0-9]*.opendev.org
grafana:
- grafana[0-9]*.open*.org
- grafana[0-9]*.openstack.org
grafana_opendev:
- grafana[0-9]*.opendev.org
graphite:
- graphite*.open*.org
health:
@@ -73,6 +75,7 @@ groups:
- etherpad[0-9]*.opendev.org
- gitea[0-9]*.opendev.org
- graphite01.opendev.org
- grafana[0-9]*.opendev.org
- insecure-ci-registry[0-9]*.opendev.org
- meetpad[0-9]*.opendev.org
- mirror[0-9]*.opendev.org


+ 5
- 0
inventory/service/host_vars/grafana01.opendev.org View File

@@ -0,0 +1,5 @@
letsencrypt_certs:
grafana01-opendev-org-main:
- grafana01.opendev.org
- grafana.opendev.org
- grafana.openstack.org

+ 1
- 0
playbooks/roles/grafana/README.rst View File

@@ -0,0 +1 @@
Run Grafana

+ 4
- 0
playbooks/roles/grafana/handlers/main.yaml View File

@@ -0,0 +1,4 @@
- name: grafana Reload apache2
service:
name: apache2
state: reloaded

+ 87
- 0
playbooks/roles/grafana/tasks/main.yaml View File

@@ -0,0 +1,87 @@
- name: Ensure docker-compose directory exists
file:
state: directory
path: /etc/grafana-docker

- name: Write settings file
template:
src: docker-compose.yaml.j2
dest: /etc/grafana-docker/docker-compose.yaml

- name: Ensure config directory exists
file:
state: directory
path: /etc/grafana

- name: Ensure secrets config directory exists
file:
state: directory
path: /etc/grafana/secrets

- name: Make admin_password
copy:
content: '{{ grafana_admin_password }}'
dest: /etc/grafana/secrets/admin_password

- name: Make admin_user
copy:
content: '{{ grafana_admin_user }}'
dest: /etc/grafana/secrets/admin_user

- name: Make secret_key
copy:
content: '{{ grafana_secret_key }}'
dest: /etc/grafana/secrets/secret_key

- name: Install apache2
apt:
name:
- apache2
- apache2-utils
state: present

- name: Apache modules
apache2_module:
state: present
name: "{{ item }}"
loop:
- rewrite
- proxy
- proxy_http
- ssl
- headers
- proxy_wstunnel

- name: Copy apache config
template:
src: grafana.vhost.j2
dest: /etc/apache2/sites-enabled/000-default.conf
owner: root
group: root
mode: 0644
notify: grafana Reload apache2

- name: Sync project-config
include_role:
name: sync-project-config

- name: Run docker-compose pull
shell:
cmd: docker-compose pull
chdir: /etc/grafana-docker/

- name: Run docker-compose up
shell:
cmd: docker-compose up -d
chdir: /etc/grafana-docker/

- name: Run docker prune to cleanup unneeded images
shell:
cmd: docker image prune -f

- name: Import dashboards to container
shell:
cmd: |
docker-compose exec -T grafana /usr/local/bin/update-grafana
chdir: /etc/grafana-docker/


+ 18
- 0
playbooks/roles/grafana/templates/docker-compose.yaml.j2 View File

@@ -0,0 +1,18 @@
# Version 2 is the latest that is supported by docker-compose in
# Ubuntu Xenial.
version: '2'

services:
grafana:
restart: always
image: docker.io/opendevorg/grafana
network_mode: host
environment:
GF_SECURITY_ADMIN_PASSWORD__FILE: '/etc/grafana/secrets/admin_password'
GF_SECURITY_ADMIN_USER__FILE: '/etc/grafana/secrets/admin_user'
GF_SECURITY_SECRET_KEY__FILE: '/etc/grafana/secrets/secret_key'
GF_AUTH_ANONYMOUS_ENABLED: 'true'
GF_USERS_ALLOW_SIGN_UP: 'false'
volumes:
- /opt/project-config:/opt/project-config
- /etc/grafana/secrets:/etc/grafana/secrets

+ 41
- 0
playbooks/roles/grafana/templates/grafana.vhost.j2 View File

@@ -0,0 +1,41 @@
<VirtualHost *:80>
ServerName {{ inventory_hostname }}
ServerAdmin webmaster@openstack.org

ErrorLog ${APACHE_LOG_DIR}/grafana-error.log

LogLevel warn

CustomLog ${APACHE_LOG_DIR}/grafana-access.log combined

Redirect / https://{{ inventory_hostname }}/

</VirtualHost>

<VirtualHost *:443>
ServerName {{ inventory_hostname }}
ServerAdmin webmaster@openstack.org

AllowEncodedSlashes On

ErrorLog ${APACHE_LOG_DIR}/grafana-ssl-error.log

LogLevel warn

CustomLog ${APACHE_LOG_DIR}/grafana-ssl-access.log combined

SSLEngine on
SSLProtocol All -SSLv2 -SSLv3
# Note: this list should ensure ciphers that provide forward secrecy
SSLCipherSuite ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:!AES256:!aNULL:!eNULL:!MD5:!DSS:!PSK:!SRP
SSLHonorCipherOrder on

SSLCertificateFile /etc/letsencrypt-certs/{{ inventory_hostname }}/{{ inventory_hostname }}.cer
SSLCertificateKeyFile /etc/letsencrypt-certs/{{ inventory_hostname }}/{{ inventory_hostname }}.key
SSLCertificateChainFile /etc/letsencrypt-certs/{{ inventory_hostname }}/ca.cer

ProxyPass / http://localhost:3000/ retry=0
ProxyPassReverse / http://localhost:3000/

</VirtualHost>


+ 3
- 0
playbooks/roles/letsencrypt-create-certs/handlers/main.yaml View File

@@ -117,6 +117,9 @@
- name: letsencrypt updated static01-zuul-ci-org
include_tasks: roles/letsencrypt-create-certs/handlers/restart_apache.yaml

- name: letsencrypt updated grafana01-opendev-org-main
include_tasks: roles/letsencrypt-create-certs/handlers/restart_apache.yaml

# nodepool

- name: letsencrypt updated nb01-opendev-org-main


+ 6
- 0
playbooks/service-grafana.yaml View File

@@ -0,0 +1,6 @@
- hosts: "grafana_opendev:!disabled"
name: "Base: configure grafana"
roles:
- iptables
- install-docker
- grafana

+ 1
- 0
playbooks/zuul/run-base.yaml View File

@@ -74,6 +74,7 @@
- host_vars/letsencrypt02.opendev.org.yaml
- host_vars/lists.openstack.org.yaml
- host_vars/gitea99.opendev.org.yaml
- host_vars/grafana01.opendev.org.yaml
- host_vars/mirror01.openafs.provider.opendev.org.yaml
- host_vars/mirror02.openafs.provider.opendev.org.yaml
- host_vars/mirror-update01.opendev.org.yaml


+ 3
- 0
playbooks/zuul/templates/host_vars/grafana01.opendev.org.yaml.j2 View File

@@ -0,0 +1,3 @@
grafana_admin_password: adminpassword
grafana_admin_user: admin
grafana_secret_key: grafanaSecretKey

+ 27
- 0
testinfra/test_grafana.py View File

@@ -0,0 +1,27 @@
# Copyright 2020 Red Hat, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.


testinfra_hosts = ['grafana01.opendev.org']


def test_grafana_container_listening(host):
grafana = host.socket("tcp://127.0.0.1:3000")
assert grafana.is_listening

def tets_grafana_proxy(host):
cmd = host.run('curl --insecure '
'--resolve grafana.opendev.org:443:127.0.0.1 '
'https://grafana.opendev.org')
assert '<title>Grafana</title>' in cmd.stdout

+ 1
- 1
zuul.d/docker-images/grafana.yaml View File

@@ -9,7 +9,7 @@
- context: docker/grafana
repository: opendevorg/grafana
files: &grafana_files
- docker/grafana
- docker/grafana/

- job:
name: system-config-upload-image-grafana


+ 17
- 0
zuul.d/infra-prod.yaml View File

@@ -509,6 +509,23 @@
- playbooks/roles/accessbot
- docker/accessbot/

- job:
name: infra-prod-service-grafana
parent: infra-prod-service-base
description: Run service-grafana.yaml playbook.
vars:
playbook_name: service-grafana.yaml
files:
- inventory/
- playbooks/service-grafana.yaml
- inventory/service/host_vars/grafana01.org.yaml
- inventory/service/group_vars/grafana
- playbooks/roles/install-docker/
- playbooks/roles/pip3/
- playbooks/roles/grafana
- playbooks/roles/logrotate
- playbooks/roles/iptables/

# Run AFS changes separately so we can make sure to only do one at a time
# (turns out quorum is nice to have)
- job:


+ 16
- 0
zuul.d/project.yaml View File

@@ -43,6 +43,11 @@
soft: true
- name: system-config-build-image-haproxy-statsd
soft: true
- system-config-run-grafana:
dependencies:
- name: opendev-buildset-registry
- name: system-config-build-image-grafana
soft: true
- system-config-run-review:
dependencies:
- name: opendev-buildset-registry
@@ -122,6 +127,11 @@
soft: true
- name: system-config-upload-image-haproxy-statsd
soft: true
- system-config-run-grafana:
dependencies:
- name: opendev-buildset-registry
- name: system-config-upload-image-grafana
soft: true
- system-config-run-review:
dependencies:
- name: opendev-buildset-registry
@@ -210,6 +220,12 @@
soft: true
- name: system-config-promote-image-etherpad
soft: true
- infra-prod-service-grafana:
dependencies:
- name: infra-prod-letsencrypt
soft: true
- name: system-config-promote-image-grafana
soft: true
- infra-prod-service-meetpad
- infra-prod-service-mirror-update
- infra-prod-service-mirror


+ 30
- 0
zuul.d/system-config-run.yaml View File

@@ -542,6 +542,36 @@
# to run this job as well.
- docker/haproxy-statsd/

- job:
name: system-config-run-grafana
parent: system-config-run-containers
description: |
Run the playbook for the etherpad servers.
timeout: 3600
requires: grafana-container-image
required-projects:
- opendev/system-config
- openstack/project-config
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic
- name: grafana01.opendev.org
label: ubuntu-focal
vars:
run_playbooks:
- playbooks/letsencrypt.yaml
- playbooks/service-grafana.yaml
files:
- playbooks/bridge.yaml
- playbooks/letsencrypt.yaml
- playbooks/service-grafana.yaml
- playbooks/roles/grafana/
- playbooks/roles/install-docker/
- playbooks/roles/pip3/
- docker/grafana/
- testinfra/test_grafana.py

- job:
name: system-config-run-meetpad
parent: system-config-run-containers


Loading…
Cancel
Save