Add a keycloak server

This adds a keycloak server so we can start experimenting with it.

It's based on the docker-compose file Matthieu made for Zuul
(see https://review.opendev.org/819745 )

We should be able to configure a realm and federate with openstackid
and other providers as described in the opendev auth spec.  However,
I am unable to test federation with openstackid due its inability to
configure an oauth app at "localhost".  Therefore, we will need an
actual deployed system to test it.  This should allow us to do so.

It will also allow use to connect realms to the newly available
Zuul admin api on opendev.

It should be possible to configure the realm the way we want, then
export its configuration into a JSON file and then have our playbooks
or the docker-compose file import it.  That would allow us to drive
change to the configuration of the system through code review.  Because
of the above limitation with openstackid, I think we should regard the
current implementation as experimental.  Once we have a realm
configuration that we like (which we will create using the GUI), we
can chose to either continue to maintain the config with the GUI and
appropriate file backups, or switch to a gitops model based on an
export.

My understanding is that all the data (realms configuration and session)
are kept in an H2 database.  This is probably sufficient for now and even
production use with Zuul, but we should probably switch to mariadb before
any heavy (eg gerrit, etc) production use.

This is a partial implementation of https://docs.opendev.org/opendev/infra-specs/latest/specs/central-auth.html

We can re-deploy with a new domain when it exists.

Change-Id: I2e069b1b220dbd3e0a5754ac094c2b296c141753
Co-Authored-By: Matthieu Huin <mhuin@redhat.com>

doc/source/keycloak.rst

@@ -0,0 +1,32 @@
+:title: Keycloak
+
+.. _keycloak:
+
+Keycloak
+########
+
+Keycloak is installed on keycloak.opendev.org.  It is in a prototype
+phase for use with the Zuul admin API, and may be used by other
+OpenDev services in the future.
+
+At a Glance
+===========
+
+:Hosts:
+  * https://keycloak.opendev.org
+:Ansible:
+  * https://opendev.org/opendev/system-config
+  * :git_file:`playbooks/roles/keycloak`
+  * :git_file:`playbooks/service-keycloak.yaml`
+:Projects:
+  * https://www.keycloak.org/
+  * https://github.com/keycloak/keycloak-containers
+:Bugs:
+  * https://storyboard.openstack.org/#!/project/748
+  * https://issues.jboss.org/browse/KEYCLOAK
+
+Overview
+========
+
+Apache is configured as a reverse proxy and there is an internal H2
+database stored at ``/var/keycloak/data``.

doc/source/systems.rst

@@ -14,6 +14,7 @@ Major Systems
    gitea
    grafana
    grafyaml
+   keycloak
    zuul
    logstash
    elastic-recheck

hiera/common.yaml

@@ -38,6 +38,7 @@ cacti_hosts:
 - jvb02.opendev.org
 - kdc03.openstack.org
 - kdc04.openstack.org
+- keycloak01.opendev.org
 - lists.openstack.org
 - logstash-worker01.openstack.org
 - logstash-worker02.openstack.org

inventory/service/groups.yaml

@@ -84,6 +84,7 @@ groups:
     - kdc03.openstack.org
   kerberos-kdc-replica:
     - kdc04.openstack.org
+  keycloak: keycloak[0-9]*.opendev.org
   letsencrypt:
     - codesearch[0-9]*.opendev.org
     - eavesdrop[0-9]*.opendev.org
@@ -93,6 +94,7 @@ groups:
     - grafana[0-9]*.opendev.org
     - graphite[0-9]*.opendev.org
     - insecure-ci-registry[0-9]*.opendev.org
+    - keycloak[0-9]*.opendev.org
     - meetpad[0-9]*.opendev.org
     - mirror[0-9]*.opendev.org
     - nb[0-9]*.opendev.org
@@ -189,6 +191,7 @@ groups:
     - grafana[0-9]*.opendev.org
     - graphite*.opendev.org
     - health[0-9]*.openstack.org
+    - keycloak[0-9]*.opendev.org
     - nb[0-9]*.opendev.org
     - nl[0-9]*.open*.org
     - paste[0-9]*.opendev.org

inventory/service/host_vars/keycloak01.opendev.org.yaml

@@ -0,0 +1,6 @@
+letsencrypt_certs:
+  keycloak01-opendev-org-main:
+    # List the service name first since that determines the filename
+    # and is referenced in the apache config.
+    - keycloak.opendev.org
+    - keycloak01.opendev.org

playbooks/roles/keycloak/README.rst

@@ -0,0 +1 @@
+Run a Keycloak server.

playbooks/roles/keycloak/handlers/main.yaml

@@ -0,0 +1,4 @@
+- name: keycloak Reload apache2
+  service:
+    name: apache2
+    state: reloaded

playbooks/roles/keycloak/tasks/main.yaml

@@ -0,0 +1,72 @@
+- name: Ensure docker-compose directory exists
+  file:
+    state: directory
+    path: /etc/keycloak-docker
+
+- name: Write settings file
+  template:
+    src: docker-compose.yaml.j2
+    dest: /etc/keycloak-docker/docker-compose.yaml
+
+- name: Ensure data directory exists
+  file:
+    state: directory
+    path: /var/keycloak/data
+    owner: "1000"
+    group: "root"
+    mode: "0755"
+
+- name: Ensure log directory exists
+  file:
+    state: directory
+    path: /var/log/keycloak
+    owner: "1000"
+    group: "root"
+    mode: "0755"
+
+- name: Install apache2
+  apt:
+    name:
+      - apache2
+      - apache2-utils
+    state: present
+
+- name: Apache modules
+  apache2_module:
+    state: present
+    name: "{{ item }}"
+  loop:
+    - rewrite
+    - proxy
+    - proxy_http
+    - ssl
+    - headers
+    - proxy_wstunnel
+
+- name: Copy apache config
+  template:
+    src: keycloak.vhost.j2
+    dest: /etc/apache2/sites-enabled/000-default.conf
+    owner: root
+    group: root
+    mode: 0644
+  notify: keycloak Reload apache2
+
+- name: Run docker-compose pull
+  shell:
+    cmd: docker-compose pull
+    chdir: /etc/keycloak-docker/
+
+- name: Run docker-compose up
+  shell:
+    cmd: docker-compose up -d
+    chdir: /etc/keycloak-docker/
+
+- name: Wait for keycloak to start
+  wait_for:
+    port: 8080
+    timeout: 60
+
+- name: Run docker prune to cleanup unneeded images
+  shell:
+    cmd: docker image prune -f

playbooks/roles/keycloak/templates/docker-compose.yaml.j2

@@ -0,0 +1,19 @@
+# Version 2 is the latest that is supported by docker-compose in
+# Ubuntu Xenial.
+version: '2'
+
+services:
+  keycloak:
+    image: docker.io/jboss/keycloak
+    network_mode: host
+    restart: always
+    environment:
+      - KEYCLOAK_USER=admin
+      - KEYCLOAK_PASSWORD="{{ keycloak_admin_password }}"
+      - DB_VENDOR=h2
+    command:
+      -Djboss.bind.address.private=127.0.0.1
+      -Djboss.bind.address=127.0.0.1
+    volumes:
+      - /var/keycloak/data:/opt/jboss/keycloak/standalone/data
+      - /var/log/keycloak:/opt/jboss/keycloak/standalone/log

playbooks/roles/keycloak/templates/keycloak.vhost.j2

@@ -0,0 +1,55 @@
+<VirtualHost *:80>
+  ServerName keycloak.opendev.org
+  ServerAdmin webmaster@openstack.org
+
+  ErrorLog ${APACHE_LOG_DIR}/keycloak-error.log
+
+  LogLevel warn
+
+  CustomLog ${APACHE_LOG_DIR}/keycloak-access.log combined
+
+  Redirect / https://keycloak.opendev.org/
+
+</VirtualHost>
+
+<VirtualHost *:443>
+  ServerName keycloak.opendev.org
+  ServerAdmin webmaster@openstack.org
+
+  AllowEncodedSlashes On
+
+  ErrorLog ${APACHE_LOG_DIR}/keycloak-ssl-error.log
+
+  LogLevel warn
+
+  CustomLog ${APACHE_LOG_DIR}/keycloak-ssl-access.log combined
+
+  SSLEngine on
+  SSLProtocol All -SSLv2 -SSLv3
+  # Note: this list should ensure ciphers that provide forward secrecy
+  SSLCipherSuite ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:!AES256:!aNULL:!eNULL:!MD5:!DSS:!PSK:!SRP
+  SSLHonorCipherOrder on
+
+  SSLCertificateFile /etc/letsencrypt-certs/keycloak.opendev.org/keycloak.opendev.org.cer
+  SSLCertificateKeyFile /etc/letsencrypt-certs/keycloak.opendev.org/keycloak.opendev.org.key
+  SSLCertificateChainFile /etc/letsencrypt-certs/keycloak.opendev.org/ca.cer
+
+  BrowserMatch "MSIE [2-6]" \
+      nokeepalive ssl-unclean-shutdown \
+      downgrade-1.0 force-response-1.0
+  # MSIE 7 and newer should be able to use keepalive
+  BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
+
+  RewriteEngine on
+
+  # Do not rewrite the /server-status URL (though by default, this
+  # is only accessible from localhost).  Connect to it with:
+  # ssh -L 8443:localhost:443 $HOSTNAME
+  # https://localhost:8443/server-status
+  RewriteRule ^/server-status$ /server-status [L]
+
+  ProxyPass  / http://localhost:8080/ retry=0
+  ProxyPassReverse / http://localhost:8080/
+
+</VirtualHost>
+

playbooks/roles/letsencrypt-create-certs/handlers/main.yaml

@@ -249,6 +249,9 @@
 - name: letsencrypt updated ethercalc02-openstack-org-main
   include_tasks: roles/letsencrypt-create-certs/handlers/restart_apache.yaml
 
+- name: letsencrypt updated keycloak01-opendev-org-main
+  include_tasks: roles/letsencrypt-create-certs/handlers/restart_apache.yaml
+
 - name: letsencrypt updated storyboard01-opendev-org-main
   include_tasks: roles/letsencrypt-create-certs/handlers/restart_apache.yaml
 

playbooks/service-keycloak.yaml

@@ -0,0 +1,6 @@
+- hosts: "keycloak:!disabled"
+  name: "Base: configure keycloak"
+  roles:
+    - iptables
+    - install-docker
+    - keycloak

playbooks/zuul/run-base.yaml

@@ -59,6 +59,7 @@
         - group_vars/gitea.yaml
         - group_vars/gitea-lb.yaml
         - group_vars/kerberos-kdc.yaml
+        - group_vars/keycloak.yaml
         - group_vars/letsencrypt.yaml
         - group_vars/meetpad.yaml
         - group_vars/jvb.yaml

playbooks/zuul/templates/group_vars/keycloak.yaml.j2

@@ -0,0 +1 @@
+keycloak_admin_password: testpassword

testinfra/test_keycloak.py

@@ -0,0 +1,22 @@
+# Copyright 2018 Red Hat, Inc.
+# Copyright 2021 Acme Gating, LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+
+testinfra_hosts = ['keycloak01.opendev.org']
+
+
+def test_keycloak_listening(host):
+    keycloak = host.socket("tcp://127.0.0.1:8080")
+    assert keycloak.is_listening

zuul.d/infra-prod.yaml

@@ -192,6 +192,21 @@
       - playbooks/roles/iptables/
       - docker/etherpad/
 
+- job:
+    name: infra-prod-service-keycloak
+    parent: infra-prod-service-base
+    description: Run service-keycloak.yaml playbook.
+    vars:
+      playbook_name: service-keycloak.yaml
+    files:
+      - inventory/base
+      - playbooks/service-keycloak.yaml
+      - inventory/service/host_vars/keycloak01.opendev.org.yaml
+      - inventory/service/group_vars/keycloak
+      - playbooks/roles/keycloak/
+      - playbooks/roles/install-docker/
+      - playbooks/roles/iptables/
+
 - job:
     name: infra-prod-service-meetpad
     parent: infra-prod-service-base

zuul.d/project.yaml

@@ -61,6 +61,7 @@
               - name: system-config-build-image-grafana
                 soft: true
         - system-config-run-graphite
+        - system-config-run-keycloak
         - system-config-run-review-3.3:
             dependencies:
               - name: opendev-buildset-registry
@@ -221,6 +222,7 @@
               - name: system-config-upload-image-grafana
                 soft: true
         - system-config-run-graphite
+        - system-config-run-keycloak
         - system-config-run-review-3.3:
             dependencies:
               - name: opendev-buildset-registry
@@ -482,6 +484,10 @@
             dependencies:
               - name: infra-prod-letsencrypt
                 soft: true
+        - infra-prod-service-keycloak: &infra-prod-service-keycloak
+            dependencies:
+              - name: infra-prod-letsencrypt
+                soft: true
         - infra-prod-service-meetpad: &infra-prod-service-meetpad
             dependencies:
               - name: infra-prod-letsencrypt
@@ -624,6 +630,7 @@
         - infra-prod-service-gitea-lb: *infra-prod-service-gitea-lb
         - infra-prod-service-grafana: *infra-prod-service-grafana
         - infra-prod-service-graphite: *infra-prod-service-graphite
+        - infra-prod-service-keycloak: *infra-prod-service-keycloak
         - infra-prod-service-meetpad: *infra-prod-service-meetpad
         - infra-prod-service-lists: *infra-prod-service-lists
         - infra-prod-service-mirror: *infra-prod-service-mirror

zuul.d/system-config-run.yaml

@@ -683,6 +683,33 @@
       - playbooks/roles/pip3/
       - testinfra/test_graphite.py
 
+- job:
+    name: system-config-run-keycloak
+    parent: system-config-run
+    description: |
+      Run the playbook for the keycloak servers.
+    timeout: 3600
+    nodeset:
+      nodes:
+        - name: bridge.openstack.org
+          label: ubuntu-bionic
+        - name: keycloak01.opendev.org
+          label: ubuntu-focal
+    vars:
+      run_playbooks:
+        - playbooks/letsencrypt.yaml
+        - playbooks/service-keycloak.yaml
+    files:
+      - inventory/service/host_vars/keycloak01.opendev.org.yaml
+      - playbooks/install-ansible.yaml
+      - playbooks/letsencrypt.yaml
+      - playbooks/service-keycloak.yaml
+      - playbooks/roles/keycloak/
+      - playbooks/roles/install-docker/
+      - playbooks/roles/iptables/
+      - playbooks/zuul/templates/group_vars/keycloak.yaml.j2
+      - testinfra/test_keycloak.py
+
 - job:
     name: system-config-run-meetpad
     parent: system-config-run