Ian Wienand 3052ff4935 kerberos-kdc: add database backups

Add a script to save a db dump to borg backups.  Add the primary KDC
to our backup list.

Change-Id: I32f4ebc1bb4c1952034aba43c75e4d2f85a1b6d3

2021-03-17 08:31:52 +11:00

5.8 KiB

Raw Blame History

title: Kerberos

Kerberos

Kerberos is a computer network authentication protocol which works on the basis of 'tickets' to allow nodes communicating over a non-secure network to prove their identity to one another in a secure manner. It is the basis for authentication to AFS.

At a Glance

Hosts

kdc*.openstack.org

Ansible

:git_file:`playbooks/service-kerberos.yaml`

Projects

http://web.mit.edu/kerberos

Bugs

Resources

OpenStack Realm

OpenStack runs a Kerberos Realm called OPENSTACK.ORG. The realm contains a Key Distribution Center or KDC which is spread across a primary and a replica, as well as an admin server which only runs on the primary.

Most of the configuration is in Ansible, but management of user accounts, known as principals, is a manual task for administrators.

Realm Creation

Realm creation is exercised by the Ansible roles during testing, but is not expected to be used in production (because we have an active realm/database).

The general process is:

create the new Kerberos database on the primary

distribute the database stash file from the primary to replicas, to allow them to unencrypt the database propogated to them. This is created from a master key kept as a secret.

create an admin user (password saved in file on primary server)

add host principals for the primary and replica servers

create keytabs on primary and replica servers (via the admin user), which allows them to authenticate to each other.

setup database propogation from primary to replicas with kprop (primary-side push) and kpropod (replica-side listen).

In a disaster recovery situation, we can provision a fresh realm and recover principals from backup dumps.

Adding A User Principal

First, ensure the user has an entry in Ansible so they have a Unix shell account on our hosts. SSH access is not necessary, but keeping track of usernames and uids with account entries is necessary.

If you are already an admin, you should authenicate with kinit <username>/admin. Otherwise you can use the kadmin.local tool (instead of kadmin) on the primary server, which by-passes authenication and writes to the database directly.

Use kadmin to add the principal like so:

kadmin: addprinc $USERNAME@OPENSTACK.ORG

Where $USERNAME is the lower-case username of their unix account in Ansible. OPENSTACK.ORG should be capitalized.

If you are adding an admin principal, use username/admin@OPENSTACK.ORG. Admins should additionally have regular user principals.

Adding A Service Principal

A service principal is one that corresponds to an application rather than a person. There is no difference in their implementation, only in conventions around how they are created and used. Service principals are created without passwords and keytab files are used instead for authentication. The program k5start can use keytab files to automatically obtain Kerberos credentials (and AFS if needed).

Add the service principal to Kerberos using kadmin (while authenticated as a Kerberos admin) or kadmin.local on the kdc:

kadmin: addprinc -randkey service/$NAME@OPENSTACK.ORG

Where $NAME is the lower-case name of the service. OPENSTACK.ORG should be capitalized.

Then save the principal's keytab:

kadmin: ktadd -k /path/to/$NAME.keytab service/$NAME@OPENSTACK.ORG

Warning

Each time ktadd is run, the key is rotated and previous keytabs are invalidated.

These keytabs are then usually converted to base-64 and stored as secret variables, and deployed to hosts via Ansible. mirror-update is probably a good example.

Resetting A User Principal's Password

If you've forgotten your user password (it happens!) then from a shell on one of the KDCs, execute sudo kadmin.local and use the cpw $USERNAME@OPENSTACK.ORG command and enter your new password twice as prompted. If you need to reset your admin principal, use cpw $USERNAME/admin@OPENSTACK.ORG instead.

No Service Outage Server Maintenance

Should you need perform maintenance on the Kerberos server that requires taking Kerberos processes offline you can do this by performing your updates on a single server at a time.

kdc03.openstack.org is our primary server and kdc04.openstack.org is the replica. Perform your maintenance on kdc04.openstack.org first. Then once that is done we can prepare for taking down the primary. On kdc03.openstack.org run:

root@kdc03:~# /usr/local/bin/run-kprop.sh

You should see:

Database propagation to kdc04.openstack.org: SUCCEEDED

Once this is done the replica is ready and we can take kdc03 offline. When kdc03 is back online rerun run-kprop.sh to ensure everything is working again.

DNS Entries

Kerberos uses the following DNS entries:

_kpasswd._udp.openstack.org.         300 IN SRV 0 0 464 kdc03.openstack.org.
_kerberos-adm._tcp.openstack.org.    300 IN SRV 0 0 749 kdc03.openstack.org.
_kerberos-master._udp.openstack.org. 300 IN SRV 0 0 88 kdc03.openstack.org.
_kerberos._udp.openstack.org.        300 IN SRV 0 0 88 kdc03.openstack.org.
_kerberos._udp.openstack.org.        300 IN SRV 0 0 88 kdc04.openstack.org.
_kerberos.openstack.org.             300 IN TXT "OPENSTACK.ORG"

Be sure to update them if kdc servers change. We also maintain a CNAME for convenience which points to the master kdc:

kdc.openstack.org. 300 IN CNAME kdc03.openstack.org.

5.8 KiB Raw Blame History