Support longer pkcs1-oaep secrets

We have run into a case where we need to store a secret longer
than 3760 bits.  We may eventually support a hybrid encryption
scheme, but for now, let's also support the alt.zuul.secrets protocol
where we split the secret into 3760 bit chunks and recombine it.

The encrypt_secret utility is updated to output a copy-pastable
YAML data structure to simplify dealing with long secrets.

Change-Id: Ied372572e5aa29fddfb7043bf07df4cd3e39566c

doc/source/user/encryption.rst

@@ -20,24 +20,41 @@ project and ``<source>`` is the name of that project's connection in
 the main Zuul configuration file.
 
 Zuul currently supports one encryption scheme, PKCS#1 with OAEP, which
-can not store secrets longer than the key length, 4096 bits.  The
-padding used by this scheme ensures that someone examining the
-encrypted data can not determine the length of the plaintext version
-of the data, except to know that it is not longer than 4096 bits.
+can not store secrets longer than the 3760 bits (derived from the key
+length of 4096 bits minus 336 bits of overhead).  The padding used by
+this scheme ensures that someone examining the encrypted data can not
+determine the length of the plaintext version of the data, except to
+know that it is not longer than 3760 bits (or some multiple thereof).
 
 In the config files themselves, Zuul uses an extensible method of
 specifying the encryption scheme used for a secret so that other
 schemes may be added later.  To specify a secret, use the
 ``!encrypted/pkcs1-oaep`` YAML tag along with the base64 encoded
-value.  For example::
+value.  For example:
+
+.. code-block:: yaml
 
   - secret:
       name: test_secret
       data:
         password: !encrypted/pkcs1-oaep |
-          BFhtdnm8uXx7kn79RFL/zJywmzLkT1GY78P3bOtp4WghUFWobkifSu7ZpaV4NeO0s71YUsi1wGZZ
+          BFhtdnm8uXx7kn79RFL/zJywmzLkT1GY78P3bOtp4WghUFWobkifSu7ZpaV4NeO0s71YUsi
           ...
 
+To support secrets longer than 3760 bits, the value after the
+encryption tag may be a list rather than a scalar.  For example:
+
+.. code-block:: yaml
+
+  - secret:
+      name: long_secret
+      data:
+        password: !encrypted/pkcs1-oaep
+          - er1UXNOD3OqtsRJaP0Wvaqiqx0ZY2zzRt6V9vqIsRaz1R5C4/AEtIad/DERZHwk3Nk+KV
+            ...
+          - HdWDS9lCBaBJnhMsm/O9tpzCq+GKRELpRzUwVgU5k822uBwhZemeSrUOLQ8hQ7q/vVHln
+            ...
+
 Zuul provides a standalone script to make encrypting values easy; it
 can be found at `tools/encrypt_secret.py` in the Zuul source
 directory.

tests/unit/test_model.py

@@ -342,16 +342,41 @@ class TestJob(BaseTestCase):
     name: pypi-credentials
     data:
       username: test-username
+      longpassword: !encrypted/pkcs1-oaep
+        - BFhtdnm8uXx7kn79RFL/zJywmzLkT1GY78P3bOtp4WghUFWobkifSu7ZpaV4NeO0s71Y
+          Usi1wGZZL0LveZjUN0t6OU1VZKSG8R5Ly7urjaSo1pPVIq5Rtt/H7W14Lecd+cUeKb4j
+          oeusC9drN3AA8a4oykcVpt1wVqUnTbMGC9ARMCQP6eopcs1l7tzMseprW4RDNhIuz3CR
+          gd0QBMPl6VDoFgBPB8vxtJw+3m0rqBYZCLZgCXekqlny8s2s92nJMuUABbJOEcDRarzi
+          bDsSXsfJt1y+5n7yOURsC7lovMg4GF/vCl/0YMKjBO5bpv9EM5fToeKYyPGSKQoHOnCY
+          ceb3cAVcv5UawcCic8XjhEhp4K7WPdYf2HVAC/qtxhbpjTxG4U5Q/SoppOJ60WqEkQvb
+          Xs6n5Dvy7xmph6GWmU/bAv3eUK3pdD3xa2Ue1lHWz3U+rsYraI+AKYsMYx3RBlfAmCeC
+          1ve2BXPrqnOo7G8tnUvfdYPbK4Aakk0ds/AVqFHEZN+S6hRBmBjLaRFWZ3QSO1NjbBxW
+          naHKZYT7nkrJm8AMCgZU0ZArFLpaufKCeiK5ECSsDxic4FIsY1OkWT42qEUfL0Wd+150
+          AKGNZpPJnnP3QYY4W/MWcKH/zdO400+zWN52WevbSqZy90tqKDJrBkMl1ydqbuw1E4ZH
+          vIs=
+        - BFhtdnm8uXx7kn79RFL/zJywmzLkT1GY78P3bOtp4WghUFWobkifSu7ZpaV4NeO0s71Y
+          Usi1wGZZL0LveZjUN0t6OU1VZKSG8R5Ly7urjaSo1pPVIq5Rtt/H7W14Lecd+cUeKb4j
+          oeusC9drN3AA8a4oykcVpt1wVqUnTbMGC9ARMCQP6eopcs1l7tzMseprW4RDNhIuz3CR
+          gd0QBMPl6VDoFgBPB8vxtJw+3m0rqBYZCLZgCXekqlny8s2s92nJMuUABbJOEcDRarzi
+          bDsSXsfJt1y+5n7yOURsC7lovMg4GF/vCl/0YMKjBO5bpv9EM5fToeKYyPGSKQoHOnCY
+          ceb3cAVcv5UawcCic8XjhEhp4K7WPdYf2HVAC/qtxhbpjTxG4U5Q/SoppOJ60WqEkQvb
+          Xs6n5Dvy7xmph6GWmU/bAv3eUK3pdD3xa2Ue1lHWz3U+rsYraI+AKYsMYx3RBlfAmCeC
+          1ve2BXPrqnOo7G8tnUvfdYPbK4Aakk0ds/AVqFHEZN+S6hRBmBjLaRFWZ3QSO1NjbBxW
+          naHKZYT7nkrJm8AMCgZU0ZArFLpaufKCeiK5ECSsDxic4FIsY1OkWT42qEUfL0Wd+150
+          AKGNZpPJnnP3QYY4W/MWcKH/zdO400+zWN52WevbSqZy90tqKDJrBkMl1ydqbuw1E4ZH
+          vIs=
       password: !encrypted/pkcs1-oaep |
-        BFhtdnm8uXx7kn79RFL/zJywmzLkT1GY78P3bOtp4WghUFWobkifSu7ZpaV4NeO0s71YUsi1wGZZ
-        L0LveZjUN0t6OU1VZKSG8R5Ly7urjaSo1pPVIq5Rtt/H7W14Lecd+cUeKb4joeusC9drN3AA8a4o
-        ykcVpt1wVqUnTbMGC9ARMCQP6eopcs1l7tzMseprW4RDNhIuz3CRgd0QBMPl6VDoFgBPB8vxtJw+
-        3m0rqBYZCLZgCXekqlny8s2s92nJMuUABbJOEcDRarzibDsSXsfJt1y+5n7yOURsC7lovMg4GF/v
-        Cl/0YMKjBO5bpv9EM5fToeKYyPGSKQoHOnCYceb3cAVcv5UawcCic8XjhEhp4K7WPdYf2HVAC/qt
-        xhbpjTxG4U5Q/SoppOJ60WqEkQvbXs6n5Dvy7xmph6GWmU/bAv3eUK3pdD3xa2Ue1lHWz3U+rsYr
-        aI+AKYsMYx3RBlfAmCeC1ve2BXPrqnOo7G8tnUvfdYPbK4Aakk0ds/AVqFHEZN+S6hRBmBjLaRFW
-        Z3QSO1NjbBxWnaHKZYT7nkrJm8AMCgZU0ZArFLpaufKCeiK5ECSsDxic4FIsY1OkWT42qEUfL0Wd
-        +150AKGNZpPJnnP3QYY4W/MWcKH/zdO400+zWN52WevbSqZy90tqKDJrBkMl1ydqbuw1E4ZHvIs=
+        BFhtdnm8uXx7kn79RFL/zJywmzLkT1GY78P3bOtp4WghUFWobkifSu7ZpaV4NeO0s71Y
+        Usi1wGZZL0LveZjUN0t6OU1VZKSG8R5Ly7urjaSo1pPVIq5Rtt/H7W14Lecd+cUeKb4j
+        oeusC9drN3AA8a4oykcVpt1wVqUnTbMGC9ARMCQP6eopcs1l7tzMseprW4RDNhIuz3CR
+        gd0QBMPl6VDoFgBPB8vxtJw+3m0rqBYZCLZgCXekqlny8s2s92nJMuUABbJOEcDRarzi
+        bDsSXsfJt1y+5n7yOURsC7lovMg4GF/vCl/0YMKjBO5bpv9EM5fToeKYyPGSKQoHOnCY
+        ceb3cAVcv5UawcCic8XjhEhp4K7WPdYf2HVAC/qtxhbpjTxG4U5Q/SoppOJ60WqEkQvb
+        Xs6n5Dvy7xmph6GWmU/bAv3eUK3pdD3xa2Ue1lHWz3U+rsYraI+AKYsMYx3RBlfAmCeC
+        1ve2BXPrqnOo7G8tnUvfdYPbK4Aakk0ds/AVqFHEZN+S6hRBmBjLaRFWZ3QSO1NjbBxW
+        naHKZYT7nkrJm8AMCgZU0ZArFLpaufKCeiK5ECSsDxic4FIsY1OkWT42qEUfL0Wd+150
+        AKGNZpPJnnP3QYY4W/MWcKH/zdO400+zWN52WevbSqZy90tqKDJrBkMl1ydqbuw1E4ZH
+        vIs=
 ''')[0]['secret']
 
         conf['_source_context'] = self.context
@@ -441,6 +466,12 @@ class TestJob(BaseTestCase):
         self.assertEqual(in_repo_job_with_inherit.auth.secrets[0].name,
                          'pypi-credentials')
         self.assertIsNone(in_repo_job_with_inherit_false.auth)
+        self.assertEqual(in_repo_job_with_inherit.auth.secrets[0].
+                         secret_data['longpassword'],
+                         'test-passwordtest-password')
+        self.assertEqual(in_repo_job_with_inherit.auth.secrets[0].
+                         secret_data['password'],
+                         'test-password')
 
     def test_job_inheritance_job_tree(self):
         tenant = model.Tenant('tenant')

tools/encrypt_secret.py

@@ -14,10 +14,13 @@
 
 import argparse
 import base64
+import math
 import os
+import re
 import subprocess
 import sys
 import tempfile
+import textwrap
 
 # we to import Request and urlopen differently for python 2 and 3
 try:
@@ -68,28 +71,70 @@ def main():
     else:
         plaintext = sys.stdin.read()
 
+    plaintext = plaintext.encode("utf-8")
+
     pubkey_file = tempfile.NamedTemporaryFile(delete=False)
     try:
         pubkey_file.write(pubkey.read())
         pubkey_file.close()
 
-        p = subprocess.Popen(['openssl', 'rsautl', '-encrypt',
-                              '-oaep', '-pubin', '-inkey',
+        p = subprocess.Popen(['openssl', 'rsa', '-text',
+                              '-pubin', '-in',
                               pubkey_file.name],
-                             stdin=subprocess.PIPE,
                              stdout=subprocess.PIPE)
-        (stdout, stderr) = p.communicate(plaintext.encode("utf-8"))
+        (stdout, stderr) = p.communicate()
         if p.returncode != 0:
             raise Exception("Return code %s from openssl" % p.returncode)
-        ciphertext = base64.b64encode(stdout)
+        output = stdout.decode('utf-8')
+        m = re.match(r'^Public-Key: \((\d+) bit\)$', output, re.MULTILINE)
+        nbits = int(m.group(1))
+        nbytes = int(nbits / 8)
+        max_bytes = nbytes - 42  # PKCS1-OAEP overhead
+        chunks = int(math.ceil(float(len(plaintext)) / max_bytes))
+
+        ciphertext_chunks = []
+
+        print("Public key length: {} bits ({} bytes)".format(nbits, nbytes))
+        print("Max plaintext length per chunk: {} bytes".format(max_bytes))
+        print("Input plaintext length: {} bytes".format(len(plaintext)))
+        print("Number of chunks: {}".format(chunks))
+
+        for count in range(chunks):
+            chunk = plaintext[int(count * max_bytes):
+                              int((count + 1) * max_bytes)]
+            p = subprocess.Popen(['openssl', 'rsautl', '-encrypt',
+                                  '-oaep', '-pubin', '-inkey',
+                                  pubkey_file.name],
+                                 stdin=subprocess.PIPE,
+                                 stdout=subprocess.PIPE)
+            (stdout, stderr) = p.communicate(chunk)
+            if p.returncode != 0:
+                raise Exception("Return code %s from openssl" % p.returncode)
+            ciphertext_chunks.append(base64.b64encode(stdout).decode('utf-8'))
+
     finally:
         os.unlink(pubkey_file.name)
 
+    output = textwrap.dedent(
+        '''
+        - secret:
+            name: <name>
+            data:
+              <fieldname>: !encrypted/pkcs1-oaep
+        ''')
+
+    twrap = textwrap.TextWrapper(width=79,
+                                 initial_indent=' ' * 8,
+                                 subsequent_indent=' ' * 10)
+    for chunk in ciphertext_chunks:
+        chunk = twrap.fill('- ' + chunk)
+        output += chunk + '\n'
+
     if args.outfile:
-        with open(args.outfile, "wb") as f:
-            f.write(ciphertext)
+        with open(args.outfile, "w") as f:
+            f.write(output)
     else:
-        print(ciphertext.decode("utf-8"))
+        print(output)
 
 
 if __name__ == '__main__':

zuul/configloader.py

@@ -223,7 +223,11 @@ class EncryptedPKCS1_OAEP(yaml.YAMLObject):
     yaml_loader = yaml.SafeLoader
 
     def __init__(self, ciphertext):
-        self.ciphertext = base64.b64decode(ciphertext)
+        if isinstance(ciphertext, list):
+            self.ciphertext = [base64.b64decode(x.value)
+                               for x in ciphertext]
+        else:
+            self.ciphertext = base64.b64decode(ciphertext)
 
     def __ne__(self, other):
         return not self.__eq__(other)
@@ -238,8 +242,14 @@ class EncryptedPKCS1_OAEP(yaml.YAMLObject):
         return cls(node.value)
 
     def decrypt(self, private_key):
-        return encryption.decrypt_pkcs1_oaep(self.ciphertext,
-                                             private_key).decode('utf8')
+        if isinstance(self.ciphertext, list):
+            return ''.join([
+                encryption.decrypt_pkcs1_oaep(chunk, private_key).
+                decode('utf8')
+                for chunk in self.ciphertext])
+        else:
+            return encryption.decrypt_pkcs1_oaep(self.ciphertext,
+                                                 private_key).decode('utf8')
 
 
 class NodeSetParser(object):