Merge "github: fallback to api_token when can't find installation"

doc/source/drivers/github.rst

@@ -12,9 +12,56 @@ installations of GitHub enterprise.
 Configure GitHub
 ----------------
 
-There are two options currently available. GitHub's project owner can either
-manually setup web-hook or install a GitHub Application. In the first case,
-the project's owner needs to know the zuul endpoint and the webhook secrets.
+Zuul needs to receive notification of events from GitHub, and it needs
+to interact with GitHub in response to those events.  There are two
+options aviable for configuring these connections.  A GitHub project's
+owner can either manually setup a web-hook or install a GitHub
+Application.  In the first case, the project's owner needs to know
+the Zuul endpoint and the webhook secrets and configure them manually.
+In the second, the project (or organization) owner need only install
+pre-existing GitHub Application into the project or organization.
+
+In general, the Application method is recommended.  Both options are
+described in the following sections.
+
+Regardless of which option is chosen, there are several types of
+authentication between Zuul and GitHub used for various purposes. Some
+are required and some are optional depending on the intended use and
+configuration.
+
+In all cases Zuul needs to authenticate messages received from GitHub
+as being valid. To do this a `webhook_token` is configured.
+
+Zuul also needs to authenticate to GitHub to make certain requests. At
+a high level, this is the sort of authentication that is required for
+various Zuul Github functionality:
+
+  * Reporting: Requires authentication with write access to the project so
+    that comments can be posted.
+  * Enqueing a pull request (including Depends-On: of a pull request): The
+    API queries needed to examine PR's so that Zuul can enqueue them
+    requires authentication with read access.
+  * :attr:`job.required-projects` listing: No authentication required.
+    For example, you may have a project where you are only interested in
+    testing against a specific branch of a GitHub project. In this case you
+    do not need any authentication to have Zuul pull the project.
+    However, note that if you will ever need to speculatively test a PR in
+    this project, you will require authenticated read access (see note above).
+
+There are two different ways Zuul can Authenticate its requests to
+GitHub. The first is the `api_token`. This `api_token` is used by the
+web-hook option for all authentication. When using the GitHub
+Application system Zuul uses an `app_id` and `app_key` which is
+used to generate an application token behind the scenes. But this only
+works against projects that have installed your application. As a
+fallback for interaction with projects that haven't installed your
+application you can also configure an `api_token` when using the
+application system. This is particularly useful for supporting
+Depends-On functionality against GitHub projects.
+
+Finally, authenticated requests receive much larger GitHub API rate limits.
+It is worthwhile to configure both an `app_id`/`app_key` and `api_token`
+when operating in application mode to avoid rate limits as much as possible.
 
 
 Web-Hook
@@ -80,10 +127,14 @@ To create a `GitHub application
 * Set Where can this GitHub App be installed to "Any account"
 * Create the App
 * Generate a Private key in the app settings page
+* Optionally configure an api_token. Please see this `article
+  <https://help.github.com/articles/creating-an-access-token-for-command-line-use/>`_
+  for more information.
 
-Then in the zuul.conf, set webhook_token, app_id and app_key.
-After restarting zuul-scheduler, verify in the 'Advanced' tab that the
-Ping payload works (green tick and 200 response)
+Then in the zuul.conf, set `webhook_token`, `app_id`, `app_key` and
+optionally `api_token`.  After restarting zuul-scheduler, verify in
+the 'Advanced' tab that the Ping payload works (green tick and 200
+response)
 
 Users can now install the application using its public page, e.g.:
 https://github.com/apps/my-org-zuul
@@ -105,9 +156,9 @@ Connection Configuration
 There are two forms of operation. Either the Zuul installation can be
 configured as a `Github App`_ or it can be configured as a Webhook.
 
-If the `Github App`_ approach is taken, the config settings ``app_id`` and
-``app_key`` are required. If the Webhook approach is taken, the ``api_token``
-setting is required.
+If the `Github App`_ approach is taken, the config settings
+``app_id``, ``app_key`` and optionally ``api_token`` are required. If
+the Webhook approach is taken, the ``api_token`` setting is required.
 
 The supported options in ``zuul.conf`` connections are:
 

releasenotes/notes/github-api-token-6070169c0d7dd1e2.yaml

@@ -0,0 +1,7 @@
+---
+upgrade:
+  - |
+    If using the GitHub driver in app mode, you should consider
+    also generating and adding an ``api_token``.  The option is not
+    strictly required but will increase rate limits and add
+    functionality for projects the app is not installed in.

tests/fakegithub.py

@@ -822,6 +822,9 @@ class FakeGithubClient(object):
     def repository(self, owner, proj):
         return self._data.repos.get((owner, proj), None)
 
+    def login(self, token):
+        pass
+
     def repo_from_project(self, project):
         # This is a convenience method for the tests.
         owner, proj = project.split('/')

tests/fixtures/zuul-github-driver.conf

@@ -15,15 +15,18 @@ driver=github
 webhook_token=0000000000000000000000000000000000000000
 app_id=1
 app_key=$APP_KEY_FIXTURE$
+api_token=ghp_51abcFzcvf3GxOJpPFUKxsT6JIL3Nnbf39E
 
 [connection github_ssh]
 driver=github
 sshkey=/home/zuul/.ssh/id_rsa
+api_token=ghp_51abcFzcvf3GxOJpPFUKxsT6JIL3Nnbf39E
 
 [connection github_ent]
 driver=github
 sshkey=/home/zuul/.ssh/id_rsa
 server=github.enterprise.io
+api_token=ghp_51abcFzcvf3GxOJpPFUKxsT6JIL3Nnbf39E
 
 [database]
 dburi=$MYSQL_FIXTURE_DBURI$

zuul/driver/github/githubconnection.py

@@ -1160,8 +1160,12 @@ class GithubClientManager:
                                                  inst_id=inst_id,
                                                  reprime=False)
 
-            self.log.error("No installation ID available for project %s",
-                           project_name)
+            if self.connection_config.get('api_token'):
+                log_severity = self.log.info
+            else:
+                log_severity = self.log.error
+            log_severity("No installation ID available for project %s",
+                         project_name)
             return ''
 
         # Consider tokens outdated 5min before the actual expiry time
@@ -1297,6 +1301,7 @@ class GithubClientManager:
                         project_name=None,
                         zuul_event_id=None):
         github = self._createGithubClient(zuul_event_id)
+        token = ''
 
         # if you're authenticating for a project and you're an integration then
         # you need to use the installation specific token.
@@ -1305,10 +1310,8 @@ class GithubClientManager:
             # case it's expired.
             token = self.get_installation_key(project_name)
 
-            # Only set the auth header if we have a token. If not, just don't
-            # set any auth header so we will be treated as anonymous. That's
-            # also what the github.login() method would do if the token is not
-            # set.
+            # Only set the auth header if we have a token. If not,
+            # falls back to the api_token specified below
             if token:
                 # To set the AppInstallationAuthToken on the github session, we
                 # also need the expiry date, but in the correct ISO format.
@@ -1329,16 +1332,18 @@ class GithubClientManager:
                 github.session.auth = AppInstallationTokenAuth(
                     token, format_expiry
                 )
+                github._zuul_project = project_name
+                github._zuul_user_id = self.installation_map.get(project_name)
 
-            github._zuul_project = project_name
-            github._zuul_user_id = self.installation_map.get(project_name)
-
-        # if we're using api_token authentication then use the provided token,
-        # else anonymous is the best we have.
-        else:
+        # If we have an api_token then we may be using webhooks or are in an
+        # application setup where the application isn't applied to the project.
+        if not token:
+            # Fall back to using the API token
             api_token = self.connection_config.get('api_token')
             if api_token:
                 github.login(token=api_token)
+            # If we have no API token we fallback further to anonymous access
+            # which is limited but the best we can do for now.
 
         return github