github: fallback to api_token when can't find installation

graphql queries (I77be4f16cf7eb5c8035ce0312f792f4e8d4c3e10) require
authentication. Enqueueing changes from GitHub (including Depends-On)
requires we run a graphql query. This means that Zuul must be able to
authenticate either via an application or api_token to support features
like Depends-On.

If the app is setup (app_id in config) but we aren't installed with
permissions on the project we're looking up, then fall back to using a
specified api_token. This will make Depends-On work.

Logging is updated to reflect whether or not we are able to fallback to
the api_token if the application is not installed. We log the lack of an
application installation at info level if we can fallback to the token,
and log at error level if we're falling back to anonymous access.

For backward compatibility we continue to fallback to anonymous access
if neither an application install or api_token are present. The reason
for this is features like Job required-projects: work fine anonymously,
and there may be Zuul installations that don't need additional
functionality.

Keep in mind that authenticated requests to GitHub get larger API rate
limits. Zuul installations should consider setting an API token even
when using an application for this reason. This gives Zuul the best
chance that fallback requests will not be rate limited.

Documentation is updated, a changelog added and several test
configuration files are padded with the required info.

Story: #2008940
Change-Id: I2107aeafc55591eea790244701567569fa6e80d4

doc/source/drivers/github.rst

@@ -12,9 +12,56 @@ installations of GitHub enterprise.
 Configure GitHub
 ----------------
 
-There are two options currently available. GitHub's project owner can either
-manually setup web-hook or install a GitHub Application. In the first case,
-the project's owner needs to know the zuul endpoint and the webhook secrets.
+Zuul needs to receive notification of events from GitHub, and it needs
+to interact with GitHub in response to those events.  There are two
+options aviable for configuring these connections.  A GitHub project's
+owner can either manually setup a web-hook or install a GitHub
+Application.  In the first case, the project's owner needs to know
+the Zuul endpoint and the webhook secrets and configure them manually.
+In the second, the project (or organization) owner need only install
+pre-existing GitHub Application into the project or organization.
+
+In general, the Application method is recommended.  Both options are
+described in the following sections.
+
+Regardless of which option is chosen, there are several types of
+authentication between Zuul and GitHub used for various purposes. Some
+are required and some are optional depending on the intended use and
+configuration.
+
+In all cases Zuul needs to authenticate messages received from GitHub
+as being valid. To do this a `webhook_token` is configured.
+
+Zuul also needs to authenticate to GitHub to make certain requests. At
+a high level, this is the sort of authentication that is required for
+various Zuul Github functionality:
+
+  * Reporting: Requires authentication with write access to the project so
+    that comments can be posted.
+  * Enqueing a pull request (including Depends-On: of a pull request): The
+    API queries needed to examine PR's so that Zuul can enqueue them
+    requires authentication with read access.
+  * :attr:`job.required-projects` listing: No authentication required.
+    For example, you may have a project where you are only interested in
+    testing against a specific branch of a GitHub project. In this case you
+    do not need any authentication to have Zuul pull the project.
+    However, note that if you will ever need to speculatively test a PR in
+    this project, you will require authenticated read access (see note above).
+
+There are two different ways Zuul can Authenticate its requests to
+GitHub. The first is the `api_token`. This `api_token` is used by the
+web-hook option for all authentication. When using the GitHub
+Application system Zuul uses an `app_id` and `app_key` which is
+used to generate an application token behind the scenes. But this only
+works against projects that have installed your application. As a
+fallback for interaction with projects that haven't installed your
+application you can also configure an `api_token` when using the
+application system. This is particularly useful for supporting
+Depends-On functionality against GitHub projects.
+
+Finally, authenticated requests receive much larger GitHub API rate limits.
+It is worthwhile to configure both an `app_id`/`app_key` and `api_token`
+when operating in application mode to avoid rate limits as much as possible.
 
 
 Web-Hook
@@ -80,10 +127,14 @@ To create a `GitHub application
 * Set Where can this GitHub App be installed to "Any account"
 * Create the App
 * Generate a Private key in the app settings page
+* Optionally configure an api_token. Please see this `article
+  <https://help.github.com/articles/creating-an-access-token-for-command-line-use/>`_
+  for more information.
 
-Then in the zuul.conf, set webhook_token, app_id and app_key.
-After restarting zuul-scheduler, verify in the 'Advanced' tab that the
-Ping payload works (green tick and 200 response)
+Then in the zuul.conf, set `webhook_token`, `app_id`, `app_key` and
+optionally `api_token`.  After restarting zuul-scheduler, verify in
+the 'Advanced' tab that the Ping payload works (green tick and 200
+response)
 
 Users can now install the application using its public page, e.g.:
 https://github.com/apps/my-org-zuul
@@ -105,9 +156,9 @@ Connection Configuration
 There are two forms of operation. Either the Zuul installation can be
 configured as a `Github App`_ or it can be configured as a Webhook.
 
-If the `Github App`_ approach is taken, the config settings ``app_id`` and
-``app_key`` are required. If the Webhook approach is taken, the ``api_token``
-setting is required.
+If the `Github App`_ approach is taken, the config settings
+``app_id``, ``app_key`` and optionally ``api_token`` are required. If
+the Webhook approach is taken, the ``api_token`` setting is required.
 
 The supported options in ``zuul.conf`` connections are:
 

releasenotes/notes/github-api-token-6070169c0d7dd1e2.yaml

@@ -0,0 +1,7 @@
+---
+upgrade:
+  - |
+    If using the GitHub driver in app mode, you should consider
+    also generating and adding an ``api_token``.  The option is not
+    strictly required but will increase rate limits and add
+    functionality for projects the app is not installed in.

tests/fakegithub.py

@@ -822,6 +822,9 @@ class FakeGithubClient(object):
     def repository(self, owner, proj):
         return self._data.repos.get((owner, proj), None)
 
+    def login(self, token):
+        pass
+
     def repo_from_project(self, project):
         # This is a convenience method for the tests.
         owner, proj = project.split('/')

tests/fixtures/zuul-github-driver.conf

@@ -15,15 +15,18 @@ driver=github
 webhook_token=0000000000000000000000000000000000000000
 app_id=1
 app_key=$APP_KEY_FIXTURE$
+api_token=ghp_51abcFzcvf3GxOJpPFUKxsT6JIL3Nnbf39E
 
 [connection github_ssh]
 driver=github
 sshkey=/home/zuul/.ssh/id_rsa
+api_token=ghp_51abcFzcvf3GxOJpPFUKxsT6JIL3Nnbf39E
 
 [connection github_ent]
 driver=github
 sshkey=/home/zuul/.ssh/id_rsa
 server=github.enterprise.io
+api_token=ghp_51abcFzcvf3GxOJpPFUKxsT6JIL3Nnbf39E
 
 [database]
 dburi=$MYSQL_FIXTURE_DBURI$

zuul/driver/github/githubconnection.py

@@ -1160,8 +1160,12 @@ class GithubClientManager:
                                                  inst_id=inst_id,
                                                  reprime=False)
 
-            self.log.error("No installation ID available for project %s",
-                           project_name)
+            if self.connection_config.get('api_token'):
+                log_severity = self.log.info
+            else:
+                log_severity = self.log.error
+            log_severity("No installation ID available for project %s",
+                         project_name)
             return ''
 
         # Consider tokens outdated 5min before the actual expiry time
@@ -1297,6 +1301,7 @@ class GithubClientManager:
                         project_name=None,
                         zuul_event_id=None):
         github = self._createGithubClient(zuul_event_id)
+        token = ''
 
         # if you're authenticating for a project and you're an integration then
         # you need to use the installation specific token.
@@ -1305,10 +1310,8 @@ class GithubClientManager:
             # case it's expired.
             token = self.get_installation_key(project_name)
 
-            # Only set the auth header if we have a token. If not, just don't
-            # set any auth header so we will be treated as anonymous. That's
-            # also what the github.login() method would do if the token is not
-            # set.
+            # Only set the auth header if we have a token. If not,
+            # falls back to the api_token specified below
             if token:
                 # To set the AppInstallationAuthToken on the github session, we
                 # also need the expiry date, but in the correct ISO format.
@@ -1329,16 +1332,18 @@ class GithubClientManager:
                 github.session.auth = AppInstallationTokenAuth(
                     token, format_expiry
                 )
+                github._zuul_project = project_name
+                github._zuul_user_id = self.installation_map.get(project_name)
 
-            github._zuul_project = project_name
-            github._zuul_user_id = self.installation_map.get(project_name)
-
-        # if we're using api_token authentication then use the provided token,
-        # else anonymous is the best we have.
-        else:
+        # If we have an api_token then we may be using webhooks or are in an
+        # application setup where the application isn't applied to the project.
+        if not token:
+            # Fall back to using the API token
             api_token = self.connection_config.get('api_token')
             if api_token:
                 github.login(token=api_token)
+            # If we have no API token we fallback further to anonymous access
+            # which is limited but the best we can do for now.
 
         return github