Unable to see the accounts listed when using the App Engine plugin in a pipeline

Issue

Service accounts do not show up in the server group. The drop down does not list the account, as seen in the following:

The Clouddriver logs may not show any access errors.  The following errors may be seen in the Clouddriver logs upon restarting the Clouddriver pod: .s.c.a.s.AppengineCredentialsInitializer : Could not load account custhub-preprod-cc6a6465-aes for App Engine

com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden

{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "The caller does not have permission",
    "reason" : "forbidden"
  } ],
  "message" : "The caller does not have permission",
  "status" : "PERMISSION_DENIED"
}

Cause

The 403 errors may be lost in the logs until the Clouddriver pod is restarted. This is because an AppEngine account with no access doesn’t stop Clouddriver from starting unlike other accounts that are set up in Clouddriver (e.g. AWS, Kubernetes, etc). AppEngine accounts are validated only on startup. If the account doesn’t have access, then Clouddriver discards it and doesn’t try to validate again, and so, the errors will only show upon restart.