Google Cloud / AppEngine debug logs

Learning my way through Google Cloud. (An aside, it’s nice! The docs are great. All the Google Cloud stuff works together like a coherent product. It’s way more approachable than the giant constellation of Amzon Web Services with all the little icons and no idea how they fit together.)

The Google Cloud Console has a nice logs viewer for your stuff running in production. But I was having a hard time finding my logs. There’s docs on stuff here, these are my own notes.

Here’s the stuff I’m finding from a Python app I’ve deployed in AppEngine. It uses the stock Python 3 logging module and configures logging.basicLogging with a level of DEBUG.

  • stderr: my app’s stderr. My log message are showing up here, just like they do when I run it locally, and I think it’s basically what the logging.basicConfig is doing. I can’t use the log level selector to filter, it’s either all or nothing. My app’s message are showing up like this:
    16:41:28.000 DEBUG:root:Connecting to database prod…
    Other stuff is showing up like this:
    16:37:28.000
    [2017-10-28 23:37:28 +0000] [1] [INFO] Starting gunicorn 19.7.1
    If I don’t configure logging at all in Python nothing shows up here. Ie: appengine doesn’t have some magic default log listener.
  • stdout: my app’s stdout. The way I write servers this should generally be empty.
  • nginx.request: basic HTTP request logs. Useful along with stdout/stderr to figure out what request triggered log entries
  • nginx.errors: errors from the nginx proxy. I’d thought this would be spam but I found a useful “upstream prematurely closed connection” error here that may be related to the bug I’m chasing (an appserver timeout).
  • nginx.health_check: spammy log of “is your server alive”. (Aside: I guess Google no longer uses /healthz, huh?)
  • activity: looks to be data related to deploys of the app.
  • vm.syslog: syslog from the machine? Mine is full of CRON and vm_runtime_init lines.
  • vm.events: this seems to contain summary stats on health checks

For debugging my own code, I think I want stderr, stdout, and nginx.request. That’s the default the console shows you and it’s a good one. The other stuff may be useful when debugging ops-level problems with the system or deployment.