The GKE Upgrade That Took Down Our Production Pods for 45 Minutes

I want to start this blog with the story that made me realise I had been running Kubernetes on GKE without actually understanding how GKE runs Kubernetes. This is about a node pool upgrade that should have been routine and wasn't. We run three GKE Standard clusters on GCP. One for production, one fo