Skip to content

Google Cloud

"Google Cloud SDK is a set of tools that you can use to manage resources and applications hosted on Google Cloud Platform. These include the gcloud, gsutil, and bq command line tools. The gcloud command-line tool is downloaded along with the Cloud SDK" - https://cloud.google.com/sdk/docs/

gcloud CLI Examples

Working with gcloud configurations

When working with several projects, it's best to use multiple configurations, with one for each project. Run gcloud init to create a new configuration. This interactive prompt will let you re-initialize the default, or create a new named config. Once you have done that, you can run gcloud config configurations list to show what you have configured. To activate a different configuration, run gcloud config configurations activate <new-config-name>.

$ gcloud config configurations list
NAME     IS_ACTIVE  ACCOUNT          PROJECT          COMPUTE_DEFAULT_ZONE  COMPUTE_DEFAULT_REGION
default  False      user@example.io  example-prod
staging  True       user@example.io  example-staging  us-west4-a            us-west4
$ gcloud config configurations activate default
Activated [default].

List google cloud projects

gcloud projects list

Switch to a different project

gcloud config set project "$project_name"

Grant a user permission to a docker registry

gsutil iam ch 'user:user@example.com:objectViewer' 'gs://artifacts.example.appspot.com/'

List google compute zones

gcloud compute zones list

List compute nodes

gcloud compute instances list

List al disks

gcloud compute disks list

Generate ssh commands for all nodes

gcloud compute instances list | awk 'NR>1 {printf "gcloud compute ssh --zone %s %s\n", $2, $1;}'

ssh to a compute node

This is useful for getting system level access to an EKS node.

ssh-to-gke-node() {
  [ "$#" -gt 0 ] || { echo "Usage: ssh-to-gke-node <node_name> [command]" ; return 1 ; }
  read -r zone host < <(gcloud compute instances list --filter="$1" | awk 'NR==2 {print $2, $1 ;}') ;
  shift
  gcloud compute ssh --tunnel-through-iap --zone "$zone" "$host" -- "${@}" ;
}

Loop through all gcloud instances and perform a command

gcloud compute instances list |
awk 'NR>1 {printf "gcloud compute ssh --zone %s %s\n", $2, $1;}' |
while read -r ssh_cmd ; do
  $ssh_cmd -- "docker images" </dev/null
done |
sort -u

Create a one-off compute node

gcloud compute instances create $USER-temp-node --zone=us-west4-a --network-interface=no-address

Leave off the --network-interface=no-address if you want a public IP address.

Delete a compute node

Sometimes autoscalers have a hard time scaling down, requiring manual termination of idle nodes. The following commands are equivalent:

gcloud compute instances delete "projects/$project_name/zones/$zone/instances/$node_name"
gcloud compute instances delete --project "$project_name" --zone="$zone" "$node_name"

To connect to VMs that don't have a public ip address, you need to give --tunnel-through-iap on the CLI and also have IAP-secured Tunnel User permission.

Add an EKS context to kubectl

https://cloud.google.com/sdk/gcloud/reference/container/clusters/get-credentials

This adds the cluster to your .kube/config with authentication done via an access token.

CLUSTER_NAME='foo-dev'
PROJECT_NAME='some-project'
REGION='us-west42'
gcloud container clusters get-credentials "${CLUSTER_NAME}" \
  --region "${REGION}" \
  --project "${PROJECT_NAME}"

Add Separate kubectl configs for different EKS clusters

This keeps each config in a different file, which is useful for requiring explicit enabling of a given environment, vs the normal behavior of inheriting the last used context.

# Set up individual kube config files for dev, prod and staging

KUBECONFIG="$HOME/.kube/foo-dev-config"
gcloud container clusters get-credentials dev-cluster --region vn-west4 --project foo-dev
kubectl config rename-context $(kubectl config current-context) foo-dev

KUBECONFIG="$HOME/.kube/foo-prod-config"
gcloud container clusters get-credentials prod-cluster --region vn-west4 --project foo-prod
kubectl config rename-context $(kubectl config current-context) foo-prod

KUBECONFIG="$HOME/.kube/foo-staging-config"
gcloud container clusters get-credentials staging-cluster --region vn-west4 --project foo-staging
kubectl config rename-context $(kubectl config current-context) foo-staging

Then setup aliases like

# ~/.bash_aliases
alias foo-k-dev="export KUBECONFIG=$HOME/.kube/foo-dev-config ; kubectl config set-context foo-dev --namespace=default ;"
alias foo-k-prod="export KUBECONFIG=$HOME/.kube/foo-prod-config ; kubectl config set-context foo-prod --namespace=default ;"
alias foo-k-stage="export KUBECONFIG=$HOME/.kube/foo-staging-config ; kubectl config set-context foo-staging --namespace=default ;"

List images available in Google Container Registry

gcloud container images list

Pull a docker container from Google Container Registry

gcloud docker -- pull gcr.io/project-id/hello-world

Control access to registries

"Container Registry uses a Cloud Storage bucket as the backend for serving container images. You can control who has access to your Container Registry images by adjusting permissions for the Cloud Storage bucket.

Caution: Container Registry only recognizes permissions set on the Cloud Storage bucket. Container Registry will ignore permissions set on individual objects within the Cloud Storage bucket.

You manage access control in Cloud Storage by using the GCP Console or the gsutil command-line tool. Refer to the gsutil acl and gsutil defacl documentation for more information." - https://cloud.google.com/container-registry/docs/access-control

Authenticate a private GCR registry in kubernetes

This is likely not copy/paste material, but the flow is generally correct.

PARTNER=other_company
PROJECT="our_company-$PARTNER"
USER=service-account-user-for-$PARTNER
EMAIL="$USER@$PROJECT.iam.gserviceaccount.com"
gcloud iam service-accounts create $USER
gcloud iam service-accounts keys create \
  --display-name "$USER" \
  --iam-account "$EMAIL" \
  key.json
gcloud projects add-iam-policy-binding "$PROJECT" \
  --member "serviceAccount:$EMAIL" \
  --role "roles/storage.objectAdmin"
done
kubectl create secret "docker-pull-$PROJECT" "$PROJECT" \
  --docker-server "https://gcr.io" \
  --docker-username _json_key \
  --docker-email "$EMAIL" \
  --docker-password "$(cat key.json)"

Then use the value of docker-pull-${PROJECT} as your ImagePullSecret.

Set cache expiration of GCP bucket items to 5 minutes

By default, GCP bucket items have 1 hour of public cache, which means items can be cached outside of the control of the GCP admin. This means that within that cache time window, any requests for the item will have unpredictable results. Set your Cache-Control max-age to something low for files that change, like page content and indexes, but long for files that never change, like images and archives.

gsutil setmeta -h "Cache-Control: public, max-age=300" gs://helm-repo.example.org/index.yaml

More info: https://medium.com/@val_deleplace/public-cache-dont-panic-11038a073a9

View the metadata of a GCP bucket item

gsutil stat gs://helm-repo.example.org/index.yaml

The output will be something like:

gs://helm-repo.example.org/index.yaml:
    Creation time:          Fri, 22 Jul 2020 23:20:11 GMT
    Update time:            Mon, 23 Jul 2020 19:17:53 GMT
    Storage class:          MULTI_REGIONAL
    Cache-Control:          public, max-age=300
    Content-Length:         3195714
    Content-Type:           application/octet-stream
    Hash (crc32c):          vA/Awm==
    Hash (md5):             2AJ32cECSriE0UQStsXxyw==
    ETag:                   COP7ew7D5+CAEoI=
    Generation:             1595460011829230
    Metageneration:         5

Set maintenance window on a GKE cluster

This syntax is a bit difficult to work with. It must be given in RFC-5545 (iCal), but GCP only supports a subset of the spec. Most frustratingly, GCP doesn't support timezones. They warn you about this, saying the "timezone will not be stored", so if you want to start something at Wednesday 03:00:00+0600 you have to specify that as Tuesday 21:00:00. 🙄

N.B.: in my testing while writing this, local TZ was used if not TZ was sent, so it looks like their docs may be wrong and maybe the conversion step can be avoided.

Convert local time to rfc-5545 UTC

So if you want to start your maintenance on Thursday at 5am Pacific time and end at Thursday 5pm Pacific time, the first step is to figure out what the rfc-5545 UTC format would be:

$ TZ=Z gdate '+%Y%m%dT%H%M%SZ %A' -d 2021-05-20T05:00:00-0800
20210520T130000Z Thursday
$ TZ=Z gdate '+%Y%m%dT%H%M%SZ %A' -d 2021-05-20T17:00:00-0800
20210521T010000Z Friday

Set and verify the maintenance schedule:

gcloud container clusters update --project="$PROJECT_NAME" --zone="$ZONE" "$CLUSTER_NAME" \
  --maintenance-window-start="20210520T130000Z" \
  --maintenance-window-end="20210521T010000Z" \
  --maintenance-window-recurrence="FREQ=WEEKLY;BYDAY=TH"

You should see Updating <cluster_name>...done. followed by some hyperlinks. Next you should validate that it was set right:

gcloud container clusters describe --format=json --project="$PROJECT_NAME" --zone="$ZONE" "$CLUSTER_NAME" |
jq .maintenancePolicy.window.recurringWindow

You should see a small json blob like:

{
  "recurrence": "FREQ=WEEKLY;BYDAY=TH",
  "window": {
    "endTime": "2021-05-21T02:00:00Z",
    "startTime": "2021-05-20T14:00:00Z"
  }
}

Grab the start time and feed it back into gdate to validate that your desired local time is set:

gdate -d "2021-05-20T14:00:00Z"  # comes out to: Thu May 20 07:00:00 PDT 2021

gcloud web console examples

Logs Explorer examples

Show namespace deletions

protoPayload.methodName="io.k8s.core.v1.namespaces.delete"

Show all images pulled by GKE

This will show pulls of all images except with substrings matching the references on line 3. The OR appears to be case sensitive.

resource.type="k8s_pod"
jsonPayload.reason="Pulling"
-jsonPayload.message : ( "Pulling image \"dade-murphy/leet-image\"" OR "Pulling image \"mr-the-plague/da-vinci\"" )

Show all k8s control plane upgrades

You can search using the same syntax at the CLI, which is great for local saving of search results so you can search them more quickly using grep and easily archive them for postmortem documentation.

gcloud beta logging read 'timestamp>="2021-01-01T00" AND protoPayload.metadata.operationType: UPGRADE_MASTER AND operation.producer: container.googleapis.com' > control-plane-upgrades.yaml