Google Cloud
"Google Cloud SDK is a set of tools that you can use to manage resources and applications hosted on Google Cloud Platform. These include the gcloud, gsutil, and bq command line tools. The gcloud command-line tool is downloaded along with the Cloud SDK" - https://cloud.google.com/sdk/docs/
Links
- https://cloud.google.com/compute/docs/ssh-in-browser
- https://cloud.google.com/container-builder/docs/build-config
- https://cloud.google.com/container-builder/docs/create-custom-build-steps
- https://cloud.google.com/container-registry/docs/quickstart
- https://cloud.google.com/docs
- https://cloud.google.com/sdk/gcloud/reference
- https://github.com/GoogleCloudPlatform/cloud-builders
- https://cloud.google.com/iam/docs/permissions-reference (Large download)
- https://cloud.google.com/compute/docs/instances/creating-instance-with-custom-machine-type
- https://cloud.google.com/container-optimized-os/docs/how-to/toolbox
gcloud CLI Examples
Working with gcloud configurations
When working with several projects, it's best to use multiple configurations, with one for each project. Run gcloud init to create a new configuration. This interactive prompt will let you re-initialize the default, or create a new named config. Once you have done that, you can run gcloud config configurations list to show what you have configured. To activate a different configuration, run gcloud config configurations activate <new-config-name>.
$ gcloud config configurations list
NAME IS_ACTIVE ACCOUNT PROJECT COMPUTE_DEFAULT_ZONE COMPUTE_DEFAULT_REGION
default False user@example.io example-prod
staging True user@example.io example-staging us-west4-a us-west4
$ gcloud config configurations activate default
Activated [default].
Alternately, you can specify the config you want to use as an environment variable.
$ gcloud config configurations list
NAME IS_ACTIVE ACCOUNT PROJECT COMPUTE_DEFAULT_ZONE COMPUTE_DEFAULT_REGION
dev False danielh@example.com example-dev-369821
staging True danielh@example.com example-staging-550891 us-east4-c us-east4
$ CLOUDSDK_ACTIVE_CONFIG_NAME=dev gcloud compute instances list
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
cat-pic-download-1 us-central1-a e2-medium 10.128.0.2 TERMINATED
gibson-access us-east4-a e2-micro 10.0.0.69 256.24.927.306 RUNNING
gke-dev-cluster-terraform-20090525165-d1058ae2-c0mc us-east4-a n1-standard-16 10.0.0.13 RUNNING
gke-dev-cluster-terraform-20210624231-9d581566-1oo9 us-east4-a n1-standard-8 10.0.0.56 RUNNING
Launch cloud shell from CLI
Google Cloud Shell is a free VM in GCP that you can use to access GCP cli tools, or do whatever else you might want to do in a free VM. The home directory is kept for like 6 months or something, so configuring your shell and storing useful scripts isn't a bad idea.
gcloud cloud-shell ssh --authorize-session
Use vscode with cloud-shell
First, make sure your cloud-shell is started by logging into it via gcloud cloud-shell ssh --authorize-session --command=uptime
Generate an ssh_config entry for your cloudshell:
gcloud cloud-shell ssh --authorize-session --dry-run |
while read -r _ _ _ port _ _ _ _ creds _ ; do
printf "Host cloud-shell\n User %s\n Port %s\n Hostname %s\n" ${creds/@*/} $port ${creds/*@/}
done
This will print something like:
Host cloud-shell
User dade_murphy
Port 53000
Hostname 5.4.3.1
You'll have to put this into your ssh_config somehow. I recommend using Include statement in ~/.ssh/config and having the above command redirect to a file with only that content so it can be updated in-place. Then use vscode like you usually do with remote ssh, or by running
code --folder-uri \
vscode-remote://ssh-remote+cloud-shell/home/dade_murphy/git_workspace/garbage_file_recovery_work
List google cloud projects
gcloud projects list
Switch to a different project
gcloud config set project "$project_name"
Grant a user permission to a docker registry
gsutil iam ch 'user:user@example.com:objectViewer' 'gs://artifacts.example.appspot.com/'
List google compute zones
gcloud compute zones list
List compute nodes
gcloud compute instances list
List all disks
gcloud compute disks list
Generate ssh commands for all nodes
gcloud compute instances list | awk 'NR>1 {printf "gcloud compute ssh --zone %s %s\n", $2, $1;}'
ssh to a compute node
This is useful for getting system level access to an EKS node.
ssh-to-gke-node() {
[ "$#" -gt 0 ] || { echo "Usage: ssh-to-gke-node <node_name> [command]" ; return 1 ; }
read -r zone host < <(gcloud compute instances list --filter="$1" | awk 'NR==2 {print $2, $1 ;}') ;
shift
gcloud compute ssh --tunnel-through-iap --zone "$zone" "$host" -- "${@}" ;
}
Loop through all gcloud instances and perform a command
gcloud compute instances list |
awk 'NR>1 {printf "gcloud compute ssh --zone %s %s\n", $2, $1;}' |
while read -r ssh_cmd ; do
$ssh_cmd -- "docker images" </dev/null
done |
sort -u
Create a one-off compute node
gcloud compute instances create $USER-temp-node --zone=us-west4-a --network-interface=no-address
Leave off the --network-interface=no-address if you want a public IP address.
Delete a compute node
Sometimes autoscalers have a hard time scaling down, requiring manual termination of idle nodes. The following commands are equivalent:
gcloud compute instances delete "projects/$project_name/zones/$zone/instances/$node_name"
gcloud compute instances delete --project "$project_name" --zone="$zone" "$node_name"
To connect to VMs that don't have a public ip address, you need to give --tunnel-through-iap on the CLI and also have IAP-secured Tunnel User permission.
Add an EKS context to kubectl
https://cloud.google.com/sdk/gcloud/reference/container/clusters/get-credentials
This adds the cluster to your .kube/config with authentication done via an access token.
CLUSTER_NAME='foo-dev'
PROJECT_NAME='some-project'
REGION='us-west42'
gcloud container clusters get-credentials "${CLUSTER_NAME}" \
--region "${REGION}" \
--project "${PROJECT_NAME}"
Add Separate kubectl configs for different EKS clusters
This keeps each config in a different file, which is useful for requiring explicit enabling of a given environment, vs the normal behavior of inheriting the last used context.
# Set up individual kube config files for dev, prod and staging
KUBECONFIG="$HOME/.kube/foo-dev-config"
gcloud container clusters get-credentials dev-cluster --region vn-west4 --project foo-dev
kubectl config rename-context $(kubectl config current-context) foo-dev
KUBECONFIG="$HOME/.kube/foo-prod-config"
gcloud container clusters get-credentials prod-cluster --region vn-west4 --project foo-prod
kubectl config rename-context $(kubectl config current-context) foo-prod
KUBECONFIG="$HOME/.kube/foo-staging-config"
gcloud container clusters get-credentials staging-cluster --region vn-west4 --project foo-staging
kubectl config rename-context $(kubectl config current-context) foo-staging
Then setup aliases like
# ~/.bash_aliases
alias foo-k-dev="export KUBECONFIG=$HOME/.kube/foo-dev-config ; kubectl config set-context foo-dev --namespace=default ;"
alias foo-k-prod="export KUBECONFIG=$HOME/.kube/foo-prod-config ; kubectl config set-context foo-prod --namespace=default ;"
alias foo-k-stage="export KUBECONFIG=$HOME/.kube/foo-staging-config ; kubectl config set-context foo-staging --namespace=default ;"
List images available in Google Container Registry
gcloud container images list
Pull a docker container from Google Container Registry
gcloud docker -- pull gcr.io/project-id/hello-world
Control access to registries
"Container Registry uses a Cloud Storage bucket as the backend for serving container images. You can control who has access to your Container Registry images by adjusting permissions for the Cloud Storage bucket.
Caution: Container Registry only recognizes permissions set on the Cloud Storage bucket. Container Registry will ignore permissions set on individual objects within the Cloud Storage bucket.
You manage access control in Cloud Storage by using the GCP Console or the gsutil command-line tool. Refer to the gsutil acl and gsutil defacl documentation for more information." - https://cloud.google.com/container-registry/docs/access-control
Authenticate a private GCR registry in kubernetes
This is likely not copy/paste material, but the flow is generally correct.
PARTNER=other_company
PROJECT="our_company-$PARTNER"
USER=service-account-user-for-$PARTNER
EMAIL="$USER@$PROJECT.iam.gserviceaccount.com"
gcloud iam service-accounts create $USER
gcloud iam service-accounts keys create \
--display-name "$USER" \
--iam-account "$EMAIL" \
key.json
gcloud projects add-iam-policy-binding "$PROJECT" \
--member "serviceAccount:$EMAIL" \
--role "roles/storage.objectAdmin"
done
kubectl create secret "docker-pull-$PROJECT" "$PROJECT" \
--docker-server "https://gcr.io" \
--docker-username _json_key \
--docker-email "$EMAIL" \
--docker-password "$(cat key.json)"
Then use the value of docker-pull-${PROJECT} as your ImagePullSecret.
Set cache expiration of GCP bucket items to 5 minutes
By default, GCP bucket items have 1 hour of public cache, which means items can be cached outside of the control of the GCP admin. This means that within that cache time window, any requests for the item will have unpredictable results. Set your Cache-Control max-age to something low for files that change, like page content and indexes, but long for files that never change, like images and archives.
gsutil setmeta -h "Cache-Control: public, max-age=300" gs://helm-repo.example.org/index.yaml
More info: https://medium.com/@val_deleplace/public-cache-dont-panic-11038a073a9
View the metadata of a GCP bucket item
gsutil stat gs://helm-repo.example.org/index.yaml
The output will be something like:
gs://helm-repo.example.org/index.yaml:
Creation time: Fri, 22 Jul 2020 23:20:11 GMT
Update time: Mon, 23 Jul 2020 19:17:53 GMT
Storage class: MULTI_REGIONAL
Cache-Control: public, max-age=300
Content-Length: 3195714
Content-Type: application/octet-stream
Hash (crc32c): vA/Awm==
Hash (md5): 2AJ32cECSriE0UQStsXxyw==
ETag: COP7ew7D5+CAEoI=
Generation: 1595460011829230
Metageneration: 5
Show a bar chart of disk usage of gcp bucket contents
The general idea here is you use gsutil du gs://whatever/somepath, use swap the first and second columns, and pipe that to termgraph. In this example I use awk to do the column swap, filter out all files so we're only checking full directory sizes, and trim the directory name and part of the filename:
$ gsutil du gs://cat-pic-downloader-backups/backups/*full* | awk '/\/$/ {gsub(/.*velero-/, "", $2) ; print $2,$1 ;}' | termgraph
full-back-up-20190128040005/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 307.60M
full-back-up-20190129040006/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 305.13M
full-back-up-20190130040007/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 298.71M
full-back-up-20190201040008/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 298.80M
full-back-up-20190202040009/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 297.77M
full-back-up-20190203040010/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 297.64M
full-back-up-20190204040011/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 297.61M
full-back-up-20190205040012/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 296.78M
full-back-up-20190206040013/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 294.78M
full-back-up-20190207040014/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 292.45M
full-back-up-20190208040015/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 292.05M
full-back-up-20190209040016/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 292.74M
full-back-up-20190210040017/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 291.43M
full-back-up-20190211040018/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 291.73M
full-back-up-20190212040019/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 291.55M
full-back-up-20190213040020/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 289.23M
full-back-up-20190214040022/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 287.54M
full-back-up-20190215040023/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 286.56M
full-back-up-20190216040024/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 286.87M
full-back-up-20190217040025/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 286.58M
full-back-up-20190218040026/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 286.01M
full-back-up-20190219040027/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 286.23M
full-back-up-20190220040028/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 285.57M
full-back-up-20190221040029/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 283.66M
full-back-up-20190222040030/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 283.62M
full-back-up-20190223040031/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 283.29M
full-back-up-20190224040032/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 283.27M
full-back-up-20190225040033/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 283.31M
full-back-up-20190226040034/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 283.19M
full-back-up-20190227040035/: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 283.44M
Set maintenance window on a GKE cluster
This syntax is a bit difficult to work with. It must be given in RFC-5545 (iCal), but GCP only supports a subset of the spec. Most frustratingly, GCP doesn't support timezones. They warn you about this, saying the "timezone will not be stored", so if you want to start something at Wednesday 03:00:00+0600 you have to specify that as Tuesday 21:00:00. 🙄
N.B.: in my testing while writing this, local TZ was used if not TZ was sent, so it looks like their docs may be wrong and maybe the conversion step can be avoided.
Convert local time to rfc-5545 UTC
So if you want to start your maintenance on Thursday at 5am Pacific time and end at Thursday 5pm Pacific time, the first step is to figure out what the rfc-5545 UTC format would be:
$ TZ=Z gdate '+%Y%m%dT%H%M%SZ %A' -d 2021-05-20T05:00:00-0800
20210520T130000Z Thursday
$ TZ=Z gdate '+%Y%m%dT%H%M%SZ %A' -d 2021-05-20T17:00:00-0800
20210521T010000Z Friday
Set and verify the maintenance schedule:
gcloud container clusters update \
--project="$PROJECT_NAME" \
--zone="$ZONE" \
"$CLUSTER_NAME" \
--maintenance-window-start="20210520T130000Z" \
--maintenance-window-end="20210521T010000Z" \
--maintenance-window-recurrence="FREQ=WEEKLY;BYDAY=TH"
You should see Updating <cluster_name>...done. followed by some hyperlinks. Next you should validate that it was set right:
gcloud container clusters describe \
--format=json \
--project="$PROJECT_NAME" \
--zone="$ZONE" \
"$CLUSTER_NAME" |
jq .maintenancePolicy.window.recurringWindow
You should see a small json blob like:
{
"recurrence": "FREQ=WEEKLY;BYDAY=TH",
"window": {
"endTime": "2021-05-21T02:00:00Z",
"startTime": "2021-05-20T14:00:00Z"
}
}
Grab the start time and feed it back into gdate to validate that your desired local time is set:
gdate -d "2021-05-20T14:00:00Z" # comes out to: Thu May 20 07:00:00 PDT 2021
gcloud web console examples
Logs Explorer examples
Show namespace deletions
protoPayload.methodName="io.k8s.core.v1.namespaces.delete"
Show all images pulled by GKE
This will show pulls of all images except with substrings matching the references on line 3. The OR appears to be case sensitive.
resource.type="k8s_pod"
jsonPayload.reason="Pulling"
-jsonPayload.message : ( "Pulling image \"dade-murphy/leet-image\"" OR "Pulling image \"mr-the-plague/da-vinci\"" )
Show all k8s control plane upgrades
You can search using the same syntax at the CLI, which is great for local saving of search results so you can search them more quickly using grep and easily archive them for postmortem documentation.
gcloud beta logging read 'timestamp>="2021-01-01T00" AND protoPayload.metadata.operationType: UPGRADE_MASTER AND operation.producer: container.googleapis.com' > control-plane-upgrades.yaml
Cloud SQL examples
Docs are here: https://cloud.google.com/sql/docs/postgres/
List instances
gcloud sql instances list
Create a new postgres db instance
In the following example, the tier is custom, with 24 cpu cores and 122880mb of memory. You can see the standard tiers that are available with gcloud sql tiers list. Also, this creates an instance of POSTGRES_14, but other versions are listed here: https://cloud.google.com/sql/docs/postgres/create-instance#gcloud
gcloud sql instances create \
--project=${GCP_PROJECT} \
--region=${INSTANCE_REGION} \
--database-version=POSTGRES_14 \
--tier=db-custom-24-122880 \
${INSTANCE_NAME}
Upgrade postgres to a new version in-place
https://cloud.google.com/sql/docs/postgres/upgrade-major-db-version-inplace
gcloud beta sql instances patch "${INSTANCE_NAME}" --database-version=POSTGRES_14
List backups for an instance
gcloud sql backups list --instance="${INSTANCE_NAME}"
Create a backup
gcloud sql backups create \
--instance="${INSTANCE_NAME}" \
--description="Pre pg14 upgrade"
Restore a backup from one instance to another
This has to be done within the same GCP project
gcloud sql backups restore \
--backup-instance="${OLD_INSTANCE}" \
--restore-instance="${NEW_INSTANCE}" \
"${BACKUP_ID}"
Restore a backup from a sql file stored in a bucket
Depending on the contents of your sql backup, you may have to create the target database first. If the db does not exist and the sql file does not working CREATE statements you'll see the ambiguous error system error occurred pre-import and an unhelpful python stack trace.
gcloud sql import sql ${INSTANCE_NAME} \
--user "${DB_USERNAME}" \
--database "${TARGET_DB}" \
gs://some-bucket-name/some_backup.sql \
--verbosity debug
Add --async if you just want it to go to the background and return you to your shell prompt.