Skip to content

kubernetes

"Kubernetes is an open-source platform for automating deployment, scaling, and operations of application containers across clusters of hosts, providing container-centric infrastructure." - https://kubernetes.io/docs/whatisk8s

Glossary

More terms in the k8s glossary: https://kubernetes.io/docs/reference/glossary/

cli usage

Learn about kubernetes

kubectl explain roles

Show what API permissions you have

$ kubectl auth can-i --list
Resources                                       Non-Resource URLs   Resource Names   Verbs
*.*                                             []                  []               [*]
                                                [*]                 []               [*]
selfsubjectaccessreviews.authorization.k8s.io   []                  []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                  []               [create]
                                                [/api/*]            []               [get]
                                                [/api]              []               [get]
                                                [/apis/*]           []               [get]
                                                [/apis]             []               [get]
                                                [/healthz]          []               [get]
                                                [/healthz]          []               [get]
                                                [/livez]            []               [get]
                                                [/livez]            []               [get]
                                                [/openapi/*]        []               [get]
                                                [/openapi]          []               [get]
                                                [/readyz]           []               [get]
                                                [/readyz]           []               [get]
                                                [/version/]         []               [get]
                                                [/version/]         []               [get]
                                                [/version]          []               [get]
                                                [/version]          []               [get]

Multiple kubernetes client configs

The default config is ~/.kube/config, but if you want to use multiple configs you can do this:

export KUBECONFIG="${HOME}/code/kubespray/artifacts/admin.conf:${HOME}/.kube/config"

I have seen weird problems when the order of configs is changed, such as certificate-authority-data and client-certificate-data being missing.

kubeadm

"kubeadm: easily bootstrap a secure Kubernetes cluster." - kubeadm --help

Show your kubeadm tokens

$ sudo kubeadm token list
TOKEN                     TTL       EXPIRES                     USAGES                   DESCRIPTION                                                EXTRA GROUPS
ubyc9a.1eq2ihwtnz7c7c9e   23h       2018-05-24T16:19:33-04:00   authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token

See sudo kubeadm token -h for more usage.

kubectl

"kubectl controls the Kubernetes cluster manager." - kubectl --help

  • https://github.com/kubernetes/kubectl

  • kubectl get - show all resource types with short-hand versions.

  • kubectl completion -h - show how to configure completion for your shell.
  • kubectl config get-contexts - show which k8s configuration contexts you can control.
  • kubectl config use-context foo - switch to the foo context.
  • kubectl get nodes - show the nodes in the k8s cluster.
  • kubectl get pods - show deployed pods. there can be many pods per deployment.
  • kubectl get pods -n kube-system - show pods in a specific namespace.
  • kubectl get pods,hpa,deployment --all-namespaces - get several resource types at once, from all namespaces
  • kubectl describe pod foo
  • kubectl get deployment
  • kubectl describe deployment foo
  • kubectl get ns - show namespaces.
  • kubectl get pv - show physical volumes.
  • kubectl get svc -n kube-system - show a table of important details about running services in the kube-system namespace.
  • kubectl get pods -o yaml - show the yaml configs for the currently running status of every pod.
  • kubectl explain pods.spec - show documentation about pod specifications.
  • kubectl describe pods/echoserver - describe the pod whose Name is echoserver.
  • kubectl get rs - show replica sets.
  • kubectl expose deployment <deployment_name> --type=NodePort - create a service for the given deployment.
  • kubectl scale deployment <deployment_name> --replicas=5 - scale a deployment to 5 pods.
  • kubectl rollout history deployment <deployment_name>
  • kubectl get cm - get a list of config maps.
  • kubectl get apiservices - get a list of api service endpoints. Show -o yaml to view status about availability, endpoint, etc..

Working with several configs

Sometimes you want to have individual configs, such as when you are using configs that are updated by other engineers and pulled down via git, and sometimes you want to have one monolithic config, such as when you are using a tool that cannot easily work with multiple configs.

Use multiple configs via alias

This is a great method for requiring explicit selection of the environment, which is good for not accidentally operating in prod. Using KUBECONFIG also allows your to set different k8s environments per terminal session, which is great for doing comparisons across clusters.

alias k8s-foo-prod="export KUBECONFIG=$HOME/.kube/foo-prod-config ; kubectl config set-context foo-prod --namespace=default ;"

See also Google Cloud for more examples like this related to GCP.

Merge several configs

This produces a monolithic file named kube_config which can be moved to ~/.kube/config. It merges the contents of your existing ~/.kube/config file.

REPO_DIR=/path/to/repo/
export KUBECONFIG="${HOME}/.kube/config"
for X in $(find "$REPO_DIR/kube_config.d" -name '*.config') ; do
    KUBECONFIG+=":$X"
done
kubectl config view --flatten > kube_config
echo "Config file successfully created at ${PWD}/kube_config"
echo "Run: mv -i ${PWD}/kube_config ${HOME}/.kube/config"

Create a KUBECONFIG env var from several config files

This produces a KUBECONFIG that looks like file1:file2:file3

REPO_DIR=/path/to/repo/
KUBECONFIG="${HOME}/.kube/config"
for config in $(find "$REPO_DIR/kube_config.d" -name '*.config') ; do
    KUBECONFIG+=":$config"
done
echo "KUBECONFIG=${KUBECONFIG}" ;

Show nodes and their taints

kubectl get nodes --output 'jsonpath={range $.items[*]}{.metadata.name} {.spec.taints[*]}{"\n"}{end}'

Drain and cordon a node

Do this before deleting or reloading a node.

kubectl drain --ignore-daemonsets --force --delete-emptydir-data "$NODE_NAME"

Drain all but the top 20 nodes in some node-pool selected by most CPU usage

kubectl top nodes --sort-by=cpu |
awk '/node-pool-identifiable-string/ {print $1}' |
tail -n +20 |
sargs kubectl drain --ignore-daemonsets --force --delete-emptydir-data

Show namespaces and how many hours old they are

kubectl get namespace --sort-by=".metadata.creationTimestamp" -o json |
jq -r '
  .items[] |
  ((now - (.metadata.creationTimestamp | fromdateiso8601))/3600 | floor) as $hours_old |
  "\(.metadata.name) \($hours_old)"
'

Show pods, sorted by creation time

Only descending sort is supported

kubectl get pods --sort-by=.metadata.creationTimestamp

To sort ascending you can use awk and tac (which is cat in reverse)

kubectl get pods --sort-by=.metadata.creationTimestamp |
awk 'NR == 1; NR > 1 {print | "tac"}'

Show pods that are not running

kubectl get pods --all-namespaces --field-selector='status.phase!=Running' --sort-by=.metadata.creationTimestamp

Show pods that are not running or did not exit cleanly

kubectl get pods --all-namespaces --field-selector='status.phase!=Running,status.phase!=Succeeded' --sort-by=.metadata.creationTimestamp

Show pods that are terminating

Unfortunately "Terminating" shows up as a status, but is not a phase, so we have to jump through some hoops to show this list. Here's one way to do this:

kubectl get pods -A |
awk '$4 == "Terminating" {print $1,$2}' |
while read -r NS POD ; do
  kubectl get pod "$POD" -n "$NS" -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,TERMINATION_GRACE_PERIOD:.spec.terminationGracePeriodSeconds
done |
column -t |
sort -u

And the output will be something like:

NAMESPACE                       NAME                                                TERMINATION_GRACE_PERIOD
otv-blazing-ray-3043            blazing-ray-3043-miner-7556f86b76-8mpdj            600
otv-gravitational-century-8705  gravitational-century-8705-miner-66b6dd97cc-c2mqq  600
otv-lunar-nova-0800             lunar-nova-0800-miner-86684cd6f8-d79wm             600

Show all images referenced by your k8s manifests

kubectl get pods --all-namespaces -o jsonpath="{..image}" |
tr -s '[[:space:]]' '\n' |
sort |
uniq -c |
sort -n

Show a list of containers formatted as pod_name container_name

kubectl get pods -n "$NS" -o json |
  jq -r '.items[] | .metadata.name as $podname | .spec.containers[] | "\($podname) \(.name)"' |
  column -t

Show a list of containers formatted as pod_manager/pod_name container_name

When you need to check all of the containers of pods in a given pod manager (sts, ds, deployment), you need a list that is formatted in a very specific way.

For instance, to get a list of containers inside the prometheus sts pods

kubectl get sts -l component=prometheus -n "$NS" -o json |
  jq -r '.items[] | .kind as $kind | .metadata.name as $name | .spec.template.spec.containers[] | "\($kind)/\($name) \(.name)"' |
  column -t

Which produces the output:

StatefulSet/demo-prometheus  configmap-reloader
StatefulSet/demo-prometheus  prometheus

This can then be fed into anything needing such syntax, for example kubectl exec to check the runtime env of these containers:

$ kubectl get sts -l component=prometheus -n $NS -o json |
  jq -r '.items[] | .kind as $kind | .metadata.name as $name | .spec.template.spec.containers[] | "\($kind)/\($name) \(.name)"' |
  while read -r p c ; do echo "$p $c $(kubectl -n $NS exec $p -c $c -- env | grep '^HOSTNAME=')" ; done ;
StatefulSet/demo-prometheus configmap-reloader HOSTNAME=demo-prometheus-1
StatefulSet/demo-prometheus prometheus HOSTNAME=demo-prometheus-1

That's obviously a contrived example, but the real learning here is that it is possible to iterate deep json data while referencing values form higher levels by storing that higher level as variables.

Decode a secret

Use built in base64 decoding like this:

kubectl get secret -n "${NAMESPACE}" "${SECRET_NAME}" -o go-template='{{ printf "%s\n" (.data.password | base64decode) }}'

Things get tricky when you have a dot in the key name:

kubectl get secret -n "${NAMESPACE}" "${SECRET_NAME}" -o go-template='{{ printf "%s\n" (index .data "pgbouncer.ini" | base64decode) }}'

Or you can use -o jsonpath with an external base64 decoder:

kubectl get secret -n "${NAMESPACE}" "${SECRET_NAME}" -o jsonpath='{.data.pgbouncer\.ini}' | base64 -d

Alternatively you can use jq, which has the cleanest syntax when accessing keys with dots in the name:

kubectl get secret -n "${NAMESPACE}" "${SECRET_NAME}" -o json | jq -r '.data["pgbouncer.ini"]' | base64 -d

Decode SSL secrets and show info about the certificates

kubectl get secret -n istio-system istio.default -o json |
jq -r '.data | keys[] as $k | "\($k) \(.[$k])"' |
grep cert |
while read -r k v ; do
  echo "------ $k ------"
  echo -n "$v" |
  base64 -d |
  openssl x509 -noout -subject -issuer -dates
done

Example output:

------ cert-chain.pem ------
subject=
issuer= /O=cluster.local
notBefore=Aug 10 13:55:50 2022 GMT
notAfter=Nov  8 13:55:50 2022 GMT
------ root-cert.pem ------
subject= /O=cluster.local
issuer= /O=cluster.local
notBefore=Sep 29 13:52:55 2021 GMT
notAfter=Sep 27 13:52:55 2031 GMT

Watch what's going on in your cluster

watch kubectl get pods --all-namespaces -o wide

or

kubectl get pods --all-namespaces -o wide -w

Show all pods and their container's requests and limits

kubectl get pods --all-namespaces -o json |
jq -r '
  .items[] |
  .metadata.namespace as $namespace |
  .metadata.name as $pod_name |
  .spec.containers[] |
  [$namespace, $pod_name, .name, (.resources | tostring)] |
  @tsv
' | column -t -s$'\t'

This will produce output like with the columns namespace, pod, container, resources as a json blob:

development  gamehouse-nats-0                  nats                {"limits":{"cpu":"250m","memory":"100Mi"},"requests":{"cpu":"75m","memory":"30Mi"}}
development  gamehouse-nats-2                  metrics             {"limits":{"cpu":"250m","memory":"100Mi"},"requests":{"cpu":"75m","memory":"30Mi"}}
development  gamehouse-nginx-85885cbb75-m5t58  nginx               {"limits":{"cpu":"100m","memory":"10Mi"},"requests":{"cpu":"80m","memory":"7Mi"}}
development  gamehouse-nginx-85885cbb75-wdmhf  nginx               {"limits":{"cpu":"100m","memory":"10Mi"},"requests":{"cpu":"80m","memory":"7Mi"}}
development  gamehouse-prometheus-0            configmap-reloader  {"limits":{"cpu":"100m","memory":"25Mi"},"requests":{"cpu":"100m","memory":"25Mi"}}
development  gamehouse-prometheus-0            prometheus          {"limits":{"cpu":"3","memory":"20Gi"},"requests":{"cpu":"1","memory":"4Gi"}}

Show daemonsets that are not up to date

kubectl get daemonset -A | awk '$3 != $6 {print}'

Watch events in a given namespace

kubectl -n kube-system get events --field-selector type=Warning -w

Or format the event messages with more useful information (really wide output)

kubectl get events -w -o custom-columns=FirstSeen:.firstTimestamp,LastSeen:lastTimestamp,Kind:involvedObject.kind,Name:.involvedObject.name,Count:.count,From:.source.component,Type:.type,Reason:.reason,Message:.message

Show all containers for each pod matching a label

kubectl -n kube-system get pod -l k8s-app=kube-dns -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\n\t"}{range .spec.containers[*]}{.name}{":\t"}{.image}{"\n\t"}{end}{"\n"}{end}'

Show a list of everything in a namespace

NS=kube-system
kubectl get all -n "$NS" --output 'jsonpath={range $.items[*]}{.kind} {.metadata.name}{"\n"}{end}' |
grep -v '^List $'  # exclude empty namespace

Show logs for a given pod since N hours ago

pod_name=httpbin
kubectl logs $pod_name --since=12h

The --since arg can take [s]econds, [m]inutes and [h]ours. Longer durations should use --since-time=<rfc3339 timestamp>

Show logs for a given pod since a given date

The --since-time arg takes RFC3339 datetime. EG: 1991-08-03T13:31:46-07:00. This format requirement is strict, and is incompatible with the GNU date --rfc-3339=seconds output, which uses a space instead of a T to separate the full date from the full time, and +%FT%F%z, which does not include a colon between hours and minutes.

pod_name=httpbin
kubectl logs $pod_name --since-time="$(date -Iseconds -d '-5 weeks')"

Output custom column names

$ kubectl get pvc --all-namespaces -o custom-columns='NAME:metadata.name,SIZE:spec.resources.requests.storage'
NAME                   SIZE
foo-logs               256Gi
test-volume-2          1Gi
some-awesome-service   5Gi

$ kubectl get pods -o custom-columns='NAME:.metadata.name,START_TIME:.status.startTime,.spec.containers[0].env[?(@.name == "GITLAB_USER_EMAIL")].value' | grep -E 'NAME|jobs'
NAME                                                 START_TIME             GITLAB_USER_EMAIL
runner-ppzmy1zx-project-11548552-concurrent-0q2pmk   2019-10-23T17:00:56Z   user2@example.com
runner-ppzmy1zx-project-11548552-concurrent-1f7nfx   2019-10-23T17:04:27Z   user1@example.com
runner-ppzmy1zx-project-11548552-concurrent-2n84rv   2019-10-23T17:04:19Z   user1@example.com

Perform a restart of a service, daemonset or statefulset

DEPLOYMENT_NAME=gibson_garbagefile_seeker
kubectl rollout restart deployment $DEPLOYMENT_NAME

Run a cronjob out of schedule

kubectl create job --from=cronjob/download-cat-pix download-cat-pix-manual-run

Create a yaml file for a resource type

You can generate yaml for a variety of entities without having to create them on the server. Each entity requires different syntax, so you have to work through the error messages to get to a final solution.

https://kubernetes.io/docs/reference/kubectl/conventions/#generators

$ kubectl create --dry-run=client -o yaml cronjob --schedule='15 * * * *' --image=image-name:1.2.3 job-name
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  creationTimestamp: null
  name: job-name
spec:
  jobTemplate:
    metadata:
      creationTimestamp: null
      name: job-name
    spec:
      template:
        metadata:
          creationTimestamp: null
        spec:
          containers:
          - image: image-name:1.2.3
            name: job-name
            resources: {}
          restartPolicy: OnFailure
  schedule: 15 * * * *
status: {}

Installations

The standard way to install k8s by yourself is to use kubeadm.

Manually on Ubuntu 16

## as root
swapoff -a # https://github.com/kubernetes/kubernetes/issues/53533
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
echo "deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable" > /etc/apt/sources.list.d/docker.list
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list
apt update
apt dist-upgrade -y
apt install -y apt-transport-https ca-certificates curl software-properties-common
apt install -y docker-ce
apt install -y kubelet kubeadm kubectl
kubeadm init

kubeadm init guide: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#instructions

DNS

Kubernetes lets you resolve resources via DNS

Enable k8s dns logging

kubectl -n kube-system edit configmap coredns
## Add 'log' to the 'Corefile' config

DNS Entity map

  • Kubernetes Service: <service>.<namespace>.svc.cluster.local. (eg: httpbin.default.svc.cluster.local.)
kubectl get svc --all-namespaces -o jsonpath='{range .items[*]}{.metadata.name}{"."}{.metadata.namespace}{".svc.cluster.local.\n"}'

Or with jq:

kubectl get svc --all-namespaces -o json |
jq -r  '.items[] | "\(.metadata.name).\(.metadata.namespace).svc.cluster.local."'

And if you want to also add port numbers:

kubectl get svc --all-namespaces -o json |
jq -r '.items[] | "\(.metadata.name).\(.metadata.namespace).svc.cluster.local." as $base | .spec.ports[] | "\($base):\(.port)"'
  • With core-dns you can run dig SRV +short *.*.svc.cluster.local. to get a list of all services.
  • Kubernetes service srv records: _${service_port_name}._${protocol}.${service}.${namespace}.svc.cluster.local. (eg: _http._tcp.httpbin.default.svc.cluster.local.)

crictl

crictl is a tool to inspect the local Container Runtime Interface (CRI)

user@k3snode:~$ sudo crictl pods
POD ID          CREATED       STATE  NAME                            NAMESPACE    ATTEMPT
688ecc2d9ce4d   2 weeks ago   Ready  log-fluentd-676d9d7c9d-ghz5x    default      8
ee1d8b0593e71   2 weeks ago   Ready  tiller-deploy-677f9cb999-rx6qp  kube-system  7
1153f4c0bd1f4   2 weeks ago   Ready  coredns-78fcdf6894-qsl74        kube-system  8
5be9c530c8cdc   2 weeks ago   Ready  calico-node-59spv               kube-system  10
d76d211830064   2 weeks ago   Ready  kube-proxy-cqdvn                kube-system  104
aa1679e0bfcca   2 weeks ago   Ready  kube-scheduler-s1               kube-system  10
ef64eea461bc0   2 weeks ago   Ready  kube-controller-manager-s1      kube-system  10
14ec5abe1e3ab   2 weeks ago   Ready  kube-apiserver-s1               kube-system  11
d4ce465a0942f   2 weeks ago   Ready  etcd-s1                         kube-system  10

Cloud Provider versions

Code snips

Show all ingresses and what hostnames they handle

This is useful when you have a lot of ingresses or very long hostnames which cause them to be hidden by an ellipse with normal or even wide output.

kubectl get ingresses -n "$NS" --output 'jsonpath={range $.items[*]}{.metadata.name}{":"}{range @.spec.rules[*]}{"\n\t"}{.host}{end}{"\n"}{end}

This will output a list like

rodent-ingress:
    mouse.example.com
    hamster.example.com
bird-ingress:
    parrot.example.com
    swallow.example.com
    owl.example.com

Parse swagger.json for API definitions

This is useful when manually writing helm chart templates to handle a range of k8s versions. (Keep an eye on https://github.com/helm/helm/issues/9765 though for hopefully a better way than manually doing this.)

# Download a range of swagger.json files named by version. EG: v1.18.0.json
for X in {15..22} ;
  do ver="v1.$X.0"
  curl -LsSo "${ver}.json" "https://raw.githubusercontent.com/kubernetes/kubernetes/${ver}/api/openapi-spec/swagger.json"
done

# Parse these into a text list of API versions supported by the version. EG: v1.18.0-definitions.txt
for X in v1.* ; do
  jq -r '.definitions | keys | .[]' $X > ${X/.json/}-definitions.txt
done

# Then you can grep for a definition to see what versions support it
grep 'Ingress$' *definitions.txt | grep -vE 'LoadBalancer'

Use jq to find zombie pods

If the base container shuts down, sometimes the istio sidecar can continue to run. You can find this condition with:

kubectl get pods -A -o json | jq '
  .items[] |
  select(.status.containerStatuses[].name == "base" and .status.containerStatuses[].state.terminated.exitCode == 0) |
  select(.status.containerStatuses[].name == "istio-proxy" and .status.containerStatuses[].state.terminated.exitCode == null) |
  {
    "name": .metadata.name,
    "namespace": .metadata.namespace,
    "status": [
      .status.containerStatuses[] |
      {
        "name": .name,
        "exit_code": .state.terminated.exitCode
      }
    ]
  }
'

Use jq to find all pods with a specific container state

kubectl get pods -A -o json | jq '
  .items[]? |
  select(.status.containerStatuses[]?.state.waiting.reason == "CreateContainerConfigError") |
  .metadata.name
'

Use jq to find pods that have problematic phases

kubectl get pods -A -o json |
jq -c '
  .items[] |
  select(.status.phase|test("Pending|Unknown")) |
  [.status.phase, .metadata.creationTimestamp, .metadata.namespace, .metadata.name]
'

linux kernel namespaces

Linux kernel namespaces are part of the magic that allows containers to run, and kubernetes pods take this a step further by allowing multiple containers to run inside a pod, and share only some of the namespaces. Which ones?

diff -t -W 65 -y ns-container-1.txt ns-container-2.txt
$ readlink /proc/$$/task/*/ns/*   $ readlink /proc/$$/task/*/ns/*
cgroup:[4026531835]               cgroup:[4026531835]
ipc:[4026532832]                  ipc:[4026532832]
mnt:[4026533233]                | mnt:[4026533326]
net:[4026532835]                  net:[4026532835]
pid:[4026533325]                | pid:[4026533328]
pid:[4026533325]                | pid:[4026533328]
user:[4026531837]                 user:[4026531837]
uts:[4026533324]                | uts:[4026533327]