Debugging Kubernetes

22/05/21 22:07 Filed in: Systems Administation
Sometimes the easiest way to understand what is happening in Kubernetes is just to have a pod that runs Linux … a general Linux … Ubuntu will do nicely.

A big shout out to Tim Downey for this article (https://downey.io/notes/dev/ubuntu-sleep-pod-yaml/) that lets you create a pod which you can exec and then just add the tools you need to to figure out what is going on from the inside of your cluster when it doesn't make sense from the outside.
Paraphrasing the approach in https://downey.io/notes/dev/ubuntu-sleep-pod-yaml/:

Create a file called debug.yml containing:
apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
  labels:
    app: ubuntu
spec:
  containers:
  - image: ubuntu
    command:
      - "sleep"
      - "604800"
    imagePullPolicy: IfNotPresent
    name: ubuntu
  restartPolicy: Always


Use the command kubectl apply -f debug.yaml to start your pod.

Use the command kubectl exec -it ubuntu -- /bin/bash to get a command line in your pod.

Some useful things to add:

  • ping: apt install iputils-ping
  • host: apt install dnsutils
  • telnet: apt install telnet

And once you have one of these running you can debug all sorts of things including external endpoints with their services:

root@ubuntu:/# host atum-ra-http
atum-ra-http.default.svc.cluster.local has address 10.152.183.71
root@ubuntu:/# telnet atum-ra-http 80
Trying 10.152.183.71...
Connected to atum-ra-http.default.svc.cluster.local.
Escape character is '^]'.
GET /
….

and see what actually comes back (or not when your declarative configuration file leads to unexpected results but parses perfectly).

Tags: