Node not ready under microk8s

I have just started monitoring my experimental Microk8s cluster via Icinga2 (a topic for another day) and encountered a critcical error. One of my nodes is not ready. This isn't the first time I have run into this issue, but this time I decided to fix it the right way i.e. diagnose what went wrong and make the minimal correction required to fix it.

Although there is nothing earth shattering or difficult in this post, what was surprising is that the answer did not turn up early in the search results if you Google:

kubelet has stopped posting node status microk8s


A bit of thinking was required to figure out the solution. Hopefully this post will make the solution more easily found.

Step by step


On logging into my Microk8s cluster I queried the state of the system

$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
10.100.0.34 Ready 59d v1.18.9
maat NotReady 178d v1.18.9

Showing that only one node was problematic. Then I looked into the state of that node

$ kubectl describe node maat
Name: maat
Roles:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=maat
kubernetes.io/os=linux
microk8s.io/cluster=true
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Thu, 23 Apr 2020 14:50:15 +1000
Taints: node.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/unreachable:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: maat
AcquireTime:
RenewTime: Sun, 18 Oct 2020 11:18:52 +1100
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure Unknown Sun, 18 Oct 2020 11:14:45 +1100 Sun, 18 Oct 2020 11:20:45 +1100 NodeStatusUnknown Kubelet stopped posting node status.
DiskPressure Unknown Sun, 18 Oct 2020 11:14:45 +1100 Sun, 18 Oct 2020 11:20:45 +1100 NodeStatusUnknown Kubelet stopped posting node status.
PIDPressure Unknown Sun, 18 Oct 2020 11:14:45 +1100 Sun, 18 Oct 2020 11:20:45 +1100 NodeStatusUnknown Kubelet stopped posting node status.
Ready Unknown Sun, 18 Oct 2020 11:14:45 +1100 Sun, 18 Oct 2020 11:20:45 +1100 NodeStatusUnknown Kubelet stopped posting node status.

At which point it becomes clear that we need to restart the kubelet. This install of Mikrok8s was done via snap. So we enquire the names of the daemons.
# snap services microk8s
Service Startup Current Notes
microk8s.daemon-apiserver enabled active -
microk8s.daemon-apiserver-kicker enabled inactive -
microk8s.daemon-cluster-agent enabled active -
microk8s.daemon-containerd enabled active -
microk8s.daemon-controller-manager enabled active -
microk8s.daemon-etcd enabled active -
microk8s.daemon-flanneld enabled active -
microk8s.daemon-kubelet enabled active -
microk8s.daemon-proxy enabled active -
microk8s.daemon-scheduler enabled active -

And restart the one used for the kubelet
root@maat:~# snap restart microk8s.daemon-kubelet