Kubernetes Monitoring

I still use Icinga2 for monitoring my home network. I haven't got annoyed enough with it to move to anything else. The plugin that monitors my microk8s cluster is a bit simplistic it simply checks a number read from the interface and if its bigger than a threshold then its a warning or an error.

The catch is that Kubernetes never resets the number of restarts of a pod … so you can't clear the error state. This is apparently by design as pods are disposable items.

Deployments come to our rescue here allowing us to recreate the pods and hence reset their restart count.

The commands


To restart all your deployments:
kubectl rollout restart deployments

To restart a single deployment:
kubectl rollout restart deployments/deployment
(Don't forget about namespaces)

Discusion


It was really only by blind luck that I found this solution. It appears in a thread about the feature I was looking for which was reseting / clearing the restart count for a pod. It turns out that others have similar reasons for wanting this feature in Kubernetes, but this is a small use case and faces some implementation challenges … so unlikely to happen in the near future. Moral of the story appears to be read ALL the comments.

Hopefully this article will make finding the answer easier for others.

References


The good idea came from
https://github.com/kubernetes/kubernetes/issues/50375#issuecomment-780778407

Icinga2
https://icinga.com (the monitoring system)

CHECK_KUBERNETES
https://github.com/agapoff/check_kubernetes (the plugin I use to monitor my cluster)

Microk8s
https://microk8s.io