Kubernetes Monitoring

I still use Icinga2 for monitoring my home network. I haven't got annoyed enough with it to move to anything else. The plugin that monitors my microk8s cluster is a bit simplistic it simply checks a number read from the interface and if its bigger than a threshold then its a warning or an error.

The catch is that Kubernetes never resets the number of restarts of a pod … so you can't clear the error state. This is apparently by design as pods are disposable items.

Deployments come to our rescue here allowing us to recreate the pods and hence reset their restart count.

Using a Microk8s Kubernetes Ingress as a Reverse Proxy for External Web Servers

Fairly early in the Kubernetes journey you are told about Ingress Controllers. They take an incoming connection (usually HTTP or HTTPS) and direct it to one or more services based on the path referred to in the incoming connection. In the case of the NGINX Ingress Controller it is literally a reverse proxy that is used as a frontend for services.

If you happen to need a reverse proxy (for example to provide authentication for your Let's Encrypt SSL certificate automatic renewal) it looks tantalisingly like it should be easy to do this from your Microk8s cluster. And with the benefit of MetalLB (Bare metal Load Balancer) it should be possible for this to keep working if one of the nodes should fail.

This turned out to be far harder than expected. Only by combining a lot of separate examples and parts was it possible to accomplish this. Furthermore, this functionality is only really intended for use while you are in process of migrating an external service into a Kubernetes cluster.


Debugging Kubernetes

Sometimes the easiest way to understand what is happening in Kubernetes is just to have a pod that runs Linux … a general Linux … Ubuntu will do nicely.

A big shout out to Tim Downey for this article (https://downey.io/notes/dev/ubuntu-sleep-pod-yaml/) that lets you create a pod which you can exec and then just add the tools you need to to figure out what is going on from the inside of your cluster when it doesn't make sense from the outside. Read More...

Kubernetes Ingress and Externals

My microk8s decided to update its Ingress component. While trying to be helpful this actually disabled my redirections to internal web sites. The reason for the failure was that the syntax for configuring the externals has changed. Read More...

Node not ready under microk8s

I have just started monitoring my experimental Microk8s cluster via Icinga2 (a topic for another day) and encountered a critcical error. One of my nodes is not ready. This isn't the first time I have run into this issue, but this time I decided to fix it the right way i.e. diagnose what went wrong and make the minimal correction required to fix it.

Although there is nothing earth shattering or difficult in this post, what was surprising is that the answer did not turn up early in the search results if you Google:

kubelet has stopped posting node status microk8s

A bit of thinking was required to figure out the solution. Hopefully this post will make the solution more easily found. Read More...

Kuber - a nice app for your Kubernetes cluster


Kuber (http://getkuber.com) is a neat little app form monitoring and controlling your cluster (available for both iOS in the App Store).

But there is a tiny hurdle to getting started … it is not obvious where your api server is on MicroK8s.

All you need to know is it is on your cluster host on port 16443 with the default install on MicroK8s and with that you are ready to go Read More...

So you want to access the kubernetes desktop on microk8s

Microk8s comes with a graphical dashboard. While not wanting to be dependent on a GUI I did want to see what it could do for me. Unfortunately this is another learning cliff.

For good security reasons the dashboard is only easily accessible on a local machine and there are several ways of providing access to the dashboard; but choose wisely as how you provide access to the dashboard affects how you authenticate to it.

The simplest approach I have found is to:
  • port forward the application to your hosts external addresses
  • use token login
Be warned if you use the API proxy method then you have to work a lot harder … or you can disable login security using skip-login (disabling security is almost certainly a very bad idea)

Ingress Server Crashing on microk8s

I have Microk8s Kubernetes running on an old server. Apparently one of the release versions of the ingress service needed sse-4.2 and would crash without it. The problem is described well at https://github.com/rancher/rancher/issues/23626. Unfortunately the solution given assumes as bit of Kubernetes knowledge. Here I fill in some of the gaps. Read More...