K8S MetalLB has moved

02/10/23 20:55 Filed in: Systems Administation

The MetalLB has moved from docker.io to quay.io



While this is actually old news it caused me an issue when rebuilding a node in my cluster. As the source of the container has moved you can't instantiate it again without some changes.

This type of issue is likely to pop up again as other components move repositories.

Apparently this is all fixed in a later version of microk8s than I am currently running … but that is a story for another day

How to recognise the issue



On the replaced / rebuilt nodes I started noticing ErrImagePull and ImagePullBackOff errors:

$ microk8s.kubectl get pods --all-namespaces --output=wide

metallb-system   speaker-ww2np                              0/1     ImagePullBackOff   0          14m      10.100.0.35    shu1   <none>           <none>


Which when examined showed

kubectl describe pod -n metallb-system speaker-ww2np

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  18m                   default-scheduler  Successfully assigned metallb-system/speaker-ww2np to shu1
  Normal   Pulling    17m (x4 over 18m)     kubelet            Pulling image "metallb/speaker:v0.9.3"
  Warning  Failed     17m (x4 over 18m)     kubelet            Failed to pull image "metallb/speaker:v0.9.3": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/metallb/speaker:v0.9.3": failed to resolve reference "docker.io/metallb/speaker:v0.9.3": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
  Warning  Failed     17m (x4 over 18m)     kubelet            Error: ErrImagePull
  Warning  Failed     16m (x6 over 18m)     kubelet            Error: ImagePullBackOff
  Normal   BackOff    3m51s (x62 over 18m)  kubelet            Back-off pulling image "metallb/speaker:v0.9.3"



The fix



The temporary fix is to patch the deployment and daemon sets:

microk8s.kubectl -n metallb-system patch deployment controller --type=json -p='[{"op":"replace","path":"/spec/template/spec/containers/0/image","value":"quay.io/metallb/controller:v0.9.3"}]'
microk8s.kubectl -n metallb-system patch daemonset speaker --type=json -p='[{"op":"replace","path":"/spec/template/spec/containers/0/image","value":"quay.io/metallb/speaker:v0.9.3"}]'



Thanks



Jon Hayes (senyahnoj) posted this solution https://github.com/canonical/microk8s/issues/3855

Tags: