Systems Administation

Power up on Power Restore

Power up on Power On is fairly important for people running servers … but its a little tricky if you are recycling Mac Minis for hosting VM hosts (eg Proxmox)

Fortunately some others have done good work and worked out the magic commands you need.
Read More...

Postgres and Moving Databases

Recently I had to recover a Postgres DB and for 'convenience' I moved it from its original Linux environment to OS X. The process had a few wrinkles most important was that the linux environment used a locale of en_US.utf8 and MAC OS only has en_US.UTF-8
Read More...

K8S MetalLB has moved

The MetalLB has moved from docker.io to quay.io



While this is actually old news it caused me an issue when rebuilding a node in my cluster. As the source of the container has moved you can't instantiate it again without some changes.

This type of issue is likely to pop up again as other components move repositories.

Apparently this is all fixed in a later version of microk8s than I am currently running … but that is a story for another day
Read More...

Kubernetes Monitoring

I still use Icinga2 for monitoring my home network. I haven't got annoyed enough with it to move to anything else. The plugin that monitors my microk8s cluster is a bit simplistic it simply checks a number read from the interface and if its bigger than a threshold then its a warning or an error.

The catch is that Kubernetes never resets the number of restarts of a pod … so you can't clear the error state. This is apparently by design as pods are disposable items.

Deployments come to our rescue here allowing us to recreate the pods and hence reset their restart count.
Read More...

Rube Goldberg would be proud ...

Rube Goldberg was a cartoonist who has a character that invented fantastic chain reaction machines. I may have been channelling his muse for inspiration in getting around my latest NBN failure:
  1. We moved house (Note: there is no such thing as an NBN relocation … its a fiction … its a cancellation and a new connection)
  2. When people order a static IP they really want the same one when they relocate it (see 1)
  3. The Sparkies came, the modem was turned off for a couple of hours … things didn't come back
On the Friday afternoon before a long weekend …

And so it begins …
Read More...

Google Updates Authenticator and loses all your data

Google Authenticator just updated on the Apple App Store today … and the update proceeded to loose all my stored authentication tokens.

There are many 'definitions' of what 2FA means

too expletive awful

is one that springs to mind today.

And the big irony. Google tells me its Cyber Sercurity month. No doubt this update is promoting better security.

If you are looking at destroying people's trust in 2FA, Google has just shown how to do it.

Read More...

Adding an NFS datastore to VMWare ESXi

VMWare ESXi supports NFS storage. All the VMWare documents talk about managing it from vSphere. Which is a pity if you don't happen to be using vSphere. Fortunately adding your NFS datastore is relatively simple, apart from the lack of documentation. Read More...

Debugging Docker

Sometimes the easiest way to understand what is happening in Docker is just to have a container that runs Linux … a general Linux … Ubuntu will do nicely.

It is easy to steal an idea from the Kubernetes world (Debugging Kubernetes) and apply it to Docker and Portainer
Read More...

Using a docker private registry with vctl

Currently I am using VMWare Fusion's vctl to run containers on my Mac. It has been surprisingly compatible with the Docker desktop, but occasionally you run into an issue.

I am currently using an unencrypted local repository and you get errors like:

% vctl login 192.168.1.1:5000
INFO Login to 192.168.1.1:5000...
Username: maurice
Password for maurice:
INFO Error logging in to v2 endpoint, trying next endpoint: Get https://192.168.1.1:5000/v2/: http: server gave HTTP response to HTTPS client
ERROR Get https://192.168.1.1:5000/v2/: http: server gave HTTP response to HTTPS client


The answer is trivial but literally buried at the end of the help for the login command:

vctl login --http my-plain-http-registry.com:5000

You will need to sprinkle the --http flag around your later commands too.

This was going to be an article about the tool skopeo (https://github.com/containers/skopeo) allows a user to copy from one repository to other formats, and I have included that for interest. Read More...

Using a Microk8s Kubernetes Ingress as a Reverse Proxy for External Web Servers

Fairly early in the Kubernetes journey you are told about Ingress Controllers. They take an incoming connection (usually HTTP or HTTPS) and direct it to one or more services based on the path referred to in the incoming connection. In the case of the NGINX Ingress Controller it is literally a reverse proxy that is used as a frontend for services.

If you happen to need a reverse proxy (for example to provide authentication for your Let's Encrypt SSL certificate automatic renewal) it looks tantalisingly like it should be easy to do this from your Microk8s cluster. And with the benefit of MetalLB (Bare metal Load Balancer) it should be possible for this to keep working if one of the nodes should fail.

This turned out to be far harder than expected. Only by combining a lot of separate examples and parts was it possible to accomplish this. Furthermore, this functionality is only really intended for use while you are in process of migrating an external service into a Kubernetes cluster.

Read More...

Debugging Kubernetes

Sometimes the easiest way to understand what is happening in Kubernetes is just to have a pod that runs Linux … a general Linux … Ubuntu will do nicely.

A big shout out to Tim Downey for this article (https://downey.io/notes/dev/ubuntu-sleep-pod-yaml/) that lets you create a pod which you can exec and then just add the tools you need to to figure out what is going on from the inside of your cluster when it doesn't make sense from the outside. Read More...

Kubernetes Ingress and Externals

My microk8s decided to update its Ingress component. While trying to be helpful this actually disabled my redirections to internal web sites. The reason for the failure was that the syntax for configuring the externals has changed. Read More...

Node not ready under microk8s

I have just started monitoring my experimental Microk8s cluster via Icinga2 (a topic for another day) and encountered a critcical error. One of my nodes is not ready. This isn't the first time I have run into this issue, but this time I decided to fix it the right way i.e. diagnose what went wrong and make the minimal correction required to fix it.

Although there is nothing earth shattering or difficult in this post, what was surprising is that the answer did not turn up early in the search results if you Google:

kubelet has stopped posting node status microk8s


A bit of thinking was required to figure out the solution. Hopefully this post will make the solution more easily found. Read More...

A collection of Mac OS X CLI commands for fixing a frozen GUI

Once in a while my very overloaded Macs have a non-responsive GUI element. Usually I can log in via ssh. I have a few command line commands that help put things right. Read More...

Kuber - a nice app for your Kubernetes cluster

Kuber



Kuber (http://getkuber.com) is a neat little app form monitoring and controlling your cluster (available for both iOS in the App Store).

But there is a tiny hurdle to getting started … it is not obvious where your api server is on MicroK8s.

All you need to know is it is on your cluster host on port 16443 with the default install on MicroK8s and with that you are ready to go Read More...

So you want to access the kubernetes desktop on microk8s

Microk8s comes with a graphical dashboard. While not wanting to be dependent on a GUI I did want to see what it could do for me. Unfortunately this is another learning cliff.

For good security reasons the dashboard is only easily accessible on a local machine and there are several ways of providing access to the dashboard; but choose wisely as how you provide access to the dashboard affects how you authenticate to it.

The simplest approach I have found is to:
  • port forward the application to your hosts external addresses
  • use token login
Be warned if you use the API proxy method then you have to work a lot harder … or you can disable login security using skip-login (disabling security is almost certainly a very bad idea)
Read More...

Ingress Server Crashing on microk8s

I have Microk8s Kubernetes running on an old server. Apparently one of the release versions of the ingress service needed sse-4.2 and would crash without it. The problem is described well at https://github.com/rancher/rancher/issues/23626. Unfortunately the solution given assumes as bit of Kubernetes knowledge. Here I fill in some of the gaps. Read More...

Resizing Root on LVM

One of the advantages of using virtual machines is that you can easily create templates which you reproduce as needed. Of course the file systems are rarely the size you need them to be.

Fortunately, if you have either XFS or ext4 installed on LVM, expanding the root file system is fairly easy. Read More...

Docker install on Ubuntu via snap

It should be obvious but I discovered the hard way that you need to create the docker group prior to installing docker via snap if you want users to be able to run docker without using the sudo command. Read More...

Safari Bypass Certificate Issue

Safari Certificate Issues


A work Around



Much as I like Safari (it really does make web pages look nicer) it's certificate handling is less than optimal. It is quite easy to get stuck in a loop with Safari complaining about the certificate letting you add it and then complaining about the certificate.



This is sometimes caused by automatically generated certificates having expired - but you don't get told this.

Read More...

OSX High Sierra - the missing utils

The missing utilities


Following the official demise of PPTP a few updates ago Apple seems to have continued retiring features in High Sierra. In particular High Sierra has dropped a couple of utilities that we dinosaurs still use:
  • telnet
  • ftp
Whilst old and not supporting the shiniest security models they are still needed for debugging and ensuring that backward compatible systems remain that way. Read More...

SCM Manager on a Mac with a Letsencrypt certificate

SCM Manager



SCM Manager (https://www.scm-manager.org) manages source code repositories. It support GIT, Mercurial and SVN. Whilst initial installation on OS X was simple there were a number of difficulties in getting to a best practice installation. In particular using external certificates from Lets Encrypt (https://letsencrypt.org) was non-trivial. Read More...

Proxying internal websites via OS X Server

Proxying internal websites via OS X Server



After rearranging my network to use an OS X server for my main web server there were a few web based systems that I need to expose to the outside world. The advice on Reverse Proxy with macOS Server (http://stationinthemetro.com/2017/05/02/reverse-proxy-with-macos-server) by Mark Boszko worked for me. Read More...

Real SSL Certificates for OS X Server

Real SSL Certificates for OS X Server



LetsEncrypt (https://letsencrypt.org) has a mission of moving unencrypted Internet traffic to encrypted Internet traffic. They do this through relatively short lived SSL certificates. So an automated process for setting up and maintaining them is vital.

On shifting my web site to OS X server I also adopted LetsEncrypt and its Certbot. Read More...

OS X Server

OS X Server



While configuring OS X server (Sierra) I had to gather together a fair number of additional facts to finish configuration:
  • Port forwarding
  • Disabling default wiki

Read More...

IPSec between a FRITZ!Box and a Mikrotik

Due to the death of my FreeBSD router out at the farm a replacement was required. A Mikrotik Router was purchased (because of its compatibility with IPv6) and then the IPSec tunnel adventure began… Read More...

P2V a Linux box into VMWare

This really should be obvious and easy … after all everyone does it … but my Google Fu kept leading me to solutions that relied on having access to old versions of VMWare Converter.

But it really is easy by combining:
  • netcat; and,
  • qemu
P2V is simple.
Read More...

FRITZ!Box VPN to FreeBSD

I recently acquired a FRITZ!Box 7272 with the aim to replace my m0nowall firewall and ADSL router. The original idea was to simplify and hence improve my IPv6 connection to Internode and use the FRITZ!Box’s phone and Fax services on my fixed line. The catch is that I run a permanent LAN to LAN VPN connection between my house and the farm … and the FRITZ!Box was going to have to do this. Read More...

Lacie 2Big NAS



I purchased a Lacie 2Big NAS to act as backup storage - it was quiet and nicely designed - it fitted into a home environment. Unfortunately, the device only offered email notifications of failure - which didn’t fit well against my Nagios based monitoring approach. A bit of protocol reverse engineering and studying their released sources resulted in a Nagios plugin that could monitor the NAS’s health.

Read More...

FSCK instructions for Rasperry Pi

It is trivially easy to shut your Pi down incorrectly and it still seems that this has not been addressed well in the Raspberian repository. Here is a summary of http://www.raspberrypi.org/forums/viewtopic.php?t=64843 which describes how to fix the dreaded:

FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.



problem Read More...

Internet Woes: Delivering the mail in spite of the weather

My ADSL connection is having difficulties at home ... one of the few services I really care about is getting email ... and without a working externally accessible IP address it is hard to deliver mail to your mail server. Many large telcos’ NAT mechanisms make it extremely difficult to connect to services behind their 3G dongles; fortunately with the help of ssh, inetd and socket a solution can be temporarily cobbled together. Read More...

PostgreSQL CLI utility with SSL

The Postgres CLI utility can sometimes prove a little painful to use when trying to connect to a server. In this case the sticking point was require SSL on the server. Read More...

IPv6 on Internode using Monowall (Update)

In my previous article on using Monowall to connect to Intenode I made an observation that turning router advertisements on on the WAN interface was necessary. Further experimentation showed that this was not the case. Read More...

IPv6 on Internode using Monowall

As a previously happy SixXS (https://www.sixxs.net) user with a Monowall firewall (http://m0n0.ch/wall/) I thought that I would move one of my IPv6 services to something a little more local than England. Internode (http://www.internode.on.net) have an IPv6 over PPPOA / PPPOE ADSL2+ service so after much trying I persuaded Monowall to provide access to Internode’s IPv6 service. Read More...

Computer Security Isn't an Afterthought

Computer security must be built in to every application, it is almost impossible to add it on at the end. Read More...

Why NAT is not a security mechanism

All too frequently I see articles that refer to Network Address Technology (NAT) as a security mechanism. Let us be clear NAT has nothing to do with strengthening security and an awful lot to do with weakening it. Read More...

UPS on FreeBSD

Last night my long serving brick of a UPS died ... and to put it bluntly I feel almost naked without a UPS protecting my home server. This left the question of what to acquire. Top of my list of priorities was to have a serial interface - USB is just too painful to contemplate after the last time I configured a UPS on FreeBSD (Another CyberPower I blogged about in January 2013) and it would be nice if it worked with Network UPS Tools (NUT).

I ended up acquiring a CyberPower Value1200eLCD - and managed to get it working on FreeBSD ... but not with NUT apparently the people at CyberPower have backed away from the openness that they once embraced and also changed their protocols.
Read More...

Mavericks

I just completed my upgrade to OS X Mavericks and so far have hit 3 glitches:
  • Install hung
  • Can’t search contacts
  • Realmac Clear for Mac goes non-responsive
Apart from these issues so far so good.
Read More...

Java and Embedded devices

Java has such promise for use in embedded devices: Just download the JVM on the client device and access the embedded device from whatever client OS or platform you wish. Unfortunately the truth is rather less than the promise. Read More...

Do you have Foxtel ...

The power saving power board people have been around … yet another piece of equipment fails due to their poor training.

There is an Australian government program which allows electricity users to receive standby power saving power boards in exchange for the carbon credits the devices are estimated to deliver. Unfortunately, the training given to the people who deliver the boards is poor. The only question they appear to ask is “Is that device for Foxtel?”, and that is the only device they plug in to an outlet that stays on all the time. Read More...

Prowl as a Nagios Notifier

Prowl (@ProwlApp) works now (Apple it seems has finally sorted out its developer portal issues in relation to certificates) and notifications are flowing. All in all the presentation and configuration of alerts is great. Read More...

Prowl

As part of improving my reporting of Nagios alerts I decided to try using the Growl framework via Prowl to my iPhone. This was precisely the wrong time to do so as Apple is having an extended down time on their developer centre and a certificate renewal and hence his push notifications are not working. I have to praise the developer of Prowl as in spite of his obvious frustrations he has used Twitter to keep his users informed in a humorous and effective way. This is one of the best examples of handling a fault that is out of your control and crippling to your business that I have seen. Read More...

Goodbye Postini, Hello SpamHero

With the impending doom of Google’s mail filtering service (the service formerly known as Postini) I have been searching for alternatives. The front runner is SpamHero. My business partner at Clarinet Internet Solutions, Daniel O’Callaghan made the transition easy and came up with an interesting hybrid solution for further enhancing its operation. Read More...

Was the BOFH sleep deprived?

Is your systems admin irritable? Could it be sleep deprivation? The BOFH (Bastard Operator from Hell) is a classic figure from computer science fiction (http://bofh.ntk.net/BOFH/) famed for his low tolerance for users disturbing his ‘work’ schedule and doing exactly what they asked but rarely what they wanted. His quick wit probably means that he wasn’t sleep deprived but I am starting to ask how prevalent are the effects of sleep deprivation and what we need to do about them. Read More...

Android on the Desktop - If the plug fits

One thing I didn’t expect when I deployed a bunch of Android based boxes in a call centre was that operators would start plugging their iPhones etc in. Apparently, if the plug fits … Read More...

Head in the Clouds

If you open any computing trade journal you could be forgiven for believing we had all moved to New Zealand and were looking out on a marvellous vista - the land of the long white cloud. Unfortunately, in spite of the great work being done by the marketing people, the world hasn’t actually changed that much. Cloud is still virtualised servers and remote hosting. The new and exciting bits are rapid on demand deployment and tear down. In this article I take a look at who and why you should use cloud from an economic management point of view.
Read More...

Fear of Command Lines

Anyone know if there is a psychological term for fear of command lines and configuration files? A name for this phobia would be incredibly useful as I keep bumping in to circumstances where supposedly technical users downgrade a tool just because it doesn’t have a GUI and assume it must be “hard to use” if it is not driven by a point and click interface. Read More...

Low cost LeoStick based Serial Switch

The farm is connected to the Internet by a WiMax based wireless service which occasionally has a problem which needs the WiMax modem / router to be rebooted.

A low cost USB powered switch connected to one of the computers at the farm solved the problem nicely.

Read More...

Android on the Desktop - Keyboard adventures

Having deployed our Android boxes we encountered a couple of interesting issues relating to the origins of the platform. Read More...

Android on the Desktop

Currently I am speccing a call centre and one of our key requirements is to reduce our software administration and maintenance overheads. We are currently trialling Android STBs as terminals for the agents. Our hope is that their browser is sufficient to work with our bespoke web site and we will not have to do any major maintenance on the boxes as they are locked down to just the browsing app. Read More...

Contact sense in 24 Hours

Lots of quite expensive equipment used in telecommunications, television and security still use contact sens outputs for reporting error conditions.This project was put together in the space of 24 hours to allow Nagios to
monitor some contact closures.

Read More...

It is better when it is green

Monitoring systems are central to my happiness as a systems administrator. They tell me when things go wrong … hopefully before the phone rings … warn me when things are getting close to failing and can provide more directly useful information than direct customer reports. The major problem with monitoring systems is getting the information from the system to the human. Nagios’s web page will make noises and can provide a text based display; but sometimes you need a simple display that prompts you to go look for a problem. This is where traffic lights and other visual indicators are useful. Read More...

The IO Blender

At the VMUG Regional User Conference (#MVMUG Feb 7, 2013), Stephen Foskett (@sfosckett, http://blog.fosketts.net) presented on the IO Blender. The essential idea was that there was a great loss of information that was exploited by the storage arrays because the data was hidden behind the hypervisor. This reminded me about some information theory and compression. Read More...

Getting rwho to work under Solaris SMF

Solaris (10+) and OpenSolaris use a relatively new mechanism for starting daemons the SMF (Service Management Framework). Adding a new entry to a manifest is surprisingly easy. We use rwhod as an example. Read More...

Wierd behaviour of Cyberpower UPS

After finally getting sick of watching RAID rebuilds I bit the bullet and installed a Cyberpower BR850ELCD UPS on my FreeBSD monitoring box. Given I bought what I could find at the instant - there was an immediate threat to the stability of my power - and there were no available serial ports on the monitoring server we took the best available compromise especially as there we could also detect power failures via the network. Naturally, things did not go smoothly... Read More...

Oops - Undeleting a file on Freebsd

A perfect storm of filename completion and inattention resulted in the deletion of a C source code file yesterday (.core starts with .c). As a result I had to find a way to “undelete” a file from a FreeBSD system. Read More...

Change management

There is a wide range of change management practices employed by companies ranging from none through to fairly rigid and long documents that need to be completed before any change can be made - no matter how trivial. Not having change management invites both disaster and avoidable errors; excessive change management is usually a reaction to too many of the former problems. Read More...

FTP on an EMC NS20 Celerra

The EMC NS20 Celerra NAS is a small enterprise grade NAS that we use to support a VMWare Cluster and Office CIFS. It seemed like to obvious candidate to provide an external FTP service; this turned out to be a little less easy than was expected. Read More...

Why sysadmins hate it when it rains …

It rained today … and rain equals pain in the world of systems administration and telecommunications. Bad things happen when it rains ... Read More...

Rsync on Windows

Rsync is probably one of the most useful tools for moving / synchronising files at reasonable bandwidth speed.

One particularly useful trick to reduce down time for daemons when moving their storage is to initially copy their file store while the daemon is running, take the daemon down, then sync the file store before bringing the daemon up. The down time of the daemon is vastly reduced as you only need to copy the changed data.

Thanks to Cygwin this tool can be enjoyed on Windows as well as Unix, FreeBSD, Linux, MacOSX etc the only trick is how do you access windows drives in a Unix like way. Read More...