AI misses stuff
06/11/26 12:20 Filed in: AI
Hold the front page AI is not perfect
While this is not new news, it can be somewhat annoying.
With the help of Claude I moved from my old Icinga2 monitoring system to CheckMk it was interesting what got left out.
Now the issue here is not that I failed to notice that the test cases were incomplete, but that it would be almost impossible for me to be sure that all the test cases were constructed in the first place. This is hard for any moderately complex system. Time pressure is only going to make this worse.
While this is not new news, it can be somewhat annoying.
With the help of Claude I moved from my old Icinga2 monitoring system to CheckMk it was interesting what got left out.
Now the issue here is not that I failed to notice that the test cases were incomplete, but that it would be almost impossible for me to be sure that all the test cases were constructed in the first place. This is hard for any moderately complex system. Time pressure is only going to make this worse.
The issue turned out that Claude missed reproducing the certificate checks in my Icinga2 system.
It monitored the web site, but didn't check that all the certificates were valid in the new configuration.
I ended up having to ask explicitly for this. This is actually due to Claude not knowing what was reported by Icinga ie the expiry showing as a warning on the website check.
While Claude was terribly apologetic about this, it seemed an unusual thing to miss. My guess is that there are underlying assumptions that were not exposed about accessibility and validity. In fact I confirmed my suspicions and yes Claude doesn't know about the underlying behaviour.
Fortunately this was a personal site, but for something more business critical it emphasises the need for extreme care in making sure that the whole job is done.
While AI has access to a lot of information it is not necessarily complete.
It monitored the web site, but didn't check that all the certificates were valid in the new configuration.
I ended up having to ask explicitly for this. This is actually due to Claude not knowing what was reported by Icinga ie the expiry showing as a warning on the website check.
While Claude was terribly apologetic about this, it seemed an unusual thing to miss. My guess is that there are underlying assumptions that were not exposed about accessibility and validity. In fact I confirmed my suspicions and yes Claude doesn't know about the underlying behaviour.
Fortunately this was a personal site, but for something more business critical it emphasises the need for extreme care in making sure that the whole job is done.
While AI has access to a lot of information it is not necessarily complete.