I guess it had to come back and bite us at least once. We are running an older version of the Host Bus Adapter (HBA) driver on one of our backup servers; that version has a bug that causes it to start throwing errors in the log after a while. We had not yet updated because, according to the vendor, the logging of errors does not cause any issues with the actual running of the HBA.

Well today we found out that it is not quite true. If you are logging those errors and one of the fibre drives goes away - it will not come back! Even worse, if you reset the HBA to try and bring that drive back, all the drives will go away! Which of course meant that an emergency reboot of that server had to be scheduled immediately.

I guess a driver update will be on the list of things to do during the next change window.