Before the beginning of April, we were spending about an hour a day every weekend to monitor what the backup environment was doing. Since the beginning of April, we have moved to a model where there is always somebody on-call during non-business hours. While this is a step in the right direction, I do not think that we have the monitoring completely set up to handle all contingencies.

Take this last weekend for example, we had a server crash Sunday morning around 3am. In the previous model, the person checking the backup environment on Sunday would have realized that the server had crashed and something would have been done about it. What happened this weekend is that nobody noticed that the server was down until Monday morning; not quite the desired result. The server crash was noticed by the monitoring, pages were sent out (but not to the person on call), and an alert was opened but no one was notified. So as you can see there are a few things that need to be addressed and that is one of the things we were doing today.

Very quiet on the home front today. We went out for fast food for dinner and brought back some donut holes. That was probably a mistake as I am sure I will eat the majority of them before they have been in the house for 24 hours - under the pretense that they will be stale if we leave them. Right...

Good night!