Well shortly after I posted the previous entry, I went to check on the backup environment at work since I am the one supposed to check the backups this weekend, and I had just glanced at it earlier in the day. Turns out that our main backup server had lost connectivity to the filesystem where the image dabatase is located - this basically means that no backups would be taking place as all of the backup executables are on that filesystem.

So I packed up the laptop, stopped by the office to pick up the badge that would let me into the datacenter and headed downtown to check on the status of the server. Fortunately, my initial fears were not realized; I thought that the disk storage must have lost a disk (which was the problem the last time we lost connectivity to the filesystem). The problem seemed to be limited to the server not being able to "see" the disk. Since it was already 1am and with a dwindling backup window I decided to take the quick road and just rebooted the server. The disk came back online and everything looked good.

I headed back to the office to turn the badge in and to make sure that the backups would resume. This is where the fun started: the jobs in the queue seemed to just be frozen there as there was nothing happening. I decided to stop the backup software (Veritas Netbackup). I had to wait for one process to finish its purge as we have been told that killing that process would result in corruption in our database.

After restarting Netbackup, the jobs that were frozen earlier all died and new jobs started. But of course, the fun did not end there. Our main media server was mounting tapes for everyone but itself - the mount requests were just being queued. So I had to restart Netbackup on that server as well and then finally backups started behaving.

It was 3:30am by the time I got back to bed and I only got 4 hours of sleep before I had to get up to get ready for church as we had a special breakfast taking place at the church where my daughters were playing special parts (Mary and the Innkeeper). So when we got back home after church and lunch I took a 3 hour nap and felt much better for it.

I have not done much else - although I did get to talk to my wife on the phone (YEAH!) and it was good to be able to do that.

Well I better close as I have to get up early to take my daughter to her work (5:35am - yuck!)

Good night!