Computers Are Evil
Yep, it's another job blog...
Thursday night, our backup drive on our main server choked. The tape got stuck, it was grinding, it wouldn't allow any writes. Pretty much, it was done. We got service in on Friday afternoon and they allegedly fixed it but to get the server to see the drive, we needed to reboot the box.
Rebooting means kicking everyone off - our web users, telecommuters, retail stores, sales and support folks. I found out at 3:30 and, since I'm the DBA, I got to make the call on when to take it down. Now, my first thought was "Screw everyone and let's take it down now and be done with it." But, I decided we should wait until 5:30 since most people would be gone for the day.
5:30 is when I had planned on leaving for the Dog Park where I was expecting 45 Parrot Heads to join me for the evening. I really didn't want to be hanging around at work but, hey, I have to earn my salary somehow.
I planned for a quick exit, though. I changed into my shorts, I ran out to the ATM for betting cash (a stop I had planned to make on the way) and shutdown everything on my PC except the bare minimums so logging off would be quick. I was ready to bolt.
Our UNIX guy took the server down at 5:35. I expected a 5 minute turnaround since when I stop and restart my database application it only takes about 2 minutes. Of course, I was wrong. The damn thing didn't get restarted until 6 AND it STILL didn't see the backup drive. While we were waiting for it decide when to restart, I start getting calls from various Parrot Heads. "What time are you going to be there?" "Where are you?" I was not a happy camper.
We knew it was back to the outside support guys to figure out so I declared no more shutdowns until today then headed to the races.
I was just starting on my second beer when my cell rang. Apparently, besides deciding not to see the new drive the server decided the date was now 7/22/05 and the time was 2 AM. Nice. Fortunately, Sugar Daddy was still at work and I talked him through all of the data corrections we needed to do and he changed the time and restarted the server. What's especially uncool about this is we had put a bug fix in on 5/31 for this very problem. Nice fix that didn't work, damn those IT people.
Of course, this took about 30 minutes of me on the phone pacing the clubhouse as people are arriving for our evening of fun and frolic. Fortunately, Steven took over my greeting duties and things look OK at the moment.
Until we take it down again today when I expect all hell will break loose.
Thursday night, our backup drive on our main server choked. The tape got stuck, it was grinding, it wouldn't allow any writes. Pretty much, it was done. We got service in on Friday afternoon and they allegedly fixed it but to get the server to see the drive, we needed to reboot the box.
Rebooting means kicking everyone off - our web users, telecommuters, retail stores, sales and support folks. I found out at 3:30 and, since I'm the DBA, I got to make the call on when to take it down. Now, my first thought was "Screw everyone and let's take it down now and be done with it." But, I decided we should wait until 5:30 since most people would be gone for the day.
5:30 is when I had planned on leaving for the Dog Park where I was expecting 45 Parrot Heads to join me for the evening. I really didn't want to be hanging around at work but, hey, I have to earn my salary somehow.
I planned for a quick exit, though. I changed into my shorts, I ran out to the ATM for betting cash (a stop I had planned to make on the way) and shutdown everything on my PC except the bare minimums so logging off would be quick. I was ready to bolt.
Our UNIX guy took the server down at 5:35. I expected a 5 minute turnaround since when I stop and restart my database application it only takes about 2 minutes. Of course, I was wrong. The damn thing didn't get restarted until 6 AND it STILL didn't see the backup drive. While we were waiting for it decide when to restart, I start getting calls from various Parrot Heads. "What time are you going to be there?" "Where are you?" I was not a happy camper.
We knew it was back to the outside support guys to figure out so I declared no more shutdowns until today then headed to the races.
I was just starting on my second beer when my cell rang. Apparently, besides deciding not to see the new drive the server decided the date was now 7/22/05 and the time was 2 AM. Nice. Fortunately, Sugar Daddy was still at work and I talked him through all of the data corrections we needed to do and he changed the time and restarted the server. What's especially uncool about this is we had put a bug fix in on 5/31 for this very problem. Nice fix that didn't work, damn those IT people.
Of course, this took about 30 minutes of me on the phone pacing the clubhouse as people are arriving for our evening of fun and frolic. Fortunately, Steven took over my greeting duties and things look OK at the moment.
Until we take it down again today when I expect all hell will break loose.
