I've Seen The Light At The End Of The Tunnel...
Unfortunately, the light is from a damned old train.
I really thought it was easing up at work. Sugar Daddy planned on taking last Thursday and Friday off (his first two vacation days this year) and things seemed calm.
Yeah, like the calm in the eye of the hurricane, maybe.
I was sitting in my jammies on Thursday morning reading the paper when my cell rang. All of the work numbers ring Animal House and I can't tell you how much I regret making myself hate that song now but that's what was playing at 7:40 AM.
I answered to hear a very plaintive "Help!" coming through the line.
Without getting too technical, I have to say that my database is really, really stable. So, when they call to say there's a problem I'm usually like, "Who screwed it up?"
Turns out the hardware that controls the disks screwed it up. For some reason (sunspots have actually been blamed), the controller that makes the disks available choked. Our Admin was on site and checking it out and he had to take the server down hard.
That's very scary to a Database Administrator (a title I still somewhat hold in my wearing of many hats at work role).
He had to call in support and there wasn't much I could do but I tried to keep up on the action while getting ready for work.
It kind of went like this:
Jump in the shower.
Run out of shower to check my Instant Messages to get an update.
Run back to the bedroom to put on my underwear.
Back to the living room to check IMs on my computer.
Call my VP and Sugar Daddy to let them know what was going on.
Back to the bathroom to dry my hair.
Back to the computer to Message.
Back to the bathroom to dress.
Answer cell phone call from VP.
Back to the computer.
Back to the bathroom to curl my hair and put on makeup while calling into work on my cell phone.
Final IM to say I was on my way into work.
While I was getting out the door, Sugar Daddy called me to say he was coming in. He was worried that the hard shutdown was going to be disastrous and that we might have to reload all of the data. That would have shut us down for the rest of the day. Yikes.
When I got to work, I was told the answer was to unplug our Baydel drives and let the battery drive so that the transaction that it as choking on would die. So, I jumped in and out of the server room to stare at the machine that kept saying "On Battery". Frakking machine!
It finally died, we recycled it and the server could see the drives again.
Unfortunately, that wasn't the end of it. Sugar Daddy tried to bring the databases back online but they no workie. We ended up on the phone with support as we coaxed each of our 20+ databases to start cooperating.
I had sent out a company wide email asking everyone to stay out of the system while we were trying to fix it. Of course, I was ignored and the users kept goofing us up because they repeatedly tried to log in. I finally sent a "DO NOT LOGIN" message that kept 95% of them out.
Did I mention it was the last day of the month? It was really helpful when people told us that...like we chose for the hardware to act up at that point in time!
We (and by "we", I mean SD ran all of the commands and pretty much refused to let me do anything including sitting in his chair while he went to the bathroom) finally got everything back up (and somehow, thanks to the database gods, with no loss of data) nearly 5 hours later. We then had to suffer through getting the relevant data back into cache before everything was better.
While all this was going on, T, one of my lunch buddies tried to blackball me from the group because I had to cancel. WTF? The company is down and I need to work on it and he's bashing me for skipping lunch? I haven't even begun to pay him back for that one.
We got through the rest of the day and I hit the sack about midnight that night because I was up late reading.
I got called at 10PM, 1 AM and 2AM (and was up for about 3 hours then - I was pleasantly surprised to find that my paper gets delivered around 3) then cat napped in 1/2 and 1 hour increments. I think I got a total of 4 hours sleep. Unfortunately, the problem that night was of my own making, damn it. I did spend more time IMing and emailing in my underwear. I'm getting quite comfortable with that.
I would have slept in but I had an 8:30 meeting. I pulled into the parking lot at 8:15 to find someone in my parking spot which didn't go over well. I parked in the south 40 of the parking lot and was about to head into my meeting when I got another "Help!" call. This time, it was a runaway process filling up our tempdb space - a known issue that I seemed to be the only one who could recall was a common cause. I made the necessary phone calls and was only 10 minutes late to my meeting. It ended up being a very long day.
Aye carumba! It just never ends...
I really thought it was easing up at work. Sugar Daddy planned on taking last Thursday and Friday off (his first two vacation days this year) and things seemed calm.
Yeah, like the calm in the eye of the hurricane, maybe.
I was sitting in my jammies on Thursday morning reading the paper when my cell rang. All of the work numbers ring Animal House and I can't tell you how much I regret making myself hate that song now but that's what was playing at 7:40 AM.
I answered to hear a very plaintive "Help!" coming through the line.
Without getting too technical, I have to say that my database is really, really stable. So, when they call to say there's a problem I'm usually like, "Who screwed it up?"
Turns out the hardware that controls the disks screwed it up. For some reason (sunspots have actually been blamed), the controller that makes the disks available choked. Our Admin was on site and checking it out and he had to take the server down hard.
That's very scary to a Database Administrator (a title I still somewhat hold in my wearing of many hats at work role).
He had to call in support and there wasn't much I could do but I tried to keep up on the action while getting ready for work.
It kind of went like this:
Jump in the shower.
Run out of shower to check my Instant Messages to get an update.
Run back to the bedroom to put on my underwear.
Back to the living room to check IMs on my computer.
Call my VP and Sugar Daddy to let them know what was going on.
Back to the bathroom to dry my hair.
Back to the computer to Message.
Back to the bathroom to dress.
Answer cell phone call from VP.
Back to the computer.
Back to the bathroom to curl my hair and put on makeup while calling into work on my cell phone.
Final IM to say I was on my way into work.
While I was getting out the door, Sugar Daddy called me to say he was coming in. He was worried that the hard shutdown was going to be disastrous and that we might have to reload all of the data. That would have shut us down for the rest of the day. Yikes.
When I got to work, I was told the answer was to unplug our Baydel drives and let the battery drive so that the transaction that it as choking on would die. So, I jumped in and out of the server room to stare at the machine that kept saying "On Battery". Frakking machine!
It finally died, we recycled it and the server could see the drives again.
Unfortunately, that wasn't the end of it. Sugar Daddy tried to bring the databases back online but they no workie. We ended up on the phone with support as we coaxed each of our 20+ databases to start cooperating.
I had sent out a company wide email asking everyone to stay out of the system while we were trying to fix it. Of course, I was ignored and the users kept goofing us up because they repeatedly tried to log in. I finally sent a "DO NOT LOGIN" message that kept 95% of them out.
Did I mention it was the last day of the month? It was really helpful when people told us that...like we chose for the hardware to act up at that point in time!
We (and by "we", I mean SD ran all of the commands and pretty much refused to let me do anything including sitting in his chair while he went to the bathroom) finally got everything back up (and somehow, thanks to the database gods, with no loss of data) nearly 5 hours later. We then had to suffer through getting the relevant data back into cache before everything was better.
While all this was going on, T, one of my lunch buddies tried to blackball me from the group because I had to cancel. WTF? The company is down and I need to work on it and he's bashing me for skipping lunch? I haven't even begun to pay him back for that one.
We got through the rest of the day and I hit the sack about midnight that night because I was up late reading.
I got called at 10PM, 1 AM and 2AM (and was up for about 3 hours then - I was pleasantly surprised to find that my paper gets delivered around 3) then cat napped in 1/2 and 1 hour increments. I think I got a total of 4 hours sleep. Unfortunately, the problem that night was of my own making, damn it. I did spend more time IMing and emailing in my underwear. I'm getting quite comfortable with that.
I would have slept in but I had an 8:30 meeting. I pulled into the parking lot at 8:15 to find someone in my parking spot which didn't go over well. I parked in the south 40 of the parking lot and was about to head into my meeting when I got another "Help!" call. This time, it was a runaway process filling up our tempdb space - a known issue that I seemed to be the only one who could recall was a common cause. I made the necessary phone calls and was only 10 minutes late to my meeting. It ended up being a very long day.
Aye carumba! It just never ends...
