How Many IT People Does It Take?

We had a problem Thursday night and here's the list of people who worked on it:
Night Operator
Sybase DBA (me)
Legacy System Programmer
IT Director (Sugar Daddy)
Unix Administrator
Former Night Operator/now PC Tech
Senior Executive VP of the company

Man, it was brutal.

I went to bed at 11 and got called by the Night guy at 11:45 because our shipping process was running slow...at the rate it was going, we would have taken a full week to process one night's orders. This is not good. I checked my server and database and the legacy system's server but couldn't find anything.

We conferenced in the Legacy guy and he had no ideas (he's pretty new).

Then, I walked over to Sugar Daddy's room and banged on his door to get him up and involved. He started rooting around and we let the Legacy guy go back to bed.

Next in was the Unix guy whose internet connection was down so he couldn't look at anything directly so I had to follow his verbal directions on some tests and relay the results back to him. He couldn't see anything so we decided we should quit the process and restart it. However, since it's all automated that means we have to modify the automation script. Since the Night guy is new, we had to call the former Night guy.

Now, there's four of us on a conference call and we're hashing stuff out when we hear.

snore...snore...snore

The former Night guy had fallen asleep on the phone. We tried yelling to wake him up but it didn't work. Then, the new Night guy tried calling his cell phone.

snore..ring...ring...snore...ring...ring...snore

At this point, we're all busting up and wishing we had a way to tape it so we could torture him with it going forward. Since he's obviously not going to be a help, the Unix guy decides to drive into the office so he can modify the automation.

Sugar Daddy and I decided to call in the Big Kahuna, our Exec VP. He's worked his way up through the company and has been our PC Tech, Legacy Programmer and IT Director and has forgotten more than most people currently know.

He was soooo excited that I called. We're always kidding him about being out of touch with IT now that he's a bigwig so it was a kick for him that we were asking for help. He wrote some test programs and finally discovered that the problem was with a completely different server and application that we use to email our customers with package tracking information. Once that was restarted, we were back up and running.

That was 2:45 AM. Nice way to spend three hours in the middle of the night. I still managed to get into work by 8:30 but I was dang tired Friday night and happy to hit the sack around 10.

Popular posts from this blog

TV At Its Finest

This Is How Fat Thinks