Well, That Was Not Very Fun
I was sitting in my office at 5:3o tonight when, pop, all of the lights went out. The only thing emitting light was my monitor because my machine is on its own UPS. The emergency lighting didn't even turn on.
We gave it a few minutes then headed to the server room to see what servers were still up. Of course, there was no light in that room, either. We checked that everything looked to be still up and running on the UPSs in there.
I got on the horn to our DBA, Unix Admin and Network Admin. While I was trying to get them up to speed, I got lots of text messages from the Big Kahuna and Sugar Daddy about the outage. Lots of multitasking going on.
Our two main buildings were both out and there were lots of employees milling around. The power came back on for a few seconds then was out again. And, stayed out.
OK, here's the deal. Our office is right by the airport. Two gridded areas went out at about the same time that President Obama was scheduled to land. He was the scheduled speaker at ASU's graduation ceremony tonight.
Coincidence or nefarious plot?
I suspect the latter with two possible sources: terrorists or the government. Either seems likely to me since I watch 24. I know what those people can do with a Blackberry and a Cisco router!
Regardless of the source, it was a major PITA. The power finally came back on and everyones PCs came back up. The users gave it about 4 minutes then they were looking for me to explain why all of the systems weren't working.
OK, just because your PC is on, that doesn't mean that all of the servers that it's connected to are also up. In fact, most of them were still down but it was hard to tell because so was our (Cisco) router.
At this point, the Unix and DBA guys were in and we started bringing machines back up. It was a tedious process. I tried to shield two of our guys - one is away for the week at a conference and the other has been out sick for two days. What I didn't realize is that one of the users would call out of town guy who in turn called sick guy so he came in. I don't think either knew that the rest of us were already on the job.
A couple of hours later, we had the majority of the systems back online and I felt good enough to head for home.
Of course, I live right by ASU and my fear was that I'd get caught in the Presidential motorcade taking Obama back to the airport. I lucked out as he was still talking while I was in transit so I made it home without any delays.
Good thing, too, as I told George that I would be home right after work. She doesn't like it when I don't warn her about coming home late.
We gave it a few minutes then headed to the server room to see what servers were still up. Of course, there was no light in that room, either. We checked that everything looked to be still up and running on the UPSs in there.
I got on the horn to our DBA, Unix Admin and Network Admin. While I was trying to get them up to speed, I got lots of text messages from the Big Kahuna and Sugar Daddy about the outage. Lots of multitasking going on.
Our two main buildings were both out and there were lots of employees milling around. The power came back on for a few seconds then was out again. And, stayed out.
OK, here's the deal. Our office is right by the airport. Two gridded areas went out at about the same time that President Obama was scheduled to land. He was the scheduled speaker at ASU's graduation ceremony tonight.
Coincidence or nefarious plot?
I suspect the latter with two possible sources: terrorists or the government. Either seems likely to me since I watch 24. I know what those people can do with a Blackberry and a Cisco router!
Regardless of the source, it was a major PITA. The power finally came back on and everyones PCs came back up. The users gave it about 4 minutes then they were looking for me to explain why all of the systems weren't working.
OK, just because your PC is on, that doesn't mean that all of the servers that it's connected to are also up. In fact, most of them were still down but it was hard to tell because so was our (Cisco) router.
At this point, the Unix and DBA guys were in and we started bringing machines back up. It was a tedious process. I tried to shield two of our guys - one is away for the week at a conference and the other has been out sick for two days. What I didn't realize is that one of the users would call out of town guy who in turn called sick guy so he came in. I don't think either knew that the rest of us were already on the job.
A couple of hours later, we had the majority of the systems back online and I felt good enough to head for home.
Of course, I live right by ASU and my fear was that I'd get caught in the Presidential motorcade taking Obama back to the airport. I lucked out as he was still talking while I was in transit so I made it home without any delays.
Good thing, too, as I told George that I would be home right after work. She doesn't like it when I don't warn her about coming home late.
