August is monsoon season up here and so we get a lot of rain and lightning this month. It was heavy enough a couple weeks ago to knock out our power for a few hours. Unfortunately, whatever bandaid they applied didn't last long, and yesturday the storm knocked out the power again. Now last time the UPS kept running everything in the basement just fine (if you don't mind a high pitched annoying whine from the alarm), and I ran my laptop for another hour before my ears started bleeding and I decided to take a break. This time however, things were not fine.
I had not been downstairs at the time of the strike since a delivery truck had just dropped off 16 pounds of legos and Max had dragged me up to check it out (he had patiently waited all week for this latest addition to his growing collection). When I checked back in a few minutes later, every open socket connection on my laptop had failed (both local and remote). I still had a link light on my laptop, and all the other local servers, switches, and routers were still up, but I couldn't talk to anything. I tried routing through a different switch, but then I noticed that the switch link to the router was dark, and my laptop couldn't sustain a connection when plugged straight into the NAT box (it kept blinking on and off).
None of the servers could see each other or the internet either, but by now the stupid UPS alarm was getting on my nerves, so I shut everthing down and went back upstairs to play legos some more. Finally, several hours later, they got the power fixed (again), and I went to survey the damage. Things still weren't working right off so I notched things up to the next level. When multiple things are suspect, its best to seperate out the tests by either trying things out individually (which in the case of a network is a little hard to do), so the next best thing is to bring in other equipment. Luckily I had a spare hub and laptop upstairs which wasn't plugged in at the time of the strike, so it was more likely to be working.
Step one was to plug my laptop and the test laptop into the test hub and see if my laptop was still working at all. Success number one was that the two laptops could talk to each other. This was a relief, as a spare mini-PCI card is not something that can be picked up locally in Flagstaff. I could also talk to the netgear box which was a relief, as previously neither my laptop nor the 3com switch could talk to it. For some reason one of my servers would show up on the 3com, but the others wouldn't, so I wasn't too hopeful about it working. I linked the hub and the switch together, and I could talk between the laptop and the one server, but the other one wouldn't turn up. When I plugged one of the other servers into the hub it worked ok, so I'm still not clear where the problem is there.
I tried plugging the one that was working in the switch into the hub, but it wouldn't work in the hub. Strange. Then I tried switching ports on the hub and it did work. Oh great, my working hub has a bad port on it too. Well, there's enough working slots that I can get my critical servers and laptop up. I'll have to tackle a long term solution when I'm more calm.
