Well, this afternoon was “fun” at work.
It was just before 4pm when I decided to plug one of the two power inputs on our main server, a Sun E250, into a new UPS we’ve bought for it. I checked the server’s status lights to make sure that there weren’t any problems and that it was running on both power inputs. So, I pulled the top cable out… and the server switched off. Oblocks!
Anyway, seeing as the server was down anyway, I transfered all the external disks onto the UPS and powered up. Everything seemed to boot up ok, so at 4:10pm I went off to get some tea. While drinking my tea a user came over and told me of a problem he was having. One of our hosts, and only one, was being denied access to all NFS exported filesystems. As far as I could tell, the only way to try to fix it meant rebooting the server again. That fixed it, so I left the room.
As I left I switched off the lights and suddenly the new UPS started beeping, so I turned around, turned the lights back on and went over to have a look. The power had tripped on one of the circuits in the machine room. It had taken out a few of the servers and all the network kit (I’ve not yet installed the UPS for the networking kit as I need some rails for the rack). Oblocks!
Now I had another problem, the trip switch is in a locked electrical cabinet, it’s 5:08pm and the only person with access to the electrical cabinet key has gone home. I had to break into the cabinet, reset the trip and then get all the equipment back on-line.
It was a complete pain.
late night dashes to ‘save the world’, you fixing things before people notice, justifyable breaking and entering… surely these are all the best bits of a sysadmins job.:)