What to do in the event of a power outage.
Exactly how you deal with a power outage will depend on how long the power is out and what problems you find. This section provides procedures to use in the fairly simple case when the power is not out long enough to outlast the UPS units on the servers, and it provides a little info on what to do with a longer power outage.
The majority of the computers in Sitterson run on a switched power grid, using the orange wall sockets. The rest, mostly servers in SN122, have some amount of backup power. The department’s switched power grid is automatically switched off when there is a significant power fluctuation. This prevents damage that would otherwise be likely to occur if the power were allowed to flicker on and off, as in a brown-out. To get the power back to the systems on conditioned power, you must switch it back on in the main utility room, SN042. Instructions for doing this are given below, after the steps which must precede turning on the power.
Before you switch the clean power back on, you must first verify that 1) power to Sitterson is back on and stable and 2) the server systems in SN122 are up. To verify that the power is on, just check any office light. If it will come on, you have power. To verify that it is stable, call Physical Plant at 962-8394 and ask. After hours, you will be switched to University Police. Ask them if it is stable or to check with Physical Plant.
To verify that the important systems are up, go into SN122. In particular, check to see if bristol and charlotte are up. Then check the PC servers in the big blue computer cabinet and make sure they are up. If all these systems are running, then the power outage was not long enough to take the servers down, in which case one person should be able to handle things. If these systems have shut down, then you will need to call in the troops, at least one PC admin and one UNIX admin, for assistance. You should not proceed to turn on the clean power until you get the servers up. In this case you should also put a preliminary sign up on the exterior doors saying that the power went down long enough to outlast the UPSs and bring the servers down.
Once you have verified that power to Sitterson is on and stable and the servers are up, you can turn on the switched power. The best way is to contact UNC Facilities, or after hours, Campus Police and ask that an electrician be dispatched to reset the grid. They should be here within 20-30 minutes. But if you need to reset it yourself, unlock the lock box in SN128 and remove the key labeled “Mech Room”, you can use that to unlock SN042. Instructions on resetting the breakers is on the north wall.
Once the switched grid is back, most computer and network switches around the building that were off will start coming back on. At this point, you need to check and make sure things came up okay. Check an office PC and see if it is up and able to see other systems in Network Neighborhood. If so, then DNS and DHCP should be working fine. Log in to a UNIX machine and verify that you can see your home directory. If so, at least one of the AFS servers is okay. Run /usr/local/bin/netcheck. This will report on network devices that are down. Don’t worry about the experimental networks (which are so indicated in the output), but if other devices report down, you should go to the indicated network closet and power cycle the device. This normally fixes it. Check a telephone and make sure it is working okay. If not, call David Musick from a phone outside Sitterson.
If things don’t come up, use your judgment about what to do and whom to call.
The last thing is to let people know the status of the systems. Post a note to firstname.lastname@example.org, and post signs on the exterior doors with the same information, being sure to include the date and time that you made the sign and your name. Depending on what is not working, it may be reasonable to go around the building and inform anyone who is there of the status of operations.
Obviously a lot of things can go wrong, but they usually do not (other than the power going off in the first place). Use your best judgment about what to do, when or if you call in help, whom to call, how to notify