| Sun Microsystems Data Center Site Planning Guide |    
|
| Sun Microsystems Data Center Site Planning Guide |
|
| C H A P T E R 6 |
|
Facility Power Requirements |
To prevent catastrophic failures, the design of the power system must ensure that adequate power, within appropriate limits, is provided to the computer hardware. Sun Microsystems suggests that all power to Sun equipment be derived from dedicated electrical distribution panels. Electrical work and installations must comply with applicable local, state and national electrical codes.
Sun Microsystems makes every effort to minimize the effects of power failures and interruptions to the hardware. However, if the computer equipment is subjected to repeated significant power interruptions and fluctuations, it is susceptible to a higher component failure rate than it would be with a stable power source. Provide a stable power source to reduce the possibility of component failures.
Power quality issues are often difficult to identify, and can be even more difficult to correct. Often, the symptoms are very similar to those caused by other environmental problems or software issues. The most effective means of mitigating power problems is through the appropriate design of the system. The goal of the power system design should be to provide power, at appropriate levels and quality, without interruption. Integral to this task is the incorporation of sufficient redundancy, and the elimination of single points of failure. While Sun Microsystems equipment is designed to be robust and tolerant of most electrical problems, a thoughtfully designed power infrastructure is an important part of any facility. The following areas should be addressed in the design of the power systems for a computer room.
Ideally, multiple utility feeds should be provided from separate sub-stations or power grids. While not essential, this provides back-up and redundancy. The criticality and extent of this issue should be determined through an examination of the size and role of the data center.
An Uninterruptible Power Supply (UPS) should be installed to carry:
On-line UPS that runs continuously should be used as opposed to an off-line unit. The on-line UPS filters, conditions, and regulates the power. Battery back-up should be capable of maintaining the critical load of the room for a minimum of 15 minutes during a power failure to allow for the transfer of power to an alternate feed or generator.
If a UPS is not used as a bare minimum, then surge suppression should be designed into the panels and a stand-alone isolation/regulation transformer should be designed into the power system to control the incoming power and protect the equipment.
Backup power generators should be installed to carry the load of both the computer equipment, as well as all necessary support equipment (such as air conditioners). Depending upon the criticality of the site, it may be acceptable to use the UPS and multiple utility feeds without generators.
The power system design should provide the means for bypassing and isolating any point of the system to allow for maintenance, repair or modification without interruption to systems operations. The system should be designed to avoid all single points of failure.
The power distribution equipment for computer applications should be installed as close as reasonable to the load, as depicted in FIGURE 6-3. All loads being supported need to be identified and evaluated for compatibility with the computer equipment. Heavy loads that are cyclic, such as elevators, air conditioners, and copy machines should not be connected directly to the same source as the computer room equipment.
|
Note - Source FIPS PUB 94, Guideline On Electrical Power for ADP Installations |
Most large commercial buildings will be supplied with three-phase power. Most data centers are comprised of both three-phase and single-phase equipment. Single phase equipment requires that steps be taken to ensure that the system is balanced (the phase currents are nearly equal). This will require forethought in wiring design, and a balancing of equipment on the phases. The phase currents will need to be monitored, often accomplished by the UPS, and taken into consideration in the planning of the electrical load distribution.
In addition to load balancing, the design must take into account the nonlinear loads of computing equipment. Most Sun AC-powered equipment has power factor correction that meets EN6100-3-2 as part of the power supply. Installations need to consider other equipment that is non power factor corrected and appropriate deratings need to be applied to the wiring, distribution equipment, and transformers.
Separately derived systems have no direct electrical connection between the output conductors and the input conductors. Separately derived systems are required by the NEC to have a load-side, neutral-ground bond that is connected to the grounding electrode system. All equipment grounding conductors, any isolated grounding conductors, neutral conductors, and the metal enclosure of the separately derived system, are required to be bonded together and bonded to the grounding electrode conductor. Visual inspection and measurements with a ground impedance tester can be used to determine the quality of these connections.
Grounding is an extremely important electrical consideration for the proper operation of electronic equipment. The ground conductor provides a designed path to ground (earth). Grounding design in a data center environment must address both the electrical service as well as the equipment. Grounding design should comply with Article 250 of the National electrical Code (NEC 250 -- Grounding) unless superseded by local codes. The National Electric Code (NEC) section #250-51, Effective Grounding Path states:
The path to ground from circuits, equipment and conductor enclosures shall: (1) be permanent and continuous; (2) have capacity to conduct safely any current likely to be imposed upon it; and (3) have sufficient low impedance to limit the voltage to ground and to facilitate the operation of the circuit protection devices in the circuit. The earth shall not be used as the sole equipment grounding conductor.
A properly designed grounding system should have as low an impedance as is practically achievable for proper operation of electronic devices as well as for safety. It is also important that the ground should be continuous from the central grounding point at the origin of the building system. Electronic equipment can be sensitive to stray currents and electronic noise. It is important to utilize a continuous, dedicated ground for the entire power system so as to avoid a ground differential between various grounds being used.
All metallic objects on the premises that enclose electrical conductors or that are likely to be energized by electrical currents (e.g., circuit faults, electrostatic discharge, and lightning) should be effectively grounded for reasons of personnel safety, fire hazard reduction, protection of the equipment itself, and performance. Solidly grounding these metallic objects will facilitate overcurrent device operation and permit return currents from EMI filters and surge suppressors, connected line-to-ground or line-to-chassis, to flow in the proper fashion.
The common point of grounding can be connected to any number of sources (water piping, driven earth rod, buried grid, building steel, etc.) at the service entrance. It is important that whatever the source, the ground is carried through the entire system from this source. Ideally, the central point of grounding at the service entrance will be connected to multiple ground sources, such as the building steel, buried grid and cold water piping. If they are connected at the same point, there is no potential for ground loops, yet a redundancy is achieved. A water pipe used for a ground could rupture, building steel could have accumulated resistance over several floors. By tying into all of these, the possibility of a disruption is greatly minimized.
Contrary to popular notion, the National Electrical Code (NEC) does not favor the use of a ground rod. Section 250-81 states that if the following items are available, they are to be used first and bonded together to form the grounding electrode system:
Section 250-83 states that if none of the above is available, then and only then can you ensure any of the following:
The most elaborate grounding system may not perform satisfactorily unless the connection of the system to earth is adequate for the particular installation. It follows, therefore, that the earth connection is one of the most important parts of the whole grounding system. It is also the most difficult to design.
The connection to earth or the electrode system, needs to have sufficiently low resistance to help permit prompt operation of the circuit protective devices in the event of a ground fault, to provide the required safety from shock to personnel, and protect the equipment from voltage gradients that may damage the equipment. Resistances in the range of 1 to 5 ohm range are generally found to be suitable for computer center installations. The 25 ohm value noted in the NEC applies to the maximum resistance for a single made electrode for safety requirements. This should not be interpreted to mean that 25 ohms is a satisfactory level for computer grounding systems.
Electronic equipment is required by the NEC and local codes to be grounded through the equipment grounding conductor and bonded to the grounding electrode system at the power source. The impedance of the equipment grounding conductor from the electronic equipment back to the source neutral-ground bonding point is a measure of the quality of the fault return path. A high impedance measurement indicates that there are poor quality connections in the equipment grounding system. Properly installed and maintained equipment grounding conductors will exhibit very low impedance levels. Recommended levels for equipment grounding conductors is to have levels that meet the code requirements and has a value of less than 0.25 ohm.
Where it is accessible, all structural steel should be electrically grounded and bonded into a single electrically conductive mass. Such grounding and bonding may be by structural means, such as welding, bolting, riveting, or by grounding and bonding jumpers. Earthing of the structural building steel system is also recommended. The structural building steel system is required to be bonded to the grounded conductor of the incoming ac supply system at the service entrance as well as to the equipment grounding conductor system and the main (metallic) cold water piping system.
The use of any separate, isolated, insulated, dedicated, clean, quiet, signal, computer, electronic, or other such improper form of earth grounding electrodes for use as a point of connection of the Isolated Ground (IG) Equipment Grounding Conductor is not recommended. These IG grounding schemes may not meet code requirements for effective grounding. The general perceived need for an isolated earth grounding electrode scheme in relation to the IG method is not based on good engineering practice. Isolated earth grounding electrode designs have no means for limiting the potential developed across the intervening impedance in the commonly shared ground medium when a current is caused to flow through it. As a result, lightning may create conditions of several thousands of volts between two or more grounding electrodes causing a safety hazard and damage to associated equipment.
The behavior of an independent grounding conductor is very different from that of a power circuit grounding conductor. These installations need to be treated specially and have each end of the metal sleeve bonded to the associated ground conductor. This requirement is due to the one way current flowing in the ground conductor acting on the metal sleeve as if it is a magnetic core of an inductor. Failure to properly bond these structures will allow for substantial voltage differences to occur on the associated equipment creating a safety hazard and potential equipment failure. These installations are commonly found on metal conduits entering the computer rooms or where physical protection is being applied to ground wires.
A signal reference grid should be designed for the computer room. This provides an equal potential plane of reference over a broad band of frequencies through the use of a network of low-impedance conductors installed throughout the facility.
Certain access floor designs can be used to provide the signal reference grid. The integrity of the grid must be maintained in order to achieve a reliable reference plane. Typically, this will involve a floor system with bolted stringers. Snap-on stringers or "stringerless" designs are also available, though these cannot provide the same integrity. A supplemental grid constructed of #4 AWG copper bonded at its intersections should be installed (FIPS PUB 94). For bolted stringer systems, 24 inch or 36 inch grids should be sufficient. In snap-on or stringerless systems where the floor grid does not provide the signal reference grid, the signal reference grid should be constructed with 24 inch intersections. The entire grid should be tied into the common point of ground.
The following is a list of recommended pratices for an SRG see FIGURE 6-4:
1. Follow applicable codes and standards for safe grounding. There is no conflict between safe grounding for people and effective high frequency grounding for sensitive electronic equipment.
2. Select a suitable SRG approach and assure that it is installed and maintained properly.
3. Permanently bond the SRG to all accessible building steel and to each metallic path crossing the plane of, or within 6 ft. of, the SRG.
4. If a single point of entry for power and grounding cables into the space exists, then single-point grounding of the area to local building steel is acceptable if this grounding system is verified periodically by skilled test personnel. The equipment protected by the SRG should be multipoint grounded to the SRG.
5. Bond the SRG to each piece of sensitive equipment.
Sun equipment is typically not classified as sensitive electrical equipment and proper application of power cord grounding meets product grounding requirements. Supplemental bonding of the equipment is at the customers discretion.
6. Bonding connections to the SRG should be as short as practical with no sharp folds or bends.
7. Connections of sensitive electronic equipment should not be made to the outermost grid conductor of the SRG. Heating, ventilation, air-conditioning equipment and panelboards should be connected to the outer most grid conductor. Critical equipment should be located and bonded to the SRG at more than 6 feet away from the building steel or other potential lightning paths.
8. All separately derived systems, serving the equipment on the SRG, should have their power grounding point bonded to the SRG.
9. All cooling, heating, ventilation, and air conditioning equipment, associated piping, metal wall studs, panelboards, switchboards, transformers, and similar equipment within the protected area, shall be bonded to the SRG.
10. No connections should be made to remote or dedicated earth grounding points or any similar attempt at separate earth ground paths.
11. All data and power cables should lay on or be very close to the SRG.
12. Documentation should be complete in all details, including the proper grounding and bonding of heating, ventilation, and air conditioning equipment, piping, raceways, and similar items. The responsible engineer should not expect the installers to complete the design.
|
Note - Source IEEE STD 1100-1992 Powering and Grounding Sensitive Electronic Equipment |
The quality of the incoming power can be a governing factor in the determination of the system components and design. A well-designed power and grounding system can be instrumental in maintaining appropriate conditions and avoiding unplanned outages. Numerous factors can disrupt, degrade or destroy electronic systems. High frequency, high amplitude noise, high ground currents, a low power factor, surges or sags in voltage, harmonic distortion or numerous other factors can all affect proper functioning of electronic components. Each individual hardware design will have specific tolerances, but the following chart from FIPS PUB 94 is a useful guide when other more specific information is not available. Additional information regarding the tolerances of specific models of Sun Microsystems hardware can be found in this Chapter.
|
Note - Source FIPS PUB 94, Guideline On Electrical Power for ADP Installations |
When the power source does not meet the equipment requirements additional hardware is required to allow the system to operate in its environment. TABLE 6-2 gives a typical cross reference of power issues and possible solutions that can be evaluated for specific problems.
|
Note - Source IEEE STD 1100-1992 Powering and Grounding Sensitive Electronic Equipment |
All of the Sun Microsystems equipment models covered in this guide are of a single phase design. Nominal voltage ranges can be 100/120 or 200/240 depending on individual unit specifications.
Most Sun equipment is designed to operate within a current frequency range of 47-63 Hz. This range should provide adequate flexibility to accommodate differences in power sources both in the united states (60 Hz) and abroad (50 and 60 Hz).
Potential harmonics problems can be caused in a data center by the interaction of the electronic equipment with the power loads or by switching power supplies. Numerous factors, such as harmonic distortion, load imbalance, high neutral current and a low power factor can all affect the power distribution system. The results can lead to decreases in efficiency and reliability that can adversely affect overall computer operations. Harmonics mitigation, however, is extremely difficult as it is contributed to by the computer hardware, and any changes in the room load or configuration can change create new problems and negate any previous corrections.
Sun Microsystems equipment has been designed to address the concerns of Harmonic Distortion, and should be generally compatible with other modern equipment of similar designs. Other equipment designs that do not have the advantages of currently accepted design corrections should ideally be isolated on separate circuits.
Each branch circuit should contain a dedicated grounding conductor that is grounded to earth at the central grounding point. While the conduit can be used as the safety ground, a separate wire is recommended. All wires must be of the same gauge, and the neutral wire must be connected to ground at the source. All national and local wiring codes must be followed. The hardware will achieve its path to ground through the power cord. It may not be necessary, but is highly recommended, to strap each unit of hardware to the signal reference grid.
Voltage spikes are rises in the voltage that are caused most often within the power distribution circuits by items turning on and off, such as the cycling of compressor motors. A UPS, and filtering on the primary side of the system, will normally stop most spikes originating upstream from the UPS. If, due to design decisions at a particular site, a UPS is not being used then some other form of regulation or surge suppression should be designed into the system. Circuits serving sensitive electronic equipment should be isolated from such influences. Large spikes can interfere with energy transfer or the associated electrical noise can cause signal corruption.
Lightning protection systems should be designed, installed and maintained in accordance with NFPA 78, Lightning Protection Code or any superseding local or National Codes. The impact of lightning on computer installations can either be direct or indirect. The effects may be on the utility power feed, directly on the equipment, or through high frequency electromagnetic interference or surge currents.
Lightning surges cannot be stopped, but they can be diverted. The plans for the data center should be thoroughly reviewed to identify any paths for surge entry into the data center. Surge arrestors can be designed into the system to help mitigate the potential for lightning damage within the data center. These should divert the power of the surge by providing a path to ground for the surge energy. Protection should be placed on both the primary and secondary side of the service transformer. It is also necessary to protect against surges through the communications lines. The specific design of the lightning protection system for the data center will be dependent on the design of the building and utilities and existing protection measures.
Sun equipment is designed to meet the surge requirements of IEC 1000-4-5.
Within the data center, a single point of disconnect for all electronic systems is required by NFPA 70 and NFPA 75 at each point of entry. Multiple disconnect means for these power systems are also acceptable, but in either case, the switches must be unobstructed and clearly marked. Protective covers may be placed over the buttons to avoid accidental contact, but access cannot be locked out. This switch should disconnect power to all computer systems, HVAC, UPS and batteries. If the UPS is located within the data center, the disconnecting means must stop power to this unit. If the unit is located remotely, the disconnecting means must stop the supply from this unit to the room. Exceptions to this rule include small UPS (750 volt-amperes or less) within the room.
Even though it is not required by code, it is still recommended that all power sources in the room, including the exceptions listed, be controlled by the disconnecting means so as to provide the greatest degree of safety to the hardware and personnel. It could be these systems, or the gear they supply, that are the cause of the fire.
All wiring and cabling should be designed and installed in accordance with the requirements outlined in this document and the National Electrical Code (NFPA 70) or superseding national or local codes or standards. In some cases, the recommendations provided in this document may exceed the minimum recommendations outlined in NFPA 70.
All wiring and cabling should be run in an orderly and efficient manner. This is particularly important beneath the raised floor. The nature of the data center requires frequent modifications, and it is extremely important that obsolete cabling be removed so as to avoid air-flow obstructions and to allow for future installations. Orderly cabling will minimize the potential for disruption due to disconnection of cables when work is taking place.
Whenever possible, data and power cables should intersect perpendicularly so as to minimize the potential for generation of disruptive electronic noise on the lines. For the same reason, try to run data and power cables at least 6 inches apart when they are parallel. Unused or partially connected cables can also act as antennas. Cables should not be cut in place and left in the subfloor void.
Temporary extension cords should not be used in the subfloor void. If these are used above the raised floor, proper safety precautions should be taken to ensure that they are not a tripping hazard, and that they will not be damaged.
Do not use power strips with switches, surge protectors and fuses within the data center. These can counter some of the designed protections of the hardware, and are not necessary. In addition, the switches on these strips can easily be tripped, causing unwanted outages. This is a particular susceptibility when they are placed on the raised floor beneath desks, or on the edges of aisles.
Electromagnetic interference (EMI) is radiated and conducted energy from electrical devices that produce electromagnetic fields, and can come from a variety of sources. The electrical noise currents associated with EMI can interfere with the signals carried by the electronic components and cabling of some equipment. Sources of EMI or Radio frequency interference (RFI) can be found within or outside the computer room environment. Airports, telecommunications or satellite centers and similar facilities are common external sources of EMI/RFI. Internal sources include the hardware itself. Sun equipment will be tolerant of most common EMI/RFI levels. If high levels are suspected, it may be necessary to conduct a study to determine whether shielding or other remedial actions are necessary.
Electrostatic Discharge (ESD) can be extremely damaging to electronic components. By definition, static electricity is an electrical charge at rest. The discharge of this built-up energy can cause numerous problems.
Today's electronic equipment has a much denser geometry, and is composed of thinner, more easily damaged, materials. Changes in the design, manufacturing process and materials used has improved ESD sensitivity considerably. While Sun equipment has been designed to be tolerant of significant ESD events, it is important to take precautions in the design of the computer room to minimize exposure to discharges. This is particularly important when unprotected components are being handled during installations or upgrades.
Damage caused by ESD can take the form of catastrophic failures, but is more often low-grade damage that may not show up during initial setup, but can make the unit more susceptible to a later failure. Cumulative degradation of the components can also occur as the result of repeated, low voltage exposures. These types of problems are very subtle, and extremely difficult to detect.
There are numerous means of controlling static generation and ESD. The following is a brief listing. A detailed site-specific evaluation should be conducted to determine the most appropriate program for each controlled area.
|
Note - Source Simco, A Basic Guide to an ESD Control Program for Electronics Manufacturers |
High speed electronic systems and equipment may be more sensitive to disturbances in the ac power system than conventional loads. The effects of power disturbances on sensitive electronic equipment can take a wide variety of forms including data errors, system halts, memory or program loss, and equipment damage. In many cases it is difficult to determine whether the system hardware and software malfunctions are actually caused by disturbances in the power system supplying the equipment. Usually some level of survey and analysis of the power system is required to determine if the power disturbances are effecting the system performance. The basic objectives of surveys and site power analyses are:
It is important to keep these approaches in mind when a site is experiencing problems that appear to be power related. All too often, corrective action is installed in a "shotgun" attempt to solve the problem. Although this method will sometimes minimize the problem, in a majority of cases, it may do little or nothing to solve the problem and can even aggravate conditions.
An effective site power survey should also include the participation of the local electric utility. Utility personnel can provide site-specific information on disturbances that can occur on the system and assist in the evaluation of equipment to meet the site requirements.
Sun recommends that large new installations and sites having problems contract with a trained professional to conduct a site survey for verification of installation and mitigation of problems that may affect the installed equipment.
| Sun Microsystems Data Center Site Planning Guide | 805-5863-13 |
|
Copyright © 2002, Sun Microsystems, Inc. All rights reserved.