Data center telecom design has moved away from traditional spanning tree networks (depicted below), which can choke the allocation of bandwidth, to fabric networks, which can provide multiple data paths. Fabric network design has had a significant impact on the cabling that runs from the top-of-rack switches in each server cabinet to the meet-me-rooms for off-site connections.

‍

Eight years ago, in a hyperscale data center, we designed around redundant backbone feeds of 288 strands of single-mode fiber-optic cable. In today’s fabric network data center, we see multiple backbone feeds of 864, 1,728, and up to 3,456 strands of fiber-optic cable. This has impacted the sizing of cable trays, the sizing of distribution rooms, and the selection of components.

On the physical layer of the network, the most significant impact has been in the layout and design of the cable distribution. Maintaining separate paths for each cable route, often referred to as divergent routing, is necessary to reduce the risk that damage to one path could affect others. When designing the cabling pathways, the data center engineer must look at a variety of options to maintain diversity.

A variety of options for divergent cable routing

The following images represent a cross-sectional area of the data center showing five options for internal cable trays and combinations of cable trays and underground conduits. Each provides four diverse routes for a fabric network with four network planes.

Option 1: Four divergently routed cable trays striped across the building. This option provides good separation between each cable route but can be difficult to implement if the building is not planned for four aisleways to access the trays.

Option 2: There are two main paths for cable tray. This is simpler to implement, with only two main aisles in the building, but presents a higher risk with trays being stacked. This option is easier to implement in design and construction.

Option 3. Two trays overhead and two trays in walkable trenches in the main corridors. This works with a data center model that has two main aisleways. There is good pathway separation, but trenches add to the cost of construction.

Option 4. Two trays overhead and two conduit banks underslab in the building. There is good separation in the paths but underslab conduits don’t work well with pre-terminated cables. Depending on the length of the duct run (over 600 feet), midpoint cable pull points may need to be added in the way of underground vaults of above-ground rooms.

Option 5. Two trays overhead and two duct banks underground outside the building footprint. There is good separation in the paths, but underslab conduits don’t work well with pre-terminated cables. It is easier to install cable pulling vaults outside the building footprint than under the building.

A decision that must be made early

Each option has benefits, challenges, risks, and cost differentials in the design and construction of the data center. The decision of which approach to employ in your facility must be made in conjunction with major programmatic and infrastructure decisions — at the earliest phase of concept and schematic design. For example, in option 1, the configuration of the main corridors in the building may determine whether it’s possible to have four separate overhead routes in the building.

Fabric network data centers present a level of complexity that requires low-voltage engineers to be involved in the design process from the very beginning. We see more savvy owners putting full multi-disciplinary design teams together in this recommended manner prior to design kickoff. Where this doesn’t occur, it falls to strong design project managers to identify these types of potential challenges and ensure that the necessary collaboration occurs to develop solutions.

Continue the conversation

This is a large topic that can be discussed in much greater detail. This post is only the beginning of that discussion. Please reach out to continue the conversation with your experience, recommendations, or challenges.