This post was written by Dr. Jeff Layton, Ph.D. | Dell HPC Enterprise Technologist
The new PowerEdge™ C8000 chassis (announced today) is the next evolution of the shared infrastructure server from Dell. It builds on the very successful PowerEdge C6100 4-in-2U server to create a more flexible and very dense server platform that works well in HPC.
A shared infrastructure server is like a cross between a blade server and a stand along rack mount server. As the name implies, the servers share infrastructure, typically power, cooling, and a chassis that houses the servers. But it stops there and doesn’t add the shared networking and management that blades typically have. The goal is to leverage the desirable infrastructure aspects of blades without having to develop networking and management hardware that can increase the cost the system.
The PowerEdge C8000 is really a family of servers that fit into the C8000 chassis. Figure 1 below shows a C8000 chassis with several different types of servers (also called “sleds” or nodes).
![]()
Figure 1 – C8000 chassis with several types of servers
The C8000 chassis itself is a 4U high chassis that is 813mm long that accommodate up to 10 “sleds.” Typically the two middle sled slots are occupied by two redundant power supplies. In Figure 1 these have white handles. Each power supply sled has two 1400 W power supplies so that up to 5600 W can be powered in a non-redundant manner (4x 1400 W). Or with full 2+2 redundancy across power supply sleds you can power up to 2800 W.
The sleds that fit into the C8000 currently are of two varieties. The first is a single-wide sled which allows up to 10 sleds to be put into the C8000 (typically two are single-width power supply sleds). There second is a double-wide sled which allows up to 5 sleds to be put into the C8000 (but again with the two center sleds occupied by power supplies, this means that you can put 4 double wide sleds into the chassis). The power supply sleds plug into a power backplane that provides power to the rest of the serves (part of the shared infrastructure design). The cooling is the classic front to back design with the fans being in the rear of the chassis.
One of the features of the C8000 family is that all cabling but the power cabling is done from the front. No longer will you have to go to the back of the chassis to disconnect a server to replace a component. Now you can do all of that from the front in the cold-aisle. That means you no longer need to use Chapstick to prevent your lips from cracking, while working on a chassis.
As I mentioned previously, the C8000 is the chassis itself, but there are a family of sled solutions that fit into the chassis. On the far left in Figure 1 is a single-width C8220 compute server which is shown below in Figure 2.
![]()
Figure 2 – C8220 single-width server sled
It is two-socket Intel® Xeon® E5-2600 server. It has 16 DIMM slots (8 per socket) for up to 256 GBs of memory per server. It also has room for two SATA 2.5” drives in the back of the sled. There is also a x8 PCI Express Gen3 customer mezzanine slot that can accommodate a variety of cards, but particularly for HPC, this can be a FDR InfiniBand (IB) card or a 10GigE or 40GigE NIC. In addition, there is a x16 PCI Express Gen3 slot that can accommodate low-profile, half height cards such as a dual-port Mellanox ConnectX-3 dual-port card or an Intel dual-port 10GigE NIC.
The C8220 also contains an embedded BMC (Baseboard Management Controller) that is IPMI 2.0 compliant. It is also Intel Node Manager 2.0 compliant. There is a single 10/100 Mbps RJ-45 connector on each C8220 sled.
Remember that with redundant power supply sleds, the C8000 can hold up to 8 C8220 servers. This gives the same density as the highly regarded Dell PowerEdge C6100 4-in-2U shared infrastructure server. However, the C8000 leverages a larger scale infrastructure simplifying the construction of HPC systems because there are fewer chassis and fewer power supplies which al reduces the number of components.
Also available is the C8220X server that builds on the C8220 server sled by adding two GPUs. In Figure 1, it is the third sled from the left or second from the right. Figure 3 below shows the sled (note that this figure shows the sled upside down so you can better see the sled).
![]()
Figure 3 – C8220X Compute/GPU sled
In Figure 3 is the compute server which is a standard C8220 sled. On the left in Figure 3 is the GPU potion of the sled which house two GPUs. The GPUs are connected to the server via two dedicated x16 PCI Express Gen3 connectors so that each GPU has full bandwidth to the server. The compute sled on the right connects to the GPU portion on the left via special cables which gives the design more freedom and density.
Currently, the C8220X ships with the NVIDIA® Tesla™ M2090 GPU computing modules. Future GPUs and other accelerators can be accommodated by the C8220X creating a very flexible accelerated compute server in a very, very compact form factor.
Recall that with redundant power supplies in the C8000, it can accommodate up to 4 double wide servers such as the C8220X. This means that in 4U of vertical rack space you can have four two-socket servers each with 2 GPUs with a full x16 PCI Express Gen3 connection to the server for a total of 8 GPUs and 8 sockets in 4U of space.
At launch there is also a storage sled called the C8000XD. In Figure 1 the storage sled is on the far right of the chassis with a handle sticking out of the front of the sled. Figure 4 below is a view of the C8000XD.
![]()
Figure 4 – C8000XD storage sled
The sled only contains disks and no compute elements. You can think of it as a JBOD (Just a Bunch of Disks) sled that connects to a compute sled for either direct disk access or RAID. As you can see in Figure 4, the disks mount vertically into the sled making connection for power and data at the bottom of the sled. The C8000XD can take up to 12x 2.5” or 3.5” SAS/SATA drives or 24x 2.5” SSD drives. All of the drive slots are hot-plug capable.
The launch configuration of the C8000XD can daisy-chain up to four C8000XD sleds giving you up to 48x 2.5” or 3.5” drives that are connected to a single compute sled. All the connections are done through the front of the sled keeping the cold aisle accessibility of the sleds.
Summary of the C8000
September 19h, 2012 is just the initial launch of the C8000. But it’s not a “point” product but rather the beginning of a family of evolving products. You will see new features and components being released over time.
At launch there will be the 4 different types of sleds in the C8000 family. The C8220 single-width compute sled, the C8220X double-wide accelerator sled, the C8000XD storage sled, the power supply sled. One of the biggest strengths of the C8000 family is flexibility. You can use any of the sleds in a single chassis, or within a rack or within a system. This means you can address a wide variety of workloads or needs within the same system.
For example, if you need a very dense pure compute platform for HPC workloads, then using the C8220 results in a very dense system with up to 80 two-socket sleds in a 42U rack.
If you need servers with large amounts of local storage for workloads such as FEM (Finite Element Methods) applications, or even Hadoop, then the C8000X can be used for large drive counts for compute servers.
If you need accelerators along with traditional CPUs, then you can use C8220X sleds. This gives a two socket server along with two GPUs that each has a dedicated x16 PCI Express Gen3 slot.
But the ways that HPC systems are being used is rapidly changing. They will always have the traditional workload of large core applications that utilize MPI (Message Passing Interface). These workloads also include large amounts of data both as input and output. But additional types of workloads such as bioinformatics - where the files are very small, but there lots of them that are used with a large number of cores - which can also include Hadoop based tools, are starting to be included in HPC systems. A second “new” workload that is also appearing on HPC systems results from researchers using tools such as Matlab, R, or Python for a wide variety of tasks including parameter studies that require thousands of cores.
These new changes to traditional HPC are reflected in a record setting system that uses the C8000 family.
Get out of the Way – it’s a Stampede!
The first system to be deployed using the C8000 family is one that you may have heard of - Stampede. The University of Texas Advanced Computing Center is deploying Stampede, one of the fastest systems in the world, achieving approximately 10 PFLOPS (PetaFLOPS) when realized for production work.
Stampede is almost entirely built using the Dell PowerEdge C8000 family. The primary compute sled is the C8220X which also contains Intel® Xeon Phi™ coprocessors and uses a Mellanox FDR InfiniBand fabric capable of 56 Gbps. Each compute sled has 32 GBs of memory and two Intel eight-core Xeon processors. There are also two power sleds for redundant power within each 4U chassis. In total there is 272 TB (Terabytes) of memory in Stampede.
Stampede also uses storage sleds for their primary storage for Stampede, using Lustre as the file system. Altogether there is 14PB of spinning disk storage.
There are also 128 NVIDIA® Tesla™ K20 GPUs for remote visualization,16 Dell servers with 1TB of memory each and two GPUs for large data analysis.
Figure 5 below is an image from Stampede.
![]()
Figure 5 – View down one row of Stampede
Altogether there are approximately 200 racks in Stampede. The non-accelerated portion of the system is capable of 2 PFLOPS which is double the performance of the largest system in the NSD XSEDE program (National Science Foundation Extreme Science and Engineering Discovery Environment). When the accelerators are included in the performance, then Stampede should finish with around 10 PFLOPS.
![]()
