With the increase in the volume of server operations, especially servers based on graphic accelerators and the introduction of powerful AMD EPYC series processors, users can access a high computational and operational volume by having only one socket in the server.
In this review, we intend to accompany you with one of the best GPU servers based on AMD EPYC processors. ASUS ESC4000A-E10 Server With support for the latest generation of AMD server-specific processors, it is possible to use eight single-slot graphics accelerators or four dual-slot graphics accelerators in 2U space.
Overview of ASUS ESC4000A-E10 server hardware
At first glance, this server has 2 units, the front part of which is almost all dedicated to storage drives. There are also four USB 3 Type-A ports, server status LED lights and a POWER button. There is also an LCD display to show POST operations to the system administrator.
Eight compartments or bays with standard dimensions of 3.5% with the possibility of installing drives of both 2.5 and 3.5 inch sizes as Hot Swap, which by default configures 4 x SATA / SAS + 4 x SATA / SAS / NVMe It is also embedded.
It goes without saying that to install or change drives, you only need to screw them to the bays, and to replace the housing, like other servers, you do not need special tools.
In order to improve airflow and better cooling along with more personalization, Asus has designed the upper part of the bays to be empty, so that if you need peripherals, it can be installed in this space. There’s also a PCIe Gen4 x8 riser here, with a power connector next to it.
Thanks to AMD’s new processors, the modern PCIe Gen 4.0 interface makes it possible to install new devices and equipment compatible with this interface on this server. This slot can support devices such as Raid Controller, PCIe NIC or newer SSDs.
The next empty slot is also a customizable slot. In this slot, it is possible to install SSD drives with 4x M.2 NVMe SSD layout. It is also possible to add an OCP NIC 3.0 (PCIe Gen4 x8) slot here, which is not found in other chassis of other servers.
On the back of the server, according to the usual 2U GPU-based servers, the graphics are located on either side of the server. In the middle of the server, the output input is located on top of two 1600 watt power supplies with a redundant arrangement.
Also in the connection section are two USB 3 Type-A ports, a VGA port, three network ports, one of which is Management and the other two are 1GbE. Asus has used the Intel i350 NIC controller for network ports. There are two other Low Profile PCIe slots in this section that we will cover in the Internal Equipment and Features section.
Take a look at the equipment and facilities inside the ASUS ESC4000A-E10
At first glance, seven large fans with Hot-Swap capability are responsible for the continuous flow of air to cool the Ram, the processor, and the airflow needed to cool the graphics. Three fans specifically direct the wind flow to the motherboard, RAMs, processor and power supplies through a strong transparent plastic duct, and the other two pairs direct air to the graphics.
Unlike other server fans, these fans are surrounded by a yellow color and integrated cover so that it can be easily distinguished on the server during the cleaning and maintenance process.
Below the transparent duct, you can see the data cabling from the front panel of the server chassis. There is also an M.2 22110 (110mm) slot here. There’s also a large motherboard socket on the back that supports AMD EPYC series processors, which we used to test the AMD EPYC 7742 64-core processor. In addition to this massive processor, we have used eight 256GB DDR4-3200 RAMs per LRDIMM module, giving us a total of 2 terabytes of RAM with just one processor.
This amount of RAM support while servers based on Intel chipset require two second generation Intel Xeon Scalable Refresh processors, the exorbitant cost of two processors may be equivalent to the entire server, given the news of the new PCIe Gen technology 4.0 is not in it either.
In the previous section, we talked about Low Profile PCIs, one of which is shown in the image below. This slot is PCIe Gen4 x16.
You can also see the second slot here. This slot is suitable for installing a variety of equipment such as 100Gbps high-speed dual network interface for InfiniBand or 100GbE.
The motherboard of this server is from Asus KRPG-U8 model. There is a dual PCIe Gen4 riser slot on the motherboard. ASPEED AST2500 BMC is also used for server management, which we will discuss in the management tools section. There is a micro SD slot next to the Intel i350 NIC heatsink.
GPU pads on the sides of the chassis; By separating the two pairs of chassis side fans, we have easier access to this part. In each section, there is a small board to distribute the current required by the graphics, which in our system requires 2 power cables for each graphics card. In this server, it is possible to install two standard two-slot cards of PCIe Gen4 x16 type on each side, which in total, four graphics cards can easily fit inside the chassis of this server.
In addition, it is possible to install four single-slot cards on each side with support for PCIe Gen4 x8 layout, which supports a total of eight cards. These pads can also support NV-Link.
Each pod is connected to the main board via two PCIe Gen4 x16 connectors, with a 32x PCIe Gen4 lane on each side of the chassis for the system administrator to install graphics or other expansion cards. We use NVIDIA Quadro RTX 6000 with PCIe Gen 3.0 interface on this server, but you can use the new NVIDIA A100 PCIe and AMD Instinct MI100 32GB CDNA GPU cards to get the most out of this server, which is very fast. They have higher.
One of the important features of this server in the field of airflow circulation is the possibility of full use of server fans and main fans of the graphics card, and in addition, the metal ducting of the chassis directs the airflow to the graphics components, which in other chassis such ducts We do not see murder.
ASUS ESC4000A-E10 Server Management
This server uses the ASPEED AST2500 BMC for internal management, which executes the MegaRAC SP-X commands. Asus calls the solution ASMB9-iKVM, which includes IPMI, WebGUI, and Redfish management for the platform.
This solution has a number of specific features, some of which we will discuss. Asus, for example, allows the system administrator to update the system BIOS through the standard web GUI interface. Supermicro servers require the purchase of a separate module for this.
Another thing is the full HTML5 iKVM support in this solution. Serial-over-LAN and JAVA iKVM are other features of this section. Dell EMC, HP and Lenovo charge extra for such features.
Asus also supports the Redfish API for management. Sometimes it may not be possible to have full access to all parts of the BIOS via the Web GUI. Redfish should be used.
Another attractive BIOS feature of this server is the ability to use the solution to increase the overall efficiency of the system. In this BIOS, features such as automatic overclocking and further optimization settings in accordance with Asus processing records in Spec.org are available to the user. In a way, by increasing the overall performance of the server, this leads to a more cost-effective overall cost of the server than its power.
Another unique feature of Asus is the IPMI Hardware Monitor or IPMI HWM feature. In order to manage this section, in addition to full access through the BIOS, Asus has made available additional software for this section.
ASUS ESC4000A-E10 block diagram
Due to the single socket of this server and the use of AMD EPYC processor, a NUMA layout, the block diagram of this server is similar to the image below, which we took from the Asus KRPG-U8 motherboard and can be said to be related to the ESC4000A-E10 server.
The most interesting part of this block diagram is the ease of connectivity that Asus design engineers have achieved with just one processor socket, drastically reducing costs and using an optimal topology.
Performance of ASUS ESC4000A-E10
Old Linux-Bench scripts were used to measure the performance of this server, in which we used an updated version of these scripts, namely Linux-Bench2. Some of our benchmarks may have been running for days at a time so that the benchmarks were the same and standard.
Python Linux 4.4.2 Kernel Compile Benchmark
In recent years, this benchmark is one of the most reputable benchmarks for measuring the power of servers. Its operation is simple, from the Linux 4.4.2 kernel with standard configuration, we tested several processors from the AMD EPYC family in this server:
C-ray benchmark 1.1
This benchmark has been used to measure the performance of Ray Tracing in recent years, which we measured from the 8K output of this server:
7-zip Compression function
One of the most common benchmarks is the 7-Zip compression benchmark, the result of which you can see on this server:
One of the most critical communication protocols between servers is OpenSSL, the different types of which we implemented in this server:
In order to validate the system and standardize it with the type of GPU used in this server and to make its performance similar to that of 4-unit servers with 8 to 10 graphical layouts, we used four NVIDIA Quadro RTX 6000 cards with Passive cooling.
The PCIe switch was not used in the system. As a result, we used the P2P transfer rate, which is AMD EPYC IO Die’s excellent topology for navigating GPU paths. Although the result will be different when installing NVLink on each pod, we examined the card server separately.
On GPU-based servers, the BIOS settings are very effective in improving the overall performance of the system, which can be changed using the Web interface in IPMI, which you can see the following results:
Power consumption ASUS ESC4000A-E10
Server power consumption depends on CPU and graphics workload. The server motherboard officially supports processors with a maximum TDP of 280W. With four graphics servers, idle time was 0.35kW with AMD EPYC 7702P, 8x 32GB DIMM, 4x Samsung PM1733 3.84TB NVMe SSD. During the full operating pressure, the consumption range of 1000 to 1400 watts was seen in this server, and the 1600 watt power supplies of this server completely overcame it.
ASUS ESC4000A-E10 Spider Diagram
This diagram simply shows the ability to understand the performance and reliability of the ESC4000A-E10 server, as expected from a GPU, which fully demonstrates its core power in operations based on GPUs.
The focus and purpose of this server is solely on the use of graphical accelerators. Although this server is not similar to the GUU of dense 1U servers, but in its kind in two units (2U) allows the user to use eight processing accelerators / GPU. In addition, there is still room to install modern and high-speed storage and networks in this server.
Finally, this server, in addition to having a powerful and cost-effective processor, is a very good alternative to servers based on common 4U graphics with 2 processor sets. In terms of support and portability, as you read in the review, the possibility of installing a maximum amount of RAM compared to Intel dual-socket servers, attractive free Asus solutions, support for new generation PCIe Gen 4.0 drives and most importantly support for 8U graphics in 2U space It is an important feature of this server.