Rdma Over Infiniband - HP 226824-001 - ProLiant - ML750 Introduction Manual

RDMA over InfiniBand

InfiniBand is a high-performance, low-latency alternative to Ethernet. InfiniBand architecture uses a
switched-fabric, channel-based design that adapts well to distributed-computing environments where
performance, infrastructure simplification, and convergence of component interconnects are key
design goals. A data center employing an InfiniBand infrastructure is less complex and easier to
manage.
Like Ethernet, InfiniBand architecture uses multi-layer processing to transfer data between nodes. Each
InfiniBand node contains a host or target channel adapter (HCA or TCA) that connects to an
InfiniBand network through a bi-directional serial link. However, InfiniBand architecture allows links to
have multiple channel pairs (4x being the current typical implementation), with each channel handling
virtual lanes of multiplexed data at a 2.5 Gbps single data rate (SDR). Additional nodes and channels
(up to 12x) and faster signaling (up to quad data rate) increase the bandwidth of a link, making an
InfiniBand infrastructure scalable with system expansion and capable of a bandwidth from 2.5 to 120
Gbps in each direction (although processor overhead and server I/O bus architectures may lower
usable bandwidth by 10 percent or more).
RDMA data transactions over InfiniBand (Figure 5) occur basically the same way as described for
Ethernet and offer the same advantages. However, InfiniBand uses a communications stack designed
to support RDMA as a core capability and therefore provides greater RDMA performance than
Ethernet.
Figure 5. RDMA data flow over InfiniBand
Sending Host
Memory Chipset
HCA
Ch 1 TX/RX Prs
Ch 2 TX/RX Prs
Ch 3 TX/RX Prs
Ch 4 TX/RX Prs
An InfiniBand HCA is similar to an Ethernet NIC in operation but uses a different software
architecture. The InfiniBand HCA uses a separate communications stack usually provided by the HCA
manufacturer or by a third-party software vendor. Therefore, operating an HCA requires prior loading
of both the software driver and the communications stack for the specific operating system.
The majority of existing InfiniBand clusters run on Linux. Drivers and communications stacks are also
available for Microsoft® Windows®, HP-UX, Solaris, and other operating systems from various
hardware and software vendors. Note that specific feature support varies among operating systems
and vendors.
CPU
IB
4x InfiniBand Link
Receiving Host
Memory Chipset
IB
HCA
CPU
9