Nvidia BlueField-3 BF3220SH User Manual
  1. Manuals
  2. Brands
  3. Nvidia Manuals
  4. Controller
  5. BlueField-3
  6. User manual
Nvidia BlueField-3 BF3220SH User Manual

Nvidia BlueField-3 BF3220SH User Manual

Dpu controller
Hide thumbs Also See for BlueField-3 BF3220SH:
Table of Contents

Advertisement

Quick Links

Download this manual
NVIDIA BlueField-3 DPU Controller User Manual

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the BlueField-3 BF3220SH and is the answer not in the manual?

Questions and answers

Related Manuals for Nvidia BlueField-3 BF3220SH

Summary of Contents for Nvidia BlueField-3 BF3220SH

  • Page 1 NVIDIA BlueField-3 DPU Controller User Manual...

  • Page 2: Table Of Contents

    Cabline CA-II Plus Harness Pinouts Technical Specifications Technical Specifications Cables and Cabling Configurations Cables and Cabling Configurations General Cooling Recommendations for Heatsink-less DPU General Cooling Recommendations for Heatsink-less DPU Controllers Controllers Document Revision History NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 3 Document Revision History About This Manual This User Manual describes NVIDIA® BlueField®-3 BF3220SH DPU Controller cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the card, hardware installation, driver installation and bring-up instructions.
  • Page 4 Customers who purchased NVIDIA products directly from NVIDIA are invited to contact us through the following methods: URL: www.nvidia.com Support E-mail: enterprisesupport@nvidia.com Customers who purchased NVIDIA M-1 Global Support Services, please see your contract for details regarding Technical Support. Customers who purchased NVIDIA products through an NVIDIA-approved reseller should first seek assistance through their reseller.
  • Page 5 Bytes. The use of Gb or Gbits (small b) indicates size in giga bits. In this document PCIe is used to mean PCI Express. Revision History A list of the changes made to this document are provided in Document Revision History. NVIDIA BlueField-3 DPU Controller User Manual...

  • Page 6: Introduction

    Introduction The NVIDIA® BlueField®-3 data processing unit (DPU) Controller is the 3rd-generation data center infrastructure-on-a-chip that enables organizations to build software-defined, hardware-accelerated IT infrastructures from cloud to core data center to edge. With 200Gb/s Ethernet or NDR200 200Gb/s InfiniBand network connectivity, BlueField-3 DPU Controller offloads, accelerates, and isolates software-defined networking, storage,...
  • Page 7 The BlueField-3 DPU Controller is shipped with Ubuntu – a Linux commercial Oper operating system – which includes the NVIDIA OFED stack (MLNX_OFED), and is ating capable of running all customer-based Linux applications seamlessly. For more Syste information, please refer to the DOCA SDK documentation or NVIDIA BlueField-3 Software User Manual.
  • Page 8 For package contents, refer to PCIe Auxiliary Card Kit. For more detailed information, refer to Specifications. Features and Benefits This section describes hardware features and capabilities. Please refer to the relevant driver and/or firmware release notes for feature availability. NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 9 200GbE / 100GbE / 50GbE / 40GbE / 25GbE / 10GbE Ethernet Protocol MAC Rate 200/100 Gigabit Ethernet IEEE802.3ck (Include ETC enhancement) IEEE802.3cd IEEE802.3bs 200/100 Gigabit Ethernet IEEE802.3cm (Include ETC enhancement) IEEE802.3cn IEEE802.3cu IEEE 802.3bj 100 Gigabit Ethernet IEEE 802.3bm NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 10 DPU_BMC Flashes: 2x 64MByte for BMC Image 512MByte for Config Data eMMC pSLC 40GB with 30K Write Cycles eMMC for SoC BIOS and OS. SSD (onboard BGA) 128GByte for user SoC OS, logs and application NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 11 NVIDIA BlueField-3 Arm cores. On the storage initiator side, the DPU Controller can prove an efficient solution for hyper- converged systems to enable the host CPU to focus on compute while all the storage interface is handled through the Arm cores.
  • Page 12 (VM) within the server. GPUDirect RDMA is a technology that provides a direct P2P (Peer-to-Peer) data path between the GPU Memory directly to/from the NVIDIA HCA devices. This provides a significant decrease in GPU-GPU communication latency and completely offloads the CPU, removing it from all GPU-GPU Direct communications across the network.

  • Page 13: Bluefield-3 Administrator Quick Start Guide

    NVIDIA DOCA Developer Quick Start Guide. Info Not sure which guide to follow? For more details on the different BlueField user types, please refer to the NVIDIA BlueField and DOCA User Types document. Prerequisites for Initial BlueField-3 Deployment...
  • Page 14 First-time Installation Procedure Content: The following illustrates the sequence of events and actions from first time power-up of the NVIDIA® BlueField® DPU in the data center environment through provisioning and maintenance. Info The numbers indicated in the sequence diagram map to the steps below the diagram.
  • Page 15 NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 16 Domain-Name (15), Static-Route (33), NTP (42), Unknown (120) Classless-Static-Route (121) MSZ (57), length 2: Hostname (12), length 7: "dpu-bmc" Vendor-Class (60), length 13: "NVIDIA/BF/BMC" END (255), length 18:18:10.565261 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto UDP (17), length 353)
  • Page 17 IP address for the DPU BMC (e.g., 10.10.1.2), and <DPU-BMC-IP> <user-password> is the chosen password to log into the DPU BMC with root privileges. The new password must comply with the following policy parameters: NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 18 Minimum number of lower-case characters: 1 Minimum number of digits: 1 Minimum number of special characters: 1 Info List of special characters: (dollar sign) (percent sign) (caret/circumflex) (ampersand) & (asterisk) (minus) (plus) (equal) (pipe) (tilde) (underscore) (comma) (period/full stop) NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 19 Examples of passwords with 5 consecutive character pairs (invalid): DcBa123456AbCd! ab1XbcYcdZdeGef! Testing_123abcgh! The following is a valid example password: HelloNvidia3D! Note The root account locks after four consecutive failed attempts and automatically unlocks after 10 minutes. For example: NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 20 Make sure to download the latest DPU BMC image available from the BlueField Runtime and Driver Downloader. Update BMC Firmware 1. Run the following Redfish command over the 1GbE out-of-band interface on the DPU BMC to trigger a secure DPU BMC firmware update: NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 21 2. To track the progress of the update, use the task received in the response above (i.e., 0) in your query and monitor the value of the task’s field: PercentComplete curl -k -u root:'<password>' -X GET https://<DPU-BMC-IP>/redfish/v1/TaskService/Tasks/<task_id> | jq -r ' .PercentComplete' Where: NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 22 – DPU BMC password <password> – eROT firmware update package path pointing to eROT <package_path> *.fwpkg binary (e.g. cec1736-ecfw-00.02.0127.0000-n02-rel-prod.fwpkg – BMC IP address <DPU-BMC-IP> After initiating the eROT secure update, a new task is created. Example: NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 23 100 PercentComplete before proceeding to other operations. Where: – DPU BMC password <password> – BMC IP address <DPU-BMC-IP> – task ID of the update process as received in the response under the <task_id> value NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 24 A new firmware update can be attempted by the Redfish client. Two firmware update The Redfish server blocks the second firmware update request requests are initiated and returns the following: HTTP code 400 "Bad Request" Redfish message based on standard registry entry UpdateInProgress NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 25 ERoT failure (not TaskStatus is set to Warning responding) Messages array in the task includes an entry describing the error The Redfish client reports the error The Redfish client may retry the firmware update. NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 26 200 MB request to retry the operation is given Step 5 – Upgrade DPU Firmware Components and BSP Upgrade the DPU firmware components (i.e., ATF, UEFI, NIC-firmware) and the BSP using the BFB image. Info NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 27 RShim device. This can be done by either running SCP directly or using the Redfish interface. Redfish Interface The following is a simple sequence diagram illustrating the flow of the BFB installation process. NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 28 The following are detailed instructions outlining each step in the diagram: 1. Confirm the identity of the remote server (i.e., host holding the BFB image) and BMC. NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 29 – remote server's public key from the remote_server_public_key ssh-keyscan response, which contains both the type and the public key with a space between the two fields (i.e., " "). <type> <public_key> – BMC IP address bmc_ip NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 30 4. If the remote server public key must be revoked, use the following command before repeating the previous step: curl -k -u root:'<password>' -H "Content-Type: application/json" -X POST -d '{"RemoteServerIP":"<remote_server_ip>"}' https://<bmc_ip>/redfish/v1/UpdateService/Actions/Oem/NvidiaUpdateService.RevokeAllRem Where: – remote server's IP address remote_server_ip NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 31 -k -u root:'<password>' -H "Content-Type: application/json" -X POST -d '{"TransferProtocol":"SCP", "ImageURI":"<image_uri>","Targets": ["redfish/v1/UpdateService/FirmwareInventory/DPU_OS"], "Username":"<username>"}' https://<bmc_ip>/redfish/v1/UpdateService/Actions/UpdateService.SimpleUpdate Info After the BMC boots, it may take a few seconds (6-8 in NVIDIA® BlueField®-2, and 2 in BlueField-3) until the DPU BSP ( ) is DPU_OS Note This command uses SCP for the image transfer, initiates a soft reset on the BlueField and then pushes the boot stream.
  • Page 32    "message": "The requested resource of type Target named '/dev/rshim0/boot' was not found." If a username or any other required field is missing: "Username@Message.ExtendedInfo": [ "@odata.type": "#Message.v1_1_1.Message", "Message": "The create operation failed because the required property Username was missing from the request.", NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 33 5 minutes with the content "Transfer is still in progress (X minutes elapsed). Please wait". Once the transfer is completed, the is set PercentComplete to 100, and the is updated to TaskState Completed NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 34 "Message": "Transfer of image '<file_name>' to '/dev/rshim0/boot' failed.", "MessageArgs": [ "<file_name>, "/dev/rshim0/boot" "MessageId": "Update.1.0.TransferFailed", "Resolution": " Unknown Host: Please provide server's public key using PublicKeyExchange ", "Severity": "Critical" … "PercentComplete": 0, "StartTime": "<start_time>", "TaskMonitor": "/redfish/v1/TaskService/Tasks/<task_id>/Monitor", "TaskState": "Exception", "TaskStatus": "Critical" Info NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 35 "PercentComplete": 0, "StartTime": "<start_time>", "TaskMonitor": "/redfish/v1/TaskService/Tasks/<task_id>/Monitor", "TaskState": "Exception", "TaskStatus": "Critical" Info In this case, verify that the BMC key has been added correctly to the file on the remote authorized_key server. If SCP fails: NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 36 "Message": " <file_name>' is being transferred to '/dev/rshim0/boot'.", "MessageArgs": [ " <file_name>", "/dev/rshim0/boot" "MessageId": "Update.1.0.TransferringToComponent", "Resolution": "Transfer is still in progress (5 minutes elapsed): Please wait", "Severity": "OK" … "PercentComplete": 0, "StartTime": "<start_time>", "TaskMonitor": "/redfish/v1/TaskService/Tasks/<task_id>/Monitor", "TaskState": "Running", "TaskStatus": "OK" NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 37 BlueField Logs" for information on dumping the which rshim.log contains the current RShim miscellaneous messages. 5. Verify that the new BFB is running by checking its version: curl -k -u root:'<password>' -H "Content-Type: application/json" -X GET https://<bmc_ip>/redfish/v1/UpdateService/FirmwareInventory/DPU_OS NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 38 Step 6 – Verify Software Component Versions Verify BlueField BSP, BlueField BMC and BlueField NIC firmware versions are up to date according to the NVIDIA BlueField BMC Software User Manual and NVIDIA BlueField DPU BSP Release Notes. 1. Use the Redfish...
  • Page 39 > curl -k -u root:<password> -H "Content-Type: application/octet-stream" -X GET https://<DPU- BMC-IP>/redfish/v1/UpdateService/FirmwareInventory/DPU_ATF "@odata.id": "/redfish/v1/UpdateService/FirmwareInventory/DPU_ATF", "@odata.type": "#SoftwareInventory.v1_4_0.SoftwareInventory", "Description": "Host image", "Id": "DPU_ATF", "Members@odata.count": 1, "Name": " "Software Inventory", "RelatedItem": [ "@odata.id": "/redfish/v1/Systems/Bluefield/Bios" "SoftwareId": "", "Status": { "Health": "OK", NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 40 For more information on deploying BlueField software from the BMC, refer to the "Deploying BlueField Software Using BFB from BMC" page of the NVIDIA BlueField DPU BSP document. Step 7 – Relate DPU to DPU BMC and NIC Data Ports on...
  • Page 41 "Description": "Management Network Interface", "FQDN": "dpu-bmc", "HostName": "dpu-bmc", "IPv4Addresses": [ "Address": "10.237.40.179", "AddressOrigin": "DHCP", "Gateway": "0.0.0.0", "SubnetMask": "255.255.0.0" "IPv4StaticAddresses": [], "IPv6AddressPolicyTable": [], "IPv6Addresses": [ "Address": "fdfd:fdfd:10:237:966d:aeff:fe17:9f5f", "AddressOrigin": "DHCPv6", "AddressState": null, "PrefixLength": 64 "Address": "fe80::966d:aeff:fe17:9f5f", "AddressOrigin": "LinkLocal", NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 42 2. Get the DPU's high-speed port's MAC addresses using the following Redfish command over the 1GbE OOB port to the DPU BMC: curl -k -u root:<password> -H "Content-Type: application/octet-stream" -X GET https://<bmc_ip>/redfish/v1/Chassis/Card1/NetworkAdapters/NvidiaNetworkAdapter/NetworkDevi "@odata.id": "/redfish/v1/Chassis/Card1/NetworkAdapters/NvidiaNetworkAdapter/NetworkDeviceFunctions/eth "@odata.type": "#NetworkDeviceFunction.v1_9_0.NetworkDeviceFunction", "Ethernet": { "MACAddress": "02:b1:b6:12:39:05", "MTUSize": 1500 "Id": "eth0f0", NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 43 -k -u root:<password> -H 'content-type: application/json' -d '{ "Attributes": { "NicMode": "DpuMode" } }' -X PATCH https://<DPU-BMC-IP>/redfish/v1/Systems/Bluefield/Bios/Settings 3. To check that the BMC recorded the change for the next UEFI reboot to apply it: curl -k -u root:<password> -H 'content-type: application/json' -X GET https://<DPU-BMC- NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 44 | python3 -m json.tool The available BlueField host privilege levels are . The default is Restricted Privileged , where the host has access to DPU. Privileged 2. Change the privilege level to Restricted Note NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 45 "Name": "UEFI Secure Boot", "SecureBootCurrentBoot": "Enabled", "SecureBootEnable": true, "SecureBootMode": "SetupMode" curl -k -u root:<password> -X PATCH https://<DPU-BMC-IP>/redfish/v1/Systems/Bluefield/SecureBoot -H 'Content-Type: application/json' -d '{"SecureBootCurrentBoot": "Enabled", "SecureBootEnable": true, "SecureBootMode": "SetupMode"}' For more information on user management, review this page. NVIDIA BlueField-3 DPU Controller User Manual...

  • Page 46: Supported Interfaces

    DPU Controller Print Side 900-9D3C6- 00CV-GA0 900-9D3C6- 00CV-DA0 Interface Description System- on-Chip 16 Arm-Cores SoC (SoC) Networkin The network traffic is transmitted through the DPU Controller g Interface QSFP112 connectors. The QSFP112 connectors allow the use of NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 47 Power An external 12V power connection through an 8-pin ATX connector Supply Connector 11 Cabline Two Cabline CA-II plus connectors are populated to allow connectivity CA-II Plus to an additional PCIe x16 Auxiliary card NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 48 NVIDIA BlueField-3 DPU Controller is a family of advanced DPU Controller IC solutions that integrate a coherent mesh of 64-bit Armv8.2+ A78 Hercules cores , an NVIDIA® ConnectX®-7 network adapter front-end, and a PCI Express switch into a single chip. The powerful DPU Controller IC architecture includes an Armv multicore processor array, enabling customers to develop sophisticated applications and highly differentiated...
  • Page 49 The network ports are compliant with the InfiniBand Architecture Specification, Release 1.5. InfiniBand traffic is transmitted through the cards' QSFP112 connectors. Networking Ports LEDs Interface One bicolor (Yellow and Green) I/O LED per port indicates speed and link status. Link Indications NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 50 The following lists PCIe interface features: PCIe Gen 5.0, 4.0, 3.0, 2.0 and 1.1 compatible 2.5, 5.0, or 8.0, 16.0 or 32.0 GT/s link rate x16 lanes Auto-negotiates to x16, x8, x4, x2, or x1 NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 51 The UART debug interface on BlueField-3 boards can be accessed through a 20-pin NC-SI connector, which is associated with the NIC BMC device. The connectivity is shown in the following table: NC-SI Connector Pin # Signal on Board UART_TX UART_RX NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 52 Note The male connector to the male Type-A cable is not included in the shipped DPU Controller card box and should be ordered separately as part of the accessories kit (P/N: MBF35-DKIT). NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 53 Two OOB management LEDs, one Green and one Yellow, behave as described in the table below. Green LED Yellow LED Link/Activity Link off 1 Gb/s link / No activity Blinking 1 Gb/s link / Activity (RX,TX) Blinking Not supported Blinking Blinking NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 54 The DPU Controller PTP solution allows you to run any PTP stack on your host. With respect to testing and measurements, selected NVIDIA DPU Controllers allow you to use the PPS-out signal from the onboard MMCX RA connecter. The DPU Controller also allows measuring PTP in scale with the PPS-In signal.
  • Page 55 If you are uncertain about your server's compatibility with the PCI ATX connection, please contact your NVIDIA representative for assistance. Mechanical Keys of the 8-pin ATX Power Cable (PCIe CEM Specification Rev 5.0) NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 56 Cabline CA-II Plus Connectors Pinouts. Integrated BMC Interface The DPU Controller incorporates an onboard integrated NIC BMC and an Ethernet switch. The BMC becomes available once the host server powers up the card. The NIC BMC can NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 57 BlueField-3 incorporates a coin type Lithium battery CR621 for RTC (Real Time Clock). eMMC Interface The eMMC is an x8 NAND flash used for Arm boot and operating system storage. Memory size is 128GB, where it is effectively pSLC 40GB. NVIDIA BlueField-3 DPU Controller User Manual...

  • Page 58: Pinouts Description

    9D3B4-00SC-EA0 only) 3.3V - (Connected in 900- JTAG - Not 3.3V 9D3B4-00CC-EA0 & 900- TRST# Connected 9D3B4-00SC-EA0 only) 3.3V - (Connected in 900- 3.3V 9D3B4-00CC-EA0 & 900- 3.3V_AUX 9D3B4-00SC-EA0 only) PERST# PCIe Reset WAKW#/RSVD NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 59 REFCLK+ Host Reference Clock REFCLK- Host Reference Clock PETP0 PETN0 PERP0 PERN0 RSVD SER_DO PETP1 PETN1 PERP1 PERN1 PETP2 PETN2 PERP2 PERN2 PETP3 PETN3 PERP3 PERN3 RSVD RSVD SER_CLK SER_CAPT PETP4 PETN4 PERP4 PERN4 NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 60 Signal Signal Description Description Name Name PETP5 PETN5 PERP5 PERN5 PETP6 PETN6 PERP6 PERN6 PETP7 PETN7 PERP7 PERN7 RSVD RSVD PETP8 PETN8 PERP8 PERN8 PETP9 PETN9 PERP9 PERN9 PETP10 PETN10 PERP10 PERN10 NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 61 PRSNT2# Present External Power Supply Connector The below table provides the External Power Supply pins of the external power supply interfaces on the DPU. For further details, please refer to External PCIe Power Supply NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 62 The below tables list the NC-SI management interface pinout descriptions. For further details, please refer to NC-SI Management Interface. Signal Signal Description Name Ground PKG_ID1 Input NC-SI PKG_ID Shoul be connected to the Primary controller NC-SI PKG_ID pins to set the appropriate package ID. NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 63 BMC and the RBT isolator to prevent the signal from floating when no card is installed. For DPUs, this pin should be connected between the connector and the RBT PHY. External termination determined by the DPU RBT PHY requirements. NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 64 100 kΩ pull down resistor to ground on the baseboard between the RBT isolator and the DPU cable connector to prevent the card-side signals from floating when the RBT signals are isolated. NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 65 3.3V UART RX signal to the baseboard Presence of DPU. Baseboard should implement a 200 Ω PRESENC series resistor and 4.7kohm pull-up resistor to 3.3V AUX. DPU should tie this to GND. Ground RBT_ARB Input NC-SI hardware arbitration output. _OUT NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 66 Cabline CA-II Plus Connectors Pinouts Component Side Pin# on Pin# Signal Name Wire Type AWG# other end GND BAR PCIE_REFCLK1_P Micro coax PCIE_REFCLK1_N Micro coax GND BAR PCIE_CPU_CX_15N Micro coax PCIE_CPU_CX_15P Micro coax GND BAR PCIE_CPU_CX_14N Micro coax NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 67 Micro coax GND BAR PCIE_CPU_CX_9N Micro coax PCIE_CPU_CX_9P Micro coax GND BAR PCIE_CPU_CX_8N Micro coax PCIE_CPU_CX_8P Micro coax GND BAR PCIE_CPU_CX_7N Micro coax PCIE_CPU_CX_7P Micro coax GND BAR CIE_CPU_CX_6N Micro coax PCIE_CPU_CX_6P Micro coax NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 68 PCIE_CPU_CX_1P Micro coax GND BAR PCIE_CPU_CX_0N Micro coax PCIE_CPU_CX_0P Micro coax GND BAR No wire Micro coax No wire Micro coax AUX_PGOOD Micro coax No wire Micro coax I2C_AUX_SCL Micro coax I2C_AUX_SDA Micro coax NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 69 Micro coax Reserved_08 Micro coax GND BAR PCIE_CPU_CX_0P Micro coax PCIE_CPU_CX_0N Micro coax GND BAR PCIE_CPU_CX_1P Micro coax PCIE_CPU_CX_1N Micro coax GND BAR PCIE_CPU_CX_2P Micro coax PCIE_CPU_CX_2N Micro coax GND BAR PCIE_CPU_CX_3P Micro coax NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 70 Micro coax GND BAR PCIE_CPU_CX_8P Micro coax PCIE_CPU_CX_8N Micro coax GND BAR PCIE_CPU_CX_9P Micro coax PCIE_CPU_CX_9N Micro coax GND BAR PCIE_CPU_CX_10P Micro coax PCIE_CPU_CX_10N Micro coax GND BAR PCIE_CPU_CX_11P Micro coax PCIE_CPU_CX_11N Micro coax NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 71 PCIE_CPU_CX_13P Micro coax PCIE_CPU_CX_13N Micro coax GND BAR PCIE_CPU_CX_14P Micro coax PCIE_CPU_CX_14N Micro coax GND BAR PCIE_CPU_CX_15P Micro coax PCIE_CPU_CX_15N Micro coax GND BAR S_PERST1_CONN_L Micro coax No wire No Wire S_PRSNT2_L Micro coax NVIDIA BlueField-3 DPU Controller User Manual...

  • Page 72: Hardware Installation And Pcie Bifurcation

    This equipment should not be operated in an area with an ambient temperature exceed ing the maximum recommended: 55°C (131°F). An airflow of 200LFM at this maximum ambient temperature is required for HCA cards NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 73 IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+ A2:20 Installation Procedure Overview The installation procedure of DPU involves the following steps: Procedure Direct Link Refer to System Check the system’s requirements. Requirements NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 74 The operating environment should meet severity level G1 as per ISA 71.04 for gaseous contamination and ISO 14644-1 class 8 for cleanliness level. The below table lists the motherboard and power supply requirements per DPU series. NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 75 Refer to System Requirements section under the Introduction section. Software Stacks -BlueField-3 DPU Controller is shipped with Ubuntu – a Linux commercial operating system – which includes the NVIDIA OFED stack (MLNX_OFED), and is capable of running all customer-based Linux applications seamlessly.
  • Page 76 2. Shut down your system if active. Turn off the power to the system, and disconnect the power cord. Refer to the system documentation for instructions. Before you install the DPU, make sure that the system is disconnected from power. NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 77 3. Insert the connector into the DPU Controller . Be careful to insert the connector straight into the cage. Do not apply any torque, up or down, to the connector cage in the DPU Controller. 4. Make sure that the connector locks in place. Note NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 78 LED indicator will turn off when the cable is unseated. 8-pin ATX Power Supply Cable Warning The 8-pin ATX power supply cable is mandatory when powering-up the following DPUs. Without a connection to the power supply NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 79 Ensure that the appropriate PCIe auxiliary power source is available, not an EPS12V power source. If you are uncertain about your server's compatibility with the PCI ATX connection, please contact your NVIDIA representative for assistance. NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 80 Ensure the cable is fully and securely connected to your DPU Controller and the connector lock is secured. Mechanical Keys of the 8-pin ATX Power Cable (PCIe CEM Specification Rev 5.0) DPU Controller Power-Up and Power-Down Sequences Power-Up Sequence NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 81 12V_ATX and 12V_PCIE voltages. 2. The 12V_ATX can be powered up after the 12V_PCIE, with a maximum delay of 2 seconds. The below graph illustrates the delay between the 12V_ATX and 12V_PCIE voltages at power-up. NVIDIA BlueField-3 DPU Controller User Manual...

  • Page 82: Pcie X16 Dpu Controller Installation Instructions

    2. The 12V_PCIE voltage must not be powered down while the 12V_ATX voltage is powered up. PCIe x16 DPU Controller Installation Instructions Installation Instructions This section provides detailed instructions on how to install your BlueField-3 card in a system. NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 83 Warning Exercise caution to avoid touching the components on the top surface of the card. Do not use excessive force when seating the card, as this may damage the system or the card's components. NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 84 5. Secure the card with the screw. 6. Close the system case. 7. Install the networking cables. For instructions, please refer to Networking Cable Installation. NVIDIA BlueField-3 DPU Controller User Manual...

  • Page 85: [Optional] Pcie Extension Connection (2X Pcie X16) Installation Instructions

    4. Holding the board from its center, gently pull the board out of the PCI Express slot. 5. When the port connectors reach the top of the chassis window, gently pull the board in parallel to the motherboard. [Optional] PCIe Extension Connection (2x PCIe x16) Installation Instructions Warning NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 86 DPU and Auxiliary PCIe card engraved with “White Cable” while the one marked with "BLACK" label should be connected to the connector on the DPU and Auxiliary PCIe card engraved with “Black Cable”. Note NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 87 Connect the DPU Controller with the Auxiliary connection card using the supplied Cabline CA-II Plus harnesses. Step 1: Slide the black and white Cabline CA-II Plus harnesses through the retention clip while making sure the clip opening is facing the plugs. NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 88 Step 2: Plug the Cabline CA-II Plus harnesses on the DPU while paying attention to the color-coding. As indicated on both sides of the card; plug the black harness to the component side and the white harness to the print side. Step 3: Verify the plugs are locked. NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 89 Step 4: Slide the retention clip latches through the cutouts on the PCB. The latches should face the annotation on the PCB. Step 4: Clamp the retention clip. Verify both latches are firmly locked. NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 90 Step 7: Verify the plugs are locked. Step 8: Slide the retention clip through the cutouts on the PCB. Make sure latches are facing "Black Cable" annotation as seen in the below picture. NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 91 Express x16 slots in the chassis. Step 1: Locate two available PCI Express x16 slots. Step 2: Applying even pressure at both corners of the cards, insert the DPU in the PCI Express slots until firmly seated. Warning NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 92 1. Remove any metallic objects from your hands and wrists. 2. It is strongly recommended to use an ESD strap or other antistatic devices. 3. Turn off the system and disconnect the power cord from the server. NVIDIA BlueField-3 DPU Controller User Manual...

  • Page 93: Pcie Bifurcation Configuration Options

    PCIe bifurcation is supported starting from DOCA 2.5 with BlueField BSP 4.5.0 (released December 2023). NVIDIA BlueField-3 Self-Hosted DPU Controller provides a range of configuration scenarios to meet the demands of environments and deployments. This section describes the various connectivity options for peripherals on the PCIe, such as NVMe SSDs as PCIe endpoints.
  • Page 94 It is recommended to establish out-of-band connectivity to the BlueField DPU Arm OS before setting any of these configurations for the first time. This enables you to reset the NVConfig parameters to their default values in case of misconfiguration. NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 95 PCIe lanes. As seen in the below visual representation of this configuration, the DPU’s Arm cores function as Root Port of all devices connected to both the Gold Fingers and the Cabline CA-II Plus connector. NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 96 -d <device> s PCI_BUS00_SPEED=4  mlxconfig -d <device> s PCI_BUS02_HIERARCHY_TYPE=2  mlxconfig -d <device> s PCI_BUS02_WIDTH=3  mlxconfig -d <device> s PCI_BUS02_SPEED=4  mlxconfig -d <device> s PCI_BUS04_HIERARCHY_TYPE=2  mlxconfig -d <device> s PCI_BUS04_WIDTH=3  mlxconfig -d <device> s PCI_BUS04_SPEED=4  NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 97 PCIe lanes. As seen in the below visual representation of this configuration, the DPU’s Arm cores function as Root Port of all downstream devices connected to both the Goldfingers and the Cabline CA-II Plus connector. NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 98 -d <device> s PCI_BUS15_HIERARCHY_TYPE=2  mlxconfig -d <device> s PCI_BUS15_WIDTH=2  mlxconfig -d <device> s PCI_BUS15_SPEED=4  mlxconfig -d <device> s PCI_BUS16_HIERARCHY_TYPE=2  mlxconfig -d <device> s PCI_BUS16_WIDTH=2  mlxconfig -d <device> s PCI_BUS16_SPEED=4  mlxconfig -d <device> s PCI_BUS17_HIERARCHY_TYPE=2  NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 99 -d <device> s PCI_BUS17_WIDTH=2  mlxconfig -d <device> s PCI_BUS17_SPEED=4  NVIDIA BlueField-3 DPU Controller User Manual...

  • Page 100: Setting High-Speed-Port Link Type

    The DHCP server instructs the PreBoot drivers to access the kernel/OS through a TFTP server, an iSCSI target, or some other service. For more information and instructions, refer to UEFI. NVIDIA BlueField-3 DPU Controller User Manual...

  • Page 101: General Troubleshooting

    Forgot password needed Refer to the latest version of BlueField DPU SW Manual and to install/upgrade the follow the instructions under "Upgrading NVIDIA BlueField DPU image DPU Software" section. NVIDIA BlueField-3 DPU Controller User Manual...

  • Page 102: Specifications

    Optional: Additional PCIe x16 Gen 5.0 @ SERDES 32GT/s Interface through the PCIe auxiliary passive card and Cabline SA-II Plus harnesses On-Board Dual-channel with 20 DDR5 + ECC (128bit + 16bit ECC) Memory for a total of 48GB @ 5200MT/s NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 103 25GBASE-R, 10GBASE-R, 10GBASE-CX4, 1000BASE-CX, CAUI-4 C2M, 25GAUI C2M, XLAUI C2M , XLPPI, SFI Voltage: 12V Electrical Electrical and thermal specifications are provided in " NVIDIA BlueField-3 Thermal DPUs and SuperNICs Electrical and Thermal Specifications " document. You Specificati can access the document either by logging into NVOnline or by contacting your NVIDIA representative.
  • Page 104 Note All dimensions are in millimeters. The PCB mechanical tolerance is +/- 0.13mm. The diagrams may differ for different cards and are provided here for illustration purposes only. Tall Bracket Mechanical Drawing and Dimensions NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 105 NVIDIA BlueField-3 DPU Controller User Manual...

  • Page 106: Monitoring

    Monitoring Thermal Sensors The thermal threshold definitions are described in the " NVIDIA BlueField-3 Networking Platforms Electrical and Thermal Specifications " document. You can access the document either by logging into NVOnline or by contacting your NVIDIA representative. Heatsink Note This section applies to 900-9D3C6-00CV-DA0 only, 900-9D3C6-00CV- GA0 is shipped without a heatsink.

  • Page 107: Finding The Guid/Mac And Serial Number On The Card

    Refer to the board label example and a table listing the different MAC addresses, offering examples and additional information to enhance clarity. DPU Controller Board Label Example Note Note: The product revisions (“Rev”) indicated on the label do not necessarily represent the latest revisions of the cards. NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 108 Port 0: 94 6D AE F5 A1 CC (same as label) Port 1: 94 6D AE F5 A1 CD (+1 from base) Port 2: 94 6D AE F5 A1 CE (+2 from base) Port 3: 94 6D AE F5 A1 CF (+3 from base) NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 109 Controller) for DPU BMC is +1 from the OOB MAC address. A unique serial number that distinguishes your DPU from S/N (Serial others. The S/N is used for Number) tracking and support purposes. NVIDIA BlueField-3 DPU Controller User Manual...

  • Page 110: Pcie Auxiliary Card Kit

    NVIDIA BlueField-3 Auxiliary Kit for additional self-hosted PCIe Gen 4.0 0035-000 x16 connection, two 350mm Cabline SA-II Plus Harnesses 930-9DHAR- NVIDIA BlueField-3 150mm Gen 5.0 FRU harness, Cabline SA-II Plus 60-pin 0001-000 micro coax 38AWG PCIe Auxiliary Card Package Contents...

  • Page 111: Channel Insertion Loss

    The Cabline CA-II Plus harness loss at 16GHz: Harness Length Channel Loss at Gen 5.0 15cm 3.8dB 35cm 7.6dB 55cm 11.4dB The above is measured data; it is recommended to add 0.5dB margins for your system (some loss variations are possible). NVIDIA BlueField-3 DPU Controller User Manual...

  • Page 112: Cabline Ca-Ii Plus Harness Pinouts

    PCIE_REFCLK1_N Micro coax Primary PCIe clock from the motherboard to the Network Adapter Main card, to be used for the x16 Cabline harness PCIe interface. This clock must meet all the PCIe NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 113 Cabline CA-II Plus cable through 220nF PCIE_CPU_CX_14P Micro coax capacitors to the x16 PCIe Tx lanes of the CPU/GPU/End-Point GND BAR PCIE_CPU_CX_13N Micro coax Connect from the Black Cabline CA-II Plus cable through 220nF NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 114 Cabline CA-II Plus cable through 220nF PCIE_CPU_CX_11N Micro coax capacitors to the x16 PCIe Tx lanes of the CPU/GPU/End-Point PCIE_CPU_CX_11P Micro coax Connect from the Black Cabline CA-II Plus cable through 220nF capacitors to the x16 NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 115 220nF PCIE_CPU_CX_9P Micro coax capacitors to the x16 PCIe Tx lanes of the CPU/GPU/End-Point GND BAR PCIE_CPU_CX_8N Micro coax Connect from the Black Cabline CA-II Plus cable through 220nF capacitors to the x16 NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 116 Micro coax capacitors to the x16 PCIe Tx lanes of the CPU/GPU/End-Point Connect from the Black Cabline CA-II Plus cable through 220nF PCIE_CPU_CX_6P Micro coax capacitors to the x16 PCIe Tx lanes of the CPU/GPU/End-Point NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 117 PCIe Tx lanes of the CPU/GPU/End-Point GND BAR Connect from the Black Cabline CA-II Plus cable through 220nF PCIE_CPU_CX_3N Micro coax capacitors to the x16 PCIe Tx lanes of the CPU/GPU/End-Point NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 118 PCIe Tx lanes of the CPU/GPU/End-Point Connect from the Black Cabline CA-II Plus cable through 220nF PCIE_CPU_CX_1P Micro coax capacitors to the x16 PCIe Tx lanes of the CPU/GPU/End-Point GND BAR NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 119 Cabline CA-II Plus interface parameters to the main-card network adapter silicon, like Cabline CA-II Plus cables length (contact NVIDIA for the format of this EEPROM). If additional optional I2C NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 120 I2C bus as well. S_PRSNT1_L Micro coax Connect this pin to No wires are connected to these pins to ensure they do not interfere with the operation of S_PRSNT2_L for the NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 121 Plus harness to the PCIe x16 Rx lanes of PCIE_CX_CPU_0P Micro coax the CPU/GPU/End- Point. 220nF caps are required on this signal on the ConnectX-7 card. PCIE_CX_CPU_0N Micro coax Connect from the White Cabline CA-II NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 122 ConnectX-7 card. GND BAR PCIE_CX_CPU_2P Micro coax Connect from the White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/End- Point. 220nF caps are required on this signal NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 123 CPU/GPU/End- Point. 220nF caps are required on this signal on the ConnectX-7 card. GND BAR PCIE_CX_CPU_4P Micro coax Connect from the White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 124 Connect from the White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of PCIE_CX_CPU_5N Micro coax the CPU/GPU/End- Point. 220nF caps are required on this signal on the ConnectX-7 card. GND BAR NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 125 ConnectX-7 card. PCIE_CX_CPU_7N Micro coax Connect from the White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/End- Point. 220nF caps are required on this signal NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 126 Micro coax the CPU/GPU/End- Point. 220nF caps are required on this signal on the ConnectX-7 card. PCIE_CX_CPU_9N Micro coax Connect from the White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 127 GND BAR Connect from the White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of PCIE_CX_CPU_11P Micro coax the CPU/GPU/End- Point. 220nF caps are required on this signal on the ConnectX-7 card. NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 128 ConnectX-7 card. GND BAR PCIE_CX_CPU_13P Micro coax Connect from the White Cabline CA-II Plus harness to the PCIe x16 Rx lanes of the CPU/GPU/End- Point. 220nF caps are NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 129 PCIe x16 Rx lanes of PCIE_CX_CPU_14N Micro coax the CPU/GPU/End- Point. 220nF caps are required on this signal on the ConnectX-7 card. GND BAR PCIE_CX_CPU_15P Micro coax Connect from the White Cabline CA-II Plus harness to the NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 130 Cabline CA-II Plus interface, this signal is driven from the motherboard side (from the CPU), to the network adapter. When connecting a GPU or an end point to the PCIe Cabline CA-II Plus interface, this NVIDIA BlueField-3 DPU Controller User Manual...

  • Page 131: Technical Specifications

    Cabline harnesses are not connected. No wires are connected to these pins to ensure they do not interfere with the operation of S_PRSNT1_L for the detection when the two Cabline harnesses are installed. Technical Specifications NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 132 EMC: CE / FCC / VCCI / ICES / RCM / KC RoHS: RoHS Compliant PCIe Auxiliary Card Mechanical Drawings and Dimensions All dimensions are in millimeters. The PCB mechanical tolerance is +/- 0.13mm. NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 133 Bracket Mechanical Drawings and Dimensions Auxiliary PCIe Connection Card Tall Auxiliary PCIe Connection Card Short Bracket Bracket Cabline CA-II Plus Harnesses Mechanical Drawing NVIDIA BlueField-3 DPU Controller User Manual...

  • Page 134: Cables And Cabling Configurations

    Debugging and loading new versions of the operating systems and firmware requires the use of Type A to Type B USB 2.0 Cable. The following figure shows an example of Type A to Type V USB Cable. NVIDIA BlueField-3 DPU Controller User Manual...

  • Page 135: Controllers

    General Cooling Recommendations for Heatsink-less DPU Controllers This section offers general cooling recommendations to NVIDIA customers who have acquired a heatsink-less DPU Controller (OPN: 900-9D3C6-00CV-GA0). Warning IMPORTANT NOTES: 1. The customer holds exclusive responsibility for the thermal design and for ensuring all board components never exceed their designated thermal operating limits.
  • Page 136 It is the responsibility of the customer to ensure proper cooling and thermal regulation for all components on the board. The provided guidelines are minimal and are based on NVIDIA's air-cooling thermal solution. Note the following: The DPU controller SoC require contact with TTP on the component side of the product.
  • Page 137 The acceptable pressure on the SoC adheres to the following specifications: 35 psi (SoC area 761.5 mm²). The center force should be applied at the center of the SoC. Component Side Print Side (without Stiffener) Item # Board Components M1394 M111-M115, M221-M225, M331-M335, M441-M445 NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 138 Ensure that the battery remains unobstructed to allow access for replacement. Furthermore, avoid any interference with the TTP. Note that the encapsulation is depicted as the maximum tolerance in the SFG and is labeled as ENC. NVIDIA BlueField-3 DPU Controller User Manual...
  • Page 139 NVIDIA BlueField-3 DPU Controller User Manual...

  • Page 140: Document Revision History

    Controllers Aug. Updated Cabline CA-II Plus Connectors Pinouts. 2023 Updated Cabline CA-II Plus Harness Pinouts. Added a note to NVMe SSD Interface. May. First release 2023 © Copyright 2024, NVIDIA. PDF Generated on 08/16/2024 NVIDIA BlueField-3 DPU Controller User Manual...

Table of Contents