Cray ClusterStor L300 Installation Manual
  1. Manuals
  2. Brands
  3. Cray Manuals
  4. Storage
  5. ClusterStor L300
  6. Installation manual

Cray ClusterStor L300 Installation Manual

Hide thumbs
Table of Contents

Advertisement

Quick Links

ClusterStor™ L300 and L300N Field
Installation Guide
(3.0.0)
H-6163

Advertisement

Table of Contents
loading
Need help?

Need help?

Do you have a question about the ClusterStor L300 and is the answer not in the manual?

Questions and answers

Summary of Contents for Cray ClusterStor L300

  • Page 1 ClusterStor™ L300 and L300N Field Installation Guide (3.0.0) H-6163...

  • Page 2: Table Of Contents

    Contents Contents 1 About the ClusterStor L300 and L300N Field Installation Guide................5 2 Install a ClusterStor L300 and L300N System......................7 3 Prerequisites to Installation.............................9 3.1 Service Area Clearances.........................10 3.2 Plan for Rack Position..........................11 3.3 Safe Handling and Precautions Check List.....................12 4 Uncrate the ClusterStor L300 and L300N Rack....................15...
  • Page 3 15 Configure an Additional MMU to Support DNE....................94 16 Reference................................95 16.1 Power Off the ClusterStor L300 and L300N with CSCLI...............95 16.2 Power Off the ClusterStor L300 and L300N System via CSSM............96 16.3 Internal Rack Cabling Reference......................97 16.3.1 ClusterStor L300 and L300N SMU Physical and Logical Port Numbering.......99 16.3.2 ClusterStor L300 and L300N MMU Physical and Logical Port Numbering......100...
  • Page 4 Contents 16.5.10 Base Rack Inverted Power Distribution for PX2-5100X2-V2..........132 16.5.11 Storage Rack Optional MMU Power Distribution for PX2-5100X2-V2.........133 16.5.12 Storage Rack Optional MMU Inverted Power Distribution for PX2-5100X2-V2....134 16.5.13 Base Rack Standard Power Distribution for PX2-5551-N4V2..........135 16.5.14 Base Rack Inverted Power Distribution for PX2-5551-N4V2..........136 16.5.15 Storage Rack Optional MMU Standard Power Distribution for PX2-5551-N4V2....137 16.5.16 Storage Rack Optional MMU Inverted Power Distribution for PX2-5551-N4V2....138 16.5.17 Base Rack Optional AMMU Standard Power Distribution for PX2-5965X3-V2....139...

  • Page 5: About The Clusterstor L300 And L300N Field Installation Guide

    XC™ Series Lustre® Administration Guide S-2648 Scope and Audience This publication is written for trained Cray technicians and provides instructions to unpack and install a ClusterStor L300 and L300N system at the customer site. It does not include information about daily operation. Typographic Conventions Monospace Indicates program code, reserved words, library functions, command-line prompts, screen output, file/path names, and other software constructs.
  • Page 6 LIBSCI, NODEKARE. The following system family marks, and associated model number marks, are trademarks of Cray Inc.: CS, CX, XC, XE, XK, XMT, and XT. The registered trademark LINUX is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a worldwide basis. Other...

  • Page 7: Install A Clusterstor L300 And L300N System

    8. Perform the first-run configuration 9. Perform the advanced MMU (AMMU) installation procedure, If the system being installed includes an AMMU. The figure shows an each of the ClusterStor L300 and L300N networks: ● Lustre client network (LCN) - Customer Lustre client network: ○...
  • Page 8 Install a ClusterStor L300 and L300N System Figure 1. ClusterStor L300 and L300N Networks Overview Lustre Client Network (LCN) Lustre Clients Lustre Client Network (x 1000s) Core Switch Base Rack AD/LDAP/NIS server* LDN Switch 1 LDN Switch 0 CSM Administrator...

  • Page 9: Prerequisites To Installation

    Prerequisites to Installation Prerequisites to Installation Verify that these prerequisites have been satisfied before installing the ClusterStor L300 and L300N system. Acclimate the System Cabinets When the system is unloaded from the climate controlled truck, the shipping environment temperature will be close to the computer room. If the system has been stored in an environment that is significantly colder than the environment in which it will be installed (15 °F [8 °C] or greater disparity), leave the system cabinets in their shipping configuration for at...

  • Page 10: Service Area Clearances

    Contact Cray Support for root access. Service Area Clearances Before unpacking the ClusterStor L300 and L300N system crate and beginning the installation, verify that recommended service area clearances are available and that personnel are aware of the safe handling procedures and precautions.

  • Page 11: Plan For Rack Position

    Figure 2. Rack Service Area Clearance Plan for Rack Position When unpacking and assembling a multi-rack ClusterStor L300 and L300N system, carefully plan the order to unpack and assemble the racks. The base rack (either as a stand-alone shipment or part of a system) is identified with “R1” & “BASE,”.

  • Page 12: Safe Handling And Precautions Check List

    Figure 3. SSU Rack Labeling Safe Handling and Precautions Check List Before unpacking the ClusterStor L300 and L300N system, complete this prescribed checklist and observe the stated precautions. CAUTION: Read and understand these instructions completely before starting any physical installation activity.
  • Page 13 ● The ClusterStor L300 and L300N rack weighs up to 2513 lbs. (1140 kg). It is possible for the rack to tip over while it is being moved. Do not lean the rack more than 5 degrees from a level surface or when rolling down an incline or ramp.
  • Page 14 Prerequisites to Installation ● The dedicated circuit must have circuit breaker or fuse protection. The PDUs provide handle-type circuit breakers. Adequate protection must be provided for the dedicated power circuit. Protection of capacity equal to the current rating of the distribution unit must be provided and must meet all applicable codes and regulations.

  • Page 15: Uncrate The Clusterstor L300 And L300N Rack

    1720 lbs (780 kg), can enable the cabinet to roll, and will not be damaged. About this task This task describes how to safely remove the ClusterStor L300 and L300N rack from the shipping crate. Two people are required to safely roll the rack off the pallet ramp.
  • Page 16 Uncrate the ClusterStor L300 and L300N Rack Procedure 1. Use a pallet jack to position the crate so that there is adequate space to roll the rack down the ramp. 2. Unlock the draw latches that secure the rear panel to the crate.

  • Page 17: Move The Rack To Its Final Position

    Move the Rack to its Final Position Move the Rack to its Final Position Prerequisites Procedures The ClusterStor L300 and L300N rack has been unpacked, rolled down the ramp, and the lower two SSUs are populated with disk drives. Tools Bubble level...
  • Page 18 Move the Rack to its Final Position Figure 5. No Side Push Label IMPORTANT: The rack may only be moved up or down a maximum incline/decline of 5 degrees. 3. Position the rack to its final location and lower the leveling feet using a 13mm wrench. 4.

  • Page 19: Install Disk Drives In A 5U84 Enclosure

    Install Disk Drives in a 5U84 Enclosure Prerequisites The ClusterStor L300 and L300N rack has been unpacked, and moved at least off the pallet and down the ramps to the floor. It may already be located in its final operating position.
  • Page 20 Install Disk Drives in a 5U84 Enclosure 2. Install the disks in the same location they were in at the factory by referencing the drive numbers, orientation (front/back, upper/lower), or SSD designation provided by manufacturing on the shipping container. From the front of the container, drive 0 is in the lower-left slot drive 41 in the upper-right slot. Drive 42 is in the lower-slot, drive 83 in the upper-right slot.
  • Page 21 Install Disk Drives in a 5U84 Enclosure Figure 7. Disk Drive in Drive Carrier Figure 8. 5U84 Drive Slot Locations NOTE: The two drive carriers with SSD drives install in slots 28 and 70. 5. Close the first drawer, open the drawer above it, and populate it with disk drives from the rear slots to the front slots, as before.
  • Page 22 Install Disk Drives in a 5U84 Enclosure Figure 9. Install the Disk Individually 8. Starting with the lower drawer in the lowest empty 5U84 enclosure, open the drawer and begin populating the drives from rear to the front, starting with slot labeled “83” and work towards the slot “42.” 9.
  • Page 23 Install Disk Drives in a 5U84 Enclosure Figure 10. Speedloader Figure 11. Speedloader Drive Slot Insertion Order 16. Pull out on the locking pin to release the locking arm side. 17. Position the rigid side of the speedloader to one side of the drives (at a slight angle), rotate the speedloader into position on the other side of the drives, and close the locking arm.
  • Page 24 Install Disk Drives in a 5U84 Enclosure 21. Lock the drives into the slot: press down while sliding the latch to the locked position, towards the rear of the enclosure. Repeat for each disk drive. 22. Note the position of the interface card, as the drives must be installed with the card facing the front of the drawer.

  • Page 25: Install Disk Drives In A 2U24 Enclosure

    Install Disk Drives in a 2U24 Enclosure Install Disk Drives in a 2U24 Enclosure Prerequisites ● A disk drive has failed and has been removed from the 2U24 enclosure ● A T10 Torx screwdriver may be required to unlock the drive carrier About this task Disk drives in the 2U24 enclosure are installed at the factory and do not need to be installed during a system installation.
  • Page 26 Install Disk Drives in a 2U24 Enclosure Figure 12. Secure the Drive Carrier 5. Seat the drive carrier by pressing fingers on the handle latch, pivoting it to a closed position, as shown. The latch clicks as it engages and holds the handle closed. 6.

  • Page 27: Cable The External Power Connections

    The PDUs must be inverted to feed the power through the top of the ClusterStor L300 and L300N rack. Note that the power sockets are always numbered from the top down; therefore, although the numbering may visually seem...
  • Page 28 Cable the External Power Connections Figure 14. ClusterStor L300 and L300N PDU Options, Bottom Feed C13 (A1) C19 (A1) C13 (A2) C19 (A1) C19 (A1) C19 (A3) C19 (A2) C19 (A2) C19 (A2) C19 (A4) C19 (A3) C19 (A3) C19 (A3)
  • Page 29 Cable the External Power Connections Figure 15. ClusterStor L300 and L300N PDU Options, Top Feed Line B Line A Line B Line C Line A Line A Line B Line A C19 (A24) C19 (A23) C13 (A22) C19 (B12) C19 (B12)
  • Page 30 1. Place the rack PDU power switches and the power switches on all the other components in the OFF position. Not all ClusterStor L300 and L300N components have power switches; in those cases the power cord controls power to the component. Verify that the power cords for those components are disconnected.

  • Page 31: Reset The Circuit Breaker

    The handle-type circuit breaker located on the ClusterStor L300 and L300N power distribution unit (PDU) may require resetting. About this task How to reset a circuit breaker on a ClusterStor L300 and L300N PDU. Each PDU has one fuse/circuit breaker per input phase. Procedure 1.
  • Page 32 Cable the External Power Connections Procedure 1. Compare the appearance of the PDU with the following figure. Figure 17. Circuit Breaker Status Conditions 2. To trip the circuit breaker when required, insert a small flat-blade screwdriver into the small slot on the OFF side of the breaker, and press.
  • Page 33 Cable the External Power Connections Figure 18. Trip the Circuit Breaker...

  • Page 34: Standardized Rack Configurations

    Standardized Rack Configurations Standardized Rack Configurations The following figures show the component positions installed in ClusterStor L300 and L300N base racks and storage racks. Three configurations are available for a ClusterStor L300 and L300N system: SSU-only, SSU+1, and SSU+2. The SSU+1 and SSU+2 configurations use expansion storage units (ESUs) and EBODs to expand the storage capacity of each SSU.
  • Page 35 Standardized Rack Configurations Typical private IP address for SMU controllers, MMU controllers, and SSU OSS nodes, and the IBoIP addresses are listed in the table. Table 4. Typical IP Addresses for SMU, MMU, OSS, and IPMI IB Net IPMI n000 172.16.2.3 10.10.100.1 172.16.0.101...

  • Page 36: Cable Inter-Rack Local Management Network (Lmn)

    Cable Inter-rack Local Management Network (LMN) 10 Cable Inter-rack Local Management Network (LMN) Inter-rack cabling in a multi-rack ClusterStor L300 and L300N system applies to only the Local Management Network LMN). Management switches in the base rack are connected to the management switches in each storage rack.
  • Page 37 Cable Inter-rack Local Management Network (LMN) Figure 20. Cable Inter-rack Local Management Network (LMN) Base Rack 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 X1 X2...

  • Page 38: Environment Connections

    11.1 Cable SMU and 36-Port InfiniBand EDR LDN with Dual 24-Port LMN Switches About this task For ClusterStor L300 and L300N inter-rack 36-port InfiniBand LDN switches, the available uplink ports are: ● Upper switch 1 (IB1) - ports 1 through 21, 23, 25, 26, 29, 31, 33, and 35 ●...
  • Page 39 Environment Connections Figure 22. Cable SMU to External Administration Network pub0 interfaces (NIC 5) To external EAC A Node n000 administration network (EAN) EAC B Node n001 Figure 23. Cable Storage Rack 36-Port IB EDR LDN 24-Port LMN Using Core Switch From base rack From base rack SW0 switch...

  • Page 40: Cable Smu And 36-Port Infiniband Edr Ldn With Dual 48-Port Lmn Switches

    11.2 Cable SMU and 36-Port InfiniBand EDR LDN with Dual 48-Port LMN Switches About this task For ClusterStor L300 and L300N inter-rack 36-port EDR InfiniBand LDN switch cabling, the available uplink ports are: ● Upper switch 1 (IB1) - ports 1 through 21, 23, 25, 26, 29, 31, 33, and 35 ●...

  • Page 41: Cable Smu And 36-Port 40Gbe Ldn And Dual 24-Port Lmn Switches

    11.3 Cable SMU and 36-Port 40GbE LDN and Dual 24-Port LMN Switches About this task For ClusterStor L300 and L300N inter-rack 40GbE LDN switch cabling, the available uplink ports are: ● Upper switch 1 (40G1) - all odd-numbered ports ●...
  • Page 42 Environment Connections ● Lower switch 0 (SW0) - 5, 7, 17, and 19 IMPORTANT: Switch port assignments in the configuration diagrams are provided for reference only. For system-specific inter-rack switch cabling, refer to the system-specific storage configuration document (SCD). Figure 26. Cable Base Rack 36-Port 40GbE LDN and Dual 24-Port LMN Using Core Switch LMN to storage racks 1, 2, 3 ...

  • Page 43: Cable Smu And 36-Port 40Gbe Ldn With Dual 48-Port Lmn Switches

    11.4 Cable SMU and 36-Port 40GbE LDN with Dual 48-Port LMN Switches About this task For ClusterStor L300 and L300N inter-rack 40GbE 36-port LDN switch cabling, the available uplink ports are: ● Upper switch 1 (40GS1) - ports 2, 4, and all odd-numbered ports ●...
  • Page 44 Environment Connections Figure 29. Base Rack 36-Port 40GbE LDN and Dual 48-Port LMN Using Core Switch LMN to storage racks 1, 2, 3 ... N SW1 switch LMN to storage X1 X2 X3 X4 X5 X6 X7 X8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24...

  • Page 45: Cable Smu And 24-Port Omni-Path Ldn With Dual 24-Port Lmn Switches

    11.5 Cable SMU and 24-Port Omni-path LDN with Dual 24-Port LMN Switches About this task For ClusterStor L300 and L300N inter-rack 24-port Omni-Path LDN switch cabling, the available uplink ports are: ● Upper switch 1 (OPA1) - 17 through 24 ●...

  • Page 46: Cable Smu And 24-Port Omni-Path Ldn Dual 48-Port Lmn Switches

    Environment Connections Figure 33. Cable Storage Rack with 24-Port OPA LDN and Dual 24-Port LMN Using Core Switch, ClusterStor L300 and L300N From base rack From base rack SW0 switch SW1 switch 9 10 11 12 13 14 15 16...
  • Page 47 Environment Connections ● Upper switch 1 (OPA1) - 17 through 24 ● Lower switch 0 (OPA2) - 17 through 24 For inter-rack LMN switch cabling, the available uplink ports are: ● Upper switch 1 (SW1) - ports 1 through 24, 29, 31, 41, and 43 ●...

  • Page 48: Cable Smu And Omni-Path 48-Port Ldn With Dual 24-Port Lmn Switches

    11.7 Cable SMU and Omni-path 48-Port LDN with Dual 24-Port LMN Switches About this task For ClusterStor L300 and L300N inter-rack Omni-Path (OPA) 48-port LDN switch cabling, the available uplink ports are: ● Upper switch 1 (OPA1) - 17 through 48 ●...
  • Page 49 Environment Connections Figure 36. Cable Base Rack with 48-Port OPA LDN and Dual 24-Port LMN Using Core Switch LMN to storage racks 1, 2, 3 ... N SW1 switch LMN to storage 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 racks 1, 2, 3 ...

  • Page 50: Cable Smu And Omni-Path 48-Port Ldn With Dual 48-Port Lmn Switches

    11.8 Cable SMU and Omni-path 48-Port LDN with Dual 48-Port LMN Switches About this task For ClusterStor L300 and L300N inter-rack 48-port Omni-Path (OPA) switch LDN cabling, the available uplink ports are: ● Upper switch 1 (OPA1) - 17 through 48 ●...
  • Page 51 Environment Connections Figure 39. Cable Base Rack with 48-Port OPA LDN and Dual 48-Port LMN Using Core Switch LMN to storage racks 1, 2, 3 ... N SW1 switch LMN to storage 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 9 10 11 12...

  • Page 52: Cable Smu And 36-Port 100Gbe Ldn With Dual 24-Port Lmn Switches

    11.9 Cable SMU and 36-Port 100GbE LDN with Dual 24-Port LMN Switches About this task For ClusterStor L300 and L300N inter-rack 36-port 100GbE LDN switch cabling, the available uplink ports are: ● Upper switch 1 (100G1) - all odd-numbered ports ●...
  • Page 53 Environment Connections Figure 42. Base Rack with 32-Port 100GbE LDN and 24-Port LMN Using Core Switch LMN to storage racks 1, 2, 3 ... N SW1 switch LMN to storage 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 racks 1, 2, 3 ...

  • Page 54: Cable Smu And 36-Port 100Gbe Ldn With Dual 48-Port Lmn Switches

    11.10 Cable SMU and 36-Port 100GbE LDN with Dual 48-Port LMN Switches About this task For ClusterStor L300 and L300N inter-rack 100GbE LDN switch cabling, the available uplink ports are: ● Upper switch 1 (100G1) - ports 2, 4, and all odd-numbered ports ●...
  • Page 55 Environment Connections Figure 45. Cable Base Rack with 32-Port 100G LDN and 48-Port LMN Using Core Switch LMN to storage racks 1, 2, 3 ... N SW1 switch LMN to storage X1 X2 X3 X4 X5 X6 X7 X8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24...
  • Page 56 IMPORTANT: Do not use the bePowerOnNodes script for ClusterStor L300 and L300N systems that are operating in Daily mode. The ClusterStor L300 and L300N ships in Daily mode and does not require the bePowerOnNodes script to be run before performing the first-run configuration.
  • Page 57 Power On the ClusterStor L300 and L300N System The MMU and each ESU have two power cooling modules (PCMs), and each SSU has four power cooling modules (PCMs) with power switches, located next to the MMU I/O modules. d. Verify that the rack PDU circuit breakers are tripped (in the OFF position).
  • Page 58 Power On the ClusterStor L300 and L300N System Figure 48. EAC for SMU Nodes Power Reset Ethernet NIC 2 mgmt0 (secondary management network) OK LED Ethernet NIC 1 mgmt1 (primary management network) Fault LED Maint ports SAS port 1 ID LED...
  • Page 59 Power On the ClusterStor L300 and L300N System Figure 50. EAC for SSU and MMU Power Reset Ethernet NIC 2 mgmt0 (secondary management network) OK LED Ethernet NIC 1 mgmt1 (primary management network) Fault LED Maint ports SAS port 1...
  • Page 60 [admin@sn11100n000]$ sudo ping IPaddress If the statement Destination host unreachable appears in the output, that node is not connected. Repeat this step until both nodes appear, or contact Cray service. 13. Verify that both nodes are started: [admin@sn11100n000]$ sudo crm_mon -r1 In some cases, node 0 will fail over to node 1.
  • Page 61 Power On the ClusterStor L300 and L300N System sn11100n000-1-ipmi-stonith (stonith:external/sea_ipmi): Started sn11100n000 sn11100n001-2-ipmi-stonith (stonith:external/sea_ipmi): Started sn11100n001 Clone Set: cln-kdump-stonith [kdump-stonith] Started: [ sn11100n000 sn11100n001 ] Clone Set: cln-ssh-10-stonith [ssh-10-stonith] Started: [ sn11100n000 sn11100n001 ] Clone Set: cln-gem-stonith [gem-stonith] Started: [ sn11100n000 sn11100n001 ]...
  • Page 62 Power On the ClusterStor L300 and L300N System prm-vip-eth0-mgmt (ocf::heartbeat:IPaddr2): Started sn11100n000 prm-vip-eth0-ipmi-sec (ocf::heartbeat:IPaddr2): Started sn11100n000 prm-vip-eth0-nfs (ocf::heartbeat:IPaddr2): Started sn11100n001 Resource Group: sn11100n000_md64-group sn11100n000_md64-raid (ocf::heartbeat:XYRAID): Started sn11100n000 sn11100n000_md64-fsys (ocf::heartbeat:XYMNTR): Started sn11100n000 sn11100n000_md64-stop (ocf::heartbeat:XYSTOP): Started sn11100n000 Resource Group: sn11188n000_md67-group sn11100n000_md67-raid (ocf::heartbeat:XYRAID):...
  • Page 63 Cluster is in 'Daily' mode. cluster_mode: name: sn11100n cluster_mode: UUID: 15fd9368-d665-11e5-9b44-0025901f832e a. If the cluster is indicated to be in Daily mode as shown, power up is complete. b. If the system is not in Daily mode, contact Cray hardware product support (HPS).
  • Page 64 Power On the ClusterStor L300 and L300N System 22. To continue system configuration or make configuration changes, return system to Pre-shipment mode: [admin@sn11100n000 network-scripts]$ sudo cscli cluster_mode --mode pre-shipment...

  • Page 65: Perform The First-Run Configuration

    13 Perform the First-Run Configuration About this task After the ClusterStor L300 and L300N system is installed and powered on, launch CSM wizard and perform the first-run configuration procedures. After all the procedures are complete, the ClusterStor L300 and L300N system is fully configured for the customer environment and ready for use.
  • Page 66 Perform the First-Run Configuration Figure 51. CCM Welcome Screen Procedure 1. Create the administrator password. The password is limited to 8 characters. 2. Re-enter the new administrator password to confirm. 3. Click Set Password and Proceed with Login. 4. Monitor progress: ●...
  • Page 67 8. Click Next. The Upload SSL Certificate screen displays (optional). ClusterStor L300 and L300N systems support only privacy enhanced mail (.PEM) format files. If the SSL certificate file is in a different format, convert it to .PEM before continuing with this procedure.

  • Page 68: Configure The Lustre Network Interfaces

    It is the administrator's choice whether to specify a single IP address range for all Lustre nodes in the ClusterStor L300 and L300N system or use multiple IP address ranges for different nodes (for example, one IP address range for the MGS and MDS nodes, and a separate IP address range for the OSS nodes). To...

  • Page 69: Troubleshoot The Lustre Network Screen

    If InfiniBand is used as the high-speed network, set the default MTU value to 65520, which supports jumbo frames (JF) traffic. This is the optimal setting for IP over InfiniBand (IPoIB). b. If 40GbE is used as the high-speed network, Cray recommends setting the MTU value to 9000 to enable support of jumbo frames over Ethernet.
  • Page 70 Perform the First-Run Configuration Some of the procedures require root system access. Contact Cray Support to acquire root access, if needed. "Expected and actual interface addresses differ, possibly some addresses are already in use" error. This error is returned after entering network settings and selecting Next.
  • Page 71 Perform the First-Run Configuration [nxxx]$ sudo mdraid-activate -d -m [nxxx]$ sudo bash -x mdraid-activate -d -m (The second command provides more information.) 2. Check to see if any hard drives have failed: [nxxx]$ sudo dm_report Review the output for the failed drive (see slot 16 in the sample output below): [admin@n005 ~]$ sudo dm_report Diskmonitor Inventory Report: Version: 1.0.x.1.5-16.x.2412 Host: fgbe05 Time: Fri Mar 14 14:10:38 2014...

  • Page 72: Run Lustre Validation Tests

    Perform the First-Run Configuration 13.4 Run Lustre Validation Tests Prerequisites About this task This procedure prepares to run both Lustre basic tests and IOZone performance tests. When the tests complete, the results display on the progress screen and are available for download. Figure 55.

  • Page 73: Run Lustre Validation Tests -- Lustre Not Mounted

    Lustre was either never mounted on the system or it has been unmounted. About this task This procedure runs both Lustre basic tests and IOZone performance tests on a ClusterStor L300 and L300N system. When the tests complete, the results display on the progress screen and are available for download.

  • Page 74: Troubleshoot The Lustre Validation Tests

    13.4.2 Troubleshoot the Lustre Validation Tests This section presents possible causes and resolutions for errors that may occur during the Lustre validation tests for ClusterStor L300 and L300N systems. IMPORTANT: Carefully inspect any errors generated by CSM. In many cases the errors reveal the exact problem through backend script output, middleware issues, etc.
  • Page 75 Perform the First-Run Configuration [admin@nxxx]$ cat /proc/mdstat Make certain the corresponding MD device is present. If the device is not present, check dmesg for mdraid assembling problems. Most likely one of the drives needed for this has failed and must be replaced.

  • Page 76: Monitor Progress Screen

    Lustre validation tests are complete and the results display on the Review Progress screen. If the tests do not run, the test will not return I/O rate information. In this case, see Post-Install Checkout on page 86 for advanced procedures. About this task When the ClusterStor L300 and L300N Figure 57. CSM Progress Screen...

  • Page 77: Review The Complete Screen

    2. Select Nodes in FS fsname (which is usually the default). 3. Click All Nodes in Filter menu tab. 4. Select Start Lustre from the drop-down menu. 3. CSM starts and the ClusterStor L300 and L300N system is ready for use.

  • Page 78: Specify Lustre User Authentication

    Perform the First-Run Configuration 13.7 Specify Lustre User Authentication Prerequisites IMPORTANT: Stop Lustre on the ClusterStor L300 and L300N system before configuring Lustre users. About this task This optional procedure specifies the settings necessary to authenticate Lustre users on CSM, which opens when the process is complete.

  • Page 79: Specify Lustre User Authentication Via The Active Directory Screen

    2. In the AD Server field, enter the AD server's hostname or IP address (if not already displayed). The AD server can be specified either as a hostname or as an IP address. ClusterStor L300 and L300N systems use service discovery (DNS-SD) to query the DNS server and discover the AD server's hostname/IP address.

  • Page 80: Specify Lustre User Authentication Via The Ldap Screen

    13.7.2 Specify Lustre User Authentication via the LDAP Screen Prerequisites ● LDAP has been selected as the Lustre user authentication method for the ClusterStor L300 and L300N system. ● If the LDAP server is being specified using the hostname, the DNS must first be configured (as part of Configure the Lustre Network Interfaces).
  • Page 81 2. In the LDAP Server field, enter the LDAP server's hostname or IP address (if not already displayed). The LDAP server can be specified either as a hostname or as an IP address. ClusterStor L300 and L300N systems use service discovery (DNS-SD) to query the DNS server and discover the LDAP server's hostname/ IP address.
  • Page 82 Perform the First-Run Configuration Figure 62. CSM LDAP Additional Servers and Ports Screen 5. To use TLS/SSL for accessing LDAP, perform the following. Figure 63. CSM LDAP Storage Cluster Authentication Checkbox Screen IMPORTANT: Use TLS/SSL port 636 if TLS/SSL is used with the LDAP. a.
  • Page 83 Perform the First-Run Configuration c. Browse to, and select, the desired file. d. Click Open. 6. In the Base DN field, enter the base DN (distinguished name) value. 7. In the User DN field, enter a valid user DN value. IMPORTANT: The User DN field remains blank until a base DN value is specified.

  • Page 84: Specify Lustre User Authentication Via The Nis Screen

    11. Click Save. 13.7.3 Specify Lustre User Authentication via the NIS Screen Prerequisites NIS has been selected as the Lustre user authentication method for the ClusterStor L300 and L300N system. Procedure 1. Select NIS on the left side of the screen.

  • Page 85: Specify Lustre User Authentication Via The Local Screen

    Click the red - button to remove NIS servers, if required. Figure 67. CSM NIS Additional Servers Screen 5. Click Save. 13.7.4 Specify Lustre User Authentication via the Local Screen Prerequisites Local has been selected as the Lustre user authentication method for the ClusterStor L300 and L300N system.

  • Page 86: Troubleshoot The Lustre Users Screen

    Some of the parameters are likely wrong. Before applying LDAP parameters to the ClusterStor L300 and L300N system, the ldapsearch command is called with the specified parameters. If ldapsearch is not able to use the parameters, LDAP will not work and the error is generated.

  • Page 87: Post-Install Checkout

    Perform the First-Run Configuration 13.8 Post-Install Checkout Prerequisites Installation of the ClusterStor L300 and L300N system is complete and the Lustre file system is running. Procedure 1. If Lustre is not running, start Lustre: [MGMT0]$ cscli mount-f file_system_name 2. Run obdfilter-survey on the primary OSS nodes:...
  • Page 88 Perform the First-Run Configuration 2. Determine the file system name. On the primary MGMT node: [admin@sn11000n000 ~]$ cscli fs_info Examples show a file system named sn11000: [admin@sn11000n000 ~]$ cscli fs_info ---------------------------------------------------------------------------------------------------- OST Redundancy style: Declustered Parity (GridRAID) Disk I/O Integrity guard (ANSI T10-PI) is not supported by hardware ---------------------------------------------------------------------------------------------------- Information about "sn11000"...
  • Page 89 10. Remove the temporary mount point: [MGMT1]# rmdir /tmp/testmount The first-run configuration of the ClusterStor L300 and L300N system is now complete. IMPORTANT: The Lustre file system remains mounted on the cluster. To perform additional updates or changes, unmount the file system first.
  • Page 90 Perform the First-Run Configuration [MGMT0]# cscli unmount –f testfs...

  • Page 91: Configure Smtp Relay, Ras Email, Remote Support, And Service Console

    Configure SMTP Relay, RAS Email, Remote Support, and Service Console 14 Configure SMTP Relay, RAS Email, Remote Support, and Service Console Prerequisites Before performing any of the procedures in this section, verify that RAS is installed on the system: [MGMT0]$ cscli service_console The system responds with the current RAS status, and provides help commands.
  • Page 92 Configure SMTP Relay, RAS Email, Remote Support, and Service Console [MGMT0]$ cscli service_console configure smtp relay --help Configure the Email Notification List This section explains how to configure a list of users to be sent notifications and how to enable/disable individual user notifications.
  • Page 93 Configure SMTP Relay, RAS Email, Remote Support, and Service Console Configure the Service Console 7. Verify that the service console is enabled: [MGMT0]$ cscli service_console enable IMPORTANT: ClusterStor L300 and L300N are shipped with service console capabilities enabled, by default. (To disable the service console:) [MGMT0]$ cscli service_console disable...

  • Page 94: Configure An Additional Mmu To Support Dne

    15 Configure an Additional MMU to Support DNE About this task An additional MMU enables the ClusterStor L300 and L300N systems to take advantage of the Lustre distributed namespace (DNE) feature. Additional MMUs are configured in manufacturing. Contact Cray Support for more information about configuring additional MMUs to support DNE.

  • Page 95: Reference

    Unmount the Lustre file system from all clients before starting the power off procedure. Failure to do so may cause clients to hang. About this task The CSCLI power off procedure stops Lustre on all ClusterStor L300 and L300N nodes. Procedure 1. SSH to the primary MGMT node: [Client]$ ssh -l admin primary_MGMT_node 2.

  • Page 96: Power Off The Clusterstor L300 And L300N System Via Cssm

    The power switches for the SSUs and ESUs are on the power supply units (PSUs). 16.2 Power Off the ClusterStor L300 and L300N System via CSSM Prerequisites Stop Lustre before performing this procedure. About this task The CSM power off procedure stops Lustre on selected ClusterStor L300 and L300N nodes.

  • Page 97: Internal Rack Cabling Reference

    Reference Procedure Log into CSSM. 2. Click on the Node Control tab. 3. Stop the Lustre file system. 4. Power off the nodes. a. Select all nodes to power off. b. Select Shutdown from the Selected Nodes drop-down menu. 5. After all of the nodes are shut down, physically power off the enclosures -- turn off the power switches (at the back of the rack).
  • Page 98 40g 100g Local data network (40GbE/100GbE) The identification for the ClusterStor L300 and L300N components are established from the bottom up. The lowest component in the rack begins numbering at 0 and increments by 1, as an example, SSU0, SSU1, SSU2 and so forth.

  • Page 99: Clusterstor L300 And L300N Smu Physical And Logical Port Numbering

    13 (or 37 on a 48-port switch) on each switch regardless of the rack type. Power Distribution Units (PDUs) are identified as PDU0 for the left side and PDU1 for the right side. 16.3.1 ClusterStor L300 and L300N SMU Physical and Logical Port Numbering ClusterStor L300 and L300N SMU physical and logical port numbering.

  • Page 100: Clusterstor L300 And L300N Mmu Physical And Logical Port Numbering

    SAS ports A, B, SAS connections (12Gbit/s) 16.3.2 ClusterStor L300 and L300N MMU Physical and Logical Port Numbering ClusterStor L300 and L300N MMU physical and logical port numbering. Figure 70. MMU Physical and Logical Port Numbering Table 8. MMU Physical and Logical Port Numbering...

  • Page 101: Cable Ammu Sas 1U Servers

    The AMMU includes two 1U servers connected via SAS cables to a 2U24 EBOD storage unit. The AMMU provides the same functions and storage resources as the MMU in a ClusterStor L300 and L300N base rack.

  • Page 102: Internal Rack Cabling - Local Management Network

    Reference Physcial Port Label Logical Port Connection Onboard NIC 3 (MGT eth3 Unused Nodes) Data Network Port I IBS 1 Data network connection Data Network Port II IBS 2 Data network connection SAS ports A, B, C, D SAS connections (6Gbit/s or 12Gbit/s) 16.3.5 Internal Rack Cabling - Local Management Network ●...
  • Page 103 Reference Loom Label Component Side/Port Switch Port White LH Side SSU1 Eth SW0 White LH Side SSU2 Eth SW0 White LH Side SSU3 Eth SW0 White LH Side SSU4 Eth SW0 White LH Side LHS PDU Eth SW0 White LH Side SSU5 Eth SW0 White LH Side...
  • Page 104 Reference Loom Label Component Side/Port Switch Port White RH Side MMU / AMMU Eth SW0 White RH Side Eth SW0 White RH Side Fabric SW 0 1 (on rear of Eth SW0 switch) Table 11. Lustre 24-Port Management Network Interconnect List - Storage Rack with Optional MMU Loom Label Component...
  • Page 105 Reference Loom Label Component Side/Port Switch Port Blue RH Side SSU3 Eth SW1 Blue RH Side SSU4 Eth SW1 Blue RH Side RHS PDU Eth SW1 Blue RH Side SSU5 Eth SW1 Blue RH Side Unused Unused Eth SW1 Blue RH Side Unused Unused Eth SW1...
  • Page 106 Reference Loom Label Component Side/Port Switch Port Blue LH Side Unused Unused Blue LH Side Unused Unused Blue LH Side MMU / AMMU Eth SW1 Blue LH Side Eth SW1 White LH Side SSU0 Eth SW0 White LH Side SSU1 Eth SW0 White LH Side SSU2...

  • Page 107: Internal Rack Cabling - Local Data Network

    Reference Loom Label Component Side/Port Switch Port White RH Side SSU4 Eth SW0 White RH Side SSU5 Eth SW0 White RH Side Unused Unused White RH Side Unused Unused White RH Side MMU / AMMU Eth SW0 White RH Side Eth SW0 White RH Side Fabric SW 0...
  • Page 108 Reference Loom Label Component Side/Port Switch Port Blue RH Side SSU1 IB SW1 Blue RH Side SSU2 IB SW1 Blue RH Side SSU3 IB SW1 Blue RH Side SSU4 IB SW1 Blue RH Side SSU5 IB SW1 Blue RH Side MMU / AMMU B I / I IB SW1...
  • Page 109 Reference Loom Label Component Side/Port Switch Port White LH Side SSU2 OPA SW0 White LH Side SSU3 OPA SW0 White LH Side SSU4 OPA SW0 White LH Side SSU5 OPA SW0 White LH Side MMU / AMMU A 1 / 1 OPA SW0 White LH Side OPA SW0...
  • Page 110 Reference Loom Label Component Side/Port Switch Port Blue RH Side Optional OPA SW1 Additional MMU Table 17. Lustre Omni-Path Data Network Interconnect List - Base Rack (48-port) Loom Label Component Side/Port Switch Port White LH Side SSU0 OPA SW0 White LH Side SSU1 OPA SW0 White LH Side...
  • Page 111 Reference Loom Label Component Side/Port Switch Port Blue RH Side SSU0 OPA SW1 Blue RH Side SSU1 OPA SW1 Blue RH Side SSU2 OPA SW1 Blue RH Side SSU3 OPA SW1 Blue RH Side SSU4 OPA SW1 Blue RH Side SSU5 OPA SW1 Blue RH Side...
  • Page 112 Additional MMU NOTE: The Arista 40GbE switch has 36 ports, however, ports 1 through 4 are 10GbE and are not used in the ClusterStor L300 and L300N system. Table 21. Lustre 40GbE Data Network Interconnect List - Base Rack (Arista 32-port)
  • Page 113 NOTE: The Arista 40GbE switch has 36 ports, however, ports 1 through 4 are 10GbE and are not used in the ClusterStor L300 and L300N system. Table 22. Lustre 40GbE Data Network Interconnect List - Storage Rack with Optional Additional MMU (Arista 32-port)
  • Page 114 Reference Loom Label Component Side/Port Switch Port White LH Side SSU4 100G SW0 White LH Side SSU5 100G SW0 White LH Side MMU / AMMU A I / I 100G SW0 White LH Side 100G SW0 Blue RH Side SSU0 100G SW1 Blue RH Side SSU1...

  • Page 115: Cable An Optional Additonal Mmu

    Reference 16.3.7 Cable an Optional Additonal MMU The Lustre Distributed Namespace (DNE) feature is configured to allow multiple MDTs operating through multiple Lustre metadata servers (MDS) nodes to operate as part of a single file system. The optional additional MMU hardware adds more MDS/MDT capability to a file system cluster.

  • Page 116: Cable Ssu Esu Sas Ports

    16.3.8 Cable SSU ESU SAS Ports Expansion storage Units (ESUs) on ClusterStor L300 and L300N systems extend the storage capacity of an SSU with controllers. Each SSU in the system is paired with 1 or 2 ESUs and each ESU includes 2 EBOD controllers.
  • Page 117 Reference Figure 75. Cabling SSU ESU SAS 6Gb/s (SSU+2) ESU 1 EBOD 0 EBOD 1 ESU 0 EBOD 1 EBOD 0 EAC 0 EAC 1 Figure 76. Cabling SSU ESU SAS 12Gb (SSU + 1) ESU 1 EBOD 0 EBOD 1 SSU+1 EAC 0 EAC 1...

  • Page 118: Cable Ammu Networks

    Each MDT is provisioned as a single RAID 10 (5+5) array, using a total of 20 HDDs with the remaining 2 HDDs configured as global hot spares. A single ClusterStor L300 and L300N file system cluster may have from 0 to 8 MMUs installed.

  • Page 119: Component Descriptions

    16.4 Component Descriptions 16.4.1 Embedded Application Controller (EAC) ClusterStor L300 and L300N uses two distinct types of embedded application controllers (EACs): one type used only for the system management unit (SMU) node, and another type is used for metadata management units...
  • Page 120 Reference Figure 79. EAC for SMU Nodes Power Reset Ethernet NIC 2 mgmt0 (secondary management network) OK LED Ethernet NIC 1 mgmt1 (primary management network) Fault LED Maint ports SAS port 1 ID LED SAS port 2 Diagnostic LEDs Ethernet NIC 5 Local data network (LDN) pub0 (external admin network) port II...

  • Page 121: Operator Control Panel - 5U84

    16.4.2 Operator Control Panel - 5U84 The front of the ClusterStor L300 and L300N 5U84 enclosure incorporates an operator’s control panel (OPS) with five LEDs and the Unit ID LCD display. The OPS panel provides a consolidated view of the operational status of the enclosure components and the enclosure's assigned ID, plus an alarm mute button.
  • Page 122 Reference the display is OFF, and the dual seven-segment display is OFF. If the VPD selects use of the display, the seven-segment display is ON and displays the number stored in VPD. Mute/Input Switch Sets the unit identification display and transitions alarm states. Power On/Standby LED (Green/Amber) Amber: ONLY standby power is available.

  • Page 123: Operator Control Panel - 2U24

    16.4.3 Operator Control Panel - 2U24 The front of the ClusterStor L300 and L300N 2U24 enclosure incorporates an operator’s control panel (OPS) with three LEDs and the Unit ID LCD display. The OPS panel provides a consolidated view of the operational status of the enclosure components and the enclosure's assigned ID, plus an alarm mute button.

  • Page 124: Power Diagrams

    16.5.1 Base Rack Standard Power Distribution for PX2-5965X3-V2 Cabling diagram for ClusterStor L300 and L300N base rack configuration PX2-5965X3-V2. Figure 84. Base Rack Standard Power Distribution for PX2-5965X3-V2 ClusterStor L300 and L300N 16.5.2 Base Rack Inverted Power Distribution for PX2-5965X3-V2...
  • Page 125 Reference Figure 85. Base Rack Inverted Power Distribution for PX2-5965X3-V2 ClusterStor L300 and L300N...

  • Page 126: Storage Rack Optional Mmu Standard Power Distribution For Px2-5965X3-V2

    Reference 16.5.3 Storage Rack Optional MMU Standard Power Distribution for PX2-5965X3-V2 Figure 86. Storage Rack Optional MMU Standard Power Distribution for PX2-5965X3-V2 ClusterStor L300 and L300N 16.5.4 Storage Rack Optional MMU Inverted Power Distribution for PX2-5965X3-V2 Cabling diagram for ClusterStor L300 and L300N storage rack optional MMU inverted power distribution for...
  • Page 127 Reference Figure 87. Storage Rack Optional MMU Inverted Power Distribution for PX2-5965X3-V2 ClusterStor L300 and L300N...

  • Page 128: Base Rack Standard Power Distribution For Px2-5104X2-V2

    Reference 16.5.5 Base Rack Standard Power Distribution for PX2-5104X2-V2 Figure 88. Base Rack Standard Power Distribution for PX2-5104X2-V2 ClusterStor L300 and L300N...

  • Page 129: Base Rack Inverted Power Distribution For Px2-5104X2-V2

    Reference 16.5.6 Base Rack Inverted Power Distribution for PX2-5104X2-V2 Figure 89. Base Rack Inverted Power Distribution for PX2-5104X2-V2 ClusterStor L300 and L300N...

  • Page 130: Storage Rack Optional Mmu Standard Power Distribution For Px2-5104X2-V2

    Reference 16.5.7 Storage Rack Optional MMU Standard Power Distribution for PX2-5104X2-V2 Figure 90. Storage Rack Optional MMU Standard Power Distribution for PX2-5104X2-V2 ClusterStor L300 and L300N...

  • Page 131: Storage Rack Optional Mmu Inverted Power Distribution For Px2-5104X2-V2

    Reference 16.5.8 Storage Rack Optional MMU Inverted Power Distribution for PX2-5104X2-V2 Figure 91. Storage Rack Optional MMU Inverted Power Distribution for PX2-5104X2-V2...

  • Page 132: Base Rack Standard Power Distribution For Px2-5100X2-V2

    Reference 16.5.9 Base Rack Standard Power Distribution for PX2-5100X2-V2 Figure 92. Base Rack Standard Power Distribution for PX2-5100X2-V2 ClusterStor L300 and L300N...

  • Page 133: Base Rack Inverted Power Distribution For Px2-5100X2-V2

    Reference 16.5.10 Base Rack Inverted Power Distribution for PX2-5100X2-V2 Figure 93. Base Rack Power Inverted Distribution for PX2-5100X2-V2 ClusterStor L300 and L300N...

  • Page 134: Storage Rack Optional Mmu Power Distribution For Px2-5100X2-V2

    Reference 16.5.11 Storage Rack Optional MMU Power Distribution for PX2-5100X2-V2 Figure 94. Storage Rack Optional MMU Power Distribution for PX2-5100X2-V2 ClusterStor L300 and L300N...

  • Page 135: Storage Rack Optional Mmu Inverted Power Distribution For Px2-5100X2-V2

    Reference 16.5.12 Storage Rack Optional MMU Inverted Power Distribution for PX2-5100X2-V2 Figure 95. Storage Rack Optional MMU Inverted Power Distribution for PX2-5100X2-V2 ClusterStor L300 and L300N...

  • Page 136: Base Rack Standard Power Distribution For Px2-5551-N4V2

    Reference 16.5.13 Base Rack Standard Power Distribution for PX2-5551-N4V2 Figure 96. Base Rack Standard Power Distribution for PX2-5551-N4V2 Sonexion 3000...

  • Page 137: Base Rack Inverted Power Distribution For Px2-5551-N4V2

    Reference 16.5.14 Base Rack Inverted Power Distribution for PX2-5551-N4V2 Figure 97. Base Rack Inverted Power Distribution for PX2-5551-N4V2 ClusterStor L300 and L300N...

  • Page 138: Storage Rack Optional Mmu Standard Power Distribution For Px2-5551-N4V2

    Reference 16.5.15 Storage Rack Optional MMU Standard Power Distribution for PX2-5551-N4V2 Figure 98. Storage Rack Optional MMU Standard Power Distribution for PX2-5551-N4V2 ClusterStor L300 and L300N...

  • Page 139: Storage Rack Optional Mmu Inverted Power Distribution For Px2-5551-N4V2

    Reference 16.5.16 Storage Rack Optional MMU Inverted Power Distribution for PX2-5551-N4V2 Figure 99. Storage Rack Optional MMU Inverted Power Distribution for PX2-5551-N4V2 ClusterStor L300 and L300N...

  • Page 140: Base Rack Optional Ammu Standard Power Distribution For Px2-5965X3-V2

    Reference 16.5.17 Base Rack Optional AMMU Standard Power Distribution for PX2-5965X3-V2 Figure 100. Base Rack Optional AMMU Standard Power Distribution for PX2-5965X3-V2 ClusterStor L300 and L300N 16.5.18 Base Rack Optional AMMU Standard Power Distribution for PX2-5104X2-V2 Figure 101. Base Rack Optional AMMU Standard Power Distribution for PX2-5104X2-V2 ClusterStor L300 and L300N...

  • Page 141: Base Rack Optional Ammu Standard Power Distribution For Px2-5100X2-V2

    Reference 16.5.19 Base Rack Optional AMMU Standard Power Distribution for PX2-5100X2-V2 Figure 102. Base Rack Optional AMMU Standard Power Distribution for PX2-5100X2-V2 ClusterStor L300 and L300N...

  • Page 142: Base Rack Optional Ammu Standard Power Distribution For Px2-5551-N4V2

    Reference 16.5.20 Base Rack Optional AMMU Standard Power Distribution for PX2-5551-N4V2 Figure 103. Base Rack Optional AMMU Standard Power Distribution for PX2-5551-N4V2 ClusterStor L300 and L300N...

  • Page 143: Base Rack Optional Ammu Inverted Power Distribution For Px2-5965X3-V2

    Reference 16.5.21 Base Rack Optional AMMU Inverted Power Distribution for PX2-5965X3-V2 Figure 104. Base Rack Optional AMMU Inverted Power Distribution for PX2-5965X3-V2 ClusterStor L300 and L300N...

  • Page 144: Base Rack Optional Ammu Inverted Power Distribution For Px2-5104X2-V2

    Reference 16.5.22 Base Rack Optional AMMU Inverted Power Distribution for PX2-5104X2-V2 Figure 105. Base Rack Optional AMMU Inverted Power Distribution for PX2-5104X2-V2 ClusterStor L300 and L300N 16.5.23 Base Rack Optional AMMU Inverted Power Distribution for PX2-5100X2-V2 Figure 106. Base Rack Optional AMMU Inverted Power Distribution for PX2-5100X2-V2 ClusterStor L300 and L300N...

  • Page 145: Base Rack Optional Ammu Inverted Power Distribution For Px2-5551-N4V2

    Reference 16.5.24 Base Rack Optional AMMU Inverted Power Distribution for PX2-5551-N4V2 Figure 107. Base Rack Optional AMMU Inverted Power Distribution for PX2-5551-N4V2 ClusterStor L300 and L300N...

This manual is also suitable for:

Clusterstor l300n

Table of Contents