From arccwiki
Jump to: navigation, search

ARCC uses the Extreme Cluster/Cloud Administration Tool for automating cluster management and deployment.

xCAT General Group Names

Following are the xCAT group names used as general group identifiers:

Group Name Usage
all Used to select all compute nodes in the cluster, except the DGX nodes.
moran Used to select all the old moran nodes in the cluster.
teton Used to select all the new teton nodes in the cluster.
tknl Knights landing nodes
mbm Moran Big memory nodes (512GB memory)
mhm Moran Huge memory nodes (1024GB memory)
tbm Teton Big memory nodes (512GB memory)
thm Teton Huge memory nodes (1024GB memory)
dbg Debug nodes
dx Used to select all the older IBM/Lenovo DX type nodes in the cluster.
nx Used to select all the newer IBM/Lenovo NX type nodes in the cluster.
rack number Used to select all nodes in a specific rack.
ipmi Used to select all IPMI enabled nodes in the cluster.
supermicro Any cluster node that is manufactured by SuperMicro.
sd530 Used to select the SD530 nodes in the cluster.

xCAT CPU Group Names

The following table details the group names used to identity a specific CPU type:

Group Name CPU Type
sandybridge Intel Sandy Bridge E5-2670
ivybridge Intel Ivy Bridge E5-2650v2, E5-2620v2
haswell Intel Haswell E5-2640v3, E5-2647v3, 2660v3
broadwell Intel Broadwell E5-2683v4
cascade Intel Cascade Lake Gold 6230 nodes

xCAT GPU Group Names

The following table details the group names used to identity GPU types within the cluster:

Group Name GPU Type
gpu Used to select all GPU enabled nodes
k20m Nvidia Tesla K20m
k20xm Nvidia Tesla K20xm
Titan Nviida Titan
Titan X Nvidia Titan X
k40c Nvidia Tesla K40c
k80 Nvidia Tesla K80
P100 Nvidia Tesla P100
v100 Nvidia Tesla V100 16GB

xCAT NX Chassis Names

NX chassis nameing currently uses the following format.


The chassis are numbered from the lowest chassis in the rack to the highest in the rack.

xCAT node Setup

In the beginning there was a new node and that node was without configuration. Then along came xCAT and the node became useful.

  1. Edit the /etc/hosts file and add the new nodes in at the appropriate places.
    1. node management IP
    2. infiniband IP
    3. IPMI IP
  2. Run "makedns" to insert the new nodes into the DNS (this will take a few minutes).
  3. run "host <node>", "host <node>-ipmi", "host <node>-ib to insure they've been set.
  4. run "nodeadd" to create a new node definition in xCAT.
    nodeadd t[465-480] groups=all,teton,g19,sd530,cascade
  5. You can run "lsdef <node>" to check it's groups.
  6. Run "makegocons" to update the console configuration file with the new nodes.
  7. Edit the xCAT switch table and add the new node and switch information. You may need to edit additional tables depending on if you are adding non Lenovo or Supermicro systems. These include nodehm, noderes and ipmi.
    1. "tabedit switch"
      1. " :$"
      2. "yy <last node>"
      3. "p" to put info
      4. change node and port
      5. " :wq!"
  8. Power on one node first to make sure the node is auto discovered. You can watch the xCAT log and messages file to see where the node is currently at in the Genesis process. You can also dump the "mac" table and see if the node MAC address has been added.
    1. Once the node has been discovered, you need to tell the node what image to boot.
      nodeset <node range> osimage=XXX
      Where XXX is the current image to boot. As of this writing it is t2018.03
    2. "cd /tftpboot/xcat/xnba/nodes" then "ls <node>*".
    3. "tail -f /var/log/messages " grep <node>".
      The node will then reboot and PXE into the specified osimage.
    4. Once the node has rebooted you should be able to ssh to the node. Verify the node is running properly and has the right network address.
      1. "cat /etc/redhat-release", "ipmitool lan print 1" (IP address is wrong) logout
      2. "cd ~/.ssh", vi "known_hosts" remove duplicate file.
    if everything looks ok go ahead a power on all the nodes. Repeat the steps above for all the nodes.
  9. "cd ~/bin" and run "setup_ipmi <node range>" to configure the IPMI settings on each node. Note: We don't use the xCAT bmcsetup script as part of the genesis process due to having issues with it setting up the IPMI interface.
  10. run "UWset_bios <node option> <node range>", this script will configure the proper BIOS settings as defined for ARCC.
  11. ssh to node and run "ipmitool lan print 1" and "ipmitool user list 1"
  12. Add the nodes to GPFS
    mmaddnode -N nodelist
    mmchlicense client --accept -N nodelist
    Make sure to specify the Infiniband address.
  13. reboot all the nodes to have the BIOS changes take effect.
  14. Verify the nodes once they all reboot.
    psh all "grep -c ^processor /proc/cpuinfo " | xcoll
  15. Add nodes to correct node group.
    "cd /apps/s/slurm/latest/etc"
    "vi nodes-teton.conf"
    add node(s) under applicable category.
  16. Add nodes to partition
    "vi partitions-general.conf"
    Under "PartitionName=teton", change the node range to add new nodes.
    1. If this is not a compute node on teton, then add node to applicable partition.
  17. Add the nodes to correct partition.
  18. Add the nodes to Slurm.
  19. After nodes are added, restart slurm control daemon
    "systemctl restart slurmctld"
  20. "scontrol reconfigure" to reconfigure slurm
  21. Add the nodes to Racktables.
  22. Add the nodes to Observium.
  23. Run test job to verify slurm connectivity and that job scheduling is available.

xCAT Installation

xCAT Creating a New Node Image

xCAT Notes and Warnings

1. If a node appears not to boot check the node definition. Could be that the console setup is incorrect in the nodehm table.

xCAT References

xCAT Wikipedia Page [1]

xCAT Documentation Page [2]