Software: Lmod

From arccwiki
Jump to: navigation, search

Lmod is a powerful piece of software that the ARCC leverages for managing dynamic user environments and access to complicated software stacks. The software was developed at the Texas Advanced Computing Center (TACC) to enhance the concepts and functionality of the older environment-modules.

Introduction

  • Our configuration of Lmod uses a hierarchy for organization of software that starts with compilers. Compilers are core to nearly all software installed on Teton. The supported compilers are Intel, GCC, and PGI. There are however some packages that are compiler-independent and they're available under the the User Core Modules section or the ARCC Core Modules and should be able to be loaded without loading a compiler module first.
  • Modules also give the ARCC administrators insights into what software is being loaded to understand the software being used on the clusters.
  • If there is software that isn't installed on the cluster that is desired and many people would benefit from, please fill out an ARCC software request.

Using Modules

Searching for modules

Searching for modules is a great way to find out if software has been installed as a system package. There are a few different methods to search for modules each with slightly different capabilities. The first method to search for modules is to use the spider subcommand.

 $ module spider 

Without any additional arguments to the above command, the output contains a list of the available software for the entire system. To narrow down individual packages and versions, you can supply the names / arguments to the command:

 $ module spider samtools 

If there is only one version, the output will contain information regarding which compilers are required to be loaded before the package can be loaded as well as provide a brief segment on help of the module. If there are multiple versions available, the output contains which versions are available and instructions to get more information on the individual version. To get information regarding a specific version of the package, include the version as part of the argument.

 $ module spider samtools/1.6 

There are opportunities to use regular expressions to search for modules. See output from module help for more information.

Additionally, using the avail subcommand will show a table of available modules that are capable of being loaded into the current environment.

 $ module avail 

Search by keyword is also an option that will search the name and whatis descriptors in the modules. An example might be:

 $ module keyword slurm 

Information About A Module

To get information regarding a module, the help subcommand is a powerful tool. This will display information regarding the help section of the module and if their are any additional information relating to the installation of the software that is pertinent.

 $ module help slurm 

Loading / Unloading Modules

Loading modules and unloading modules how Lmod dynamically change the environment on the cluster. To access software available through modules, the specific module must be loaded. There are several modules that are loaded automatically on the system as defaults. They include arcc, slurm, and singularity.

New modules can be loaded with the load subcommand. I.e., to load the Intel compilers with the most up to date version:

 $ module load intel 

To remove the software from the environment, using the unload subcommand will remove the module from the environment. If the module specified isn't loaded, this becomes a no-op (no operation).

 $ module unload intel 

It is important to note that when loading modules that are dependent on a compiler and the compiler module is unloaded, Lmod will deactivate the dependent modules until the correct compiler is loaded or that an appropriate compiler is loaded that has alternative versions of the software available.

Module Collections

Module collections are a great method to keep certain software stacks in order and switch between them regularly and quickly. They allow for all modules to be loaded, then saved, and recalled later for use.

To save a module collection:

 $ module save 

By default, this is will save the loaded modules as the default collection. You can optionally supply a name to the command to give a custom name and have many collections available:

 $ module save NAME 

Swapping between collections requires the restore subcommand is used. Without any additional arguments, the module system will load your default collection. If there is no default collection defined, the system default collection will be loaded. Use a NAME as an argument to load a specific collection.

 $ module restore 
 $ module restore NAME 

A special condition to load the system defined collection is done with the reset subcommand allowing a fresh start of configured modules. Note, this doesn't take into account what is inserted into shell startup files (i.e., .bashrc, .cshrc, etc.)

 $ module reset 

or optionally:

 $ module restore system 

To query the system and determine which module collections have been created, use the savelist subcommand.

 $ module savelist 

It's important to use names that are memorable, but that doesn't necessarily describe all the contents of the collection. To list which modules are part of the collection, use the describe subcommand with an optional argument to list contents of a certain NAME collection.

 $ module describe 
 $ module describe NAME 

To disable a module collection that is no longer needed, use the disable subcommand.

 $ module disable NAME 

Automatically loading modules

Some core modules are automatically loaded when logging in. On Teton, these automatically-loaded modules do not include a compiler module such as gcc or intel that start the hierarchy for organization of other modules, since for different users, different choices of default compiler would make sense. However, a user can customize the automatically-loaded modules by adding module load commands to the .bashrc file in their home directory.

Examples

To automatically load one of the compilers when logging in, append the following lines to your .bashrc file:

For gcc
module load swset/2018.05
module load gcc/7.3.0
For intel
module load swset/2018.05
module load intel/18.0.1

To see if different versions of the compilers are available, and which swset version a particular compiler version depends on, use the module avail and module spider commands described above on this page.

Module shorthand with ml

Lmod provides a shorthand interface to the modules command as ml. The command is quite powerful and allows quicker access to the modules system. The default action of ml is to list the loaded modules. Providing a module as an argument will load the module.

 $ ml intel 

Modules can quickly be unloaded as well by supplying a nearly identical command, but with the - operator before the argument:

 $ ml -intel 

Querying the system of available modules is easy:

 $ ml av 
 $ ml spider 

The ml command is very useful and should be explored.

Requesting Software with Module

All software requested on ARCC HPC clusters generally comes with a modulefile as the primary method of access. If the software is deemed either too restrictive or is better suited to be offered from the Operating System's package manager, it will be noted in the software request.

Requesting software installation on HPC cluster

Creating Private Modules (Advanced)

Using Miniconda to install software