Here are the rules of the game. Ignorance of the rules is not an acceptable excuse for bad behavior.
- If you are not familiar with HPC scheduling and the various parameters that determine which job goes next, I gave a presentation at the Biomedical Informatics Seminar on March 15th which gives an introduction to HPCs and scheduling that might be helpful.
- The easiest way to access Grace is through her web-based portal (Viewpoint), found at https://portal.hpc.uams.edu, which allows you to manipulate files (e.g. in your home directory), submit jobs and monitor their progress (see videos at https://youtu.be/6tCuE6K8RqI and/or https://youtu.be/R3b2BvdyDl0), and create job templates, which simplify submission of similar jobs. Viewpoint documentation is at http://docs.adaptivecomputing.com/9-1-3/Viewpoint/viewpoint.htm. You may need to log into Grace at least once by ssh to set your password before using Viewpoint.
- Many researchers access Grace by using ssh directed to login.hpc.uams.edu. The login node is not meant for computation. Compiling programs is allowed, if you have no facilities on your personal computer for compiling programs.
- The system currently utilizes Moab/Torque as the scheduler/resource manager, hence the msub command gives the most options for job control. The qsub command also works.
- The scheduler/resource manager works best if your wallclock, memory, and processing core requests are close to real. If you are unsure what to use, please try one or two small sample jobs, and then check what the actual runtime, memory, and core usage were, and use that to guide your estimates. Note that your job priorities are penalized if you consistently overestimate these values.
- The compute nodes are outfitted with the OpenHPC libraries. Many options require issuing a module command to load appropriate versions of software libraries before use.
- If you need special software configuration (i.e. the OpenHPC libraries are not enough) we recommend using Singularity containers. Most Docker containers can run under Singularity, and there are a lot of Singularity and Docker containers available on their respective hubs. If you can’t find a suitable container, you can choose a base container, and then configure to your liking on your local workstation before loading the container to the HPC.
- Each HPC user gets a home directory, which is visible to all of the compute nodes and the login node, with a 1 TB quota.
- Users may temporarily store data in the /scratch directory, which is also shared across all nodes. Note that data in the /scratch directory is not considered permanent, and could be purged of older data at any time. So do not count on keeping things in /scratch for more than 14 days after a job ends. Hence the strong recommendation that users utilize the data staging options in Moab to copy output data to outside storage (e.g. the research NAS or the object store or the user’s personal computer) at the end of a job.
- Each compute job also has the option to use local scratch storage (/tmp or /local-scratch) on each node. Note that local scratch areas are not shared between nodes (i.e. they are on a disk local to a particular compute node). Their big advantage over /scratch is that accessing local scratch storage incurs no network overhead, so can be faster than /scratch if a program is doing lots of small, random reads and writes. The shared /scratch area may be faster for long sequential reads and writes.
- A good habit to get into is to use the Moab data staging options to move data into a scratch (local or shared, as needed) area prior to the start of a job, and to copy data out of the scratch areas after the end of the job. Note that data staging, if done through the Moab data staging options, does not count against a job’s wall clock time.
- There is a shared area of library files (files common to a number of projects) currently called /storage. (We may rename it.) Please do not store personal files there. At some point in the future, /storage will be made read-only, and personal files will be moved to the user’s home directory, potentially impacting that user’s quota.
- There are three categories of processors, selectable via feature requests when submitting jobs, in Grace:
- Xeon Phi (phi) – ideal for massively parallel operations that have lots of threads. Each Xeon Phi node has 64 physical cores and either 384 GB of memory (80 of the Xeon Phi nodes), or 192 GB of memory (16 of the nodes). Although the clock speed of each individual processor is slower than a regular xeon, since there are more than double the number of cores on a Xeon Phi compared to regular Xeon, appropriately configured multi-threaded jobs can run faster on the Xeon Phi. A program optimized for Xeon Phi can gain significant speed advantage over regular Xeon. Optimized programs also typically will run faster on regular Xeon as well, though the speedup is not as dramatic.
- Regular Xeon (xeon) – use where single thread performance is critical. Each regular Xeon node has 28 physical cores and 128 GB of memory.
- Nvidia Tesla GPU nodes (gpu) – for jobs that can capitalize on GPU performance. Each node has 24 regular Xeon cores, 128 GB of memory, and a pair of 12GB P100 GPUs. We have Nvidia CUDA 10 drivers installed on the GPU nodes. The GPU nodes are in a special class (a.k.a. queue) named “gpu”, separate from the other nodes. So when submitting jobs that depend on the GPU, make sure to select the “gpu” class, not the default “batch” class. Otherwise your job could sit in limbo asking for GPU resources on nodes that don’t have them.
- Generally, if your program is not geared specifically for one of the above categories, it is best to just let the scheduler decide (i.e. don’t put in a feature request), so that the job gets routed to the first available processor.