Blog

Blog

The NeSI Auckland equipment (includes Pan cluster) is migrating to the new Tamaki Data Centre to allow room for expansion.

The time-line so far:

  • 7 December - Friday
    • 09:00 Login node closed and job queues drained
    • 11:00 Power off all equipment
    • 12:00 Disconnecting cables
    • 17:00 Trucks move equipment
    • 19:00 Equipment braced in TDC
    • 20:00 Cabling commences

 

  

  • 8 December - Saturday
    • 17:00 Power-up of racks
    • 21:30 Network functionality restored


  • 9 December - Sunday
    • 08:30 IB working (with only one lost node at the moment)
    • 08:30 3 GPFS servers found
    • 13:00 Storage looking ok
    • 13:00 Temperatures holding
    • 14:00 LoadLeveler testing commences
    • 16:00 Benchmarks run - looking good 

 

  • 10 December - Monday
    • Most of the day spent fault finding and checking functionality
    • 17:00 Pan cluster available for researchers
December Update

Important Changes

Multithreaded and parallel job descriptions changes

Loadleveler configuration has been changed. The new configuration requires mutltithreaded and mixed jobs to be defined in a slightly different way. Now multithreaded jobs will need to have job_type set to serial and the amount of threads requested should be set via parallel_threads= directive:

#@ job_type=serial
#@ parallel_threads=4

Mixed jobs (those that use MPI and multiple threads per rank) should also use parallel_threads directive to reserve the number of threads per rank:

#@ job_type=parallel
#@ node=2
#@ tasks_per_node=2
#@ parallel_threads=4

CPU affinity binding

With new configuration affinity control has become available. Affinity-based scheduling means that Loadleveler now knows what cores exactly are given to the users and these cores can be tied up to the user application. In order to use affinity control an application must be launched through smpexec program:

/share/apps/smpexec matlab -f mycode.m

An additional perk of using smpexec is delivered by using -s key:

/share/apps/smpexec -s matlab -f mycode.m

If -s is used, smpexec will print out application statistics once the application finishes. The statistics includes maximum amount of memory used and the number of threads the application spawned.

Number of cores per node is reduced

One of the effects of using affinity scheduling is the reduced number of cores per node. Now one core per node is reserved for I/O operations and as such the maximum number of allocatable cores in Westmere nodes is 11 and in Sandy Bridge nodes is 15.

llstatus output interpretation

Due to the introduction of new way to define multithreaded jobs llstatus output can now be misleading, showing very low number of tasks while jobs are being held due to the resource constraints. The reason for it is that llstatus reports only running tasks, but not allocated cores. In order to assess cores availability llstatus -R command should be used that displays the list of available/accessible cores per node.

Documentation

New documentation on how to install and use the NeSI Tools software suite has been created and posted at the following link:https://wiki.auckland.ac.nz/display/CERES/NeSI+Pan+Cluster+Login

This software allows remote computers (Windows, Mac and UNIX) to login and use the Pan cluster. The suite includes MobaXterm, Grisu Template Client and Gricli.  The guides show installation and uploading and downloading files to/from the Pan cluster. 

News

Cluster Relocation

On December 7 at 9 am, the NeSI Pan cluster will be shutdown and relocated to the new Tamaki Data Centre. We aim to have all services available no later than by the end of Monday December 10. The relocation will make room for additional hardware as part of the University of Otago's recent co-investment in NeSI. Over the next few weeks, an additional 63 nodes will be made available to researchers, increasing the total capacity of Pan to over 3,000 CPU cores. The expansion will also increase the number of available NVIDIA 'Tesla' M2090 GPUs to a total of 32. As part of this addition, three high memory nodes (20 CPU cores, 512 GB RAM) will also be made available to researchers.

Planned Improvements

Looking forward, the Centre for eResearch is planning another investment into the NeSI Pan cluster for 2013 Q1 and would appreciate any feedback and suggestions with regard to hardware needs. We are considering purchasing new GPUs based on NVIDIA's Kepler architecture as well as the recently released Intel Xenon Phi accelerator. These devices will primarily be of interest to researchers who develop their own code but we would also like to know about compatible applications that you would like to use if these accelerators were made available. Please send any feedback to eresearch-support@auckland.ac.nz

Due to restricted expansion room in OGG DC, the Pan cluster will move to TDC. The planning is still underway, but the planned duration will be from Friday the 7th of December until Monday the 10th.

Due to a storage array problem the cluster file system needs to be shut down and restarted. We will use the opportunity to also update firmware versions and a LoadLeveler reconfiguration.

Expected duration is from 9 am until 5 pm.

Peter Mayes (IBM UK) is conducting LoadLeveler training at Centre for eResearch for end-users.

Please indicate your interest to Gene (g.soudlenkov@auckland.ac.nz, ext 89834). Limited seating.

New BLAST software

We have installed the following BLAST tools and invite those with an interest in bioinformatics to try them out:

BLAST+ 2.2.26 - for use on a single node with one or more cores.
mpiBLAST 1.6.0 - for use across multiple nodes.
gpuBLAST 1.1 - for use on a single node with a GPU. Protein sequences (blastp) only.
NCBI BLAST database - full FASTA and BLAST+ formatted databases. Updated on the first of every month.

Please let us know if you would like any assistance using these tools. Documentation can be found in our bioinformatics software usage guides.

Octave installed

We now have Octave available which can be used as an alternative to Matlab for some use cases.

Package information can be found here:

https://wiki.auckland.ac.nz/display/CERES/Octave+usage

SAS 9.2 installed

SAS® statistics provides a wide range of statistical analysis software, ranging from traditional analysis of variance to exact methods and dynamic data visualization techniques. Using statistical software can provide new information that improves processes, drives development and revenues, and helps organizations retain valued and satisfied customers.

The University of Auckland has a site license. All researchers affiliated with the University of Auckland that have an account at the Centre for eResearch can use SAS on the Pan cluster.

Find more details in the SAS usage guide.

The program relax is a software package designed for the study of molecular dynamics through the analysis of experimental NMR data. Organic molecules, proteins, RNA, DNA, sugars, and other biomolecules are all supported. It was originally written for the model-free analysis of protein dynamics, though its scope has been significantly expanded.

You can find relax installed on the Pan cluster.

/share/apps/relax/relax

Documentation is available on the relax website

OpenBUGS is a software package for performing Bayesian inference Using Gibbs Sampling and is now available on the Pan cluster.

For more details please see the OpenBUGS usage page

FSL now available

The FMRIB neuroimaging software suite FSL has been installed on the Pan cluster and adapted for use with LoadLeveler.

For more information on how to use FSL to process MRI data in parallel, please see the FSL usage guide.

If you have any questions please contact us.

A new login server has been setup at login.uoa.nesi.org.nz for use as a submission and build node for the NeSI Pan cluster.

As of Monday 28 May, login1.uoa.nesi.org.nz wil no longer be available as a submission and build node.

Gricli will also be made available on Monday 28 May on the new login server.

The most recent version of the Amber molecular dynamics suite, Amber 12, is now available on Pan.

There are two versions available. One, the conventional (CPU) version, has been built using the PGI compilers. The other, the GPU version built with the Intel compilers, uses cutting-edge CUDA technology for very fast molecular dynamics.

For a usage guide, see Amber 12 usage.

The latest version of PGI C/C++ and Fortran compilers are now available on the NeSI cluster.
The compilers are installed in the /share/apps/pgi directory and include both 32- and 64-bit linux versions.

To prepare the environment for the 64-bit Linux version:

module load pgi/12.4/64bit

Similarly, for the 32-bit Linux version:

module load pgi/12.4/32bit

Also the Intel Composer XE 2011 compilers are now available on the NeSI cluster.
The compiler pack includes the Intel C, C++ and Fortran compilers and is installed in the directory /share/apps/intel/2011.
Before running the Intel compilers, it is strongly recommended to configure your shell environment. You can do so by running one of the following scripts:
For the 64-bit compiler:

module load intel/2011-64bit

For the 32-bit compiler:

module load intel/2011-32bit

We're pleased to announce that we've installed a new version of the OpenMPI parallel computing framework. This new version (1.5.5) will be available alongside existing versions.

This new version includes explicit support for LoadLeveler (the scheduler used on Pan), as well as MPI 2 with spawning functionality.

At present, we have not made 1.5.5 the system default; we've left the default at 1.4.3 for the sake of compatibility with previously built applications.

To switch to OpenMPI 1.5.5, you should to complete the following steps:

  1. Run the program "mpi-selector-menu" at your command prompt, and, when prompted, select the option for openmpi_gcc-1.5.5, for "user" (as opposed to "system").
  2. Log out and log back in.

Please note that applications built against other versions of MPI will need to be recompiled against OpenMPI 1.5.5, otherwise OpenMPI 1.5.5 will not run them correctly.

Also, in LoadLeveler job files, only if using OpenMPI 1.5.5:

  • The job_type directive should be set to "MPICH" instead of "parallel"
  • Your executable must still be run through mpirun, but mpirun will no longer require the "-hostfile ${LOADL_HOSTFILE}" command-line option to mpirun