Software Development

The following Intel Software Development Products are available as shown:
            Fortran
            Fortran 9.1.032
                    setenv TMP /temp/`id -nu`
                    source /opt/
intel/fce/9.1.032/bin/ifortvars.csh (ifort command)
            Fortran 9.1.045
                    setenv TMP /temp/`id -nu`
                    source /opt/
intel/fce/9.1.045/bin/ifortvars.csh (ifort command)
            Fortran 10.1.021
                    setenv TMP /temp/`id -nu`
                    source /opt/intel/fce/10.1.021/bin/ifortvars.csh
(ifort command)
            C / C++
           C++ 9.1
                    setenv TMP /temp/`id -nu`
                    source /opt/
intel/cce/9.1.049/bin/iccvars.csh (icpc command)
           C++ 10.1
                    setenv TMP /temp/`id -nu`
                    source /opt/intel/cce/10.1.021/bin/iccvars.csh
(icpc command)
            VTune Performance Analyzer
            Integrated Performance Primitives

    Intel® C++ Compiler 8.1 for Linux* Getting Started Guide
    Intel® C++ Compiler 8.1 for Linux* Release Notes
    Intel® C++ Compiler for Linux* User's Guide

    Intel® Fortran Compiler 8.1 for Linux* Getting Started Guide
    Intel® Fortran Compiler 8.1 for Linux* Release Notes
    Intel® Fortran Compiler For Linux* Systems User's Guide
    Intel® Fortran Programmer’s Reference
    Intel® Fortran Libraries Reference

Host Compiler Version
aries cce 9.1.042
aries cce 9.1.049
aries cce 10.0.025
aries cce 10.1.021
aries fce 9.1.032
aries fce 9.1.045
aries fce 10.1.021
athena cce 9.1.042
athena cce 9.1.049
athena cce 10.0.025
athena cce 10.1.021
athena fce 9.1.032
athena fce 9.1.045
athena fce 10.1.021

The Portland Group PGHPF Workstation compiler is available on athena
                   
source /apps/pgi/pgivars.csh enables: pghpf, pgf90, pgf77, pgprof, pgdbg
                    If you get "error while loading shared libraries: libnuma.so: cannot open shared object file: No such file or directory", then compile with -mp=nonuma option
                    Also available is the PGI User's Guide
                    And the PGI Fortran Reference

GNU Compiler Collection gcc version 4.1.2  is available on aries and version 4.4.7 on athena. This includes Fortran 95 as f95/gfortran

To compile, ssh to aries or athena and run the commands shown above

Libraries

Intel has Math Kernel Library Reference Manual and Technical User Notes - Linux

High-Performance BLAS by Kazushige Goto

 

Message Passing Interface

The Message Passing Interface (MPI) standard

* aries *

Open MPI is installed on aries.
MPI/compiler choices: (use qlc/qlogic for Infiniband)
mvapich2_gcc-1.4.1
mvapich2_gcc_qlc-1.4.1
mvapich2_gnu-1.4
mvapich2_intel-1.4
mvapich2_intel_qlc-1.4.1
mvapich2_pathscale_qlc-1.4.1
mvapich2_pgi-1.4
mvapich2_pgi_qlc-1.4.1
mvapich_gcc-1.2.0
mvapich_gcc_qlc-1.2.0
mvapich_gnu-1.2.0
mvapich_intel-1.2.0
mvapich_intel_qlc-1.2.0
mvapich_pathscale_qlc-1.2.0
mvapich_pgi-1.2.0
mvapich_pgi_qlc-1.2.0
openmpi_gcc-1.4.1
openmpi_gcc_qlc-1.4.1
openmpi_intel_qlc-1.4.1
openmpi_pathscale_qlc-1.4.11
openmpi_pgi_qlc-1.4.11
qlogicmpi_gnu-0.1.01
qlogicmpi_intel-0.1.01
qlogicmpi_pathscale-0.1.01
qlogicmpi_pgi-0.1.01

gcc compiler:

source /usr/mpi/gcc/openmpi-1.4.1-qlc/bin/mpivars.csh

And LD needs the library:
setenv LD_LIBRARY_PATH /usr/mpi/gcc/openmpi-1.4.1-qlc/lib64:$LD_LIBRARY_PATH

Sample PBS script:
#!/bin/tcsh
#PBS -q para -l select=1:ncpus=16:ariesall=True:mpiprocs=16
setenv MPIDIR /usr/mpi/gcc/openmpi-1.4.1-qlc/bin
source $MPIDIR/mpivars.csh
cd myproblemdir
setenv HOSTFILE ./hostfile
sort $PBS_NODEFILE|uniq >$HOSTFILE
setenv NUMPROCS `wc -l < $PBS_NODEFILE`
$MPIDIR/mpirun -n $NUMPROCS -machinefile $HOSTFILE ./greetings

intel compiler:

source /usr/mpi/intel/openmpi-1.4.1-qlc/bin/mpivars.csh

At run time, the default MPI/compiler choice is: openmpi_gcc_qlc-1.4.1
You may change it with the mpi-selector command:
mpi-selector --set mvapich2_intel_qlc-1.4.1 --yes

You must also set up MPI variables and your PATH with:
source /usr/mpi/intel/mvapich2-1.4.1-qlc/bin/mpivars.csh

You may also use the Intel MPI library as shown below.

* athena *

Open MPI is installed on athena.

gcc compiler:

For gcc compiler use the mpif90 command. mpif90 is Open MPI Fortran 90 wrapper compiler.
And LD may need the library:
setenv LD_LIBRARY_PATH /opt/openmpi/lib:$LD_LIBRARY_PATH


Intel MPI is installed on athena.
IntelĀ® MPI Library - Documentation is at this link

To run programs already compiled by Intel, we will use Intel MPI. Here is a sample script:
#!/bin/sh
#PBS -q para -l select=1:ncpus=X:athena=True:mpiprocs=X
source /opt/intel/impi/4.1.0/bin64/mpivars.sh
# multiple programs on same node were pinned to the same cpus.
export I_MPI_PIN=off
export I_MPI_DEVICE=sock
export HOSTFILE=mpd.hosts
sort $PBS_NODEFILE|uniq >$HOSTFILE
export NUMPROCS=`wc -l < $PBS_NODEFILE`
mpirun -n $NUMPROCS . . . .

intel compiler:

To compile programs, instead of the instructions at the top of the page run:
setenv TMP /temp/`id -nu`
source /apps/msc/msc20131/linux64/intel/bin64/mpivars.csh (mpiifort and mpiicpc commands)


OPENMP

The OpenMP standard

OpenMP uses a portable, scalable model that gives programmers a simple and flexible interface for developing parallel applications for platforms ranging from the standard desktop computer to the supercomputer. [Wikipedia]

    Links:
        Wikipedia OpenMP Page
       
Livermore Computing OpenMP Tutorial Page

Compiler options to compile OPENMP commands:
GNU: -fopenmp
PGI: -mp
Intel: -openmp or -parallel for auto-parallelization

Use "setenv OMP_NUM_THREADS X" in your PBS script where X is the number of processors requested.



Hybrid MPI + OPENMP

Try these web sites:
SuperComputing13 Tutorial
Golden Energy Computing Workshop shows examples for Intel and PGI compilers

Example using gfortran where Makefile contains:
CC = mpif90 -fopenmp
LIBS = -L/usr/lib64/ -llapack -lblas -L/opt/openmpi/lib

And using PBSPro to run:
iVEC Introduction, Examples, and Debugging
Note how ncpus = mpiprocs * ompthreads for 1 chunk. select is used to specify how many chunks. Thus select * ncpus is the total number of cpus.