Running Elan MPI jobs on QsNet Clusters --------------------------------------- If built with --with-qshell or --with-mqshell, pdsh may be used to run MPI jobs on a QsNet interconnect. This option requires that you have the Elan user space libraries installed (qsnetlibs or qswelan-r RPM for Linux) and that your kernel be patched to run the 'elan3' or 'elan4' and 'rms' device drivers. Pdsh can run independently of the RMS product (the 'rms' kernel module, which is used by pdsh, is a distinct facility from the RMS product). Quadrics has provided a PDSH FAQ which may answer some common questions about getting the qshell module to run MPI jobs. Please see http://web1.quadrics.com/twiki/bin/view/FAQs/SetupPDSH rms pdsh module --------------------------------------- Pdsh can also be run via the Quadrics RMS 'allocate' command such that allocate takes care of the node reservations and passes a batch ID through to pdsh via the RMS_RESOURCEID environment variable. Pdsh retrieves the list of allocated nodes out of the RMS database using the rmsquery command. This functionality is provided by the "rms" pdsh module (--with-rms). slurm pdsh module --------------------------------------- Similar to the rms pdsh module, the slurm module allows pdsh to target nodes based on SLURM allocations, either targetting an already running job or by running under ``srun --allocate'' The SLURM jobid can be passed to pdsh using the `-j' option provided by the module, or via the SLURM_JOBID environment variable, which is set by --allocate. The `/etc/elanhosts' config file --------------------------------------- Pdsh uses a simple config file, /etc/elanhosts, to describe hosts containing Elan adapters (and on which Elan MPI jobs may be run). The config file is also used by the daemons qshd and mqshd to initialize the Elan network error resolver thread. Parsing of the /etc/elanhosts file is accomplished by using the libelanhosts(3) library, upon which pdsh depends (when building for QsNet). See the elanhosts(5) man page for a description of the /etc/elanhosts file format. The libelanhosts package may be obtained from ftp://ftp.llnl.gov/pub/linux/libelanhosts/
