% -*- latex -*-
%
% Copyright (c) 2001-2004 The Trustees of Indiana University.  
%                         All rights reserved.
% Copyright (c) 1998-2001 University of Notre Dame. 
%                         All rights reserved.
% Copyright (c) 1994-1998 The Ohio State University.  
%                         All rights reserved.
% 
% This file is part of the LAM/MPI software package.  For license
% information, see the LICENSE file in the top level directory of the
% LAM/MPI source distribution.
%
% $Id: lam-ssi.tex,v 1.25 2004/03/06 21:30:36 jsquyres Exp $
%

\chapter{Available LAM Modules}
\label{sec:lam-ssi}

There is currently only type of LAM module that is visible to users:
\kind{boot}, which is used to start the LAM run-time environment, most
often through the \icmd{lamboot} command.  The \cmd{lamboot} command
itself is discussed in Section~\ref{sec:commands-lamboot}
(page~\pageref{sec:commands-lamboot}); the discussion below focuses on
the boot modules that make up the ``back end'' implementation of
\cmd{lamboot}.  

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Booting the LAM Run-Time Environment}
\label{sec:lam-ssi-boot}
\index{boot SSI modules|(}
\index{SSI boot modules|see {boot SSI modules}}

LAM provides a number of modules for starting the \cmd{lamd} control
daemons.  In most cases, the \cmd{lamd}s are started using the
\icmd{lamboot} command.  In previous versions of LAM/MPI,
\icmd{lamboot} could only use \icmd{rsh} or \icmd{ssh} for starting
the LAM run-time environment on remote nodes.  In LAM/MPI \lamversion,
it is possible to use a variety of mechanisms for this process
startup.  The following mechanisms are available in LAM/MPI
\lamversion:

\begin{itemize}
\item BProc
\item Globus (beta-level support)
\item \cmd{rsh} / \cmd{ssh}
\item OpenPBS / PBS Pro / Torque (using the Task Management interface)
\changebegin{7.1}
\item SLURM (using its native interface)
\changeend{7.1}
\end{itemize}

These mechanisms are discussed in detail below.  Note that the
sections below each assume that support for these modules have been
compiled into LAM/MPI.  The \icmd{laminfo} command can be used to
determine exactly which modules are supported in your installation
(see Section~\ref{sec:commands-laminfo},
page~\pageref{sec:commands-laminfo}).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Boot Schema Files (a.k.a., ``Hostfiles'' or
  ``Machinefiles'')}
\label{sec:lam-ssi-boot-schema}
\index{boot schema}
\index{hostfile|see {boot schema}}
\index{machinefile|see {boot schema}}
\cmdindex{lamboot}{boot schema file}

Before discussing any of the specific boot SSI modules, this section
discusses the boot schema file, commonly referred to as a ``hostfile''
or a ``machinefile''.  Most (but not all) boot SSI modules require a
boot schema, and the text below makes frequent mention of them.
Hence, it is worth discussing them before getting into the details of
each boot SSI.

A boot schema is a text file that, in its simplest form, simply lists
every host that the LAM run-time environment will be invoked on.  For
example:

\lstset{style=lam-shell}
\begin{lstlisting}
# This is my boot schema
inky.cluster.example.com
pinky.cluster.example.com
blinkly.cluster.example.com
clyde.cluster.example.com
\end{lstlisting}

Lines beginning with ``{\tt \#}'' are treated as comments and are
ignored.  Each non-blank, non-comment line must, at a minimum, list a
host.  Specifically, the first token on each line must specify a host
(although the definition of how that host is specified may vary differ
between boot modules).  

However, each line can also specify arbitrary ``key=value'' pairs.  A
common global key is ``{\tt cpu}''.  This key takes an integer value
and indicates to LAM how many CPUs are available for LAM to use.  If
the key is not present, the value of 1 is assumed.  This number does
{\em not} need to reflect the physical number of CPUs -- it can be
smaller then, equal to, or greater than the number of physical CPUs in
the machine.  It is solely used as a shorthand notation for
\icmd{mpirun}'s ``C'' notation, meaning ``launch one process per CPU
as specified in the boot schema file.''  For example, in the following
boot schema:

\lstset{style=lam-shell}
\begin{lstlisting}
inky.cluster.example.com cpu=2
pinky.cluster.example.com cpu=4
blinkly.cluster.example.com cpu=4
# clyde doesn't mention a cpu count, and is therefore implicitly 1
clyde.cluster.example.com
\end{lstlisting}

\noindent issuing the command ``{\tt mpirun C foo}'' would actually
launch 11 copies of \cmd{foo}: 2 on \host{inky}, 4 on \host{pinky}, 4
on \host{blinky}, and 1 on \host{clyde}.

Note that listing a host more than once has the same effect as
incrementing the CPU count.  The following boot schema has the same
effect as the previous example (i.e., CPU counts of 2, 4, 4, and 1,
respectively):

\lstset{style=lam-shell}
\begin{lstlisting}
# inky has a CPU count of 2
inky.cluster.example.com
inky.cluster.example.com
# pinky has a CPU count of 4
pinky.cluster.example.com
pinky.cluster.example.com
pinky.cluster.example.com
pinky.cluster.example.com
# blinky has a CPU count of 4
blinkly.cluster.example.com 
blinkly.cluster.example.com 
blinkly.cluster.example.com 
blinkly.cluster.example.com 
# clyde only has 1 CPU
clyde.cluster.example.com
\end{lstlisting}

Other keys are defined on a per-boot-SSI-module, and are described
below.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Minimum Requirements}
\label{sec:lam-ssi-boot-min-reqs}

In order to successfully launch a process on a remote node, several
requirements must be met.  Although each of the boot modules have
different specific requirements, all of them share the following
conditions for successful operation:

\begin{enumerate}
\item Each target host must be reachable and operational.
  
\item The user must be able to execute arbitrary processes on the
  target.
  
\item The LAM executables must be locatable on that machine.  This
  typically involves using: the shell's search path, the
  \ienvvar{LAMHOME} environment variable, or a boot-module-specific
  mechanism.

\item The user must be able to write to the LAM session directory
  (typically somewhere under \file{/tmp}; see
  Section~\ref{sec:misc-session-directory},
  page~\pageref{sec:misc-session-directory}).
  
\item All hosts must be able to resolve the fully-qualified domain
  name (FQDN) of all the machines being booted (including itself).
  
\item Unless there is only one host being booted, any host
  resolving to the IP address 127.0.0.1 cannot be included in the list
  of hosts.
\end{enumerate}

If all of these conditions are not met, \cmd{lamboot} will fail.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Selecting a \kind{boot} Module}

Only one \kind{boot} module will be selected; it will be used for the
life of the LAM universe.  As such, module priority values are the
only factor used to determine which available module should be
selected.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{\kind{boot} SSI Parameters}

On many kinds of networks, LAM can know exactly which nodes should be
making connections while booting the LAM run-time environment, and
promiscuous connections (i.e., allowing any node to connect) are
discouraged.  However, this is not possible in some complex network
configurations and promiscuous connections {\em must} be enabled.
  
By default, LAM's base \kind{boot} SSI startup protocols disable
promiscuous connections.  However, this behavior can be overridden
when LAM is configured and at run-time.  If the SSI parameter
\issiparam{boot\_\-base\_\-promisc} set to an empty value, or set to
the integer value 1, promiscuous connections will be accepted when
than LAM RTE is booted.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{The \boot{bproc} Module}
\index{bproc boot SSI module@\boot{bproc} boot SSI module}
\index{boot SSI modules!bproc@\boot{bproc}}

The Beowulf Distributed Process Space (BProc)
project\footnote{\url{http://bproc.sourceforge.net/}} is set of kernel
modifications, utilities and libraries which allow a user to start
processes on other machines in a Beowulf-style cluster.  Remote
processes started with this mechanism appear in the process table of
the front-end machine in a cluster.

LAM/MPI functionality has been tested with BProc version 3.2.5.  Prior
versions had a bug that affected at least some LAM/MPI functionality.
It is strongly recommended to upgrade to at least version 3.2.5 before
attempting to use the LAM/MPI native BProc capabilities.

%%%%%

\subsubsection{Minimum Requirements}

Several of the minimum requirements listed in
Section~\ref{sec:lam-ssi-boot-min-reqs} will already be met in a BProc
environment because BProc will copy \cmd{lamboot}'s entire environment
(including the \envvar{PATH}) to the remote node.  Hence, if
\cmd{lamboot} is in the user's path on the local node, it will also
[automatically] be in the user's path on the remote node.

However, one of the minimum requirements conditions (``The user must
be able to execute arbitrary processes on the target'') deserves a
BProc-specific clarification.  BProc has its own internal permission
system for determining if users are allowed to execute on specific
nodes.  The system is similar to the user/group/other mechanism
typically used in many Unix filesystems.  Hence, in order for a user
to successfully \cmd{lamboot} on a BProc cluster, he/she must have
BProc execute permissions on each of the target nodes.  Consult the
BProc documentation for more details.

%%%%%

\subsubsection{Usage}

In most situations, the \cmd{lamboot} command (and related commands)
should automatically ``know'' to use the \boot{bproc} boot SSI module
when running on the BProc head node; no additional command line
parameters or environment variables should be required.  
%
Specifically, when running in a BProc environment, the \boot{bproc}
module will report that it is available, and artificially inflate its
priority relatively high in order to influence the boot module
selection process.
%
However, the BProc boot module can be forced by specifying the
\issiparam{boot} SSI parameter with the value of
\issivalue{boot}{bproc}.

Running \cmd{lamboot} on a BProc cluster is just like running
\cmd{lamboot} in a ``normal'' cluster.  Specifically, you provide a
boot schema file (i.e., a list of nodes to boot on) and run
\cmd{lamboot} with it.  For example:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ lamboot hostfile
\end{lstlisting}
% stupid emacs mode: $

Note that when using the \boot{bproc} module, \cmd{lamboot} will only
function properly from the head node.  If you launch \cmd{lamboot}
from a client node, it will likely either fail outright, or fall back
to a different boot module (e.g., \cmd{rsh}/\cmd{ssh}).

It is suggested that the \file{hostfile} file contain hostnames in the
style that BProc prefers -- integer numbers.  For example,
\file{hostfile} may contain the following:

\lstset{style=lam-shell}
\begin{lstlisting}
-1
 0
 1
 2
 3
\end{lstlisting}

\noindent which boots on the BProc front end node (-1) and four slave
nodes (0, 1, 2, 3).  Note that using IP hostnames will also work, but
using integer numbers is recommended.

%%%%%

\subsubsection{Tunable Parameters}

Table~\ref{tbl:lam-ssi-boot-bproc-ssi-params} lists the SSI parameters
that are available to the \boot{bproc} module.

\begin{table}[htbp]
  \begin{ssiparamtb}
%
    \ssiparamentry{boot\_\-bproc\_\-priority}{50}{Default priority level.}
  \end{ssiparamtb}
  \caption{SSI parameters for the \boot{bproc} boot module.}
  \label{tbl:lam-ssi-boot-bproc-ssi-params}
\end{table}

%%%%%

\subsubsection{Special Notes}

After booting, LAM will, by default, not schedule to run MPI jobs on
the BProc front end.  Specifically, LAM implicitly sets the
``no-schedule'' attribute on the -1 node in a BProc cluster.  See
Section~\ref{sec:commands-lamboot}
(page~\pageref{sec:commands-lamboot}) for more detail about this
attribute and boot schemas in general, and
\ref{sec:commands-lamboot-no-schedule} (page 
\pageref{sec:commands-lamboot-no-schedule}).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{The \boot{globus} Module}
\index{globus boot SSI module@\boot{globus} boot SSI module}
\index{boot SSI modules!globus@\boot{globus}}

LAM/MPI \lamversion\ includes beta support for Globus.  Specifically,
only limited types of execution are possible.  The LAM Team would
appreciate feedback from the Globus community on expanding Globus
support in LAM/MPI.

%%%%%

\subsubsection{Minimum Requirements}

LAM/MPI jobs in Globus environment can only be started on nodes using
the ``fork'' job manager for the Globus gatekeeper.  Other job
managers are not yet supported.

%%%%%

\subsubsection{Usage}

Starting the LAM run-time environmetn in Globus environment makes use
of the Globus Resource Allocation Manager (GRAM) client
\icmd{globus-job-run}.
%
The Globus boot SSI module will never run automatically; it must
always be specifically requested setting the \issiparam{boot} SSI
parameter to \issivalue{boot}{globus}.  Specifically, although the
\boot{globus} module will report itself available if
\icmd{globus-job-run} can be found in the \envvar{PATH}, the default
priority will be quite low, effectively ensuring that it will not be
selected unless it is the only module available (which will only occur
if the \ssiparam{boot} parameter is set to \issivalue{boot}{globus}).

LAM needs to be able to find the Globus executables.  This can be
accompilshed either by adding the appropriate directory to your path,
or by setting the \ienvvar{GLOBUS\_\-LOCATION} environment variable.

Additionally, the \ienvvar{LAM\_\-MPI\_\-SESSION\_\-SUFFIX}
environment variable should be set to a unique value.  This ensures
that this instance of the LAM universe does not conflict with any
other, concurrent LAM universes that are running under the same
username on nodes in the Globus environment.  Although any value can
be used for this variable, it is probably best to have some kind of
organized format, such as {\tt
  <your\_\-username>-<some\_\-long\_\-random\_\-number>}.

Next, create a boot schema to use with \cmd{lamboot}.
%
Hosts are listed by their Globus contact strings (see the Globus
manual for more information about contact strings).  In cases where
the Globus gatekeeper is running as a \cmd{inetd} service on the node,
the contact string will simply be the hostname.  If the contact string
contains whitespace, the {\em entire} contact string must be enclosed
in quotes (i.e., not just the values with whitespaces).
%
For example, if your contact string is:

\centerline{\tt host1:port1:/O=xxx/OU=yyy/CN=aaa bbb ccc}

Then you will need to have it listed as:
    
\centerline{\tt "host1:port1:/O=xxx/OU=yyy/CN=aaa bbb ccc"}

The following will not work:
  
\centerline{\tt host1:port1:/O=xxx/OU=yyy/CN="aaa bbb ccc"}
  
Each host in the boot schema must also have a ``{\tt
  lam\_\-install\_\-path}'' key indicating the absolute directory
where LAM/MPI is installed.  This value is mandatory because you
cannot rely on the \ienvvar{PATH} environment variable in Globus
environment because users' ``dot'' files are not executed in Globus
jobs (and therefore the \envvar{PATH} environment variable is not
provided).  Other keys can be used as well; {\tt
  lam\_\-install\_\-path} is the only mandatory key.

Here is a sample Globus boot schema:

\changebegin{7.0.5}
\lstset{style=lam-shell}
\begin{lstlisting}
# Globus boot schema
``inky.mycluster:12853:/O=MegaCorp/OU=Mine/CN=HPC Group'' prefix=/opt/lam cpu=2
``pinky.yourcluster:3245:/O=MegaCorp/OU=Yours/CN=HPC Group'' prefix=/opt/lam cpu=4
``blinky.hiscluster:23452:/O=MegaCorp/OU=His/CN=HPC Group'' prefix=/opt/lam cpu=4
``clyde.hercluster:82342:/O=MegaCorp/OU=Hers/CN=HPC Group'' prefix=/software/lam
\end{lstlisting}
\changeend{7.0.5}

Once you have this boot schema, the \cmd{lamboot} command can be used
to launch it.  Note, however, that unlike the other boot SSI modules,
the Globus boot module will never be automatically selected by LAM --
it must be selected manually with the \issiparam{boot} SSI parameter
with the value \issivalue{boot}{globus}.

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ lamboot -ssi boot globus hostfile
\end{lstlisting}
% stupid emacs mode: $

%%%%%

\subsubsection{Tunable Parameters}

Table~\ref{tbl:lam-ssi-boot-globus-ssi-params} lists the SSI
parameters that are available to the \boot{globus} module.

\begin{table}[htbp]
  \begin{ssiparamtb}
%
    \ssiparamentry{boot\_\-globus\_\-priority}{3}{Default priority level.}
  \end{ssiparamtb}
  \caption{SSI parameters for the \boot{globus} boot module.}
  \label{tbl:lam-ssi-boot-globus-ssi-params}
\end{table}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{The \boot{rsh} Module (including \cmd{ssh})}
\index{rsh (ssh) boot SSI module@\boot{rsh} (\cmd{ssh}) boot SSI module}
\index{boot SSI modules!rsh (rsh/ssh)@\boot{rsh} (\cmd{rsh}/\cmd{ssh})}

The \cmd{rsh}/\cmd{ssh} boot SSI module is typically the ``least
common denominator'' boot module.  When not in an otherwise
``special'' environment (such as a batch scheduler), the
\cmd{rsh}/\cmd{ssh} boot module is typically used to start the LAM
run-time environment.

%%%%%

\subsubsection{Minimum Requirements}

In addition to the minimum requirements listed in
Section~\ref{sec:lam-ssi-boot-min-reqs}, the following additional
conditions must also be met for a successful \cmd{lamboot} using the
\cmd{rsh} / \cmd{ssh} boot module:

\begin{enumerate}
\item The user must be able to execute arbitrary commands on each
  target host without being prompted for a password.
  
\item The shell's start-up script must not print anything on standard
  error. The user can take advantage of the fact that \cmd{rsh} /
  \cmd{ssh} will start the shell non-interactively. The start-up
  script can exit early in this case, before executing many commands
  relevant only to interactive sessions and likely to generate output.

  \changebegin{7.1}
  This has now been changed in version 7.1; if the SSI parameter
  \issiparam{boot\_\-rsh\_\-ignore\_\-stderr} is nonzero, any output
  on standard error will {\em not} be treated as an error.
  \changeend{7.1}
\end{enumerate}

Section~\ref{sec:getting-started} (page~\pageref{sec:getting-started})
provides a short tutorial on using the \cmd{rsh} / \cmd{ssh} boot
module, including tips on setting up ``dot'' files, setting up
password-less remote execution, etc.

%%%%%

\subsubsection{Usage}

Using \cmd{rsh}, \cmd{ssh}, or other remote-execution agent is
probably the most common method for starting the LAM run-time
execution environment.  The boot schema typically lists the hostnames,
CPU counts, and an optional username (if the user's name is different
on the remote machine).  

\changebegin{7.1}

The boot schema can also list an optional ``prefix'', which specifies
the LAM/MPI installatation to be used on the particular host listed in
the boot schema. This is typically used if the user has mutliple
LAM/MPI installations on a host and want to switch between them
without changing the dot files or \envvar{PATH} environment variables,
or if the user has LAM/MPI installed under different paths on
different hosts.  If the prefix is not specified for a host in the
boot schema file, then the LAM/MPI installation which is available in
the \envvar{PATH} will be used on that host, or if the \cmdarg{-prefix
  $<$/lam/install/path$>$} option is specified for \cmd{lamboot}, the
$<$/lam/install/path$>$ installation will be used.  The prefix option
in the boot schema file however overrides any prefix option specified
on the \cmd{lamboot} command line for that host.

For example:

\lstset{style=lam-shell}
\begin{lstlisting}
# rsh boot schema
inky.cluster.example.com cpu=2
pinky.cluster.example.com cpu=4 prefix=/home/joe/lam7.1/install/
blinky.cluster.example.com cpu=4
clyde.cluster.example.com user=jsmith
\end{lstlisting}

\changeend{7.1}

The \cmd{rsh} / \cmd{ssh} boot module will usually run when no other
boot module has been selected.  It can, however, be manually selected,
even when another module would typically [automatically] be selected
by specifying the \issiparam{boot} SSI parameter with the value of
\issivalue{boot}{rsh}.  For example:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ lamboot -ssi boot rsh hostfile
\end{lstlisting}
% stupid emacs mode: $

%%%%%

\subsubsection{Tunable Parameters}

\changebegin{7.1}

Table~\ref{tbl:lam-ssi-boot-rsh-ssi-params} lists the SSI parameters that
are available to the \boot{rsh} module.

\changeend{7.1}

\begin{table}[htbp]
  \begin{ssiparamtb}
%
    \ssiparamentry{boot\_\-rsh\_\-agent}{From configure}{Remote shell
      agent to use.}
%
    \ssiparamentry{boot\_\-rsh\_\-fast}{0}{If nonzero, assume that the
      shell on the remote node is the same as on the origin (i.e., do
      not check).}
% 
   \ssiparamentry{boot\_\-rsh\_\-ignore\_\-stderr}{0}{If nonzero,
      ignore output from \file{stderr} when booting; don't treat it as
      an error.}  \ssiparamentry{boot\_\-rsh\_\-priority}{10}{Default
      priority level.}
%
    \ssiparamentry{boot\_\-rsh\_\-no\_\-n}{0}{If nonzero, don't use
      ``\cmd{-n}'' as an argument to the boot agent}
%
    \ssiparamentry{boot\_\-rsh\_\-no\_\-profile}{0}{If nonzero, don't
      attempt to run ``\file{.profile}'' for Bourne-type shells.}
%
    \ssiparamentry{boot\_\-rsh\_\-username}{None}{Username to use if
      different than login name.}
  \end{ssiparamtb}
  \caption{SSI parameters for the \boot{rsh} boot module.}
  \label{tbl:lam-ssi-boot-rsh-ssi-params}
\end{table}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{The \boot{slurm} Module}
\index{batch queue systems!SLURM boot SSI module}
\index{slurm boot SSI module@\boot{slurm} boot SSI module} 
\index{boot SSI modules!slurm@\boot{slurm}}

\changebegin{7.1}

As its name implies, the Simple Linux Utility for Resource Management
(SLURM)\footnote{http://www.llnl.gov/linux/slurm/} package is commonly
used for managing Linux clusters, typically in high-performance
computing environments.  SLURM contains a native system for launching
applications across the nodes that it manages.  When using SLURM,
\cmd{rsh}/\cmd{ssh} is not necessary to launch jobs on remote nodes.
Instead, the \boot{slurm} boot module will automatically use SLURM's
native job-launching interface to start LAM daemons.

The advantages of using SLURM's native interface are:

\begin{itemize}
\item SLURM can generate proper accounting information for all nodes in
  a parallel job.

\item SLURM can kill entire jobs properly when the job ends.  
  
\item \icmd{lamboot} executes significantly faster when using SLURM as
  compared to when it uses \cmd{rsh} / \cmd{ssh}.
\end{itemize}

%%%%%

\subsubsection{Usage}

SLURM allows running jobs in multiple ways.  The \boot{slurm} boot
module is only supported in some of them:

\begin{itemize}
\item ``Batch'' mode: where a script is submitted via the \icmd{srun}
  command and is executed on the first node from the set that SLURM
  allocated for the job.  The script runs \icmd{lamboot},
  \icmd{mpirun}, etc., as is normal for a LAM/MPI job.
  
  This method is supported, and is perhaps the most common way to run
  LAM/MPI automated jobs in SLURM environments.
  
\item ``Allocate'' mode: where the ``\cmdarg{-A}'' option is given to
  \icmd{srun}, meaning that the shell were \icmd{lamboot} runs is
  likely to {\em not} be one of the nodes that SLURM has allocated for
  the job.  In this case, LAM daemons will be launched on all nodes
  that were allocated by SLURM as well as the origin (i.e., the node
  where \cmd{lamboot} was run.  The origin will be marked as
  ``no-schedule,'' meaning that applications launched by \cmd{mpirun}
  and \cmd{lamexec} will not be run there unless specifically
  requested (see See Section~\ref{sec:commands-lamboot},
  page~\pageref{sec:commands-lamboot}, for more detail about this
  attribute and boot schemas in general).

  This method is supported, and is perhaps the most common way to run
  LAM/MPI interactive jobs in SLURM environments.
  
\item ``\icmd{srun}'' mode: where a script is submitted via the
  \icmd{srun} command and is executed on {\em all} nodes that SLURM
  allocated for the job.  In this case, the commands in the script
  (e.g., \icmd{lamboot}, \icmd{mpirun}, etc.) will be run on {\em all}
  nodes simultaneously, which is most likely not what you want.

  This mode is not supported.
\end{itemize}

When running in any of the supported SLURM modes, LAM will
automatically detect that it should use the \boot{slurm} boot module
-- no extra command line parameters or environment variables should be
necessary.
%
Specifically, when running in a SLURM job, the \boot{slurm} module
will report that it is available, and artificially inflate its
priority relatively high in order to influence the boot module
selection process.
%
However, the \boot{slurm} boot module can be forced by specifying the
\issiparam{boot} SSI parameter with the value of
\issivalue{boot}{slurm}.

Unlike the \cmd{rsh}/\cmd{ssh} boot module, you do not need to specify
a hostfile for the \boot{slurm} boot module.  Instead, SLURM itself
provides a list of nodes (and associated CPU counts) to LAM.  Using
\icmd{lamboot} is therefore as simple as:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ lamboot
\end{lstlisting}
% stupid emacs mode: $

\changebegin{7.1}

Note that in environments with multiple TCP networks, SLURM may be
configured to use a network that is specifically designated for
commodity traffic -- another network may exist that is specifically
allocated for high-speed MPI traffic.  By default, LAM will use the
same hostnames that SLURM provides for all of its traffic.  This means
that LAM will send all of its MPI traffic across the same network that
SLURM uses.

However, LAM has the ability to boot using one set of hostnames /
addresses and then use a second set of hostnames / addresses for MPI
traffic.  As such, LAM can redirect its TCP MPI traffic across a
secondary network.  It is possible that your system administrator has
already configured LAM to operate in this manner.

If a secondary TCP network is intended to be used for MPI traffic, see
the section entitled ``Separating LAM and MPI TCP Traffic'' in the
LAM/MPI Installation Guide.  Note that this functionality has no
effect on non-TCP \kind{rpi} modules (such as Myrinet, Infiniband,
etc.).

\changeend{7.1}

%%%%%

\subsubsection{Tunable Parameters}

Table~\ref{tbl:lam-ssi-boot-slurm-ssi-params} lists the SSI parameters
that are available to the \boot{slurm} module.

\begin{table}[htbp]
  \begin{ssiparamtb}
%
    \ssiparamentry{boot\_\-slurm\_\-priority}{50}{Default priority level.}
  \end{ssiparamtb}
  \caption{SSI parameters for the \boot{slurm} boot module.}
  \label{tbl:lam-ssi-boot-slurm-ssi-params}
\end{table}

%%%%%

\subsubsection{Special Notes}

Since the \boot{slurm} boot module is designed to work in SLURM jobs,
it will fail if the \boot{slurm} boot module is manually specified and
LAM is not currently running in a SLURM job.

The \boot{slurm} module does not start a shell on the remote node.
Instead, the entire environment of \cmd{lamboot} is pushed to the
remote nodes before starting the LAM run-time environment.
  
\changeend{7.1}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{The \boot{tm} Module (OpenPBS / PBS Pro / Torque)}
\index{batch queue systems!OpenPBS / PBS Pro / Torque (TM) boot SSI module}
\index{tm boot SSI module@\boot{tm} boot SSI module} 
\index{boot SSI modules!tm (PBS / Torque)@\boot{tm} (PBS / Torque)}

Both OpenPBS and PBS Pro (both products of Altair Grid Technologies,
LLC), contain support for the Task Management (TM) interface.  Torque,
the open source fork of the Open MPI product, also contains the TM
interface.  When using TM, \cmd{rsh}/\cmd{ssh} is not necessary to
launch jobs on remote nodes.

The advantages of using the TM interface are:

\begin{itemize}
\item PBS/Torque can generate proper accounting information for all
  nodes in a parallel job.
  
\item PBS/Torque can kill entire jobs properly when the job ends.
  
\item \icmd{lamboot} executes significantly faster when using TM as
  compared to when it uses \cmd{rsh} / \cmd{ssh}.
\end{itemize}

%%%%%

\subsubsection{Usage}

When running in a PBS/Torque batch job, LAM will automatically detect
that it should use the \boot{tm} boot module -- no extra command line
parameters or environment variables should be necessary.
%
Specifically, when running in a PBS/Torque job, the \boot{tm} module
will report that it is available, and artificially inflate its
priority relatively high in order to influence the boot module
selection process.
%
However, the \boot{tm} boot module can be forced by specifying the
\issiparam{boot} SSI parameter with the value of \issivalue{boot}{tm}.

Unlike the \cmd{rsh}/\cmd{ssh} boot module, you do not need to specify
a hostfile for the \boot{tm} boot module.  Instead, PBS/Torque itself
provides a list of nodes (and associated CPU counts) to LAM.  Using
\icmd{lamboot} is therefore as simple as:

\lstset{style=lam-cmdline}
\begin{lstlisting}
shell$ lamboot
\end{lstlisting}
% stupid emacs mode: $

The \boot{tm} boot modules works in both interactive and
non-interactive batch jobs.

\changebegin{7.1}

Note that in environments with multiple TCP networks, PBS / Torque may
be configured to use a network that is specifically designated for
commodity traffic -- another network may exist that is specifically
allocated for high-speed MPI traffic.  By default, LAM will use the
same hostnames that the TM interface provides for all of its traffic.
This means that LAM will send all of its MPI traffic across the same
network that PBS / Torque uses.

However, LAM has the ability to boot using one set of hostnames /
addresses and then use a second set of hostnames / addresses for MPI
traffic.  As such, LAM can redirect its TCP MPI traffic across a
secondary network.  It is possible that your system administrator has
already configured LAM to operate in this manner.

If a secondary TCP network is intended to be used for MPI traffic, see
the section entitled ``Separating LAM and MPI TCP Traffic'' in the
LAM/MPI Installation Guide.  Note that this has no effect on non-TCP
\kind{rpi} modules (such as Myrinet, Infiniband, etc.).

\changeend{7.1}

%%%%%

\subsubsection{Tunable Parameters}

Table~\ref{tbl:lam-ssi-boot-tm-ssi-params} lists the SSI parameters
that are available to the \boot{tm} module.

\begin{table}[htbp]
  \begin{ssiparamtb}
%
    \ssiparamentry{boot\_\-tm\_\-priority}{50}{Default priority level.}
  \end{ssiparamtb}
  \caption{SSI parameters for the \boot{tm} boot module.}
  \label{tbl:lam-ssi-boot-tm-ssi-params}
\end{table}

%%%%%

\subsubsection{Special Notes}

Since the \boot{tm} boot module is designed to work in PBS/Torque
jobs, it will fail if the \boot{tm} boot module is manually specified
and LAM is not currently running in a PBS/Torque job.

The \boot{tm} module does not start a shell on the remote node.
Instead, the entire environment of \cmd{lamboot} is pushed to the
remote nodes before starting the LAM run-time environment.
  
Also note that the Altair-provided client RPMs for PBS Pro do not
include the \icmd{pbs\_\-demux} command, which is necessary for proper
execution of TM jobs.  The solution is to copy the executable from the
server RPMs to the client nodes.
  
Finally, TM does not provide a mechanism for path searching on the
remote nodes, so the \cmd{lamd} executable is required to reside in
the same location on each node to be booted.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% Close of index

\index{boot SSI modules|)}

