# n2001.lbl.gov
mpirun [mpirun_options...] <progname> [options...]

  mpirun_options:
    -arch <architecture>
            specify the architecture (must have matching machines.<arch>
            file in /usr/local/pkg/mpich-1.2.5/share) if using the execer
    -h      This help
    -machine <machine name>
            use startup procedure for <machine name>
            Currently supported:
              paragon
              p4
              sp1
              ibmspx
              anlspx
              sgi_mp
              ipsc860
              inteldelta
              cray_t3d
              execer
              smp
              symm_ptx

    -machinefile <machine-file name>
            Take the list of possible machines to run on from the
            file <machine-file name>.  This is a list of all available
            machines; use -np <np> to request a specific number of machines.
    -machinedir <directory>
            Look for the machine files in the indicated directory
    -np <np>
            specify the number of processors to run on
    -nodes <nodes>
            specify the number of nodes to run on (for SMP systems, 
	    currently only ch_mpl device supports this)
    -nolocal
            don't run on the local machine (only works for 
            ch_p4 jobs)
    -all-cpus, -allcpus
            Use all available CPUs on all the nodes.
    -all-local
            Run all processes on the master node.
    -exclude <list>
            Exclude nodes in a colon delimited list.
    -map <list>
            Use the colon delimited list to specify which rank
            runs on which nodes.
    -stdin filename
            Use filename as the standard input for the program.  This
            is needed for programs that must be run as batch jobs, such
            as some IBM SP systems and Intel Paragons using NQS (see 
            -paragontype below).
	    use 
                  -stdin /dev/null 
            if there is no input and you intend to run the program in the
            background.  An alternate is to redirect standard input from
            /dev/null, as in 
                  mpirun -np 4 a.out < /dev/null
    -t      Testing - do not actually run, just print what would be
            executed
    -v      Verbose - throw in some comments
    -dbg    The option '-dbg' may be used to select a debugger.  For example,
            -dbg=gdb invokes the mpirun_dbg.gdb script located in the 
	    'mpich/bin' directory.  This script captures the correct arguments,
            invokes the gdb debugger, and starts the first process under gdb 
            where possible.  There are 4 debugger scripts; gdb, xxgdb, ddd, 
	    totalview.  These may need to be edited depending on your system.
            There is another debugger script for dbx, but this one will always
            need to be edited as the debugger commands for dbx varies between 
            versions.  You can also use this option to call another debugger; 
            for example, -dbg=mydebug.  All you need to do is write a script 
            file, 'mpirun_dbg.mydebug', which follows the format of the 
            included debugger scripts, and place it in the mpich/bin directory.
    -ksq    Keep the send queue. This is useful if you expect later
            to attach totalview to the running (or deadlocked) job, and
	    want to see the send queues. (Normally they are not maintained 
            in a way which is visible to the debugger).

      Special Options for NEC - CENJU-3:

    -batch  Excecute program as a batch job (using cjbr)

    -stdout filename
            Use filename as the standard output for the program.
    -stderr filename
            Use filename as the standard error  for the program.
    -jid    Jobid from Job-Scheduler EASY.
            If this option is specified, mpirun directly executes
            the parallel program using this jobid.
            Otherwise, mpirun requests np nodes from the Job-Scheduler
                       in interactive or batch mode.
                       In interactive mode (i.e. option -batch is not
                       specified), mpirun waits until the processors are
                       allocated, executes the parallel program and
                       releases the processors.
	          
	          
    Special Options for Batch Environments:

    -mvhome Move the executable to the home directory.  This 
            is needed when all file systems are not cross-mounted
            Currently only used by anlspx
    -mvback files
            Move the indicated files back to the current directory.
            Needed only when using -mvhome; has no effect otherwise.
    -maxtime min
            Maximum job run time in minutes.  Currently used only
            by anlspx.  Default value is 15 minutes.
    -nopoll Do not use a polling-mode communication.
            Available only on IBM SPx.
    -mem value
	   This is the per node memory request (in Mbytes).  Needed for some
           CM-5s. ( Default 32. )
    -cpu time
	   This is the the hard cpu limit used for some CM-5s in
	   minutes. (Default 15 minutes.)

    Special Options for IBM SP2:

    -cac name
            CAC for ANL scheduler.  Currently used only by anlspx.
            If not provided will choose some valid CAC.

On exit, mpirun returns a status of zero unless mpirun detected a problem, in
which case it returns a non-zero status (currently, all are one, but this 
may change in the future).

Multiple architectures may be handled by giving multiple -arch and -np 
arguments.  For example, to run a program on 2 sun4s and 3 rs6000s, with 
the local machine being a sun4, use

    /usr/local/pkg/mpich-1.2.5/bin/mpirun -arch sun4 -np 2 -arch rs6000 -np 3 program

This assumes that program will run on both architectures.  If different
executables are needed, the string '%a' will be replaced with the arch name.
For example, if the programs are program.sun4 and program.rs6000, then the
command is

    /usr/local/pkg/mpich-1.2.5/bin/mpirun -arch sun4 -np 2 -arch rs6000 -np 3 program.%a

If instead the executables are in different directories; for example, 
/tmp/me/sun4 and /tmp/me/rs6000, then the command is

    /usr/local/pkg/mpich-1.2.5/bin/mpirun -arch sun4 -np 2 -arch rs6000 -np 3 /tmp/me/%a/program

It is important to specify the architecture with -arch BEFORE specifying
the number of processors.  Also, the FIRST -arch command must refer to the
processor on which the job will be started.  Specifically, if -nolocal is 
NOT specified, then the first -arch must refer to the processor from which
mpirun is running.