From: mccalpin@frakir.engr.sgi.com (John McCalpin)
Newsgroups: comp.parallel
Subject: Re: Origin2000 Shared Memory
Date: 9 May 1999 04:48:04 GMT
Organization: Silicon Graphics, Inc.
Approved: bigrigg@cs.cmu.edu
Message-Id: <7h3424$j47$1@goldenapple.srv.cs.cmu.edu>
Originator: bigrigg@ux6.sp.cs.cmu.edu
Xref: ukc comp.parallel:15580


In article <7goeiq$m7i$1@goldenapple.srv.cs.cmu.edu>,
John Roberts  <johnr@jhu.edu> wrote:
>When I run a parallel shared memory (as opposed to MPI) program on an
>Origin 2000 how do I specify and determine how many processors I'm
>using? 

The traditional invocation has been 
	setenv MP_SET_NUMTHREADS $(THREADS)

With the newer compilers (7.2.1 and following), the MP_SET_NUMTHREADS
still works, but we have added a new variable
	setenv OMP_NUM_THREADS $(THREADS)
to make it more clear that this variable is setting the number
of threads for OpenMP parallelism.

It is also possible to set the number of threads within the
running program by a call of the form:
	CALL MP_SET_NUMTHREADS (ITHREADS)

>If I run 8 threads, what's to say that all 8 threads aren't time
>sharing on the same CPU? 

The O/S works very hard to ensure that the 8 threads are running
on different cpus, but cpus that are close to each other in the
system interconnect topology.

>If I get 8 CPU's do I have exclusive access
>to them, or can other threads time-share with mine? 

Other threads can time-share on "your" cpus unless you are running
with reserved cpus using Miser (a space-sharing scheduler).  The O/S
spreads the load across the system as evenly as possible, so if your
threads having been running for a while, then it is unlikely that it
will schedule other threads on "your" cpus if there are idle cpus
elsewhere in the system.

>Can my threads be
>migrated to other processors? How do I know when this happens?

Unless the threads have been "pinned", they might be moved to 
other processors.  The O/S generally does not want to do this
(cache affinity is a very strong factor in scheduling), but on
heavily loaded systems, or on systems with many short-running
jobs, it may happen.

To "pin" a shared-memory job, you can set an environment variable
	setenv _DSM_MUSTRUN 1

or you can launch your job using "dplace -mustrun a.out" (try 
"man dplace" for details).

I believe that "par" can be used to monitor all system scheduler
events, such as moving your process to another cpu.  There is a
relatively steep learning curve associated with understanding "par"
output.

As a final comment, the newer versions of the O/S (6.5 & following)
combined with the newer compilers (7.2.1 & following), have a 
default of "dynamic threads".  If this is active (as it is by default),
the O/S will reduce the number of threads available to your job
if the machine becomes oversubscribed.  So your 8 thread job may
find itself told to reduce to 4 threads if another 8 thread job
is started on the same 8-cpu system.  The OpenMP support library
takes care of the details. 

To prevent this behaviour, you can turn dynamic threads off
for your session with
		setenv OMP_DYNAMIC FALSE

Reading "man mp" is definitely a good idea -- just skip the
sections on the older SGI parallelization directions and the
PCF directives -- these have all been superceded by the OpenMP
directives (they still work, but are not the recommended way
to develop new codes).
-- 
--
John D. McCalpin, Ph.D.      Principal Scientist
System Architecture Group    http://reality.sgi.com/mccalpin/
SGI                          mccalpin@sgi.com  650-933-7407

--
Articles to bigrigg+parallel@cs.cmu.edu (Admin: bigrigg@cs.cmu.edu)
Archive: http://www.hensa.ac.uk/parallel/internet/usenet/comp.parallel