MC# Programming System for Linux
        	  USER'S GUIDE
          
1. INTRODUCTION

This package contains MC# 3.0 Programming System for Linux.
MC# Programming System can be used on both multicore (or SMP - 
symmetric multiprocessing) machines and clusters.
The rules of programming for multicore machines (= concurrent prigramming)
and for clusters (= distributed programming) can be found in
"Introduction to programming in MC# language" document.

2. SYSTEM REQUIREMENTS

1) Operating systems: all ones which support Mono (www.mono-project.com)
- an open source implementation of .NET for Unix-like systems.
2) Mono >= 2.10
3) Libgdiplus libarary >= 2.10 (to support the graphical applications)
4) rsh or ssh configutred on the cluster nodes for password free logins

3. INSTALLATION

To install MC# on your machine you need to do the following steps:

1) Untar the gzip-archive into the corresponding directory, for example
			~/mcsharp
2) Add the following environment variables:
	$MCSHARPPATH - main directory of MC# Programming System
	$MONO_PATH - in that directory Mono will search needed dlls;
	             you must add here the $MCSHARPPATH/lib and
	             $MCSHARPPATH/bin directories
	$PATH - add here the directory $MCSHARPPATH/bin
	$LD_LIBRARY_PATH - you must add here $MCSHARPPATH/lib
		
For example, if you use bash shell, you need to add the following lines into
the file .bashrc:

export MCSHARPPATH=<path to your home dir>/mcsharp
export MONO_PATH=$MCSHARPPATH/lib:$MCSHARPPATH/bin:$MONO_PATH
export PATH=$MCSHARPPATH/bin:$PATH
export LD_LIBRARY_PATH=$MCSHARPPATH/lib:$LD_LIBRARY_PATH

3) Make MC# libraries and modules:

cd ~/mcsharp
make

NOTE.
-------------

To run the distribute MC# programs on cluster, the installer 
will create in ~/.mcsharp directory two files

		.rmport
		.wnport
		
and will write there some free ports on your system. 
These ports are needed to Reource Manager (RM) and WorkNode (WN)
components of MC# Programming System. RM uses port from .rmport-file
to listen connections from WN-services. WNs are running on all cluster
nodes and use port from .wnport-file to listen connections from RM.
You can change these ports manually if needed.
Also the installer will create in ~/.mcsharp directory the file

		.accesstype
		
in which you need to point out a command type to enter to the cluster
nodes ("rsh" value is used by default).

4. COMPILATION OF MC# APPLICATIONS

Compilation of MC# applications is performed by "mcsc" compiler. 
There are two stages of compilation:
	- compilation from MC# to C#, and then
	- compilation from C# to bytecode.								
The latter is performed by standard "gmcs" compiler from the Mono package.

The source files with MC# code must have .mcs extensions.

Compilation command usage:

>mcsc myprogram.mcs

You can use any parameters for C# compiler as usual:

>mcsc /t:library /out:mylibrary.dll myprogram.mcs

You can mix an arbitrary number of MC# and C# files to compile:

>mcsc /r:mylibrary.dll a.mcs b.mcs c.cs d.cs

Compilation of concurrent and distributed (intended to run on a cluster)
doesn't differ in any way.

5. RUNNING OF MC# PROGRAMS

To run a concurrent MC# program (i.e., a program which uses async-methods)
you need to perform a command like

>mono myprogram.exe arg1 ... argN

This kind of MC# program running we call a running in "local mode".
In particular, if you run a distributed MC# program (i.e., a program
which uses movable methods) in local mode, then the movable methods will
be interpreted as async-methods executing on local machine. You may
use this as a convenient tool to debug the didstributed programs before 
running them on cluster.

To run a distribited MC# program on a cluster, you nedd to do 3 steps:

1) boot the MC# Runtime system,
2) run the program in distributed mode,
3) halt the MC# Runrime system.

To boot the Runtime-system you need to perform 

>mcsboot <file containing names of nodes>

(by default mcsboot uses $MCSHARPPATH/bin/nodes file).

In the file with names, each name must be unique and be placed on 
the separate line, for example

	node-21
	node-22
	node-31
	node-32
	
A line which contains the symbol "#" in the first position
is considered as a comment.

To use a current machine where an application is starting as a worknode,
you may use "localhost" name in the file with names.

Often a file with names of the nodes is provided by some batch system 
running on the cluster. In this case, mcsboot has some additional switches 
to account for the format of such file.

The switch -d will lead to ignoring of multiply occurences of some names in the file.
So if a batch system provides the file with the names in the format like

	node-1   -|  number of occurences is equal to number	
	 ...      |  processors on the node
	node-1   -|
	node-2
	 ...
	node-2
	
	 ...
	
	node-n
	 ...
	node-n 

you need to start mcsboot as

>mcsboot -d <my nodes file>

If a batch system provides the file where the number of processors
on the node follows by the colon like

	node-1:8
	node-2:8
	 ...
	node-n:8
	
you need to apply the switch -n as

>mcsboot -n <my nodes file>

In fact, there are two different possibilities to use the main machine
(frontend) and work nodes of the cluster for running of distributed
MC# programs:
	1) to run an application on the frontend and use the work nodes
	   for running of movable methods, and
	2) to run an application on the one of the work nodes with
	   running movable methods on the other work nodes.
In the last case, it os reasonable don't use a work node running the
application as a node to execute movable methods. In similar cases,
we may to ignore a first name in the file with names by applying
the switch "-f" as follows

>mcsboot -f <my nodes file>

As usually, you may apply a several switches to the same file as below

>mcsboot -d -f <my nodes file>

If the booting of MC# Runtime system is successful, then you may start
a distributed MC# program by specifying the additional switch /np
to the command line:

>mono myprogram.exe arg1 .. argn /np N

where N is a number of nodes you like use in performing of your program.
In the most cases, N is equal to the number of nodes which were
enumerated in the files with names for mcsboot.

By using the switch /n you may point out the number of processors
(cores) you wish to use on the each node:

>mono myprogram.exe /np 8 /n 2

This may be useful for debugging and measurement purposes.

The switch /completeboot forces the Runtime to wait when WN services
has been started on the all nodes and only then to start an application.
Otherwise, a distribution of movable methods between the nodes can be
strated before some WN services become ready. This may lead to uneven
workload between the nodes.

The switch /withlog forces to save the outputs of movable methods 
running on the work nodes. The outputs are saved in the log files -
one file for the one node.

The switch /showstats  forces to output a some statistics about 
running of distributed MC# program.

To stop the MC# runtime system you need to perform the command

>mcshalt

or

>mcshalt -force

The last command allows to terminate normally the failed MC# applications
in some cases.

6. RUNNING MC# PROGRAMS in BATCH MODE

Here we show how to run MC# programs on clusters with the different
batch systems.

1) PBS/Torque

Below is a pattern of the script to run MC# programs under PBS/Torque
batch system.

#!/bin/sh
#PBS -N <job name>
#PBS -j n
#PBS -o <full path to output file>
#PBS -l walltime=<limit time to run the job>
#PBS -l nodes=<number of nodes>:ppn=<number processors on the node>
#PBS -q batch
set -x
cd <path to the directory where MC# .exe-module resides>
cat $PBS_NODEFILE > hosts
mcsboot -d -f hosts
mono <name of .exe-module> <arguments> <MC# switches> /np 3 <number of nodes>
mcshalt -force
exit

To start such script, you need to place it in file, say, mcsharp_job
and perform the command 

>qsub mcsharp_job

Here is a some example of the script:

#!/bin/sh
#PBS -N mcsharp_job
#PBS -j n
#PBS -o /home/myname/output_file.txt
#PBS -l walltime=1:00:00
#PBS -l nodes=4:ppn=4
#PBS -q batch
set -x
cd /home/myname
cat $PBS_NODEFILE > hosts
mcsboot -d -f hosts
mono myprogram.exe /withlog /showstats /completeboot /np 3
mcshalt -force
exit

2) A task running management system (for MVS-100K cluster)

A task running management system (TRMS) is a batch system
developed in the Keldysh Institute of Applied Mathematics
(Russian Academy of Sciences). This system is used on MVS-100K cluster 
in the Joint SuperComputer Center of Russian Academy of Sciences.

Here is a pattern of the script to run MC# programs under TRMS
batch system:

#!/bin/bash
getnodes -np <number of processors> -maxtime <limit time> <job name>
taskname=`mlt`
rc=1
while [ 0 -ne $rc ]; do
mqtest -rcode $taskname
rc=$?
#
sleep 1
done
echo "getnodes DONE dir=" $taskname
cd $taskname
mcsboot -n .hosts
#
mono < name of .exe-module> <arguments> <MC# switches> /np <number of nodes> > <output file>
#
mcshalt -force
mkill $taskname

To start such kind script, you need to perform the command

>./<script file>

Below is a sample script to start the job on the 160 nodes:

#!/bin/bash
getnodes -np 1280 -maxtime 1400 mcsharp_job
taskname=`mlt`
rc=1
while [ 0 -ne $rc ]; do
mqtest -rcode $taskname
rc=$?
#
sleep 1
done
echo "getnodes DONE dir=" $taskname
cd $taskname
cat .hosts
mcsboot -n .hosts
#
mono myprogram.exe /showstats /completeboot /withlog /np 160 > myprogram.out
#
mcshalt -force
mkill $taskname

3) Cleo ( "SKIF-MGU" cluster )

Cleo is a batch system developed in the Scientific Research 
Computational Center of Moscow State University.
The system is used on the "SKIF-MGU" and on some other clusters.

The Cleo system supports the running of MC# programs directly
(without any additional scripts) - you need only to put the 
value of -as switch as "mcs".

Here is the example of job running on the 160 nodes:

mpirun -np 1280 -as mcs -maxtime 2500 -stdout /home/myname/myprog.out \
       myprog.exe /withlog /showstats /completeboot /np 160
       
       
6. Using GPU in MC#-programs

To use GPU in your MC#-programs, you need 

a) CUDA package have been installed on your machine;
in particular, you need to set the environment variables like as

export LD_LIBRARY_PATH=/opt/cuda/lib64:$LD_LIBRARY_PATH
export PATH=/opt/cuda/bin:$PATH

b) moreover, you need to set the environment variables pointing to the corresponding libraries

export MONO_PATH=$MCSHARPPATH/gpu/lib:$MCSHARPPATH/bin
export LD_LIBRARY_PATH=$MCSHARPPATH/gpu/lib:$LD_LIBRARY_PATH