Cluster: Difference between revisions

From RedwoodCenter
Jump to navigationJump to search
No edit summary
No edit summary
Line 1: Line 1:
== General Information ==
= General Information =
 
== Directory setup ==


=== home directory quota ===
=== home directory quota ===
Line 26: Line 28:
* press enter
* press enter
* the 7 digit password is given (without the dash)
* the 7 digit password is given (without the dash)
=== setup environment ===
* put all your customizations into your .bashrc
* for login shells, .bash_profile is used, which in turn loads .bashrc


=== ssh to the gateway computer (hadley) ===
=== ssh to the gateway computer (hadley) ===
Line 39: Line 36:


and use your crypto password
and use your crypto password
=== Setup environment ===
* put all your customizations into your .bashrc
* for login shells, .bash_profile is used, which in turn loads .bashrc


== Useful commands ==
== Useful commands ==
Line 116: Line 118:
   mpirun -np 4 -mca btl ^openib sample_executables/mpi_hello
   mpirun -np 4 -mca btl ^openib sample_executables/mpi_hello


=== Matlab ===
= Software =
 
== Matlab ==


'''note: remember to start an interactive session before starting matlab!'''
'''note: remember to start an interactive session before starting matlab!'''
Line 128: Line 132:
   matlab -nojvm -nodesktop
   matlab -nojvm -nodesktop


=== Python ===
== Python ==
 
We have several Python Distributions installed: The Enthought Python Distribution (EPD), the Source Python Distribution (SPD) and Sage. The easiest way to get started is probably to use EPD (see below).
 
=== Enthought Python Distribution (EPD) ===
 
We have the Source Python Distribution installed [[http://www.enthought.com/products/epd.php EPD]]. In order to use it, you have to follow the following steps:
 
* login to the gateway server using "ssh -Y" (see above)
* start an interactive session using "qsub -I -X" (see above)
* load the python environment module:
 
  module load python/epd
 
* start ipython:
 
  ipython -pylab
 
* run the following commands inside ipython to test the setup:
 
  from enthought.mayavi import mlab
  mlab.test_contour3d()
 
=== Source Python Distribution (SPD) ===


We have the Source Python Distribution installed [[http://code.google.com/p/spdproject/ SPD]]. In order to use python with ipython/numpy/scipy, you have to first load the python environment module:
We have the Source Python Distribution installed [[http://code.google.com/p/spdproject/ SPD]]. In order to use it, you have to first load the python environment module:


   module load python/spd
   module load python/spd
Line 172: Line 199:
--Amir
--Amir


=== CUDA ===
== CUDA ==


I've installed CUDA 2.2 toolkit here:
I've installed CUDA 2.2 toolkit here:
Line 192: Line 219:
--Amir
--Amir


== Support Requests ==
= Support Requests =


* If you have a problem that is not covered on this page, you can send an email to our user list:
* If you have a problem that is not covered on this page, you can send an email to our user list:

Revision as of 21:24, 21 August 2009

General Information

Directory setup

home directory quota

There is a 10GB quota limit enforced on $HOME directory (/global/home/users/username) usage. Please keep your usage below this limit. There will NETAPP snapshots in place in this file system so we suggest you store only your source code and scripts in this area and store all your data under /clusterfs/cortex (see below).

data

For large amounts of data, please create a directory

 /clusterfs/cortex/scratch/username

and store the data inside that directory.

Connect

get a password

  • press the PASS WORD button on your crypto card
  • enter passoword
  • press enter
  • the 7 digit password is given (without the dash)

ssh to the gateway computer (hadley)

note: please don't use the gateway for computations (e.g. matlab)!

 ssh -Y neuro-calhpc.berkeley.edu (or hadley.berkeley.edu) 

and use your crypto password

Setup environment

  • put all your customizations into your .bashrc
  • for login shells, .bash_profile is used, which in turn loads .bashrc

Useful commands

Start interactive session on compute node

  • start interactive session:
 qsub -X -I
  • start interactive session on particular node (nodes n0000.cortex and n0001.cortex have GPUs):
 qsub -X -I -l nodes=n0001.cortex

Perceus commands

The perceus manual is here

  • listing available cluster nodes:
 wwstats
  • list cluster usage
 wwtop
  • to restrict the scope of these commands to cortex cluster, add the following line to your .bashrc
 export NODES='*cortex'
  • module list
  • module avail
  • module help


Resource Manager PBS

  • Job Scheduler MOAB
  • List running jobs:
 qstat -a
  • List jobs of a given node:
 qstat -n 98
  • sample script
 #!/bin/bash
 
 #PBS -q cortex
 #PBS -l nodes=1:ppn=2:cortex
 #PBS -l walltime=01:00:00
 #PBS -o path-to-output
 #PBS -e path-to-error
 cd /global/home/users/kilian/sample_executables
 cat $PBS_NODEFILE
 mpirun -np 8 /bin/hostname
 sleep 60
  • submit script
 qsub scriptname
  • interactive session
 qsub -I -q cortex -l nodes=1:ppn=2:cortex -l walltime=00:15:00
  • list nodes that your job is running on
 cat $PBS_NODEFILE
  • run the program on several cores
 mpirun -np 4 -mca btl ^openib sample_executables/mpi_hello

Software

Matlab

note: remember to start an interactive session before starting matlab!

In order to use matlab, you have to load the matlab environment:

 module load matlab

Once the matlab environment is loaded, you can start a matlab session by running

 matlab -nojvm -nodesktop

Python

We have several Python Distributions installed: The Enthought Python Distribution (EPD), the Source Python Distribution (SPD) and Sage. The easiest way to get started is probably to use EPD (see below).

Enthought Python Distribution (EPD)

We have the Source Python Distribution installed [EPD]. In order to use it, you have to follow the following steps:

  • login to the gateway server using "ssh -Y" (see above)
  • start an interactive session using "qsub -I -X" (see above)
  • load the python environment module:
 module load python/epd
  • start ipython:
 ipython -pylab
  • run the following commands inside ipython to test the setup:
 from enthought.mayavi import mlab
 mlab.test_contour3d()

Source Python Distribution (SPD)

We have the Source Python Distribution installed [SPD]. In order to use it, you have to first load the python environment module:

 module load python/spd

Afterwards, you can run ipython

 % ipython -pylab

At the moment, we have numpy, scipy, and matplotlib installed. If you would like to have additional modules installed, let me know [kilian]

Sage

Sage is http://sagemath.org. In order to use sage, you have to first load the sage environment module

 module load python/sage

After loading the sage module, if you want to have a scipy environment (run ipython, etc) in your interactive session, first do:

 % sage -sh

then you can run:

 % ipython

or you can just do:

 % sage -ipython

This is a temporary solution for people wanting use scipy with mpi on the cluster. It was built against the default openmpi (1.2.8) (icc) and mpi4py 1.1.0. For those using hdf5, I also built hdf5 1.8.3 (gcc) and h5py 1.2.

Sample pbs and mpi script is here:

 ~amirk/test

You can run it as:

 % mkdir -p ~/jobs
 % cd ~amirk/test
 % qsub pbs

--Amir

CUDA

I've installed CUDA 2.2 toolkit here:

 /clusterfs/cortex/software/cuda-2.2 

The SDK is here:

 /clusterfs/cortex/software/cuda-2.2/sdk

To your PATH, add:

 /clusterfs/cortex/software/cuda-2.2/bin

To your LD_LIBRARY_PATH, add:

 /clusterfs/cortex/software/cuda-2.2/lib

--Amir

Support Requests

  • If you have a problem that is not covered on this page, you can send an email to our user list:
 redwood_cluster@lists.berkeley.edu
  • If you need additional help from the LBL group, send an email to their email list. Please always cc our email list as well.
 scs@lbl.gov
  • In urgent cases, you can also email Krishna Muriki (LBL User Services) directly.