Cluster-Software: Difference between revisions

From RedwoodCenter
Jump to navigationJump to search
 
(4 intermediate revisions by 2 users not shown)
Line 30: Line 30:


The Anaconda Python 2.7 or 3.4 Distributions can be loaded through
The Anaconda Python 2.7 or 3.4 Distributions can be loaded through
   module load python/anaconda2/anaconda2
   module load python/anaconda2
or
or
   module load python/anaconda3/anaconda3
   module load python/anaconda3
respectively. This distribution has NumPy and SciPy built against the Intel MKL BLAS library (multicore BLAS). You will need to get an [https://store.continuum.io/cshop/academicanaconda academic license] from Continuum and copy it to the cluster.
respectively. This distribution has NumPy and SciPy built against the Intel MKL BLAS library (multicore BLAS). You will need to get an [https://store.continuum.io/cshop/academicanaconda academic license] from Continuum and copy it to the cluster.


Line 52: Line 52:
== CUDA ==
== CUDA ==


CUDA is a library to use the graphics processing units (GPU) on the graphics card for general-purpose computing. We have a separate wiki page to collect information on how to do general-purpose computing on the GPU: [[GPGPU]].
CUDA is a library to use the graphics processing units (GPU) on the graphics card for general-purpose computing.
The --constraint={cortex_k40, cortex_fermi} option must be used in order to schedule a node with a GPU.
The  
We have installed the CUDA 6.5 driver and toolkit.
 
  #SBATCH--constraint={cortex_k40, cortex_fermi}  
  #SBATCH --gres=gpu:1
 
options must be used in order to schedule a node with a GPU.


In order to use CUDA, you have to load the CUDA environment:
In order to use CUDA, you have to load the CUDA environment:
Line 65: Line 69:
   module unload intel
   module unload intel


Theano caches certain compiled libraries and these will sometimes cause errors when Theano gets updated. If you are experiencing problems with Theano, you can try clearing the cache with
Theano caches certain compiled functions and these will sometimes cause errors when Theano or CUDA gets updated. If you are experiencing problems with Theano, you can try clearing the cache with
   theano-cache clear
   theano-cache clear
and if you still have problems you can delete the .theano folder from your home directory.
and if you still have problems you can delete the .theano folder from your home directory.
Line 77: Line 81:


   [global]
   [global]
  root = /global/software/sl-6.x86_64/modules/langs/cuda/6.5/
   device = gpu
   device = gpu
   floatX = float32
   floatX = float32
Line 87: Line 90:
==== Using the CPU ====
==== Using the CPU ====


Theano can also run on the CPU. Any of the CPU nodes will work. You will want to have Theano build against the MKL BLAS library that comes with Anaconda and so your .theanorc might look like
Theano can also run on the CPU. Any of the CPU nodes will work. You will want to have Theano use the MKL BLAS library that comes with Anaconda and so your .theanorc might look like


   [global]
   [global]
Line 93: Line 96:
   floatX = float32
   floatX = float32
   ldflags = -lmkl_rt
   ldflags = -lmkl_rt
=== Obtain GPU lock for Jacket in Matlab ===
If you are using Matlab, you can obtain a GPU lock by running
  addpath('/clusterfs/cortex/software/gpu_lock');
  addpath('/clusterfs/cortex/software/jacket/engine');
  gpu_id = obtain_gpu_lock_id();
  gselect(gpu_id);
By default, obtain_gpu_lock() throws an error when all gpu cards are taken.
There is another option: obtain_gpu_lock_id(true) will return -1 in case there
is no card available and you can then write your own code to deal with that
fact.
ginfo tells you which gpu card you are using.
The following lines should also be in your .bashrc
  ## jacket stuff!
  module load cuda
  export LD_LIBRARY_PATH=/clusterfs/cortex/software/jacket/engine/lib64:$LD_LIBRARY_PATH

Latest revision as of 17:52, 12 October 2016

Software

Matlab

In order to use matlab, you have to load the matlab environment:

 module load matlab/R2013a

Once the matlab environment is loaded, you can start a matlab session by running

 matlab -nodesktop

An example SLURM script for running matlab code is

 #!/bin/bash -l
 #SBATCH -p cortex
 #SBATCH --time=03:30:00
 #SBATCH --mem-per-cpu=2G
 module load matlab/R2013a
 matlab -nodesktop -r "scriptname. $variable1 $variable2"

The above script takes a matlab job with scriptname = scriptname and accepts two variables $variable1 and $variable2

If you would like to see who is using matlab licenses, enter

 lmstat

Python

Anaconda Python Distribution

The Anaconda Python 2.7 or 3.4 Distributions can be loaded through

 module load python/anaconda2

or

 module load python/anaconda3

respectively. This distribution has NumPy and SciPy built against the Intel MKL BLAS library (multicore BLAS). You will need to get an academic license from Continuum and copy it to the cluster.

On the cluster

 cd
 mkdir .continuum

On the machine where you downloaded the license file

 scp file_name username@hpc.brc.berkeley.edu:/global/home/users/username/.continuum/.

Local Install of Anaconda Python Distribution

If you want to manage your own python distribution the Anaconda Python is a very good distribution. To get it, go the the Continuum downloads page and select the linux distribution (penguin). Copy the download link address, and then in a terminal on the cluster run:

 wget paste_link_here

This should download a .sh file that can be run with

 bash Anaconda-version_info.sh

CUDA

CUDA is a library to use the graphics processing units (GPU) on the graphics card for general-purpose computing. The

 #SBATCH--constraint={cortex_k40, cortex_fermi} 
 #SBATCH --gres=gpu:1

options must be used in order to schedule a node with a GPU.

In order to use CUDA, you have to load the CUDA environment:

 module load cuda

Using Theano

By default, Theano expects the default compiler to be gcc, so you'll need to unload the intel compiler.

 module unload intel

Theano caches certain compiled functions and these will sometimes cause errors when Theano or CUDA gets updated. If you are experiencing problems with Theano, you can try clearing the cache with

 theano-cache clear

and if you still have problems you can delete the .theano folder from your home directory.

Using the GPU

You must request a GPU node. The Anaconda Python distribution comes with a version of Theano that should work. If you need new Theano features, the development version of Theano can be obtained from the github repository, installed locally, and added to your PYTHONPATH if you are using the preinstalled Python verions. If you have a local python install you can install theano with

 python setup.py develop

from the repository folder. Theano must be configured to use the GPU. General information can be found in the Theano documentation, but a working (June 2015) version is to create a .theanorc file in your HOME directory with the contents:

 [global]
 device = gpu
 floatX = float32
 force_device=True
 
 [nvcc]
 fastmath = True

Using the CPU

Theano can also run on the CPU. Any of the CPU nodes will work. You will want to have Theano use the MKL BLAS library that comes with Anaconda and so your .theanorc might look like

 [global]
 device = cpu
 floatX = float32
 ldflags = -lmkl_rt