Here, you will find some useful notes about working on the DKRZ (Deutsches Klimarechenzentrum) computer "Mistral" and running ECHAM5. It is assumed that you are comfortable working with terminal emulators and have knowledge of basic shell commands. These notes are meant to ease you into using ECHAM on Mistral and should be regarded as supplemental material to the existing official documentations for the climate model and the DKRZ facilities.
For questions: This email address is being protected from spambots. You need JavaScript enabled to view it.
1. What you need
A username, accountname and password will be given to you by the DKRZ. Also, you need a version of ECHAM - this should include the actual FORTRAN codes as well as controlling shell scripts, include files, etc. Furthermore, you need the .nc input files. These exist for different resolutions (T63, T85, etc.).
2. Login
You login to the DKRZ computers using the command:
ssh [username]@mistral.dkrz.de
The username starts with a 'b', followed by a 6 digit number. Note that the nodes may be used for compiling and submitting jobs, but not for interactive job processing or data storage!
3. SCP files into home directory
Secure-copy the model into your home directory on Mistral (your location after login)
/pf/b/[username]/
Only copy the input files of desired resolution. They are quite large and you should be economical with the storage space in your home directory.
4. Set the right environment
Useful aliases
To avoid the loss of important shell scripts or files by accidental or careless removal, you might want to customise your remove commands like this:
alias del= 'mv \!* /tmp'
del [FILENAME]
'rm' [FILENAME] will permanently delete the file.
These should be defined in the .cshrc or .bashrc files (depending on which shell you are using) in your home folder.
5. Useful commands
It is assumed that the user is familiar with the basic commands, so only a handfull of useful options and commands that you should know when working on Mistral are listed here. The most important is:
scancel [job ID]
This command will cancel a job. When you notice that a model is not running correctly (anymore), you can stop it using this command. This will prevent it from unneccessarily using up node hours and wasting storage space with wrong output. It is recommended you check up on your model on a regular basis using:
squeue -u [username]
This command lets you submit a job in form of a job shellscript:
sbatch [name of shell script]
All the commands above are part of the SLURM (Simple Linux Utility for Resource Management) workload manager currently in use on Mistral. In the jobscript/control script you submit, the model executable should be run with srun.
To quickly check on your output results (to make sure nothing went wrong), you can view the contents of netCDF output directly using the ncview application by typing:
ncview [filename.nc]
6. Compilation
This section describes how to configure and compile ECHAM. If you have a compiled version of the model (compiled for use on Mistral!), you can skip this step.
6.1 Configuration
Configuration files for different operating systems and compilers can be found in the config directory. To find out about your system, run config.guess like this:
sh config.guess
This will allow you to figure out, which of the config files you need to edit. Files starting with mh- are machine specific files. Here, you need to make sure, paths are set correctly. For example, the NETCDFROOT path variable needs to be given the correct value:
NETCDFROOT = /location/of/your/netcdf/root/folder
Then, run the file 'configuration' (which should be found in your ECHAM root folder):
sh configuration
When compiling with the intel fortran compiler, you need to set your locale in the .bashrc
6.2 Compilation
After successful configuration, run gmake to compile ECHAM - simply type 'make' in the folder containing your Makefile.
7. Running ECHAM
To run ECHAM, you send a run-script to the scheduler as a "job". In the run-script you define all paths to in- and output directories and set all options for your simulation. Your script needs to write the values for the option variables in the "ECHAM namelists". These will be passed on to the model once it starts. You also need to write the scheduler options in your run-script, i.e. what resources you wish to use for your simulation, what happens when errors occur, etc. The format of these options depends on the scheduler used (in the case of Mistral it is SLURM). You are free to write the run-script itself as a csh-, bash- or ksh script.
Before you submit the script, make sure of the following:
- You have the input files
- You defined the paths correctly to your input files, model executable, etc.
- There is plenty of space in your output directory - expect 0.1-1.5 TB of output, depending on your output interval, number of model years and levels
- You know how to send a job cancelling command to the scheduler (see commands section)
A simple textfile "rerun.txt" should be placed in the same folder as the jobscript and contain three values in one line:
.false. [start year] [start month]
where the 1st boolean value indicates whether it is the start of a simulation (consisting of multiple submitted "runs") or not. When it is set to false, the script will create a directory structure for your simulation and not look for any rerun files (i.e. previous output used as starting conditions). The job should submit itself after a set number of years/months are finished (make sure it is set in the jobscript so that it can be completed within the 8h wallclock limit on Mistral). The rerun.txt file will be updated automatically.
The file below is a heavily commented example script for running an ECHAM5-wiso Miocene simulation on Mistral.
8. Transferring model output to tape archives
Your simulation generated output and it looks reasonable? Wonderful! If you want to clear the space in your output directory for another simulation and store the old output in the DKRZ HPSS filesystem, follow the instructions below.
8.1 Organising your data
One large file is preferable over many small files. You might want to organise your individual output files into a tar archive for that reason by typing:
tar - cvf [filename.tar] [directory you wish to tar]
The options c,v and f mean:
c - create new archive
v - be verbose (displays which file is currently added to the archive) - useful to monitor the progress
f - specify filename
If you also want to compress your archive with gzip, add the z option to the command
tar - cvzf [filename.tar.gz] [directory you wish to tar and gzip]
The preferred size for transferred files is 10GB - 50GB and 500GB is the maximum. If you have several TB of output, you might want to separate them into several files, e.g. experiment01_1950-1969.tar, experiment01_1970-1999.tar.
8.2 Data transfer
First, go to the location of your tar or tar.gz file on Mistral. Then, type pftp to gain access to the tape archives.
It will ask you for your user password. Once you have entered those, you should be logged in and find yourself in the directory /hpss/arch. You can check your location using the pwd command. Now, go to your HPSS project directory /hpss/arch/[your account/project number]. Feel free to create subdirectories for your experiments. To transfer a file from your current Mistral folder to your current HPSS folder, simply type:
put filename.tar
To transfer a file from the HPSS archives to your Mistral folder, use the command get instead of put.
To log out and return to Mistral, type quit.
Below is an example of how to transfer filename.tar.gz from the Mistral directory /work/account_no/experiment_01 to hpss/arch/account_no
username@blizzard1$ cd /work/account_no/experiment_01 # go to location of yusername
username@blizzard1$ ls # check if file exists
filename.tar.gz
username@blizzard1$ pftp
[...]
Name (tape:username): # press enter if displayed username is correct
Password: # enter your user password
ftp> cd account_no # got to project directory
ftp> put filename.tar.gz # transfer command
[...]
ftp> ls # check if file is on archives
[...]
filename.tar.gz
ftp> quit # quit and return to Blizzard
username@blizzard1$
For more information, visit:
http://www.dkrz.de/Nutzerportal-en/doku/hpss