Here, you will find some notes about running ECHAM on BWHPC (ESD nodes). It is assumed that you are comfortable working with terminal emulators and have knowledge of basic shell commands.
For questions: This email address is being protected from spambots. You need JavaScript enabled to view it.
1. What you need
A username, accountname and password will be given to you by the ZDV upon request. Also, you need a version of ECHAM - this should include the actual FORTRAN codes as well as controlling shell scripts, include files, etc. Furthermore, you need the .nc input files. These exist for different resolutions (T63, T85, etc.). All this is archieved on the ESD server.
The latest code and Makefiles are in:
/esd/esd02/data/climate_models/echam/echam_code/echam5-wiso.v2.3_20210601
The input files for ECHAM can be found in:
/esd/esd02/data/climate_models/echam/echam_input
2. Login
You login to HPC using the command:
$ ssh -X -l [user name] glogin.hpc.uni-tuebingen.de
The X-forwarding option (-X) is optional and only needed if you wish to display visuals from the terminal.
3. SCP files into home directory
Secure-copy (scp) the entire model folder (e.g. "/esd/esd02/data/climate_models/echam/echam_code/echam5-wiso.v2.3_20210601") into your home directory on HPC. Create a separate folder for model input and model output. Then, scp the input files into the model input folder.
4. Set the right environment
Useful aliases
It is useful to customise your .bashrc file in your home directory to make work on Mistral faster and more comfortable. (If it doesn't exist, create one). To avoid the loss of important shell scripts or files by accidental or careless removal, you might want to customise your remove commands like this:
alias del= 'mv \!* /tmp'
del [FILENAME]
'rm' [FILENAME] will permanently delete the file.
These should be defined in the .bashrc files (depending on which shell you are using) in your home folder.
Running the model
Before you run the model, you will have to change a few lines in the default run script (included in the model folder listed above). The default run script is located in [model folder]/run/test/. At the top ("user declaration" section) of the run script, change the paths to fit your own directory structure. The script is well commented, so this should not be a problem.
Note that the current version of the model is compiled with specific versions of the intel compiler, openmpi and netCDF. The former are available through loading the modules:
$ module load chains/ALL
$ module load compiler/intel/13.1
$ module load mpi/openmpi/1.10-intel-13.1
The latter (netCDF) is found in:
/opt/bwhpc/tu/geo/echam5/share
Since it is not available through modules, the lib path has to be manually added to the LD_LIBRARY_PATH variable. All of this is already implemented in the run script.
5. Useful commands
It is assumed that the user is familiar with the basic commands, so only a handfull of useful options and commands that you should know when working on BWHPC are listed here. The most important is:
$ qdel [job ID]
This command will cancel a job. When you notice that a model is not running correctly (anymore), you can stop it using this command. This will prevent it from unneccessarily using up node hours and wasting storage space with wrong output. It is recommended you check up on your model on a regular basis using:
$ qstat -u [job ID]
This command lets you submit a job in form of a job shellscript:
$ qsub [name of shell script]
All the commands above are part of the TORQUE scheduler currently in use. In the jobscript/control script you submit, scheduler options are listed at the top.
To quickly check on your output results (to make sure nothing went wrong), you can view the contents of netCDF output directly using the ncview application by typing:
$ ncview [filename.nc]
For a non-visual look into the model output, you can use the ncdump command. It's part of the netCDF installation, so loading any of the netCDF modules will add the respective version of ncdump executable to your bin path for the session you are currently in. To check the header of an .nc file, use:
$ ncdump -h [filename.nc]
To check the actual values for a specific variable like 2m air temperature ("temp2" in ECHAM output), use:
$ ncdump -v [variable name] [filename.nc]
6. Compilation
If you wish to recompile ECHAM with the same configuration, follow the steps below.
1. Navigate to the model folder
2. Rename (or copy) Makefile_vSM20210601 to Makefile. Makefile_vSM20210601 is a copy of the last working makefile. The command "make" for compiling will only consider a file named Makefile.
$ cp Makefile_vSM20210601 Makefile
3. Replace the prefix value in the Makefile with your model directory, so it compiles the model into your own folder.
4. Load required modules (from glogin):
$ module load chains/ALL
$ module load compiler/intel/13.1
$ module load mpi/openmpi/1.10-intel-13.1
5. (You don't load any netCDF libs, cause they are explicitly included in the Makefile)
6. Compile:
$ make clean
$ make