Start running GEMCLIM

To run GEM one needs appropriate executables as well as a set of scripts and configuration files.
You will find some examples at the end of this page.

Content

1)-3)	Set up the environment
4)	The executables
5)	The scripts
6)	The configuration files
7)	Launching the model
8)	What happens while the model is running
10)	What to do with the model output

Examples:
1) Create your own executables
2) Set up and start a model run
3) Modify some decks

1) Create the directories 'gemclim' and 'listings'

You will have to create a few directories before you can start running the model.
You only need to do this step once!

You need to create the directories 'gemclim' and 'listings' directly in your ${HOME}. In each of these two directories you need to create soft links with the names of the machines you will use to run the model. These links should point to directories on a disk on which you have enough disc space.

First create the directories 'gemclim' and 'listings' in your ${HOME}:

mkdir
~/gemclim


mkdir ~/listings

Now create a directory under which the model will be running. You only need to create a directory for the machine, 'mach', on which you will run the model.

Ouranos/UQAM:

At Ouranos/UQAM you will be running on 'mach=headnode':mkdir -p /local/fiber1/${USER}/gemclim/EXECDIR; ln -s /local/fiber1/${USER}/gemclim/EXECDIR ~/gemclim/headnode

Dorval:

In Dorval you could run on 'maia', 'naos', a Linux machine or cluster, pollux... For now let's use 'maia':mach=maiaGo on 'maia'! The disk on which you have a lot of temporary disk space will probably be either be '/fs/dev/mrb/02' or '/fs/dev/mrb/03'. Find the disk you have a directory on, i.e.: /fs/dev/mrb/03/armn/${USER}
mkdir -p /fs/dev/mrb/03/armn/${USER}/gemclim/EXECDIR; ln -s /fs/dev/mrb/03/armn/${USER}/gemclim/EXECDIR ~/gemclim/${mach}

Now create directories for the model, script and transfer listings during run time.
You need to create subdirectories for all the machines you will be using to run the entry, the model and the post processing and for the machine on which you will finally store the model output.

Ouranos/UQAM:

At Ouranos/UQAM you will run everything on the 'headnode' and also store the output there:mkdir -p /local/sata1/${USER}/gemclim/listings; ln -s /local/sata1/${USER}/gemclim/listings ~/listings/headnode

Dorval:

In Dorval you could, in LAM mode, run the entry on a Linux machine, i.e. 'copernic', run the model and post processing on an AIX cluster, i.e. 'maia', and then store the output on another Linux machine, i.e. 'kepler'.
'pollux' is the machine handling the transfer of the model in and output to/from maia from/to the machine on which you process or store your data. Therefore you also need to create a listings directory for 'pollux'.

This means you have to create directories for:
   copernic, maia, kepler and pollux!
Go on the, in this case, four machines into the directory '~/listings' and create links like i.e.this one:
   cd ~/listings
   ln -s /fs/mrb/03/armn/armnkwi/listings maia

At the end of each job all listings get zipped and transfered to arch_mach in your archdir as specified in your 'configexp.dot.cfg'.

2) Export RDIAG and EDITFST in your '.profile_usr'

Dorval:

When running in Dorval export these two variables in you '.profile_usr':
export RDIAG=/usr/local/env/armnlib/scripts/latest.r.diag
export EDITFST=editfst_6.02

3) Set the model environment

Every time you start the model or any of the scripts you have to set the model environment:

. r.sm.dot gemclim 3.3.0

Note that this adds two (resp. three or four if you have scripts or bin directories under ~/modeles/GEMCLIM/v_3.3.0) directories to your environment variable ${PATH} and also sets up a new environment variable 'gemclim' which points to the directory under which you find the official model environment (source decks, scripts, etc.).

4) The executables

Usually GEM(CLIM) needs two absolutes: one for the entry ('...ntr...') and one for the model ('...dm...').

Their names are:
'maingemclimntr_${ARCH}_modelversion.Abs' for the entry,
'maingemclimdm_${ARCH}_modelversion.Abs' for the model.

To find out about how to create the absolutes you can either have a look at the general introduction to GEMDM web page or follow the little example below.

In climate mode the absolutes always need to be on the machine from which the model gets launched.

Dorval:

If the model is run in LAM mode the entry is usually ran on another machine with another architecture than the model. In this case there need to be an absolute for the entry with the architecture of the entry machine and an absolute for entry and model with the architecture of the machine were the model will be running. All three absolutes have to be on the machine from which the model will be launched.

5) The scripts

Usually one can use the scripts from the environment. This is always a closed version.

But you might also want to ask either Bernard Dugas (Bernard.Dugas@ec.gc.ca) or Katja Winger (Katja.Winger@ec.gc.ca) if there is a newer updated version of the scripts.

There are two types of scripts:

version-dependent, that change with model versions: ${gemclim}/scripts/* and
version-invariant, that do not change: ${ARMNLIB}/scripts/Climat_*

In case you want to modify and use some of these scripts you must put your (modified) copies of any version-dependent scripts into the '~/modeles/GEMCLIM/v_3.3.0/scripts' directory and any version-invariant scripts into the '~/ovbin' directory.

After having created the directory '~/modeles/GEMCLIM/v_3.3.0/scripts' you need to set the model environment again so that this directory gets added to your 'PATH'!

You find more detailed explanations about the scripts here.

6) The configuration files

GEMCLIM needs three configuration files: 'configexp.dot.cfg', 'gemclim_settings.nml' and 'outcfg.out'

configexp.dot.cfg: file to control where to send the batch run, how much cpu time allowed, which machine to run the job on, which model version to use, which machine to do the post processing on, where to place the stdout (listings) and the model output, etc.

Click on the following links to get more information on the general variables and on the special climate variables.

gemclim_settings.nml: This file contains all the namelists to control the model grid, the different schemes and parameters for the entry program (gemclimntr) and the main program (gemclimdm).

Click on the following links to get more information on the general namelist variables and on the physics namelist variables.

outcfg.out: file to control the RPN standard file output from the run such as frequency of output, which fields, at what levels, etc. If you have set 'strip_phy=1;' in your 'configexp.dot.cfg' this file will get created automatically otherwise you have to provide it for the first job. In any case the first and last time step for the output ('steps=#,step,<start,end,inc>';) will get updated automatically for following jobs.

Click on the following link to get more information on the outcfg.out file variables.

7) Launching the model

If you run the model in LAM mode you need to start it from the machine on which you want to do the entry otherwise go on the machine on which you want to run the model.
Go in the directory where you have your config files 'configexp.dot.cfg' and 'gemclim_settings.nml'.
Set the model environment:
. r.sm.dot gemclim 3.3.0
Launch the model:
Um_launch

Or have a look at the example below.

8) What happens while the model is running

You can also have a look at the flowchart showing all major jobs, which listings they produce and where their output goes.

During its execution, 'Um_launch' messages will appear on your screen telling you how the config files got updated, that the executables got copied in your execution directory on $mach and finally that the entry was submitted.
At that point you should see the entry running (or being queued). The entry runs on only 1 cpu and takes only a few minutes if the model is not run in LAM mode; but it can take up to several minutes per month in LAM mode.
After the entry, the model will be started on $mach usually running on multiple tiles/cpu's.
The model output will be written to the directory '${outrep}/current_last_step'. Each tile will write to its own sub-directory, 00-00, 00-01, ... The domain geometry (actual tile number and configuration) is specified in the in the namelist 'ptopo' of 'gemclim_settings.nml'.

        There are three (next version four) types of output files:

        dp... : dynamics on pressure level
        dm... : dynamics on model levels
        pm... : physics on model levels
        pp... : physics on pressure levels (does not exist yet but will in version 4.0.0 !!!)

The post processing always gets started at the end of a job/month. This means that the model output will get moved from the output-directory 'current_last_step' into 'last_step_xxx' (actually the directory just gets renamed).
Dorval:

The post processing can also get started while the model is still running. This can be useful when there is not enough disk space since this early post processing reduces the output when ${strip_phy}=1.
The post processing is then started every 'cleanup' days (this number is specified by the variable 'climat_cleanup' in your 'configexp.dot.cfg').

Dorval:

When the post processing is done on another machine (${mach} not equal ${xfer}) a job (..._FT_...) will be send to pollux which moves the files from ${mach} to ${xfer}.

The post processing job (..._PP_...) gets launched on ${xfer}. This is a little 1 cpu job which determines what kind of post processing is to be done, i.e. saving of analysis or pilot files, the usual post processing, diagnostics. If no file transfer, saving of analysis or pilot files needs to be done, this job will be run as part of the model job.
Then the reassemble job will start using up to 8 cpu's as specified by ${climat_pp_cpus}. This will first put all tiles together (using 'd2z') and put the output into a sub-directory in your post processing working directory (specified by the variable 'xfer' in your 'configexp.dot.cfg') called '${exp}/WORKING_ON_last_step_xxx_files'.

Short post processing: 'strip_phy=0;'

Your output files will get moved in the directory ${xfer}/${exp}.

Extended post processing: ' strip_phy=1;'

Your output will get re-organized: The 3-D physic fields will be interpolated from model to pressure levels, a few new variables will get calculated and in the end there will be only two types of files:
md-files : fields on model levels (now only 2-D fields)
pr-files : fields on pressure levels
all of which will be put one directory up into ${xfer}/${exp}.

Diagnostics: 'diagnos=1;'

The diagnostics will start in ${xfer}, again on up to 8 cpu's. But this job is only launched after the model has run for the requested (interval) number of months, as specified in your 'configexp.dot.cfg' and after the last post processing job has finished. The diagnostic job creates the monthly means, time series, variances and several covariances for each individual months in this interval.
After this step, the diagnostic files, as well as the md- and pr-files, will be archived and transfered to 'arch_mach' into the directory 'archdir', again as specified in your 'configexp.dot.cfg'.

At the end of this, all listings and jobs/scripts (created during the run) from this job are collected and archived on 'arch_mach' in 'archdir'.
At the end of each model job, the restart files is archived and transfered to 'arch_mach' into 'archdir'. The name of the archive will be '${exp}step${laststep}.ca.gz'. If the run continues, the "old" restart files will get erased from ${mach}. If the run is finished, they will stay on ${mach}.

Note the model does not depend on the post processing and diagnostics. So even if one of these jobs aborts, the model will continue running and at the end of a job (in case the run continues) submit the next job automatically. You will then get another directory in your ${HOME}/gem/${mach} for the new job.

10) What to do with the model output

There are several ways to look at the model output, which is in the RPN standard file format, and to modify it.
Have a look at this web page for some examples.

Examples

We assume you did already point 1) and 2) of the section above.

1) Create your own executables:

Go on the machine for which you want to create the absolutes into the directory in which you would have your own decks (preferably under your ${HOME}). mkdir -p ~/gemclim/v_3.3.0/Abs/bcmk; cd ~/gemclim/v_3.3.0/Abs/bcmk

Set up an active experiment:

Ouranos/UQAM:

ouv_exp base -RCSPATH "/home/armnlib/env/armnlib/modeles/GEMCLIM/v_3.3.0/RCS_DYN /home/armnlib/env/armnlib/modeles/GEMCLIM/v_3.3.0/RCS_PHY"

Dorval:

ouv_exp v_3.3.0 -RCSPATH "/usr/local/env/armnlib/modeles/GEMCLIM/v_3.3.0/RCS_DYN /usr/local/env/armnlib/modeles/GEMCLIM/v_3.3.0/RCS_PHY"

This command will create and RCS directory and the file '.exper_cour' in which the paths of the dynamic and physics routines.

To avoid limited disc space problems you may want to create some soft links.
malib${ARCH} is the directory to hold future object files created from recompiling new or modified routines (by you) from the GEMCLIM library. It is also recommended to create soft links for the GEM* binaries (main*.Abs) as they are also written in the working directory.

Ouranos/UQAM:

absdir=/local/sata1/${USER}/gemclim/Abs/v_3.3.0/bcmk

Dorval:

It is up to you where you want to put your absolutes and object files. On 'maia' you could put them i.e. here:
absdir=/fs/dev/mrb/03/armn/${USER}/ABS/v_3.3.0/bcmk

mkdir
-p ${absdir}/malib${ARCH}; ln -s ${absdir}/malib${ARCH}

ln -s ${absdir}/maingemclimdm_${ARCH}_3.3.0.Abs ln -s ${absdir}/maingemclimntr_${ARCH}_3.3.0.Abs

Set the model environment: . r.sm.dot gemclim 3.3.0
Build a makefile: r.make_exp This command creates the make file 'Makefile', the file 'arbre_de_dependance' containing a list of all the functions/decks with their included comdecks as well as the file 'make_cdk' containing a list of all the comdecks with the names of the functions/decks they are included in.
Build GEMCLIM: make gemclim
If you wish to make modifications to the GEMCLIM source code, see the expanded description below.

2) Set up and start a model run

Build and enter a configuration directory: mkdir -p ~/gemclim/v_3.3.0/Configs/bcmk; cd ~/gemclim/v_3.3.0/Configs/bcmk
Copy the benchmark configuration scripts into the current directory:

Ouranos/UQAM:

cp /local/sata1/data/configs/gemclim_3.3.0-lam/* .

Dorval:

At the moment there is no example - sorry!
Replace the model configuration user name with your own user name: perl -pi -e 's/FOO/$ENV{USER}/g' configexp.dot.cfgDorval:

Make sure all paths are set correctly!
Launch the model run: Um_launch

3) Modify some decks:

You will find all the RCS model decks of the dynamics in ${gemclim}/RCS_DYN and of the physics in ${gemclim}/RCS_PHY.

Return to your experiment directory: cd ~/gemclim/v_3.3.0/Abs/bcmk
In case you did not do that already you need to set the model environment with: . r.sm.dot gemclim 3.3.0This will set the environment variable $gemclim to the model path.
Get an RCS repository listing for the dynamics segment of the model: ls ${gemclim}/RCS_DYN
Get an RCS repository listing for the physics segment of the model: ls ${gemclim}/RCS_PHY
Extract the function or comdeck you want, i.e. the CCCma radiation driver source from the physics RCS repository: omd_exp cccmarad.ftn

Use your favorite text editor to have a look at (and modify if you'd like) the source for cccmarad.ftn.
In case you include a new comdeck or change function calls you need to recreate the make file: r.make_exp Compile the modified (or unmodified) subprogram (create *.f and *.o): make cccmarad.o If you modify a comdeck or want to compile all the routines you have in the directory you should instead use: make objloc
(Optional) Directory cleanup:
After having compiled a function or comdeck you will also have all their included decks and '*.f' in your directory. They all will have the permission '-r--r--r--'. To erase them you can use: 'make clean'
This will erase all decks with the permission '-r--r--r--'. So be careful!!! If you write-protect a deck to not accidentally erase it 'make clean' WILL ERASE it!
make clean
Recompile the whole model to use your newly-compiled subprogram: make gemclim To only recompile the model you can also use: make gemclimdm To only recompile the entry you can also use: make gemclimntr
Return to your benchmark configuration directory: cd ../../Configs/bcmk
And launch you model:Um_launch

Author: Katja Winger and Ron McTaggart-Cowan
Last update: June 2008