How to run on ARCHER2

How to run on Archer2

How to run on Archer2

Here is a tutorial on how to run NEMO4.2-MEDUSA on ARCHER2.

Environment

Luckily, Andrew Coward Already wrote a complete note on that subject here

In short :

load the right modules, the way it's done here : /work/n01/shared/nemo/setup_new_gnu (full cabinet). Currently (January 2024) the modules are :

module swap PrgEnv-cray/8.3.3 PrgEnv-gnu/8.3.3
module load cray-mpich/8.1.23
module load cray-hdf5-parallel/1.12.2.1
module load cray-netcdf-hdf5parallel/4.9.0.1

Get MEDUSA

Go to your work-directory :

cd /work/n01/n01/${userID}/

To get MEDUSA, see the full description with explanation on the main page of the git project, but roughly :

git clone --recurse-submodules git@git.noc.ac.uk:acc/medusa_4.2.x.git
### or
git clone --recurse-submodules https://git.noc.ac.uk/acc/medusa_4.2.x.git
###  put MEDUSA at the right places :
cd medusa_4.2.x
./add_msa_to_nemo

Compile MEDUSA

Compile : on Archer2 (full cabinet) \
- move to the nemo directory :
```
cd nemo
```

Before compiling

First some corrections listed in Andrews documentations here :

Update ext/FCM/lib/Fcm/Config.pm by changing FC_MODSEARCH => '', into FC_MODSEARCH => '-J',

Add this arch file into your arch directory :

cp /work/n01/shared/nemo/ARCH/arch-X86_ARCHER2-Gnu_4.2.fcm arch/

Compiling

Now you can compile as follows (With MY_NEW_MEDUSA your MEDUSA config if you want/need to create one) :
```
./makenemo -m X86_ARCHER2-Gnu_4.2 -r ORCA2_MEDUSA -n ${MY_NEW_MEDUSA} -j 16 
```

Subsequently when you compile you can instead run :

./makenemo -m X86_ARCHER2-Gnu_4.2 -r ${MY_NEW_MEDUSA} -j 16

Get your running directory ready

Once compiled, prepare your running directory, with ${running_dir} the name of your running directory :

mkdir cfgs/${MY_NEW_MEDUSA}/${running_dir}
cp /work/n01/n01/julpal/SETTING_RUNNING_DIR/eORCA1_ARCHER2/COPY_ME__NEMOMED-4.2_ORCA1_UKESM1_405.sh cfgs/ORCA2_MEDUSA/${running_dir}/.

Then go to your running directory and run this shell script. It should copy and link all necessary files in your running directory.
```
cd cfgs/${MY_NEW_MEDUSA}/${running_dir}
./COPY_ME__NEMOMED-4.2_ORCA1_UKESM1_405.sh
```

Run your model

The submission script, hetjob_ORCA1, will likely need some edits. The only parts that should need changing in hetjob_ORCA1 are in the head of the file :

export OMP_NUM_THREADS=1
export EXP_NAM=RIVER2
export RUN_LEN_YR=1
export TOT_RUN_LEN_YR=10
export TS=2700
export MEAN_DIR=MEANS_RIV1
export MEAN_ROOT=$(grep "cn_exp" namelist_cfg | awk -F"'" '{print $2}')
export RUN_SCRIPT=hetjob_ORCA1

With RUN_LEN_YR the length (in year) of the submitted section of the run and TOT_RUN_LEN_YR the total run length in year. TS it the model time-step in seconds. Also, Make sure that cn_exp value in namelist_cfg is between ' and not ", the running script would not get the experiment name otherwise, and the updating script would not work properly.

The submission script will automatically update the restart and time-step variables in the namelist to start and restart the run.
Finally, the namelists should be edited as needed for the experiment. Remember that there are pairs of files for each component :
1. namelist_ref, the reference namelist for the default configuration
2. namelist_cfg, the configuration namelist for the experiment configuration The second should be used to make changes that override the default settings and parameter values in the first.
To submit the job, use sbatch hetjob_ORCA1.
To monitor the job, use squeue -u ${userid}. (If you can't remember who you are, type whoami from the prompt!)
You can restart (not starting from scratch, but from restart files), by simply adding an argument, like sbatch hetjob_ORCA1 2.
- To restart the run from scratch, don't forget to clean the RESTART and PREV directories before submitting the job. No need to reinitialize the namelists, it is done by the submission script.
Don't forget, the namelists here are adapted to run NEMO4.2-MEDUSA on eORCA1, forced with UKESM atmosphere and rivers ; starting on the 1850/01/01. You can change the namelists if you need to change anything to that.