How-to-run-on-ARCHER2.md

+[[_TOC_]]
+
+# How to run on Archer2
+
+Here is a tutorial on how to run `NEMO4.2-MEDUSA` on `ARCHER2`.
+
+## Environment
+
+Luckily, Andrew Coward Already wrote a complete note on that subject [here](https://docs.archer2.ac.uk/research-software/nemo/nemo/#setting-up-the-correct-environment)
+
+In short :
+
+load the right modules, the way it's done here : `/work/n01/shared/nemo/setup_gnu` (full cabinet). Currently (January 2022) the modules are :
+```sh
+module swap PrgEnv-cray/8.0.0 PrgEnv-gnu/8.1.0
+module swap craype-network-ofi craype-network-ucx
+module swap cray-mpich cray-mpich-ucx
+module load cray-hdf5-parallel/1.12.0.7
+module load cray-netcdf-hdf5parallel/4.7.4.7
+module load libfabric
+```
+
+## Get MEDUSA
+
+* Go to your work-directory :
+```sh
+cd /work/n01/n01/${userID}/
+```
+
+To get `MEDUSA`, see the full description with explanation on [the main page of the git project](https://git.noc.ac.uk/acc/medusa_4.2.x), but roughly :   
+
+```sh
+git clone --recurse-submodules git@git.noc.ac.uk:acc/medusa_4.2.x.git
+### or
+git clone --recurse-submodules https://git.noc.ac.uk/acc/medusa_4.2.x.git
+###  put MEDUSA at the right places :
+cd medusa_4.2.x
+./add_msa_to_nemo
+```
+
+## Compile MEDUSA
+
+ * Compile :  on Archer2 (full cabinet) \
+    * move to the nemo directory :
+      ```sh
+      cd nemo
+      ```
+### Before compiling
+ First some corrections listed in Andrews documentations [here](https://docs.archer2.ac.uk/research-software/nemo/nemo/#setting-up-the-correct-environment) :
+* Update `ext/FCM/lib/Fcm/Config.pm` by changing `FC_MODSEARCH => '',` into `FC_MODSEARCH => '-J',`
+* Add this arch file into your `arch` directory :
+  ```sh 
+  cp /work/n01/shared/nemo/ARCH/arch-X86_ARCHER2-Gnu.fcm arch/
+  ```
+### Compiling
+* Now you can compile as follows (With `MY_NEW_MEDUSA` your MEDUSA config if you want/need to create one) : 
+  ```sh 
+  ./makenemo -m X86_ARCHER2-Gnu -r ORCA2_MEDUSA -n ${MY_NEW_MEDUSA} -j 16 
+  ```
+* Subsequently when you compile you can instead run :
+  ```sh
+  ./makenemo -m X86_ARCHER2-Gnu -r ${MY_NEW_MEDUSA} -j 16
+  ```
+
+## Get your running directory ready
+
+* Once compiled, prepare your running directory, with `${running_dir}` the name of your running directory : 
+  ```sh 
+  mkdir cfgs/${MY_NEW_MEDUSA}/${running_dir}
+  cp /work/n01/n01/julpal/SETTING_RUNNING_DIR/eORCA1_ARCHER2/COPY_ME__NEMOMED-4.2_ORCA1_UKESM1_405.sh cfgs/ORCA2_MEDUSA/${running_dir}/.
+  ```
+* Then go to your running directory and run this shell script. It should copy and link all necessary files in your running directory.
+  ```sh
+  cd cfgs/${MY_NEW_MEDUSA}/${running_dir}
+  ./COPY_ME__NEMOMED-4.2_ORCA1_UKESM1_405.sh
+  ```
+
+## Run your model
+
+* The submission script, `hetjob_ORCA1`, will likely need some edits. The only parts that should need changing in `hetjob_ORCA1` are in the head of the file : 
+
+```sh
+export OMP_NUM_THREADS=1
+export EXP_NAM=RIVER2
+export RUN_LEN_YR=1
+export TOT_RUN_LEN_YR=10
+export TS=2700
+export MEAN_DIR=MEANS_RIV1
+export MEAN_ROOT=$(grep "cn_exp" namelist_cfg | awk -F"'" '{print $2}')
+export RUN_SCRIPT=hetjob_ORCA1
+```
+
+With `RUN_LEN_YR` the length (in year) of the submitted section of the run and `TOT_RUN_LEN_YR` the total run length in year. TS it the model time-step in seconds. Also, Make sure that `cn_exp` value in `namelist_cfg` is between `'` and not `"`, the running script would not get the experiment name otherwise, and the updating script would not work properly.
+
+* The submission script will automatically update the restart and time-step variables in the namelist to start and restart the run.
+* Finally, the namelists should be edited as needed for the experiment. Remember that there are pairs of files for each component : 
+  1. `namelist_ref`, the reference namelist for the default configuration 
+  2. `namelist_cfg`, the configuration namelist for the experiment configuration 
+ The second should be used to make changes that override the default settings and parameter values in the first.
+* To submit the job, use `sbatch hetjob_ORCA1`. 
+* To monitor the job, use `squeue -u ${userid}`. (If you can't remember who you are, type `whoami` from the prompt!)
+* You can restart (not starting from scratch, but from restart files), by simply adding an argument, like `sbatch hetjob_ORCA1 2`.
+  * To restart the run from scratch, don't forget to clean the `RESTART` and `PREV` directories before submitting the job. No need to reinitialize the namelists, it is done by the submission script.
+* Don't forget, the namelists here are adapted to run NEMO4.2-MEDUSA on eORCA1, forced with UKESM atmosphere and rivers ; starting on the 1850/01/01. You can change the namelists if you need to change anything to that.
\ No newline at end of file