GitLab

Hello,

I took the liberty to do several changes in order to accommodate the new icoads.utils package that complements the scripts in rscripts. All the changes have been motivated to reduce repetition of code in functions like: get_isolated.R. And to remove all source() in the code routines of rscripts.

The current code has been tested for the entire 1970. But requires in depth test to see if we end up with the same output of duplicates/results. I can't particular test this since now there are less variables in the code and if I compare two output files to see if they are equal I won't get a positive result since my previous output has all variables accounted for the first version of the code. The files wont get a match. It will be good idea to design a test for a particular year that we know the output off, so we can implement a CI/CD setting on this project as well.

Along the way I also fixed the following issues:

1. #14 (closed), see issue for explanation.

2. #13 (closed) : get_isolated.R has been removed according to my suggestions on the issue. As a result of this clean_data.R is a bit longer.... but we don't have repetitive code in icoads.utils. This has been tested for the entire 1970 in JASMIN.

3. #12 (closed): There is a function to run a profiler in icoads.utils/profile_code.R and a place to store the output in config.yml This needs @dyb revision to see if he wants to add something else. I will be adding an issue in the icoads.utils repository to remind him.

4. #11 (closed) All non used code is not there anymore and files like read_imma1.R and remove_output_files.R have been moved to the /rscripts/ folder as agreed on this issue discussion.

5. #10 (closed) print_id_match_info.R is now a function in icoads.utils/print_id_match_info.R

Instructions on how to install the new toolbox can be found in the updated documentation Wiki. For gathering SLURM run statistics I have added a new section in the wiki at JASMIN tips see here

Cheers