Skip to content

GitLab

Closed
Created by brivas@brecinosrivasMaintainer

need to reduce memory by dropping unused variables

Hello,

I have now identified a list of variables from the .Rda data frames that are not called by "var_name" or by $var_name through out the entire code (/rscripts and /rutils).

See the following list:

not_used <- c("a", "af", "anc", "aqa", "aqz", "attc",
               "b1", "b10", "bnc", "bsi", 
               "c2", "ch", "cl", "cm", "cnc", "cor", 
               "di", "dnc", "dpti", "dqa", "dqz", "dupc",
               "enc", "eot", 
               "fnc", 
               "gnc", 
               "h", "hi", "hoa", 
               "im", "it", 
               "kov", 
               "li", 
               "nd", 
               "op", "opm", "os", "otv", "otz", 
               "pb", "pf", "pnc", "ppp", "pqa", "pqz", "puid", 
               "qc1", "qc10", "qc11", "qc12", "qc13", "qc14", 
               "qc15", "qc16", "qc17", "qc18", "qc19", "qc2", 
               "qc20", "qc22", "qc23", "qc24", "qc25", "qc27", 
               "qc28", "qc29", "qc3", "qc4", "qc5", "qc6", "qc7", 
               "qc8", "qc9", "qce", "qcz", 
               "rf", "rhi", "rn1", "rn2", "rn3", "rsa", 
               "sd", "sf", "sh", "snc", "sp", "sqa", "sqz",
               "tc", "ti", "tnc", "tob", "toh", "tot", 
               "uf", "uqa", "uqz", 
               "vf", "vi", "vqa", "vqz", 
               "w2", "wbti", "wd", "wi", "wnc", "wp", 
               "xnc", 
               "ync", 
               "znc")

I have double checked this with the little bash code below. It search's for these var names wrap by either "" or $.

The rest of variables that do get use, will be documented in the Wiki API.

But if we drop these at the first stage of the workflow the amount of memory will be significantly reduce. I still need to do more test on this (More comments to come below this one).

Text files used in the code below:

var_not_used.txt

var_not_used_dollar.txt

#!/bin/bash

Elogs=/home/bea/R_projects/output.txt
Errors=/home/bea/R_projects/var_not_used.txt
for i in `cat $Errors`; do
    #Get Error Logs
    grep "$i" /home/bea/R_projects/ICOADS/rscripts/* >> $Elogs
done

Elogs=/home/bea/R_projects/output_rutils.txt
Errors=/home/bea/R_projects/var_not_used.txt
for i in `cat $Errors`; do
    #Get Error Logs
    grep "$i" /home/bea/R_projects/ICOADS/rutils/* >> $Elogs
done

Elogs=/home/bea/R_projects/output_dollar.txt
Errors=/home/bea/R_projects/var_not_used_dollar.txt
for i in `cat $Errors`; do
    #Get Error Logs
    grep "$i" /home/bea/R_projects/ICOADS/rscripts/* >> $Elogs
done

Elogs=/home/bea/R_projects/output_rutils_dollar.txt
Errors=/home/bea/R_projects/var_not_used_dollar.txt
for i in `cat $Errors`; do
    #Get Error Logs
    grep "$i" /home/bea/R_projects/ICOADS/rutils/* >> $Elogs
done

Note: this script is semi-automatic it still needs manual checkout for the output sometimes $shipname is confused with $sh for example. But is faster than a fully manual search.

Edited by brivas