| ... | ... | @@ -8,24 +8,24 @@ ____________ |
|
|
|
|
|
|
|
Script | Description
|
|
|
|
-------- |:------------
|
|
|
|
`split_by_type.R` | Separates records according to the different platform types. This is done following a [selection criteria](Workflow/data-selection).
|
|
|
|
`simple_dup.R` | Duplicate record identification among ship data and the rest of the different platform types for which the records show a matching `date` , `time` and position `(lat, lon)`.
|
|
|
|
`ship2plat.R` | Excludes non-ship data. Uses additional information from WMO Publication 47 to flag non-ship data. [Kent. *et al.,* (2007)](https://doi.org/10.1175/JTECH1949.1)
|
|
|
|
`process_ids.R` | Re-format of ship IDs that comes from different data sources (`SID`) to enable linking of data from the same ship (same ship name) across different `DCK`s. And to also enable linking to metadata information in WMO Publication No. 47.
|
|
|
|
`process_shipdata.R` | Corrects dates and times errors noticed in some `DCK`. Largely arise from confusion over historical definitions of the marine day and conversions between local time and UTC.
|
|
|
|
`new_get_pairs.R` | Duplicate record identification within the ship data. Pairs the reports as duplicate if they have associated ship `ID`s. The candidate pairs are then selected according to i) the number of matching elements (similar content of variables within a specific tolerance), ii) the `DCK`s, and iii) a comparison of the `ID`s.
|
|
|
|
`new_get_dups.R` | Counts the number of duplicated records and flags the best according to a [quality control criteria](Workflow/quality-control). Groups duplicated records by common `callsings`.
|
|
|
|
`new_merge_ids_year.R` | Links `ID`'s into classes. Ship tracks of the linked `ID`'s are then checked. Reports that fail the track check are flagged as the worst duplicate. Uses a shipping tracking alogrithm from [**imma**.](https://git.noc.ac.uk/brecinosrivas/icoads-r-hostace/-/wikis/How-to-install#install-dependencies-with-conda-all-platforms)
|
|
|
|
`clean_data.R` | ---.
|
|
|
|
`clean2track.R` | Forms ship tracks for linked `ID`s. --.
|
|
|
|
[`split_by_type.R`](https://git.noc.ac.uk/brecinosrivas/icoads-r-hostace/-/blob/master/rscripts/split_by_type.R) | Separates records according to the different platform types. This is done following a [selection criteria](Workflow/data-selection).
|
|
|
|
[`simple_dup.R`](https://git.noc.ac.uk/brecinosrivas/icoads-r-hostace/-/blob/master/rscripts/simple_dup.R) | Duplicate record identification among ship data and the rest of the different platform types for which the records show a matching `date` , `time` and position `(lat, lon)`.
|
|
|
|
[`ship2plat.R`](https://git.noc.ac.uk/brecinosrivas/icoads-r-hostace/-/blob/master/rscripts/ship2plat.R) | Excludes non-ship data. Uses additional information from WMO Publication 47 to flag non-ship data. [Kent. *et al.,* (2007)](https://doi.org/10.1175/JTECH1949.1)
|
|
|
|
[`process_ids.R`](https://git.noc.ac.uk/brecinosrivas/icoads-r-hostace/-/blob/master/rscripts/process_ids.R) | Re-format of ship IDs that comes from different data sources (`SID`) to enable linking of data from the same ship (same ship name) across different `DCK`s. And to also enable linking to metadata information in WMO Publication No. 47.
|
|
|
|
[`process_shipdata.R`](https://git.noc.ac.uk/brecinosrivas/icoads-r-hostace/-/blob/master/rscripts/process_shipdata.R) | Corrects dates and times errors noticed in some `DCK`. Largely arise from confusion over historical definitions of the marine day and conversions between local time and UTC.
|
|
|
|
[`new_get_pairs.R`](https://git.noc.ac.uk/brecinosrivas/icoads-r-hostace/-/blob/master/rscripts/new_get_pairs.R) | Duplicate record identification within the ship data. Pairs the reports as duplicate if they have associated ship `ID`s. The candidate pairs are then selected according to i) the number of matching elements (similar content of variables within a specific tolerance), ii) the `DCK`s, and iii) a comparison of the `ID`s.
|
|
|
|
[`new_get_dups.R`](https://git.noc.ac.uk/brecinosrivas/icoads-r-hostace/-/blob/master/rscripts/new_get_dups.R) | Counts the number of duplicated records and flags the best according to a [quality control criteria](Workflow/quality-control). Groups duplicated records by common `callsings`.
|
|
|
|
[`new_merge_ids_year.R`](https://git.noc.ac.uk/brecinosrivas/icoads-r-hostace/-/blob/master/rscripts/new_merge_ids_year.R) | Links `ID`'s into classes. Ship tracks of the linked `ID`'s are then checked. Reports that fail the track check are flagged as the worst duplicate. Uses a shipping tracking alogrithm from [**imma**.](https://git.noc.ac.uk/brecinosrivas/icoads-r-hostace/-/wikis/How-to-install#install-dependencies-with-conda-all-platforms)
|
|
|
|
[`clean_data.R`](https://git.noc.ac.uk/brecinosrivas/icoads-r-hostace/-/blob/master/rscripts/clean_data.R) | ---.
|
|
|
|
[`clean2track.R`](https://git.noc.ac.uk/brecinosrivas/icoads-r-hostace/-/blob/master/rscripts/clean2track.R) | Forms ship tracks for linked `ID`s. --.
|
|
|
|
|
|
|
|
**Utils**
|
|
|
|
____________
|
|
|
|
|
|
|
|
Function | Description
|
|
|
|
-------- |:------------
|
|
|
|
`add_date2.R ` |
|
|
|
|
`add_dck_priority.R ` |
|
|
|
|
[`add_date2.R `](https://git.noc.ac.uk/brecinosrivas/icoads-r-hostace/-/blob/master/rutils/add_date2.R) | Adds a date variable to a file (YY_MM.Rda) based on yr, mo, dy, hr. Invalid values are set to missing. It generates a date variable with missing hour set to local noon and adds a **date.flag.** <br><br> `0 = valid date & time`<br> `1 = invalid date or time (time not missing)` <br> `2 = valid date, hr missing, 12 local added`
|
|
|
|
[`add_dck_priority.R `](https://git.noc.ac.uk/brecinosrivas/icoads-r-hostace/-/blob/master/rutils/add_dck_priority.R) | Function use to identify duplicates. During the identification procedure priorities are assigned to data from each`dck` (See [source](https://git.noc.ac.uk/brecinosrivas/icoads-r-hostace/-/blob/master/rutils/add_dck_priority.R)), data expected to be of best quality is assigned a priority of 1, data with larger priority numbers will be flagged as the worst duplicate if identified as potential matches. Priority values are based on those from [ICOADS.](https://icoads.noaa.gov/)
|
|
|
|
`add_ID_class.R ` |
|
|
|
|
`add_shipnames.R` |
|
|
|
|
`assess_match.R` |
|
| ... | ... | @@ -50,6 +50,8 @@ Function | Description |
|
|
|
`write_dup_func.R` |
|
|
|
|
|
|
|
|
**ICOADS variables used**
|
|
|
|
|
|
|
|
Visit the [ICOADS main website](https://icoads.noaa.gov/) or check th [IMMA report](https://icoads.noaa.gov/e-doc/imma/R3.0-imma1.pdf)
|
|
|
|
________________
|
|
|
|
Code | Description
|
|
|
|
-----: |:------------
|
| ... | ... | @@ -85,54 +87,54 @@ Code | Description |
|
|
|
`mds` | metadata source
|
|
|
|
`mo` | month UTC
|
|
|
|
`n` | cloud amount
|
|
|
|
`ne` |
|
|
|
|
`nh` |
|
|
|
|
`nhe` |
|
|
|
|
`nid` |
|
|
|
|
`oav` |
|
|
|
|
`oaz` |
|
|
|
|
`ocv` |
|
|
|
|
`ocz` |
|
|
|
|
`onv` |
|
|
|
|
`onz` |
|
|
|
|
`oov` |
|
|
|
|
`ooz` |
|
|
|
|
`ophv` |
|
|
|
|
`ophz` |
|
|
|
|
`opv` |
|
|
|
|
`opz` |
|
|
|
|
`osiv` |
|
|
|
|
`osiz` |
|
|
|
|
`osv` |
|
|
|
|
`osz` |
|
|
|
|
`pt` |
|
|
|
|
`qci` |
|
|
|
|
`rh` |
|
|
|
|
`ri` |
|
|
|
|
`rrr` |
|
|
|
|
`rs` |
|
|
|
|
`sa` |
|
|
|
|
`sbi` |
|
|
|
|
`si` |
|
|
|
|
`sid` |
|
|
|
|
`sim` |
|
|
|
|
`slp` |
|
|
|
|
`sme` |
|
|
|
|
`smf` |
|
|
|
|
`smv` |
|
|
|
|
`sst` |
|
|
|
|
`sx` |
|
|
|
|
`uh` |
|
|
|
|
`uid` |
|
|
|
|
`um` |
|
|
|
|
`vs` |
|
|
|
|
`vv` |
|
|
|
|
`w` |
|
|
|
|
`w1` |
|
|
|
|
`wbt` |
|
|
|
|
`wh` |
|
|
|
|
`wmi` |
|
|
|
|
`ww` |
|
|
|
|
`wwe` |
|
|
|
|
`wx` |
|
|
|
|
`yr` | |
|
|
\ No newline at end of file |
|
|
|
`ne` | total cloud amount
|
|
|
|
`nh` | amt. of lowest clouds
|
|
|
|
`nhe` | lower cloud amount
|
|
|
|
`nid` | national source indic.1
|
|
|
|
`oav` | alkalinity value
|
|
|
|
`oaz` | alkalinity depth
|
|
|
|
`ocv` | total chlorophyll value
|
|
|
|
`ocz` | total chlorophyll depth
|
|
|
|
`onv` | nitrate value
|
|
|
|
`onz` | nitrate depth
|
|
|
|
`oov` | dissolved oxygen
|
|
|
|
`ooz` | dissolved oxygen depth
|
|
|
|
`ophv` | pH value
|
|
|
|
`ophz` | pH depth
|
|
|
|
`opv` | phosphate value
|
|
|
|
`opz` | phosphate depth
|
|
|
|
`osiv` | silicate value
|
|
|
|
`osiz` | silicate depth
|
|
|
|
`osv` | salinity value
|
|
|
|
`osz` | salinity depth
|
|
|
|
`pt` | platform type
|
|
|
|
`qci` | quality control indic.
|
|
|
|
`rh` | relative humidity
|
|
|
|
`ri` | relative lunar illuminance
|
|
|
|
`rrr` | amount of precip
|
|
|
|
`rs` | rate of Is (ice accretion on ship)
|
|
|
|
`sa` | solar altitude
|
|
|
|
`sbi` | sky-brightness indicator
|
|
|
|
`si` | SST meas. method ? or indic. for SST meas.
|
|
|
|
`sid` | source ID
|
|
|
|
`sim` | SST measurement method
|
|
|
|
`slp` | air pressure
|
|
|
|
`sme` | source meta. element
|
|
|
|
`smf` | source metadata file
|
|
|
|
`smv` | source format version
|
|
|
|
`sst` | sea surface temperature
|
|
|
|
`sx` | swell period indicator
|
|
|
|
`uh` | NOL high amount
|
|
|
|
`uid` | unique report ID
|
|
|
|
`um` | NOL middle amount
|
|
|
|
`vs` | ship's average speed
|
|
|
|
`vv` | visibility
|
|
|
|
`w` | wind speed
|
|
|
|
`w1` | past weather
|
|
|
|
`wbt` | wet-bulb temperature
|
|
|
|
`wh` | wave height
|
|
|
|
`wmi` | indic. for wave measurement
|
|
|
|
`ww` | present weather
|
|
|
|
`wwe` | present weather
|
|
|
|
`wx` | wave period indicator
|
|
|
|
`yr` | year UTC |
|
|
\ No newline at end of file |
