Skip to content

GitLab

Last edited by Beatriz Recinos
Page history
This is an old version of this page. You can view the most recent version or browse the history.

Workflow

Data input

  • ICOADS v3.0. Freeman. et al., (2017).
  • Metadata from WMO Publication 47. Kent. et al., (2007)
  • CLIWOC logbook IDs. (couldn't find the link)
  • Inventory of ship names in the US Maury Collection
  • generate_id (by Dave... not so clear the source)
  • Precision criteria: an estimate of the precision of each key variable (e.g. sst, lat, lon) per DCK, year and or SID. This precision criteria is require in order to set tolerances when comparing variables from ICOADS. Comparison of variables allows for a match between reports in the duplicate identification procedure.
  • Json files containing ITU call sign prefixes associated with a country.
  • seq IDS.

Processing stages

The following diagram is a summary of the data processing workflow follow by the shell scripts defined in scr. Each block represents a main task done by one script inrscripts. The corresponding .R file name is highlight in grey between each stage.

Orange blocks represent pre-processing tasks done to the ICOADS data base, in order to:

  1. Select data taken only by commercial ships, excluding specialist ship data sources, such as research vessels (For more information see the selection criteria).
  2. Preprocessing of IDs to improve duplicate identification and linking of IDs between each pair of duplicate reports.
  3. Preformed quality control on the data to point out the best duplicate.
graph TB
A1[rscripts]

 id1[(ICOADS v3.0)] --> |split_by_type.R|id2[Separate records according <br> to the different platform types.]
 id2 --> |simple_dup.R|id3[First duplicate identification between <br> ship data and the different platform types. <br> Considers the records as duplicates if they <br> show matching date, time & position.] 
 id3 --> |ship2plat.R|id4[Exclude non-ship data.] 
 id4 --> id5[(ICOADS SHIP data)]
 id5 --> |process_ids.R|id6[Homogenize and re-format <br> ship IDs from different decks. <br> Links metadata from Pub 47 & logbooks <br> to formed a plausible ship track.]
 id6 -->  |process_shipdata.R|id7[Process ship data: <br> correction of dates & times.] 
 id7 --> |new_get_pairs.R|id8[Second duplicate identification. <br> Pairs the reports as duplicate if <br> they have associated ship IDs. <br> Reports that fail the track check <br> are flagged as the worst.] 
 id8 --> |new_get_dups.R|id9[Counts the number of duplicates and flags the best.]
 id9 --> |new_merge_ids_year.R|id10[Links ID's into classes.]
 id10 --> |clean_data.R|id11[Clean of ship data.] 
 id11 --> |clean2track.R|id12[Forms ship tracks for linked IDs.] 
 id12 --> id13[(Output data)]

classDef pre-processing fill:#fcc679,stroke:#333,stroke-width:1px,font-size:16px,font-weight:100,text-align:center
classDef scripts fill:#8C929D,stroke:#333,stroke-width:1px,font-size:16px,font-weight:100,text-align:center
classDef rest fill:#e8eaf6,stroke:#333,stroke-width:1px,font-size:16px,font-weight:100,text-align:center
class id2,id3,id4 pre-processing;
class A1,id1,id5,id13 scripts;
class id6,id7,id8,id9,id10,id11,id12 rest