Skip to content

GitLab

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
I ICOADS R HOSTACE
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 7
    • Issues 7
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
  • Merge requests 0
    • Merge requests 0
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Operations
    • Operations
    • Incidents
    • Environments
  • Analytics
    • Analytics
    • CI/CD
    • Repository
    • Value Stream
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Members
    • Members
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • brivas
  • ICOADS R HOSTACE
  • Wiki
  • API Reference

Last edited by Elizabeth Kent Jul 26, 2021
Page history

API Reference

Note: still not complete! needs checking from liz

This page lists all available scripts, functions and variables from ICOADS used in this repository.

Workflow

Scripts here follow the same order as in the Workflow diagram.

Script Description
read_imma1.R Reads ICOADS Release3 IMMA1 files to Rda format.
split_by_type.R This script splits up the Rda versions of ICOADS R3 Total files by platform type (ship, moored buoy etc.) adding or correcting the PT flag when necessary. The separation of records into the different platform types is done following a selection criteria.
simple_dup.R Duplicate record identification among ship data and the rest of the different platform types for which the records show a matching date , time and position (lat, lon). The script finds reports in the ship files output that are also contained in the files output from platforms, coastal data, moored buoys or drifters.
ship2plat.R Assesses potential matches and identify reports that are "not ship". Uses additional information from WMO Publication 47 to flag non-ship data. Kent. et al., (2007). These reports are then written to directory MFILES_NOTSHIP to be excluded from further SHIP processing.
process_ships.R The script adds various ID info and flags to the ICOADS shipdata. Reformat of ship id's coming from different data sources (sid) to enable linking of data from the same ship (same ship name) across different dck's. And to also enable linking to metadata information in WMO Publication No. 47. More information in Processing of id's. Corrects dates and times errors noticed in some dck. Largely arise from confusion over historical definitions of the marine day and conversions between local time and UTC.
get_pairs.R Duplicate record identification within the ship data. Pairs the reports as duplicate if they have associated ship id's. The candidate pairs are then selected according to i) the number of matching elements (similar content of variables within a specific tolerance), ii) the dck's, and iii) a comparison of the id's.
get_dups.R Counts the number of duplicated records and flags the best according to a quality control criteria. Groups duplicated records by common callsigns.
merge_ids_year.R Links id's into classes. Ship tracks of the linked id's are then checked. Reports that fail the track check are flagged as the worst duplicate. Uses a shipping tracking alogrithm from r-imma.
nrt_dup.R ---.
clean_data.R ---.
clean2track.R Copies data with missing/generic id`'s after processing to directory TRACK_INPUT for use as input to Carella. et al., (2017).
remove_output_files.R Clean of output_data/output_files before re-running.

icoads.utils package description

The following functions are in alphabetical order.

Function Description
add_ID_class.R Adds an specific id class to the report. Based on the ICOADS doc. and other earlier sources, the id's have been classified as to their "id type" (logbook number, ship number, etc.) and validity. Where an ITU callsign is found, the country for that callsign is also identified.
add_date2.R Adds a date variable to the data frame based on yr, mo, dy, hr.
Invalid values are set to missing. For these reports the code generates a date variable with a missing hour set to local noon and adds a date.flag.

0 = valid date & time
1 = invalid date or time (time not missing)
2 = valid date, hr missing, 12 local added
add_dck_priority.R Function use to identify duplicates. During the identification procedure priorities are assigned to the data from each dck (See source), data expected to be of best quality is assigned a priority of 1, data with larger priority numbers will be flagged as the worst duplicate if identified as potential match. Priority values are based on those from ICOADS.
add_match_id.R Tests whether paired id's are allow to match according a matching criteria and by using the Damerau–Levenshtein (DL) distance formula.
add_shipnames.R Adds ship names according to dck numbers.
find_gap_func.R -------
flag_id_dups.R Adds a flag to reports that fail the IMMA ship track check. Such reports are flagged as the worst duplicate.
get_gap_pos.R -----
get_groups.R Assigns a group number to a specific type of dck or dck's. The group is appended to the record id to avoid mixing data with the same id from different types of data (e.g. Japanese data, UK Navy).
get_id_class.R Used in add_ID_class.R.
Assigns an id class to callsigns listed in metadata of Pub 47 (Not sure).
get_itu_country.R Gets the ITU callsign prefix associated with a country.
get_matchedids.R ------
get_mis.R Selects candidate pairs of duplicate reports according to i) the number of matching elements (similar content of variables within a specific tolerance).
get_prec.R Get's the dck/sid precision of measurement for selected variables (sst,slp,at,dpt,w,d,n,ww,w1,vv)
get_pub47.R function to process monthly files from Pub47.
get_speed.R Contains several utility functions to convert between radians and degrees, calculate time and speed from coordinates and calculate the geodesic distance between two points specified by radian latitude/longitude using the spherical law of cosines (slc).
homog_ids.R Uses standard linkages between id's to add id homogenisation information to each records.
id_group_func.R Makes sure that all id's grouped in get_groups.R are properly associated.
liz_merge.R Merges two data frames by specific columns.
print_id_match_info.R Prints ICOADS duplicates information and id matching results.
read_rdsfiles.R A collection of functions to read all different types of data files (e.g rds, .txt) used through out the code.
write_dup_func.R Writes duplicate information in a pipe-delimited year-month output format. The report uid is followed by the report id then a flag with value 1 if the id has been changed, 0 if it remains the same.
An example of the format is:
ICOADS-30-0Y0HJK | 32024 | 0
ICOADS-30-0Y0HJL | 14 00117 | 1

ICOADS variables used

Visit the ICOADS main website or check th IMMA report.

Variable names are in alphabetical order.


Code Description
ah high cloud amount
am middle cloud amount
at air temperature
c1 country recruited ship
c1m recruiting country
cce change code
che high cloud type
cle low cloud type
cme middle cloud type
d wind direction (true)
dck deck
dpt dew-point temperature
ds ship course
dups dup status
dy day UTC
eoh exposure of hygrometer
es thickness of Is (ice accretion on ship)
he
hop height of visual observation platform
hr hour UTC1
id identification/callsign
ii ID indicator
immv IMMT version (International Maritime Meteorological Tape)
ir indic. for precip. data
irf intermediate reject flag
is ice accretion on ship
lat latitude
lon longitude
lz 2°×2° landlocked flag
mds metadata source
mo month UTC
n cloud amount
ne total cloud amount
nh amt. of lowest clouds
nhe lower cloud amount
nid national source indic.1
oav alkalinity value
oaz alkalinity depth
ocv total chlorophyll value
ocz total chlorophyll depth
onv nitrate value
onz nitrate depth
oov dissolved oxygen
ooz dissolved oxygen depth
ophv pH value
ophz pH depth
opv phosphate value
opz phosphate depth
osiv silicate value
osiz silicate depth
osv salinity value
osz salinity depth
pt platform type
qci quality control indic.
rh relative humidity
ri relative lunar illuminance
rrr amount of precip
rs rate of Is (ice accretion on ship)
sa solar altitude
sbi sky-brightness indicator
si SST meas. method ? or indic. for SST meas.
sid source ID
sim SST measurement method
slp air pressure
sme source meta. element
smf source metadata file
smv source format version
sst sea surface temperature
sx swell period indicator
uh NOL high amount
uid unique report ID
um NOL middle amount
vs ship's average speed
vv visibility
w wind speed
w1 past weather
wbt wet-bulb temperature
wh wave height
wmi indic. for wave measurement
ww present weather
wwe present weather
wx wave period indicator
yr year UTC
Clone repository

Wiki pages

Home

Introduction
Installation
JASMIN tips

Workflow
- Data selection
- Processing of ID's
- Matching criteria
- Quality control
- Duplicate identification

API Reference

Releases

Examples