Skip to content

GitLab

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
I ICOADS R HOSTACE
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 7
    • Issues 7
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
  • Merge requests 0
    • Merge requests 0
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Operations
    • Operations
    • Incidents
    • Environments
  • Analytics
    • Analytics
    • CI/CD
    • Repository
    • Value Stream
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Members
    • Members
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • brivas
  • ICOADS R HOSTACE
  • Wiki
  • Workflow

Workflow · Changes

Page history
added links to technical report authored Jun 11, 2020 by bearecinos's avatar bearecinos
Hide whitespace changes
Inline Side-by-side
Showing with 11 additions and 13 deletions
+11 -13
  • Workflow.md Workflow.md +11 -13
  • No files found.
Workflow.md
View page @ d88c54c9
**Data input** **Data input**
-------------- --------------
The following data is required by the scripts of this repository: The following data is required to run the scripts in this repository:
- [ICOADS v3.0](https://icoads.noaa.gov/r3.html). [Freeman. *et al.,* (2017)](https://doi.org/10.1002/joc.4775). - [ICOADS v3.0](https://icoads.noaa.gov/r3.html). [Freeman. *et al.,* (2017)](https://doi.org/10.1002/joc.4775).
- Metadata from WMO Publication 47. - Metadata from WMO Publication 47.
[Kent. *et al.,* (2007)](https://doi.org/10.1175/JTECH1949.1) [Kent. *et al.,* (2007)](https://doi.org/10.1175/JTECH1949.1)
- CLIWOC logbook IDs. (**needs a link**) - [CLIWOC logbook IDs](https://stvno.github.io/page/cliwoc/)
- Inventory of ship names in the - Inventory of ship names in the
[US Maury Collection](https://icoads.noaa.gov/software/transpec/maury/mauri_out) [US Maury Collection](https://icoads.noaa.gov/software/transpec/maury/mauri_out)
- generate_id (**needs description**) - generate_id (**needs description**)
- **Precision criteria file**. An estimate of the precision of each key variable (e.g. `sst, lat, lon`) per `dck`, - **Precision criteria file**. An estimate of the precision of each key variable (e.g. `sst, lat, lon`) per `dck`,
`yr` and or `sid`. This precision criteria is require in order to set tolerances `yr` and or `sid`. This precision criteria is require in order to set tolerances when comparing variables from ICOADS (See the [list of ICOADS variables](https://git.noc.ac.uk/brecinosrivas/icoads-r-hostace/-/wikis/API-Reference#icoads-variables-used) used in this repository). Comparison of variables allows for
when comparing variables from ICOADS (See the [list of ICOADS variables](https://git.noc.ac.uk/brecinosrivas/icoads-r-hostace/-/wikis/API-Reference#icoads-variables-used) used in this repository). Comparison of variables allows for a match between reports in the [duplicate identification](Workflow/duplicate-identification) procedure.
a match between reports in the duplicate identification procedure.
- **Json files** containing ITU callsign prefixes associated with a country. - **Json files** containing ITU callsign prefixes associated with a country.
- **seq IDS.** (**needs description**) - **seq IDS.** (**needs description**)
**Processing stages** **Processing stages**
-------------------- --------------------
The diagram below is a summary of the data processing workflow followed by The diagram below is a summary of the data processing workflow followed by the shell scripts defined in ```scr```. Each block
the shell scripts defined in ```scr```. Each block
represents a main task done by one script in ```rscripts```. represents a main task done by one script in ```rscripts```.
The corresponding `.R` file name has been added in grey between each block. For more information on eah `.R` script, please look into the [API reference page.](api-reference) The corresponding `.R` file name has been added in grey between each block. For more information on eah `.R` script, please look into the [API reference page.](api-reference)
...@@ -31,13 +29,12 @@ in order to: ...@@ -31,13 +29,12 @@ in order to:
1. Select data taken only by commercial ships, excluding 1. Select data taken only by commercial ships, excluding
specialist ship data sources, such as research vessels specialist ship data sources, such as research vessels
(For more information see the [selection criteria](Workflow/data-selection)). (For more information see the [selection criteria](Workflow/data-selection)).
2. [Preprocessing of ID's](Workflow/processing-of-ids) to improve duplicate 2. [Preprocessing of ID's](Workflow/processing-of-ids) to improve [duplicate identification](Workflow/duplicate-identification) and linking of `id`'s between each pair of duplicate reports.
identification and linking of `id`'s between each pair of duplicate reports. 3. Preformed [quality control](Workflow/quality-control) on the data to point out the best duplicate.
3. Preformed [quality control](Workflow/quality-control) on the data to point
out the best duplicate.
- The rest of the blocks represent processing scripts that concentrate in the duplicates - The rest of the blocks represent processing scripts that concentrate in the duplicates identification and [matching of reports ID's](Workflow/matching-criteria).
identification and [matching of reports ID's](Workflow/matching-criteria).
More details on the data processing can be found in this [technical report]().
```mermaid ```mermaid
graph TB graph TB
...@@ -68,3 +65,4 @@ class id6,id7,id8,id9,id10,id11,id12 rest ...@@ -68,3 +65,4 @@ class id6,id7,id8,id9,id10,id11,id12 rest
**Output data** **Output data**
-------------- --------------
Maybe here we can add some of the plots that you created as output.
Clone repository

Wiki pages

Home

Introduction
Installation
JASMIN tips

Workflow
- Data selection
- Processing of ID's
- Matching criteria
- Quality control
- Duplicate identification

API Reference

Releases

Examples