| ... | ... | @@ -31,4 +31,28 @@ specialist ship data sources, such as research vessels |
|
|
|
identification and linking of IDs between each pair of duplicate reports.
|
|
|
|
3. Preformed [quality control](quality-control) on the data to point out the best duplicate.
|
|
|
|
|
|
|
|
[](https://mermaid-js.github.io/mermaid-live-editor/#/edit/eyJjb2RlIjoiZ3JhcGggVEJcbkExW3JzY3JpcHRzXVxuXG4gaWQxWyhJQ09BRFMgdjMuMCldIC0tPiB8c3BsaXRfYnlfdHlwZS5SfGlkMltTZXBhcmF0ZSByZWNvcmRzIGFjY29yZGluZyA8YnI-IHRvIHRoZSBkaWZmZXJlbnQgcGxhdGZvcm0gdHlwZXMuXVxuIGlkMiAtLT4gfHNpbXBsZV9kdXAuUnxpZDNbRmlyc3QgZHVwbGljYXRlIGlkZW50aWZpY2F0aW9uIGJldHdlZW4gPGJyPiBzaGlwIGRhdGEgYW5kIHRoZSBkaWZmZXJlbnQgcGxhdGZvcm0gdHlwZXMuIDxicj4gQ29uc2lkZXJzIHRoZSByZWNvcmRzIGFzIGR1cGxpY2F0ZXMgaWYgdGhleSA8YnI-IHNob3cgbWF0Y2hpbmcgZGF0ZSwgdGltZSAmIHBvc2l0aW9uLl0gXG4gaWQzIC0tPiB8c2hpcDJwbGF0LlJ8aWQ0W0V4Y2x1ZGUgbm9uLXNoaXAgZGF0YS5dIFxuIGlkNCAtLT4gaWQ1WyhJQ09BRFMgU0hJUCBkYXRhKV1cbiBpZDUgLS0-IHxwcm9jZXNzX2lkcy5SfGlkNltIb21vZ2VuaXplIGFuZCByZS1mb3JtYXQgPGJyPiBzaGlwIElEcyBmcm9tIGRpZmZlcmVudCBkZWNrcy4gPGJyPiBMaW5rcyBtZXRhZGF0YSBmcm9tIFB1YiA0NyAmIGxvZ2Jvb2tzIDxicj4gdG8gZm9ybWVkIGEgcGxhdXNpYmxlIHNoaXAgdHJhY2suXVxuIGlkNiAtLT4gIHxwcm9jZXNzX3NoaXBkYXRhLlJ8aWQ3W1Byb2Nlc3Mgc2hpcCBkYXRhOiA8YnI-IGNvcnJlY3Rpb24gb2YgZGF0ZXMgJiB0aW1lcy5dIFxuIGlkNyAtLT4gfG5ld19nZXRfcGFpcnMuUnxpZDhbU2Vjb25kIGR1cGxpY2F0ZSBpZGVudGlmaWNhdGlvbi4gPGJyPiBQYWlycyB0aGUgcmVwb3J0cyBhcyBkdXBsaWNhdGUgaWYgPGJyPiB0aGV5IGhhdmUgYXNzb2NpYXRlZCBzaGlwIElEcy4gPGJyPiBSZXBvcnRzIHRoYXQgZmFpbCB0aGUgdHJhY2sgY2hlY2sgPGJyPiBhcmUgZmxhZ2dlZCBhcyB0aGUgd29yc3QuXSBcbiBpZDggLS0-IHxuZXdfZ2V0X2R1cHMuUnxpZDlbQ291bnQgdGhlIG51bWJlciBvZiBkdXBsaWNhdGVzIGFuZCBmbGFnIHRoZSBiZXN0Ll1cbiBpZDkgLS0-IHxuZXdfbWVyZ2VfaWRzX3llYXIuUnxpZDEwW0xpbmtzIG9mIElEJ3MgaW50byBjbGFzc2VzLl1cbiBpZDEwIC0tPiB8Y2xlYW5fZGF0YS5SfGlkMTFbQ2xlYW5zIG9mIHNoaXAgZGF0YS5dIFxuIGlkMTEgLS0-IHxjbGVhbjJ0cmFjay5SfGlkMTJbRm9ybXMgc2hpcCB0cmFja3MgZm9yIGxpbmtlZCBJRHMuXSBcbiBpZDEyIC0tPiBpZDEzWyhPdXRwdXQgZGF0YSldXG5cbmNsYXNzRGVmIHByZS1wcm9jZXNzaW5nIGZpbGw6I2ZjYzY3OSxzdHJva2U6IzMzMyxzdHJva2Utd2lkdGg6MXB4XG5jbGFzc0RlZiBzY3JpcHRzIGZpbGw6IzhDOTI5RCxzdHJva2U6IzMzMyxzdHJva2Utd2lkdGg6MXB4XG5jbGFzc0RlZiByZXN0IGZpbGw6I2U4ZWFmNixzdHJva2U6IzMzMyxzdHJva2Utd2lkdGg6MXB4XG5jbGFzcyBpZDIsaWQzLGlkNCBwcmUtcHJvY2Vzc2luZztcbmNsYXNzIEExLGlkMSxpZDUsaWQxMyBzY3JpcHRzO1xuY2xhc3MgaWQ2LGlkNyxpZDgsaWQ5LGlkMTAsaWQxMSxpZDEyIHJlc3QiLCJtZXJtYWlkIjp7InRoZW1lIjoiZGVmYXVsdCJ9LCJ1cGRhdGVFZGl0b3IiOmZhbHNlfQ) |
|
|
\ No newline at end of file |
|
|
|
```mermaid
|
|
|
|
graph TB
|
|
|
|
A1[rscripts]
|
|
|
|
|
|
|
|
id1[(ICOADS v3.0)] --> |split_by_type.R|id2[Separate records according <br> to the different platform types.]
|
|
|
|
id2 --> |simple_dup.R|id3[First duplicate identification between <br> ship data and the different platform types. <br> Considers the records as duplicates if they <br> show matching date, time & position.]
|
|
|
|
id3 --> |ship2plat.R|id4[Exclude non-ship data.]
|
|
|
|
id4 --> id5[(ICOADS SHIP data)]
|
|
|
|
id5 --> |process_ids.R|id6[Homogenize and re-format <br> ship IDs from different decks. <br> Links metadata from Pub 47 & logbooks <br> to formed a plausible ship track.]
|
|
|
|
id6 --> |process_shipdata.R|id7[Process ship data: <br> correction of dates & times.]
|
|
|
|
id7 --> |new_get_pairs.R|id8[Second duplicate identification. <br> Pairs the reports as duplicate if <br> they have associated ship IDs. <br> Reports that fail the track check <br> are flagged as the worst.]
|
|
|
|
id8 --> |new_get_dups.R|id9[Count the number of duplicates and flag the best.]
|
|
|
|
id9 --> |new_merge_ids_year.R|id10[Links of ID's into classes.]
|
|
|
|
id10 --> |clean_data.R|id11[Cleans of ship data.]
|
|
|
|
id11 --> |clean2track.R|id12[Forms ship tracks for linked IDs.]
|
|
|
|
id12 --> id13[(Output data)]
|
|
|
|
|
|
|
|
classDef pre-processing fill:#fcc679,stroke:#333,stroke-width:1px
|
|
|
|
classDef scripts fill:#8C929D,stroke:#333,stroke-width:1px
|
|
|
|
classDef rest fill:#e8eaf6,stroke:#333,stroke-width:1px
|
|
|
|
class id2,id3,id4 pre-processing;
|
|
|
|
class A1,id1,id5,id13 scripts;
|
|
|
|
class id6,id7,id8,id9,id10,id11,id12 rest
|
|
|
|
|
|
|
|
``` |
|
|
\ No newline at end of file |
