Skip to content

GitLab

Commit cf9e619c authored by Beartriz Recinos Rivas's avatar Beartriz Recinos Rivas
Browse files

fixed typo on notebook

parent ee617296
master mdf_reader
No related merge requests found
Hide whitespace changes
Inline Side-by-side
Showing with 96 additions and 306 deletions
docs/notebooks/CLIWOC_datamodel.ipynb
View file @ cf9e619c
......@@ -16,7 +16,7 @@
"name": "stderr",
"output_type": "stream",
"text": [
"2021-01-08 15:42:25,535 - root - INFO - init basic configure of logging success\n"
"2021-01-08 17:05:40,504 - root - INFO - init basic configure of logging success\n"
]
}
],
......@@ -37,19 +37,19 @@
"name": "stderr",
"output_type": "stream",
"text": [
"2021-01-08 15:42:38,945 - root - INFO - READING DATA MODEL SCHEMA FILE...\n",
"2021-01-08 15:42:38,950 - root - INFO - EXTRACTING DATA FROM MODEL: imma1\n",
"2021-01-08 15:42:38,951 - root - INFO - Getting data string from source...\n",
"2021-01-08 15:42:38,964 - root - INFO - Extracting and reading sections\n",
"2021-01-08 15:42:38,965 - root - INFO - Processing section partitioning threads\n",
"2021-01-08 15:42:38,965 - root - INFO - 1000 ...\n",
"2021-01-08 15:42:38,978 - root - INFO - done\n",
"2021-01-08 15:42:38,979 - root - INFO - 211000 ...\n",
"2021-01-08 15:42:39,000 - root - INFO - done\n",
"2021-01-08 15:42:39,001 - root - INFO - 29211000 ...\n",
"2021-01-08 15:42:39,016 - root - INFO - done\n",
"2021-01-08 15:42:39,017 - root - INFO - 2929211000 ...\n",
"2021-01-08 15:42:39,022 - root - INFO - done\n"
"2021-01-08 17:05:41,566 - root - INFO - READING DATA MODEL SCHEMA FILE...\n",
"2021-01-08 17:05:41,571 - root - INFO - EXTRACTING DATA FROM MODEL: imma1\n",
"2021-01-08 17:05:41,572 - root - INFO - Getting data string from source...\n",
"2021-01-08 17:05:41,584 - root - INFO - Extracting and reading sections\n",
"2021-01-08 17:05:41,585 - root - INFO - Processing section partitioning threads\n",
"2021-01-08 17:05:41,586 - root - INFO - 1000 ...\n",
"2021-01-08 17:05:41,600 - root - INFO - done\n",
"2021-01-08 17:05:41,601 - root - INFO - 211000 ...\n",
"2021-01-08 17:05:41,614 - root - INFO - done\n",
"2021-01-08 17:05:41,615 - root - INFO - 29211000 ...\n",
"2021-01-08 17:05:41,629 - root - INFO - done\n",
"2021-01-08 17:05:41,630 - root - INFO - 2929211000 ...\n",
"2021-01-08 17:05:41,635 - root - INFO - done\n"
]
},
{
......@@ -66,21 +66,7 @@
"Reading section c95\n",
"Reading section c96\n",
"Reading section c97\n",
"Reading section c98\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2021-01-08 15:42:39,980 - root - WARNING - Data numeric elements with missing upper or lower threshold: ('c1', 'BSI'),('c1', 'AQZ'),('c1', 'AQA'),('c1', 'UQZ'),('c1', 'UQA'),('c1', 'VQZ'),('c1', 'VQA'),('c1', 'PQZ'),('c1', 'PQA'),('c1', 'DQZ'),('c1', 'DQA'),('c5', 'OS'),('c5', 'OP'),('c5', 'FM'),('c5', 'IMMV'),('c5', 'IX'),('c5', 'W2'),('c5', 'WMI'),('c5', 'SD2'),('c5', 'SP2'),('c5', 'IS'),('c5', 'RS'),('c5', 'IC1'),('c5', 'IC2'),('c5', 'IC3'),('c5', 'IC4'),('c5', 'IC5'),('c5', 'IR'),('c5', 'RRR'),('c5', 'TR'),('c5', 'NU'),('c5', 'QCI'),('c5', 'QI1'),('c5', 'QI2'),('c5', 'QI3'),('c5', 'QI4'),('c5', 'QI5'),('c5', 'QI6'),('c5', 'QI7'),('c5', 'QI8'),('c5', 'QI9'),('c5', 'QI10'),('c5', 'QI11'),('c5', 'QI12'),('c5', 'QI13'),('c5', 'QI14'),('c5', 'QI15'),('c5', 'QI16'),('c5', 'QI17'),('c5', 'QI18'),('c5', 'QI19'),('c5', 'QI20'),('c5', 'QI21'),('c5', 'QI22'),('c5', 'QI23'),('c5', 'QI24'),('c5', 'QI25'),('c5', 'QI26'),('c5', 'QI27'),('c5', 'QI28'),('c5', 'QI29'),('c5', 'RHI'),('c5', 'AWSI'),('c6', 'FBSRC'),('c6', 'MST'),('c7', 'OPM'),('c7', 'LOT'),('c9', 'CCe'),('c9', 'WWe'),('c9', 'Ne'),('c9', 'NHe'),('c9', 'He'),('c9', 'CLe'),('c9', 'CMe'),('c9', 'CHe'),('c9', 'SBI'),('c95', 'DPRO'),('c95', 'DPRP'),('c95', 'UFR'),('c95', 'ASIR'),('c96', 'ASII'),('c97', 'ASIE')\n",
"2021-01-08 15:42:39,981 - root - WARNING - Corresponding upper and/or lower bounds set to +/-inf for validation\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Reading section c98\n",
"Reading section c99\n"
]
},
......@@ -88,8 +74,10 @@
"name": "stderr",
"output_type": "stream",
"text": [
"2021-01-08 15:42:41,021 - root - INFO - Wrapping output....\n",
"2021-01-08 15:42:41,128 - root - INFO - CREATING OUTPUT DATA ATTRIBUTES FROM DATA MODEL\n"
"2021-01-08 17:05:42,553 - root - WARNING - Data numeric elements with missing upper or lower threshold: ('c1', 'BSI'),('c1', 'AQZ'),('c1', 'AQA'),('c1', 'UQZ'),('c1', 'UQA'),('c1', 'VQZ'),('c1', 'VQA'),('c1', 'PQZ'),('c1', 'PQA'),('c1', 'DQZ'),('c1', 'DQA'),('c5', 'OS'),('c5', 'OP'),('c5', 'FM'),('c5', 'IMMV'),('c5', 'IX'),('c5', 'W2'),('c5', 'WMI'),('c5', 'SD2'),('c5', 'SP2'),('c5', 'IS'),('c5', 'RS'),('c5', 'IC1'),('c5', 'IC2'),('c5', 'IC3'),('c5', 'IC4'),('c5', 'IC5'),('c5', 'IR'),('c5', 'RRR'),('c5', 'TR'),('c5', 'NU'),('c5', 'QCI'),('c5', 'QI1'),('c5', 'QI2'),('c5', 'QI3'),('c5', 'QI4'),('c5', 'QI5'),('c5', 'QI6'),('c5', 'QI7'),('c5', 'QI8'),('c5', 'QI9'),('c5', 'QI10'),('c5', 'QI11'),('c5', 'QI12'),('c5', 'QI13'),('c5', 'QI14'),('c5', 'QI15'),('c5', 'QI16'),('c5', 'QI17'),('c5', 'QI18'),('c5', 'QI19'),('c5', 'QI20'),('c5', 'QI21'),('c5', 'QI22'),('c5', 'QI23'),('c5', 'QI24'),('c5', 'QI25'),('c5', 'QI26'),('c5', 'QI27'),('c5', 'QI28'),('c5', 'QI29'),('c5', 'RHI'),('c5', 'AWSI'),('c6', 'FBSRC'),('c6', 'MST'),('c7', 'OPM'),('c7', 'LOT'),('c9', 'CCe'),('c9', 'WWe'),('c9', 'Ne'),('c9', 'NHe'),('c9', 'He'),('c9', 'CLe'),('c9', 'CMe'),('c9', 'CHe'),('c9', 'SBI'),('c95', 'DPRO'),('c95', 'DPRP'),('c95', 'UFR'),('c95', 'ASIR'),('c96', 'ASII'),('c97', 'ASIE')\n",
"2021-01-08 17:05:42,554 - root - WARNING - Corresponding upper and/or lower bounds set to +/-inf for validation\n",
"2021-01-08 17:05:43,402 - root - INFO - Wrapping output....\n",
"2021-01-08 17:05:43,498 - root - INFO - CREATING OUTPUT DATA ATTRIBUTES FROM DATA MODEL\n"
]
}
],
......@@ -175,28 +163,28 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'99 0 NAN NATIONAAL ARCHIEF OF THE NETHERLANDS DEN HAAG NEDERLAND 1.04.02 152 VOC 5137 VOC_152_5137 DUTCH VOC_152_5137_0071 0191500N1020200E185900N 2 4TENERIFE NOORDELIJKSTE LAND NTW ZUIDELIJKSTE WTN 10 0 21776100112 3 VM 8UNKNOWN TEMPEL DUTCH VOC C. OVERBEEK ASMUS HENDRIK STERRENBERG SCHIPPER TEXEL KASSERAIJEN 0 4259.317760102NW 11.50 UNKNOWN UNKNOWN 0 DUITSE DUITSE 360 DEGREES N-NNW SLAPPE-LABBER BRAMZEILSKOELTE GOED WEER OVERDRIJVENDE DEINSIGE LUCHT 00000000CLIWOC VERSION 2.0'"
"'99 0 AGI ARCHIVO GENERAL DE INDIAS SEVILLE SPAIN CORREOS, 193A R5 SPANISH 0155100S 3481100E 1 2TENERIFE 0 21776100112 UNKNOWN DILIGENCIA SPANISH FRAGATA CORREO ANDR�S V�LEZ CAPITAN LA CORU�A MONTEVIDEO 0 17760816 129.00 UNKNOWN -1 NM 360 DEGREES E FRESQUITO LLANA 00000000CLIWOC VERSION 1.0'"
]
},
"execution_count": 5,
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"line = data_raw.data['c99'].iloc[0].values[0]\n",
"line = data_raw.data['c99'].iloc[3].values[0]\n",
"line"
]
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": 5,
"metadata": {},
"outputs": [
{
......@@ -205,7 +193,7 @@
"'/Users/brivas/c3s_work/mdf_reader/data_models/lib/imma1_d730'"
]
},
"execution_count": 6,
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
......@@ -219,32 +207,32 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2021-01-08 15:43:38,121 - root - INFO - READING DATA MODEL SCHEMA FILE...\n",
"2021-01-08 15:43:38,126 - root - INFO - EXTRACTING DATA FROM MODEL: /Users/brivas/c3s_work/mdf_reader/data_models/lib/imma1_d730\n",
"2021-01-08 15:43:38,127 - root - INFO - Getting data string from source...\n",
"2021-01-08 15:43:38,139 - root - INFO - Extracting and reading sections\n",
"2021-01-08 15:43:38,140 - root - INFO - Processing section partitioning threads\n",
"2021-01-08 15:43:38,141 - root - INFO - 1000 ...\n",
"2021-01-08 15:43:38,154 - root - INFO - done\n",
"2021-01-08 15:43:38,155 - root - INFO - 211000 ...\n",
"2021-01-08 15:43:38,169 - root - INFO - done\n",
"2021-01-08 15:43:38,170 - root - INFO - 29211000 ...\n",
"2021-01-08 15:43:38,190 - root - INFO - done\n",
"2021-01-08 15:43:38,190 - root - INFO - 3029211000 ...\n",
"2021-01-08 15:43:38,194 - root - INFO - done\n",
"2021-01-08 15:43:38,195 - root - INFO - 303029211000 ...\n",
"2021-01-08 15:43:38,199 - root - INFO - done\n",
"2021-01-08 15:43:38,200 - root - INFO - 30303029211000 ...\n",
"2021-01-08 15:43:38,204 - root - INFO - done\n",
"2021-01-08 15:43:38,205 - root - INFO - 3030303029211000 ...\n",
"2021-01-08 15:43:38,207 - root - INFO - done\n"
"2021-01-08 17:06:06,666 - root - INFO - READING DATA MODEL SCHEMA FILE...\n",
"2021-01-08 17:06:06,671 - root - INFO - EXTRACTING DATA FROM MODEL: /Users/brivas/c3s_work/mdf_reader/data_models/lib/imma1_d730\n",
"2021-01-08 17:06:06,673 - root - INFO - Getting data string from source...\n",
"2021-01-08 17:06:06,685 - root - INFO - Extracting and reading sections\n",
"2021-01-08 17:06:06,686 - root - INFO - Processing section partitioning threads\n",
"2021-01-08 17:06:06,687 - root - INFO - 1000 ...\n",
"2021-01-08 17:06:06,707 - root - INFO - done\n",
"2021-01-08 17:06:06,708 - root - INFO - 211000 ...\n",
"2021-01-08 17:06:06,723 - root - INFO - done\n",
"2021-01-08 17:06:06,724 - root - INFO - 29211000 ...\n",
"2021-01-08 17:06:06,737 - root - INFO - done\n",
"2021-01-08 17:06:06,738 - root - INFO - 3029211000 ...\n",
"2021-01-08 17:06:06,742 - root - INFO - done\n",
"2021-01-08 17:06:06,743 - root - INFO - 303029211000 ...\n",
"2021-01-08 17:06:06,746 - root - INFO - done\n",
"2021-01-08 17:06:06,747 - root - INFO - 30303029211000 ...\n",
"2021-01-08 17:06:06,750 - root - INFO - done\n",
"2021-01-08 17:06:06,751 - root - INFO - 3030303029211000 ...\n",
"2021-01-08 17:06:06,753 - root - INFO - done\n"
]
},
{
......@@ -264,21 +252,7 @@
"Reading section c98\n",
"Reading section c99_sentinal\n",
"Reading section c99_logbook\n",
"Reading section c99_voyage\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2021-01-08 15:43:39,541 - root - WARNING - Data numeric elements with missing upper or lower threshold: ('c1', 'BSI'),('c1', 'AQZ'),('c1', 'AQA'),('c1', 'UQZ'),('c1', 'UQA'),('c1', 'VQZ'),('c1', 'VQA'),('c1', 'PQZ'),('c1', 'PQA'),('c1', 'DQZ'),('c1', 'DQA'),('c5', 'OS'),('c5', 'OP'),('c5', 'FM'),('c5', 'IMMV'),('c5', 'IX'),('c5', 'W2'),('c5', 'WMI'),('c5', 'SD2'),('c5', 'SP2'),('c5', 'IS'),('c5', 'RS'),('c5', 'IC1'),('c5', 'IC2'),('c5', 'IC3'),('c5', 'IC4'),('c5', 'IC5'),('c5', 'IR'),('c5', 'RRR'),('c5', 'TR'),('c5', 'NU'),('c5', 'QCI'),('c5', 'QI1'),('c5', 'QI2'),('c5', 'QI3'),('c5', 'QI4'),('c5', 'QI5'),('c5', 'QI6'),('c5', 'QI7'),('c5', 'QI8'),('c5', 'QI9'),('c5', 'QI10'),('c5', 'QI11'),('c5', 'QI12'),('c5', 'QI13'),('c5', 'QI14'),('c5', 'QI15'),('c5', 'QI16'),('c5', 'QI17'),('c5', 'QI18'),('c5', 'QI19'),('c5', 'QI20'),('c5', 'QI21'),('c5', 'QI22'),('c5', 'QI23'),('c5', 'QI24'),('c5', 'QI25'),('c5', 'QI26'),('c5', 'QI27'),('c5', 'QI28'),('c5', 'QI29'),('c5', 'RHI'),('c5', 'AWSI'),('c6', 'FBSRC'),('c6', 'MST'),('c7', 'OPM'),('c7', 'LOT'),('c9', 'CCe'),('c9', 'WWe'),('c9', 'Ne'),('c9', 'NHe'),('c9', 'He'),('c9', 'CLe'),('c9', 'CMe'),('c9', 'CHe'),('c9', 'SBI'),('c95', 'DPRO'),('c95', 'DPRP'),('c95', 'UFR'),('c95', 'ASIR'),('c96', 'ASII'),('c97', 'ASIE'),('c99_voyage', 'drLatDeg'),('c99_voyage', 'drLatMin'),('c99_voyage', 'drLatSec'),('c99_voyage', 'drLonDeg'),('c99_voyage', 'drLonMin'),('c99_voyage', 'drLonSec'),('c99_voyage', 'LatDeg'),('c99_voyage', 'LatMin'),('c99_voyage', 'LatSec'),('c99_voyage', 'LonDeg'),('c99_voyage', 'LonMin'),('c99_voyage', 'LonSec'),('c99_voyage', 'LMdistance1'),('c99_voyage', 'LMdistance2'),('c99_voyage', 'LMdistance4'),('c99_voyage', 'TimeOB'),('c99_voyage', 'Glasses'),('c99_data', 'AT_outside'),('c99_data', 'SST'),('c99_data', 'attached_tem')\n",
"2021-01-08 15:43:39,542 - root - WARNING - Corresponding upper and/or lower bounds set to +/-inf for validation\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Reading section c99_voyage\n",
"Reading section c99_data\n"
]
},
......@@ -286,8 +260,10 @@
"name": "stderr",
"output_type": "stream",
"text": [
"2021-01-08 15:43:48,904 - root - INFO - Wrapping output....\n",
"2021-01-08 15:43:49,012 - root - INFO - CREATING OUTPUT DATA ATTRIBUTES FROM DATA MODEL\n"
"2021-01-08 17:06:08,027 - root - WARNING - Data numeric elements with missing upper or lower threshold: ('c1', 'BSI'),('c1', 'AQZ'),('c1', 'AQA'),('c1', 'UQZ'),('c1', 'UQA'),('c1', 'VQZ'),('c1', 'VQA'),('c1', 'PQZ'),('c1', 'PQA'),('c1', 'DQZ'),('c1', 'DQA'),('c5', 'OS'),('c5', 'OP'),('c5', 'FM'),('c5', 'IMMV'),('c5', 'IX'),('c5', 'W2'),('c5', 'WMI'),('c5', 'SD2'),('c5', 'SP2'),('c5', 'IS'),('c5', 'RS'),('c5', 'IC1'),('c5', 'IC2'),('c5', 'IC3'),('c5', 'IC4'),('c5', 'IC5'),('c5', 'IR'),('c5', 'RRR'),('c5', 'TR'),('c5', 'NU'),('c5', 'QCI'),('c5', 'QI1'),('c5', 'QI2'),('c5', 'QI3'),('c5', 'QI4'),('c5', 'QI5'),('c5', 'QI6'),('c5', 'QI7'),('c5', 'QI8'),('c5', 'QI9'),('c5', 'QI10'),('c5', 'QI11'),('c5', 'QI12'),('c5', 'QI13'),('c5', 'QI14'),('c5', 'QI15'),('c5', 'QI16'),('c5', 'QI17'),('c5', 'QI18'),('c5', 'QI19'),('c5', 'QI20'),('c5', 'QI21'),('c5', 'QI22'),('c5', 'QI23'),('c5', 'QI24'),('c5', 'QI25'),('c5', 'QI26'),('c5', 'QI27'),('c5', 'QI28'),('c5', 'QI29'),('c5', 'RHI'),('c5', 'AWSI'),('c6', 'FBSRC'),('c6', 'MST'),('c7', 'OPM'),('c7', 'LOT'),('c9', 'CCe'),('c9', 'WWe'),('c9', 'Ne'),('c9', 'NHe'),('c9', 'He'),('c9', 'CLe'),('c9', 'CMe'),('c9', 'CHe'),('c9', 'SBI'),('c95', 'DPRO'),('c95', 'DPRP'),('c95', 'UFR'),('c95', 'ASIR'),('c96', 'ASII'),('c97', 'ASIE'),('c99_voyage', 'drLatDeg'),('c99_voyage', 'drLatMin'),('c99_voyage', 'drLatSec'),('c99_voyage', 'drLonDeg'),('c99_voyage', 'drLonMin'),('c99_voyage', 'drLonSec'),('c99_voyage', 'LatDeg'),('c99_voyage', 'LatMin'),('c99_voyage', 'LatSec'),('c99_voyage', 'LonDeg'),('c99_voyage', 'LonMin'),('c99_voyage', 'LonSec'),('c99_voyage', 'LMdistance1'),('c99_voyage', 'LMdistance2'),('c99_voyage', 'LMdistance4'),('c99_voyage', 'TimeOB'),('c99_voyage', 'Glasses'),('c99_data', 'AT_outside'),('c99_data', 'SST'),('c99_data', 'attached_tem')\n",
"2021-01-08 17:06:08,028 - root - WARNING - Corresponding upper and/or lower bounds set to +/-inf for validation\n",
"2021-01-08 17:06:17,135 - root - INFO - Wrapping output....\n",
"2021-01-08 17:06:17,244 - root - INFO - CREATING OUTPUT DATA ATTRIBUTES FROM DATA MODEL\n"
]
}
],
......@@ -299,7 +275,7 @@
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": 7,
"metadata": {},
"outputs": [
{
......@@ -377,7 +353,7 @@
"4 99 0 NaN"
]
},
"execution_count": 8,
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
......@@ -390,7 +366,7 @@
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": 8,
"metadata": {},
"outputs": [
{
......@@ -531,7 +507,7 @@
"4 ENGLISH NaN 0 "
]
},
"execution_count": 10,
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
......@@ -542,7 +518,7 @@
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": 9,
"metadata": {},
"outputs": [
{
......@@ -740,7 +716,7 @@
"[5 rows x 58 columns]"
]
},
"execution_count": 11,
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
......@@ -753,12 +729,14 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"So for the ship type on this deck for example we will have tons of ship types in different languages. There is no code table for this. Would this information will be enough (e.g. for things that Liz is doing)?"
"e.g. the ship types on this deck will be given in a tons of different languages. There is no code table for this variable in the CLIWOC website. \n",
"\n",
"Would this information will be enough (e.g. for things that Liz is doing)?"
]
},
{
"cell_type": "code",
"execution_count": 13,
"execution_count": 10,
"metadata": {},
"outputs": [
{
......@@ -772,7 +750,7 @@
"Name: Ship_type, dtype: object"
]
},
"execution_count": 13,
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
......@@ -783,7 +761,7 @@
},
{
"cell_type": "code",
"execution_count": 15,
"execution_count": 12,
"metadata": {},
"outputs": [
{
......@@ -817,33 +795,7 @@
" <th>Distance_units</th>\n",
" <th>Distance_units_to_landmark</th>\n",
" <th>Distance_units_travelled</th>\n",
" <th>Longitude_units</th>\n",
" <th>units_of_measurement</th>\n",
" <th>humidity_units</th>\n",
" <th>water_at_pump_units</th>\n",
" <th>wind_scale</th>\n",
" <th>BARO_type</th>\n",
" <th>BARO_brand</th>\n",
" <th>API</th>\n",
" <th>Humidity_method</th>\n",
" <th>compas_error</th>\n",
" <th>compas_correction</th>\n",
" <th>AT_outside</th>\n",
" <th>SST</th>\n",
" <th>AP</th>\n",
" <th>wind_dir</th>\n",
" <th>current_dir</th>\n",
" <th>current_speed</th>\n",
" <th>attached_tem</th>\n",
" <th>pump_water</th>\n",
" <th>Humidity</th>\n",
" <th>wind_force</th>\n",
" <th>weather</th>\n",
" <th>prcp_descriptor</th>\n",
" <th>sea_state</th>\n",
" <th>shape_coulds</th>\n",
" <th>dir_coulds</th>\n",
" <th>Clearness</th>\n",
" <th>...</th>\n",
" <th>cloud_fraction</th>\n",
" <th>gusts</th>\n",
" <th>Rain</th>\n",
......@@ -869,33 +821,7 @@
" <td>NaN</td>\n",
" <td>DUITSE</td>\n",
" <td>DUITSE</td>\n",
" <td>360 DEGREES</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>N-NNW</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>SLAPPE-LABBER BRAMZEILSKOELTE</td>\n",
" <td>GOED WEER</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>OVERDRIJVENDE DEINSIGE LUCHT</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
......@@ -919,33 +845,7 @@
" <td>NaN</td>\n",
" <td>LEAGUES</td>\n",
" <td>NM</td>\n",
" <td>180 DEGREES</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NEBN, NE, NNE, NEBE</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>PLEASANT BREEZE, FRESH BREEZE</td>\n",
" <td>NaN</td>\n",
" <td>SPITS OF RAIN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>CLEAR THEN CLOUDY</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
......@@ -969,33 +869,7 @@
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>UNKNOWN</td>\n",
" <td>360 DEGREES</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>E</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>FRESCACH�N</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>GRUESA</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
......@@ -1019,33 +893,7 @@
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NM</td>\n",
" <td>360 DEGREES</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>E</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>FRESQUITO</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>LLANA</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
......@@ -1069,33 +917,7 @@
" <td>NaN</td>\n",
" <td>LEAGUES</td>\n",
" <td>NM</td>\n",
" <td>UNKNOWN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>SSW</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>MODERATE</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>FAIR</td>\n",
" <td>...</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
......@@ -1109,6 +931,7 @@
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 47 columns</p>\n",
"</div>"
],
"text/plain": [
......@@ -1126,70 +949,36 @@
"3 NaN UNKNOWN -1 NaN \n",
"4 UNKNOWN UNKNOWN -17 NaN \n",
"\n",
" Distance_units_to_landmark Distance_units_travelled Longitude_units \\\n",
"0 DUITSE DUITSE 360 DEGREES \n",
"1 LEAGUES NM 180 DEGREES \n",
"2 NaN UNKNOWN 360 DEGREES \n",
"3 NaN NM 360 DEGREES \n",
"4 LEAGUES NM UNKNOWN \n",
"\n",
" units_of_measurement humidity_units water_at_pump_units wind_scale \\\n",
"0 NaN NaN NaN NaN \n",
"1 NaN NaN NaN NaN \n",
"2 NaN NaN NaN NaN \n",
"3 NaN NaN NaN NaN \n",
"4 NaN NaN NaN NaN \n",
"\n",
" BARO_type BARO_brand API Humidity_method compas_error compas_correction \\\n",
"0 NaN NaN NaN NaN NaN NaN \n",
"1 NaN NaN NaN NaN NaN NaN \n",
"2 NaN NaN NaN NaN NaN NaN \n",
"3 NaN NaN NaN NaN NaN NaN \n",
"4 NaN NaN NaN NaN NaN NaN \n",
" Distance_units_to_landmark Distance_units_travelled ... cloud_fraction \\\n",
"0 DUITSE DUITSE ... NaN \n",
"1 LEAGUES NM ... NaN \n",
"2 NaN UNKNOWN ... NaN \n",
"3 NaN NM ... NaN \n",
"4 LEAGUES NM ... NaN \n",
"\n",
" AT_outside SST AP wind_dir current_dir current_speed \\\n",
"0 NaN NaN NaN N-NNW NaN NaN \n",
"1 NaN NaN NaN NEBN, NE, NNE, NEBE NaN NaN \n",
"2 NaN NaN NaN E NaN NaN \n",
"3 NaN NaN NaN E NaN NaN \n",
"4 NaN NaN NaN SSW NaN NaN \n",
" gusts Rain Fog Snow Thunder Hail Sea_ice Trivial_correction \\\n",
"0 0 0 0 0 0 0 0 0 \n",
"1 0 1 0 0 0 0 0 0 \n",
"2 0 0 0 0 0 0 0 0 \n",
"3 0 0 0 0 0 0 0 0 \n",
"4 0 0 0 0 0 0 0 0 \n",
"\n",
" attached_tem pump_water Humidity wind_force weather \\\n",
"0 NaN NaN NaN SLAPPE-LABBER BRAMZEILSKOELTE GOED WEER \n",
"1 NaN NaN NaN PLEASANT BREEZE, FRESH BREEZE NaN \n",
"2 NaN NaN NaN FRESCACH�N NaN \n",
"3 NaN NaN NaN FRESQUITO NaN \n",
"4 NaN NaN NaN MODERATE NaN \n",
" Release \n",
"0 CLIWOC VERSION 2.0 \n",
"1 CLIWOC VERSION 2.0 \n",
"2 CLIWOC VERSION 2.0 \n",
"3 CLIWOC VERSION 1.0 \n",
"4 CLIWOC VERSION 1.1 \n",
"\n",
" prcp_descriptor sea_state shape_coulds dir_coulds \\\n",
"0 NaN NaN OVERDRIJVENDE DEINSIGE LUCHT NaN \n",
"1 SPITS OF RAIN NaN NaN NaN \n",
"2 NaN GRUESA NaN NaN \n",
"3 NaN LLANA NaN NaN \n",
"4 NaN NaN NaN NaN \n",
"\n",
" Clearness cloud_fraction gusts Rain Fog Snow Thunder Hail Sea_ice \\\n",
"0 NaN NaN 0 0 0 0 0 0 0 \n",
"1 CLEAR THEN CLOUDY NaN 0 1 0 0 0 0 0 \n",
"2 NaN NaN 0 0 0 0 0 0 0 \n",
"3 NaN NaN 0 0 0 0 0 0 0 \n",
"4 FAIR NaN 0 0 0 0 0 0 0 \n",
"\n",
" Trivial_correction Release \n",
"0 0 CLIWOC VERSION 2.0 \n",
"1 0 CLIWOC VERSION 2.0 \n",
"2 0 CLIWOC VERSION 2.0 \n",
"3 0 CLIWOC VERSION 1.0 \n",
"4 0 CLIWOC VERSION 1.1 "
"[5 rows x 47 columns]"
]
},
"execution_count": 15,
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.options.display.max_columns = None\n",
"data.data[[\"c99_data\"]].c99_data.head()"
]
},
......@@ -1197,12 +986,12 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"What about the different scales of wind force from different languages?"
"What about the different scales for the wind force, given different languages?"
]
},
{
"cell_type": "code",
"execution_count": 16,
"execution_count": 13,
"metadata": {},
"outputs": [
{
......@@ -1216,7 +1005,7 @@
"Name: wind_force, dtype: object"
]
},
"execution_count": 16,
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
......@@ -1229,16 +1018,17 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"For some of this we would have to make a nested .json file I think ... with the key as the language in which the information is written and then find the wind force scale that corresponds to that description.\n",
"For some of this data there are some code tables but according to the mdf reader we need a single code table. \n",
"\n",
"\n",
"More ideas on how to do this are welcome."
"We would have to make a nested .json file I think ... with the main key as the language. More ideas on how to do this are welcome."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"I have also taken done some of the easy key tables including the Meridian one. You can find those in this [link under ICOADS.C99.Variable.json]()"
"I have done some of the easy key tables including the Meridian one. You can find those [under ICOADS.C99.Variable.json](https://git.noc.ac.uk/brecinosrivas/mdf_reader/-/tree/master/data_models/lib/imma1_d730/code_tables)"
]
}
],
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment