GitLab

The main function in import_data().main has no attribute to pass an specific encoding to the following lines

TextParser = pd.read_fwf(source,widths=[properties.MAX_FULL_REPORT_WIDTH],header = None, delimiter="\t", skiprows = skiprows, chunksize = chunksize, quotechar='\0',escapechar='\0')

This requires to also modify the read.py

by adding

encoding = schema['header'].get('encoding')

to line 265 and modify the TextParser arguments in line 275

TextParser = import_data.main(source, encoding=encoding, chunksize = chunksize, skiprows = skiprows)

This fixed the problem as shown below...

data.data[["c99_header"]].iloc[15]

c99_header  folder_ID                    974
            ship_name             GLÃœCK AUF
            rig                       BARQUE
            commander                    NaN
            from_city                HAMBURG
            to_city               VALPARAISO
            voyage_begin_month             5
            voyage_begin_day               7
            voyage_begin_year           1857
            voyage_end_month               6
            voyage_end_day                12
            voyage_end_year             1857
            page_number                  053
Name: 15, dtype: object

Let me know what you think @dyb

added encoding label

It also works if you just leave encoding as None since the default of pd.read_fwf us utf-8

mentioned in commit d13675c0

mentioned in merge request !1 (merged)

closed via merge request !1 (merged)

mentioned in commit 60ea212a