wirewall-processing
A python package that can process wirewall data either from CSV files or direct from ERDDAP
Overview
This is a python implementation of a Matlab script design to calulate overtopping events from raw wirewall data. This version can either be used via a Command Line Interface (CLI) or embedd into your python project.
Running Locally
First we need to setup a virual env and then download the packaage.
mkdir wirewall-processing/
cd wirewall-processing
python -m venv wirewall
source ./wirewall/scripts/active.sh # for windows that would be ./wirewall/Scripts/Active.ps1
pip install wirewall_processing --extra-index-url=https://repo.bodc.me/repository/pypi-bodc/simple/
we now have the package install into our new enviroment. We can now checkout the CLI via --help command.
wirewall --help
and here we can see that output. ` Usage: wirewall [OPTIONS]
This cli will process csv based wirewall data and output a csv of overtopping events.
Either a file path or a URL can be given for the location of the (csv) data. This data is then check and if needed reformated and cleaned. Overtopping and flagging information will be produced and saved to a csv file on completion of the script.
Inputs: [exactly 1 required] -f, --file_location FILE Location of the csv file to read -u, --url TEXT URL for the csv file to use
Other options: -s, --site_id TEXT [required] -d, --dataset_id TEXT [required] -o, --output_location DIRECTORY Location to save the output to as a csv file [required] --help Show this message and exit. `
Running with Docker
The app has been dockerized, and can be ran on the command line once you have the image. The commands below assume an image called 'wirewall', and are setup specifically for P03, so these may need to be changed accordingly.
The <DIRECTORY_OF_CSV_FILE> needs to be populated with the directory where the CSV file is, and the output file will also be put in the same location.
For running on the command line
docker run --rm
-v <DIRECTORY_OF_CSV_FILE>:/data
wirewall
-f /data/PenzanceFarEnd.csv
-s P03
-d CreamT_890g_ty66_3qso
-o /data/
For Powershell
docker run --rm -v <DIRECTORY_OF_CSV_FILE>:/data
wirewall -f /data/PenzanceFarEnd.csv
-s P03 -d CreamT_890g_ty66_3qso
-o /data/
Example Run commands
$ wirewall --file_location 'home/examples/CreamT_890g_ty66_3qso_a004_a83c_580f.csv' -s P03 -d CreamT_890g_ty66_3qso -o /Tmp/
-----------
**Documentation**
PROVIDE A LINK TO THE DOCUMENTATION HERE
-------------
**Development**
PROVIDE A LINK TO ANY DEVELOPMENT INFORMATION HERE
-----------
**Package Installation**
Poetry is used to manage the building of this package (.whl & .tar.gz files), and Poetry can be used to install the package
dependencies for you. The dependencies are in the pypoject.toml file, to install them run:
- ``poetry install``
To update your installed dependencies, run:
- ``poetry update``
To install a new dependency and add it to the projects pyproject.toml file, run:
- ``poetry add package_name``
To view your projects dependencies, run:
- ``poetry show``
To view the specific dependencies for a particular package, run:
- ``poetry show package_name``
**Virtual Environments**
This project uses ``tox`` to manage virtual environments. This allows us to have the same environment configuration
for all users, as well as in our GitLab CI/CD pipelines.
To run a ``tox`` environment (for example the linting checks), you can use::
(wirewall-processing-tox) $ tox -e lint
The ``(wirewall-processing-tox) $`` indicates that we're running with the conda environment activated.
The available tox environments and their uses are:
py310
Run the tests from the ``tests`` directory.
``lint``
Run the static analysis of the code to check for possible errors and style requirements.
``build``
Build the Python package to check that it will be able to be uploaded.
``docs``
Build the documentation from the ``docs/source`` directory and output the HTML to ``docs/build``.
``format``
Format the Python code using ``black``.
Troubleshooting
+++++++++++++++
If you have made changes to the repository that don't seem to be updating in the tox environments,
you can recreate an environment using the ``-r`` option. For example::
(wirewall-processing-tox) $ tox -r -e 310
Releasing a new version
-----------------------
Versions of the package are denoted by tags in git.
To create a new tag, you can use the GitLab UI by following these steps:
#. Go the repository tags page
#. Click **New tag**
#. Enter the tag name. There are four options for the format for this tag:
#. Alpha release (development release): ``vX.Y.ZaW``, for example ``v1.0.2a3``
#. Beta release (development release): ``vX.Y.ZbW``, for example ``v2.3.0b1``.
#. Release candidate (test release): ``vX.Y.ZrcW``, for example ``v1.10.9rc2``.
#. Full release (production release): ``vX.Y.Z``, for example ``v3.0.11``.
#. Select the branch to create the tag from, this will normally be ``main``
#. Enter a message for the tag, this is required for the CI/CD pipeline to function correctly
#. Click **Create tag**
#. This will trigger a CI/CD pipeline