Source
...
Target
Commits (122)
__pycache__/
*.pyc
include:
- project: communications-backbone-system/backbone-infrastructure-config
ref: master
file: gitlab/all.yml
variables:
DOCKER_IMAGE_NAME: backbone-message-format
TEST_DOCKER_COMPOSE: 1
\ No newline at end of file
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
## [Unreleased]
## [v0.1.0] - 2023-03-24
### Added
JSON schema definitions for the SoAR project
- outer wrapper MESSAGE definition
- header object
- payload object
- acknowledgement
- mission_plan (raw and encoded)
- observation (raw and encoded)
- platform_status (raw and encoded)
- planning_configuration
Example messages matching the schema for each partner
[v0.1.0]: https://git.noc.ac.uk/communications-backbone-system/backbone-message-format/compare/9e6ce245...v0.1.0
[unreleased]: https://git.noc.ac.uk/communications-backbone-system/backbone-message-format/compare/v0.1.0...dev
# Contributing to Backbone Message Formats
_Last updated: 01 Dec 2022_
<br/>
Guidelines for our project partners to collaborate in this workspace specifically for message formats that interact with each partner's software components and the Communications Backbone.
<br/><br/>
## Where To Find A Brief on Message Formats
Please refer to the [**README.md**](https://git.noc.ac.uk/communications-backbone-system/backbone-message-format/-/blob/master/README.md) file that summarises the `message types` and `message flows`.
> _Info: [Quick Links](#quick-links) might be helpful too!_
<br/>
## Commenting
A great way to collaborate and refine the message formats is by commenting on the schema definitions/examples (within the code itself) or on an `issue` card.
### **Commenting on Code**
Go to [`Merge Requests`](https://git.noc.ac.uk/communications-backbone-system/backbone-message-format/-/merge_requests), add your comment and click `Add comment now`. Be sure to **tag partners** who should response. **Use `@`** followed by the **partner's username** or **tag a partner's** organisation by **adding a label**.
P.S. The initial working branch is [`create-initial-message-formats`](https://git.noc.ac.uk/communications-backbone-system/backbone-message-format/-/merge_requests/2/diffs).
**But _HOW_ Do I Add A Comment?**
> A. To **comment directly on the** [**`master` branch**](https://git.noc.ac.uk/communications-backbone-system/backbone-message-format/-/tree/master) directly, hover over the left column that shows the line numbers, and click on the **paperclip icon**. This copies the specific line of code. You can then proceed to pasting this link into a new issue as per [Raising New Ideas or Questions](#raising-new-ideas-or-questions).
> B. To **comment on a Merge Request**, hover over the left column that shows the line numbers, and click on the _paperclip_ or _comment bubble_ icon. This copies the line of code. You can then proceed to pasting this link into a new issue as per [Raising New Ideas or Questions](#raising-new-ideas-or-questions).
<br/>
### **Commenting on Issue Cards**
Go to the [issue](https://git.noc.ac.uk/communications-backbone-system/backbone-message-format/-/issues), scroll to the bottom of the page and add your comment. Once done, click on `Comment`.
<br/><br/>
## Raising New Ideas or Questions
If you have an idea or a question for our collaborators, it's simple! [`Create a new issue`](https://git.noc.ac.uk/communications-backbone-system/backbone-message-format/-/issues/new?issue), assign a person to respond/take action under `Assignee`, and add a partner under `Labels` .
<br/><br/>
## Labels Summary
To add/edit labels, go to [manage labels](https://git.noc.ac.uk/communications-backbone-system/backbone-message-format/-/labels).
| Category | Labels Available | Description |
| -------- | ---------------------------------------------------------- | ----------------------------------------------------------------------- |
| Project | `SoAR` | Associated project |
| Partner | `Planet Ocean`, `Hydro-Surv`, `NOC`, `RHU` | Associated partner |
| Asset | `ah1`, `ecosub`, `reav`, `autonomy engine` | Associated platform |
| Status | `In Sprint Backlog`, `In Progress`, `In Review`, `BLOCKED` | Status of resolving issue |
| Weight | `0`, `1`, `2`, `3`, `5`, `8` | Complexity of issue (0 - quick task and 8 - full 2 weeks worth of work) |
| - | `bug`, `feature` | Type of issue i.e. new feature, bug |
<br/>
## Quick Links
1. [Schema Fields Definitions](https://git.noc.ac.uk/communications-backbone-system/backbone-message-format/-/tree/7-message-formats-initial/formats)
2. [JSON Schema Examples](https://git.noc.ac.uk/communications-backbone-system/backbone-message-format/-/tree/7-message-formats-initial/examples)
### Project-Specific: SoAR Project
* **[Reference SoAR Message Format Slides](https://planetocean15.sharepoint.com/sites/IUKSoAR/Shared%20Documents/Forms/AllItems.aspx?id=%2Fsites%2FIUKSoAR%2FShared%20Documents%2FTechnical%2FTechnical%20meetings%2FDiscussions%E2%80%94%20Backbone%20Message%20Formats%2Epdf&viewid=1d649f5f%2Dd30e%2D482f%2Dbd77%2D9316a0023bf9&parent=%2Fsites%2FIUKSoAR%2FShared%20Documents%2FTechnical%2FTechnical%20meetings)**
* To view schema docs via Swagger UI, run the command below and go to `http://127.0.0.1:5000`
```
python3 docs/generate_swagger.py
```
*Disclaimer: These are *not* endpoints. Purely for schema representation purposes.
> TEMPORARY: Pregenerated Swagger docs can be [viewed here](https://planetocean15.sharepoint.com/sites/IUKSoAR/Shared%20Documents/Forms/AllItems.aspx?id=%2Fsites%2FIUKSoAR%2FShared%20Documents%2FTechnical%2FTechnical%20meetings%2Finitial%2Dswagger%2Ddocs%2Ddraft%2Epdf&viewid=1d649f5f%2Dd30e%2D482f%2Dbd77%2D9316a0023bf9&parent=%2Fsites%2FIUKSoAR%2FShared%20Documents%2FTechnical%2FTechnical%20meetings).
<br/>
## Getting Started
> Notes for this section is in PROGRESS
1. Clone this gitlab repository:
```
git clone https://git.noc.ac.uk/communications-backbone-system/backbone-message-format.git
```
2. Install the dependencies from `requirements.txt`.
\ No newline at end of file
MIT License
Copyright (c) 2022 National Oceanography Centre
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
# backbone-message-format
# Overview
This project repository is a collaborative workspace. It consists of all messages transferred into and out of the Communications Backbone. Message type schemas will be developed once reviewing each platform's data and statuses defined by each partner.
Message format(s) for the Communications Backbone System
\ No newline at end of file
The schemas are using OpenApi 3.0 (due to the `payload` sub-schemas that vary according to the `message_type`).
## Message Types
Each message below will be wrapped in a `payload` field and will have a `header` with message metadata information:
* `mission_plan`: these would be two message types, i. encoded (platform-specific serialized message) and ii. parsed, human-readable message.
* `platform_status`: these would be two message types, i. encoded (platform-specific serialized message) and ii. parsed, human-readable message.
* `observation`: this would be desired scientific data sent by the platform
* `acknowledgement`: level of acknowledgment where an acknowledgement is sent when a message is i. received, ii. sent to the next destination (e.g. platform in the water).
* `planning_configuration`: sent from the GUI to initialise the AI model (autonomy engine).
## Topics
Messages are routed using either a publish/subscribe or broadcast mechanism.
For published messages, the message is published for a given a topic.
Clients create a subscription (stored in their config) to a topic pattern which may include wildcards.
It's important to note that the client config documents the subscription. To change a client
subscription it must be updated with the backbone.
Published messages are delivered to all clients where
the message topic matches the client's subscription
topic pattern.
Broadcast messages are delivered to all clients regardless of their subscription.
### Field
The topic should be set in the `message.header.destination` for published messages.
The `destination` field should be set to `broadcast` for any messages sent using broadcast.
Because the `destination` field is a topic it defines what the message relates to - not
who the intended recipient is.
This means that the clients are not typically represented in the topics. A client publishes a message about a platform and subscribers interested in that project, platform or message type will receive that message.
### Structure
`<prefix>.<operator>.<platform_type>.<platform_identifier>.<to_platform|from_platform>.<message_type>`
e.g
* `soar.noc.autosub.ah1.from_platform.platform_status`
* `soar.planet-ocean.ecosub.eco1.to_platform.mission_plan`
* `soar.hydrosurv.reav.reav1.to_platform.mission_plan`
### Terms
* `prefix` - an agreed hard-coded/config setting prefix shared by all adapters and client subscriptions eg soar
* `operator` - human readable organisation name for the platform operator one of [noc|planet-ocean|hydrosurv]
* `platform_type` - one of [autosub|ecosub|reav?]
* `platform_identifier` - human readable identifier for the platform eg ah1, eco1, reav1
* `to_platform|from_platform` - eg ..to_platform.mission_plan, ..from_platform.platform_status
missions are sent `to_platform` to be executed and status updates are received `from_platform` containing
information about its position and health status
* `message_type` - one of `[acknowledgement|mission_plan|observation|planning_configuration|platform_status...]`
### Subscriptions
Subscriptions may contain multiword wildcards (#) or single word wildcards (*)
eg
* `soar.#` - everything
* `soar.planet-ocean.#` - anything relating to the planet-ocean operator
* `soar.*.*.*.from_platform.*` - all messages inbound from platforms (autonomy engine)
* `soar.*.*.*.to_platform.*` - all messages outbound to platforms (hermes)
* `soar.*.slocum.#` - all messages relating to a slocum platform type
## Run Docs
Run the command below and go to `http://127.0.0.1:5000`
```
python3 generate_schema_config.py
```
## Run Tests
Run the command below
```
python3 -m unittest tests/test_schemas.py
```
## Quick Links
1. [Generated Swagger Docs (recommended to look at this)](https://git.noc.ac.uk/communications-backbone-system/backbone-message-format/-/blob/dev/project/soar/swagger.json)
2. [Schema Fields Definitions](https://git.noc.ac.uk/communications-backbone-system/backbone-message-format/-/tree/dev/formats)
3. [JSON Schema Examples](https://git.noc.ac.uk/communications-backbone-system/backbone-message-format/-/tree/dev/examples)
4. Ongoing Project: [SoAR README.md](https://git.noc.ac.uk/communications-backbone-system/backbone-message-format/-/blob/dev/project/soar/SOAR_README.md)
## Installing
You can install the message formats as a package with pip
```requirements.txt
backbone-message-format @ git+https://git.noc.ac.uk/communications-backbone-system/backbone-message-format.git@v0.1.0#egg=backbone-message-format
```
### Importing the formats
```python
import backbone_formats
```
### Importing the compiled schema
```python
import importlib.resources
import soar_schema
import json
with importlib.resources.open_text(soar_schema, "swagger.json") as file:
schema = json.load(file)
```
\ No newline at end of file
from flask_restx import Api
from flask import Flask
__all__ = [
"formats",
"examples",
"project",
"tests",
]
app = Flask(__name__)
api = Api(app)
FROM python:3.9.16-alpine
WORKDIR /app
COPY requirements-dev.txt requirements-dev.txt
RUN pip install -r requirements-dev.txt
\ No newline at end of file
FROM python:3.9.16-alpine
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
\ No newline at end of file
version: '3.8'
services:
soar_flasgger_test:
build:
context: ..
dockerfile: docker/Dev.Dockerfile
restart: unless-stopped
ports:
- "5000:5000"
container_name: soar_flasgger_test
environment:
- PYTHONDONTWRITEBYTECODE=1
volumes:
- ../:/app
command: "python3 -m unittest tests/test_schemas.py"
\ No newline at end of file
version: '3.8'
services:
soar_flasgger:
build:
context: ..
dockerfile: docker/Dockerfile
restart: unless-stopped
ports:
- "5000:5000"
container_name: soar_flasgger
environment:
- FLASK_HOST=0.0.0.0
volumes:
- ../:/app
command: "python generate_schema_config.py"
\ No newline at end of file
{
"header":{
"message_ID": "b427003c-0000-11aa-a1eb-bvcdfghjgfdd",
"timestamp": "2022-11-16T00:00:00Z",
"version": 1,
"source": "hydrosurv_adapter",
"destination": "soar.hydrosurv.reav.reav1.from_platform.acknowledgement",
"delivery_type": "publish",
"encoded": false
},
"payload":{
"message_type": "acknowledgement",
"autonomy_engine_plan_ID": 1,
"platform_ID": "reav-x-1",
"approved": false
}
}
\ No newline at end of file
{
"header": {
"message_ID": "b427003c-0000-11aa-a1eb-bvcdfghjgfdd",
"timestamp": "2022-11-16T00:00:00Z",
"version": 2,
"source": "autonomy_engine",
"destination": "soar.noc.autosub.alr-52.to_platform.mission_plan",
"delivery_type": "publish",
"encoded": false
},
"payload":{
"message_type": "mission_plan",
"platform_ID": "5-ah1",
"autonomy_engine_plan_ID": 1,
"plan": [
{
"action": "move",
"start_point_latitude": -3.007143188645706,
"start_point_longitude": 50.37072283932642,
"target_waypoint_latitude": -3.237143188645706,
"target_waypoint_longitude": 52.37072283932642,
"depth": 0.0,
"activate_payload": false,
"timeout": 200
},
{
"action": "dive",
"start_point_latitude": -3.237143188645706,
"start_point_longitude": 52.37072283932642,
"target_waypoint_latitude": -3.337143188645706,
"target_waypoint_longitude": 56.37072283932642,
"altitude": 10,
"activate_payload": false,
"timeout": 200
},
{
"action": "scanline",
"start_point_latitude": -3.337143188645706,
"start_point_longitude": 56.37072283932642,
"target_waypoint_latitude": -0.237143188645706,
"target_waypoint_longitude": 60.00000000000000,
"altitude": 10,
"activate_payload": true,
"timeout": 200
},
{
"action": "climb",
"start_point_latitude": -0.237143188645706,
"start_point_longitude": 60.00000000000000,
"target_waypoint_latitude": -0.237143188645706,
"target_waypoint_longitude": 52.37072283932642,
"depth": 0.0,
"activate_payload": false,
"timeout": 200
}
]
}
}
\ No newline at end of file
{
"header": {
"message_ID": "b427003c-0000-11aa-a1eb-bvcdfghjgfdd",
"timestamp": "2022-11-16T00:00:00Z",
"version": 2,
"source": "autonomy_engine",
"destination": "soar.planetocean.ecosub.ecosub-1.to_platform.mission_plan",
"delivery_type": "publish",
"encoded": false
},
"payload":{
"message_type": "mission_plan",
"platform_ID": "ecosub-2",
"autonomy_engine_plan_ID": 1,
"plan": [
{
"action": "move",
"start_point_latitude": -3.007143188645706,
"start_point_longitude": 50.37072283932642,
"target_waypoint_latitude": -3.237143188645706,
"target_waypoint_longitude": 52.37072283932642,
"depth": 0.0,
"activate_payload": false,
"timeout": 300
},
{
"action": "dive",
"start_point_latitude": -3.237143188645706,
"start_point_longitude": 52.37072283932642,
"target_waypoint_latitude": -3.337143188645706,
"target_waypoint_longitude": 56.37072283932642,
"altitude": 10,
"activate_payload": false,
"timeout": 300
},
{
"action": "scanline",
"start_point_latitude": -3.337143188645706,
"start_point_longitude": 56.37072283932642,
"target_waypoint_latitude": -0.237143188645706,
"target_waypoint_longitude": 60.00000000000000,
"altitude": 10,
"activate_payload": true,
"timeout": 300
},
{
"action": "climb",
"start_point_latitude": -0.237143188645706,
"start_point_longitude": 60.00000000000000,
"target_waypoint_latitude": -0.237143188645706,
"target_waypoint_longitude": 52.37072283932642,
"depth": 0.0,
"activate_payload": false,
"timeout": 300
}
]
}
}
\ No newline at end of file
{
"header":{
"message_ID": "b427003c-0000-11aa-a1eb-bvcdfghjgfdd",
"timestamp": "2022-11-16T00:00:00Z",
"version": 2,
"source": "autonomy_engine",
"destination": "soar.hydrosurv.reav-60.reav-60-1.to_platform.mission_plan",
"delivery_type": "publish",
"encoded": false
},
"payload":{
"message_type": "mission_plan",
"platform_ID": "reav-60-1",
"autonomy_engine_plan_ID": 1,
"plan": [
{
"target_waypoint_latitude": -4.187143188645706,
"target_waypoint_longitude": 50.37072283932642
},
{
"target_waypoint_latitude": -3.187143188645706,
"target_waypoint_longitude": 51.37072283932642
},
{
"target_waypoint_latitude": -3.237143188645706,
"target_waypoint_longitude": 52.37072283932642
}
]
}
}
\ No newline at end of file
{
"header": {
"message_ID": "t1237003c-0000-11aa-a1eb-bvcdfghjgfdd",
"timestamp": "2023-03-16T00:00:00Z",
"version": 2,
"source": "autonomy_engine",
"destination": "soar.hermes.autosub.alr-52.to_platform.mission_plan_encoded",
"delivery_type": "publish",
"encoded": true
},
"payload": {
"message_type": "mission_plan_encoded",
"data": "SDQke4uwyP/YQQAgAhA2AND/nu8nvQAAAAAAAAAACtejPa5HHUGkcBlB/tRYQW3nez5HWYFBAAD+zwBySUAAAADVek72v1N2lUQAAAABAQAAAAAAAAAAAAAAAAAAIAAAAQAAAAAAAAAA9P2cP166ab+9cg==",
"file_name": "ah1_03837434286438.sbd",
"mime_type": "application/gzip",
"is_binary": true
}
}
{
"header": {
"message_ID": "e1237003c-0000-11aa-a1eb-bvcdfghjgfdd",
"timestamp": "2023-03-16T00:00:00Z",
"version": 2,
"source": "hermes.ecosub2",
"destination": "soar.planet-ocean.ecosub.ecosub-2.from_platform.observation_encoded",
"delivery_type": "publish",
"encoded": true
},
"payload": {
"message_type": "observation_encoded",
"data": "T1N2lUQAAAABAQAAAAAAAAAAAAAAAAAAIAAAAQAAAAAAAAAA9P2cP166ab+9cg==",
"file_name": "ecosub1_200002114.bin",
"mime_type": "application/gzip",
"is_binary": true
}
}
{
"header":{
"message_ID": "b427003c-0000-11aa-a1eb-bvcdfghjgfdd",
"timestamp": "2022-11-16T00:00:00Z",
"version": 2,
"source": "gui",
"destination": "soar.all.all.all.from_platform.planning_configuration",
"delivery_type": "publish",
"encoded": false
},
"payload":{
"message_type": "planning_configuration",
"planning_config_ID": 3,
"exclusion_zones": [
{
"geometry_coordinates": [
[
[
-4.187143188645706,
50.37072283932642
],
[
-4.202697005964865,
50.368816892405874
],
[
-4.203156724702808,
50.365640144076906
],
[
-4.19449868846155,
50.362267670845654
]
]
]
}
],
"squads": [
{
"squad_ID": 1,
"no_of_platforms": 1,
"squad_mission_type": "tracking",
"squad_state": "active",
"platforms": [
{
"model": "reav",
"platform_ID": "reav-60-1",
"emergency": {
"additional_data": {},
"target_waypoint_latitude": -7.432,
"target_waypoint_longitude": 50.365,
"safe_command": "go_home",
"target_depth": 10.0
},
"max_velocity": 0.9,
"min_altitude": 15.2,
"min_velocity": 0.1,
"additional_data": {}
}
],
"region_of_interest": {
"geometry_coordinates": [
[
[
-4.187143188645706,
50.37072283932642
],
[
-4.202697005964865,
50.368816892405874
],
[
-4.203156724702808,
50.365640144076906
],
[
-4.19449868846155,
50.362267670845654
]
]
]
}
},
{
"squad_ID": 2,
"no_of_platforms": 3,
"squad_mission_type": "survey",
"squad_state": "active",
"platforms": [
{
"platform_ID": "ecosub-1",
"model": "ecosub",
"emergency": {
"additional_data": {},
"target_waypoint_latitude": -7.432,
"target_waypoint_longitude": 50.365,
"safe_command": "go_home",
"target_depth": 10.0
},
"max_velocity": 0.9,
"min_altitude": 15.2,
"min_velocity": 0.1,
"additional_data": {
"scan_type": "DVL",
"swath_width": 10
}
},
{
"platform_ID": "ecosub-2",
"model": "ecosub",
"emergency": {
"additional_data": {},
"target_waypoint_latitude": -0.432,
"target_waypoint_longitude": 20.365,
"safe_command": "go_home",
"target_depth": 0.0
},
"max_velocity": 0.9,
"min_altitude": 15.2,
"min_velocity": 0.1,
"additional_data": {
"scan_type": "DVL",
"swath_width": 10
}
}
]
},
{
"squad_ID": 3,
"no_of_platforms": 1,
"squad_mission_type": "inspection",
"squad_state": "active",
"platforms": [
{
"platform_ID": "ah-1",
"model": "autosub",
"emergency": {
"additional_data": {},
"target_waypoint_latitude": 20.432,
"target_waypoint_longitude": 50.365,
"safe_command": "abort_now",
"target_depth": 0.0
},
"max_velocity": 0.9,
"min_altitude": 15.2,
"min_velocity": 0.1,
"additional_data": {
"scan_type": "MBES"
}
}
]
}
]
}
}
\ No newline at end of file
{
"header":{
"message_ID": "b427003c-0000-11aa-a1eb-bvcdfghjgfdd",
"timestamp": "2022-11-16T00:00:00Z",
"version": 2,
"source": "hermes",
"destination": "soar.planet-ocean.ecosub.ecosub-5.from_platform.platform_status",
"delivery_type": "publish",
"encoded": false
},
"payload":{
"message_type": "platform_status",
"platform_ID": "ecosub-5",
"platform_timestamp": "2022-12-21T00:00:00Z",
"status_source": "usbl",
"latitude": 178.2,
"longitude": -10.122,
"depth": 50.0,
"altitude": 20.0
}
}