Create initial message schemas

Closes #7 (closed)

examples/ecosub_c2/platform_status-from_usbl_example.json

+{
+    "message_ID": "b427003c-0000-11aa-a1eb-bvcdfghjgfdd",
+    "timestamp": "2022-11-16T00:00:00Z",
+    "version": 2,
+    "source": "hermes",
+    "destination": "autonomy_engine",
+    "delivery_type": "publish",
+    "encoded": false,
+    "message_type": "platform_status",
+    "platform_ID": 1,

examples/ecosub_c2/observation.json

+{
+    "message_ID": "b427003c-0000-11aa-a1eb-bvcdfghjgfdd",
+    "timestamp": "2022-11-16T00:00:00Z",
+    "version": 2,
+    "source": "ecosub_c2",
+    "destination": "autonomy_engine",
+    "delivery_type": "publish",
+    "encoded": false,
+    "message_type": "observation",
+    "platform_ID": 3,

examples/gui/planning_configuration.json

+              "safe_command": "go_home",
+              "target_depth": 10.0
+            },
+            "max_velocity": 0.9,
+            "min_altitude": 15.2,
+            "min_velocity": 0.1,
+            "additional_data": {
+                "scan_type": "DVL",
+                "swath_width": 10
+            }
+          },
+          {
+            "platform_ID": 10,
+            "serial": "ecosub-5",
+            "model": "ecosub",
+            "emergency": {

formats/mission_plan.py

+            "example": 15.0,
+        },
+        "activate_payload": {
+            "type": "boolean",
+            "description": "To activate/deactivate sensor for Autosub "
+            + "Hover-1 --> `MBES` sensor and for EcoSUB --> `Sidescan`",
+            "example": True,
+        },
+        "send_environmental_data": {
+            "type": "boolean",
+            "description": "To trigger the platform to send list of"
+            + "  observations if any found",
+            "example": False,
+        },
+    },
+    "required": [

formats/mission_plan.py

+action_schema = {
+    "type": "object",
+    "properties": {
+        "action": {
+            "type": "string",
+            "description": "Autonomy Engine's action from `move`, `payload`,"
+            + " `dive`, `send_hits`, `scanline`, `scanpoint`.",
+            "example": "move",
+        },
+        "flight_style": {
+            "type": "string",
+            "description": "Platform-specific modes/flight styles to perform"
+            + " next action",
+            "example": "orbit",
+        },
+        "latitude_waypoint": {

formats/platform_status.py

+            "example": {"payload": [1.2, 434]},
+        },
+    },
+    "required": [],
+}
+
+platform_status_message_schema = {
+    "allOf": [{"$ref": "#/components/schemas/Message"}],
+    "type": "object",
+    "properties": {
+        "platform_ID": {
+            "type": "integer",
+            "description": "Identifier for platform",
+            "example": 1,
+        },
+        "status_source": {

formats/planning_configuration.py

+            "description": "Minimum altitude set for squad.",
+            "example": 15.2,
+        },
+        "min_velocity": {
+            "type": "number",
+            "format": "float",
+            "description": "Minimum velocity set for squad.",
+            "example": 0.1,
+        },
+        "max_velocity": {
+            "type": "number",
+            "format": "float",
+            "description": "Maximum altitude set for squad.",
+            "example": 0.9,
+        },
+        "additional_data": {

requirements.txt

+Flask
+flask-restx
+flasgger
+flask-marshmallow

tests/test.py

+"""
+schema, spec_url = read_from_filename("fixtures/swagger.json")
+errors_iterator = openapi_v3_spec_validator.iter_errors(schema)
+print(errors_iterator)
+print(openapi_v30_spec_validator.validate(schema))
+
+"""
+Test schema - Hydrosurv Messages
+"""
+message_types = ["acknowledgement", "mission_plan", "platform_status"]
+for item in message_types:
+    f = open("mock_data/hydrosurv_adapter/%s.json" % item)
+    mock_data = json.load(f)
+    header_data = mock_data["header"]
+    payload_data = mock_data["payload"]
+    print(validate(header_data, message_header, format_checker=FormatChecker()))

examples/all/mission_plan.json

-{
-    "message_ID": "b427003c-0000-11aa-a1eb-bvcdfghjgfdd",
-    "timestamp": "2022-11-16T00:00:00Z",
-    "version": 2,
-    "source": "autonomy_engine",
-    "destination": "hydrosurv_adapter",
-    "delivery_type": "publish",
-    "encoded": false,
-    "message_type": "mission_plan",
-    "plan_ID": 1,
-    "plan": [
-        {
-          "action": "move",
-          "flight_style": "orbit",
-          "activate_payload": true,
-          "altitude": 15,
-          "depth": 15,
-          "latitude_waypoint": -4.187143188645706,
-          "longitude_waypoint": 50.37072283932642,
-          "send_environmental_data": false
-        }
-      ],  
-    "platform_serial": "reav-60"
-}

examples/all/observation.json

-{
-    "message_ID": "b427003c-7bc8-11ed-a1eb-0242ac120002",
-    "timestamp": "2022-11-16T00:00:00Z",
-    "version": 2,
-    "source": "ecosub_c2",
-    "destination": "autonomy_engine",
-    "encoded": false,
-    "type": "observation",
-    "payload": {
-        "platform_serial":"ecosub-3",
-        "points_of_interest": [],
-        "region_surveyed": [],
-        "quality_of_points": 0.98,
-        "additional_data": {
-            "sensor_x": false
-        }
-    }
-}
\ No newline at end of file

examples/all/planning_configuration.json

-{
-    "message_ID": "b427003c-7bc8-11ed-a1eb-0242ac120002",
-    "timestamp": "2022-11-16T00:00:00Z",
-    "version": 2,
-    "source": "autonomy_engine",
-    "destination": "ah-1",
-    "encoded": false,
-    "type": "platform_status",
-    "payload": {
-        "ID": 3,
-        "squads": {
-            "squad_ID": 23,
-            "no_of_platforms": 3,
-            "platforms": [
-            {
-                "platform_ID": 1,
-                "serial": "ah-1",
-                "model": "ah-1",
-                "constraints": {
-                "min_altitude": 15.2,
-                "min_velocity": 0.1,
-                "max_velocity": 0.9
-                },
-                "active": true
-            }
-            ],
-            "squad_mission_type": "survey",
-            "squad_state": false,
-            "region_of_interest": [
-            {
-                "region": {
-                "type": "FeatureCollection",
-                "features": [
-                    {
-                    "type": "Feature",
-                    "properties": {},
-                    "geometry": {
-                        "coordinates": [
-                        [
-                            [
-                            -4.187143188645706,
-                            50.37072283932642
-                            ],
-                            [
-                            -4.202697005964865,
-                            50.368816892405874
-                            ],
-                            [
-                            -4.203156724702808,
-                            50.365640144076906
-                            ],
-                            [
-                            -4.19449868846155,
-                            50.362267670845654
-                            ]
-                        ]
-                        ],
-                        "type": "Polygon"
-                    }
-                    }
-                ]
-                }
-            }
-            ],
-            "exclusion_zones": [
-            {
-                "region": {
-                "type": "FeatureCollection",
-                "features": [
-                    {
-                    "type": "Feature",
-                    "properties": {},
-                    "geometry": {
-                        "coordinates": [
-                        [
-                            [
-                            -4.187143188645706,
-                            50.37072283932642
-                            ],
-                            [
-                            -4.202697005964865,
-                            50.368816892405874
-                            ],
-                            [
-                            -4.203156724702808,
-                            50.365640144076906
-                            ],
-                            [
-                            -4.19449868846155,
-                            50.362267670845654
-                            ]
-                        ]
-                        ],
-                        "type": "Polygon"
-                    }
-                    }
-                ]
-                }
-            }
-            ]
-        }
-    }
-}
\ No newline at end of file

examples/all/platform_status.json

-{
-    "message_ID": "b427003c-0000-11aa-a1eb-bvcdfghjgfdd",
-    "timestamp": "2022-11-16T00:00:00Z",
-    "version": 2,
-    "source": "hydrosurv_adapter",
-    "destination": "autonomy_engine",
-    "delivery_type": "publish",
-    "encoded": false,
-    "message_type": "platform_status",
-    "active": true,
-    "autonomy_plan_ID": 1,
-    "battery_remaining_capacity": 80.2,
-    "gps_data": [
-        {
-            "altitude": 20,
-            "depth": 50,
-            "gps_source": "internal",
-            "latitude": 178.2,
-            "latitude_type": "string",
-            "longitude": -10.122,
-            "longitude_type": "string"
-        }
-    ],
-    "health_status": "Warning",
-    "localisation_error": null,
-    "mission_action_ID": 1,
-    "mission_track_ID": 4,
-    "platform_ID": 1,
-    "platform_state": "ABORT",
-    "platform_timestamp": "2022-12-21T00:00:00Z",
-    "range_to_go": 124.3,
-    "sensor_config": {
-      "additional_data": {},
-      "sensor_serial": null,
-      "sensor_status": null
-    },
-    "speed_over_ground": 124.3,
-    "thrust_applied": 124.3,
-    "usbl_fix_seconds_ago": null,
-    "water_current_velocity": 124.3
-  }

examples/ecosub_c2/mission_plan.json

 {
-    "message_ID": "b427003c-7bc8-11ed-a1eb-0242ac120002",
-    "timestamp": "2022-11-16T00:00:00Z",
-    "version": 2,
-    "source": "autonomy_engine",
-    "destination": "ecosub_c2",
-    "encoded": false,
-    "type": "mission_plan",
-    "payload": {
-        "plan_ID": 3,
-        "platform_serial": "ecosub-4",
-        "plan": [
-            {
-                "action": "move",
-                "flight_style": "dive",
-                "latitude_waypoint": -4.187143188645706,
-                "longitude_waypoint": 50.37072283932642,
-                "altitude": 20,
-                "depth": null,
-                "activate_payload": true,
-                "send_environmental_data": false
-            }
-        ]
+  "message_ID": "b427003c-0000-11aa-a1eb-bvcdfghjgfdd",
+  "timestamp": "2022-11-16T00:00:00Z",
+  "version": 2,
+  "source": "autonomy_engine",
+  "destination": "ecosub_c2",
+  "delivery_type": "publish",
+  "encoded": false,
+  "message_type": "mission_plan",
+  "platform_ID": 1,
+  "autonomy_engine_mission_plan_ID": 1,
+  "plan": [
+    {
+      "action": "move",
+      "flight_style": "go to waypoint",
+      "latitude_waypoint": -3.237143188645706,
+      "longitude_waypoint": 52.37072283932642,
+      "altitude": null,
+      "depth": 0.0,
+      "activate_payload": false,
+      "send_environmental_data": false
+    },
+    {
+      "action": "dive",
+      "flight_style": "dive to depth",
+      "latitude_waypoint": -3.237143188645706,
+      "longitude_waypoint": 52.37072283932642,
+      "altitude": 10,
+      "depth": null,
+      "activate_payload": false,
+      "send_environmental_data": false
+    },
+    {
+      "action": "scanline",
+      "flight_style": "go to waypoint",
+      "latitude_waypoint": -0.237143188645706,
+      "longitude_waypoint": 60.00000000000000,
+      "altitude": 10,
+      "depth": null,
+      "activate_payload": true,
+      "send_environmental_data": false
+    },
+    {
+      "action": "climb",
+      "flight_style": "go to surface",
+      "latitude_waypoint": -0.237143188645706,
+      "longitude_waypoint": 52.37072283932642,
+      "altitude": null,
+      "depth": 0,
+      "activate_payload": false,
+      "send_environmental_data": true
     }
+  ]
 }
\ No newline at end of file

examples/ecosub_c2/observation.json

 {
-    "message_ID": "b427003c-7bc8-11ed-a1eb-0242ac120002",
+    "message_ID": "b427003c-0000-11aa-a1eb-bvcdfghjgfdd",
     "timestamp": "2022-11-16T00:00:00Z",
     "version": 2,
     "source": "ecosub_c2",
     "destination": "autonomy_engine",
+    "delivery_type": "publish",
     "encoded": false,
-    "type": "platform_status",
-    "payload": {
-        "platform_serial":"ecosub-3",
-        "points_of_interest": [],
-        "region_surveyed": [],
-        "quality_of_points": 0.98,
-        "additional_data": {
-            "sensor_x": false
+    "message_type": "observation",
+    "platform_ID": 3,
+    "points_of_interest": [
+        {
+            "latitude": 178.2,
+            "longitude": -10.122,
+            "quality_of_point": 0.98
+        },
+        {
+            "latitude": 50.1382,
+            "longitude": -11.122,
+            "quality_of_point": 0.50
         }
-    }
+    ],
+    "region_surveyed": ""
 }
\ No newline at end of file

examples/ecosub_c2/platform_status-from_usbl_example.json

     "message_ID": "b427003c-0000-11aa-a1eb-bvcdfghjgfdd",
     "timestamp": "2022-11-16T00:00:00Z",
     "version": 2,
-    "source": "autonomy_engine",
-    "destination": "ah1",
+    "source": "hermes",
+    "destination": "autonomy_engine",
     "delivery_type": "publish",
     "encoded": false,
     "message_type": "platform_status",
-    "acknowledged_message_ID": "02125022255-7bc8-11ed-a1eb-0242ac999999",
-    "status": "executed by platform"
-}
\ No newline at end of file
+    "platform_ID": 1,
+    "platform_timestamp": "2022-12-21T00:00:00Z",
+    "status_source": "usbl",
+    "latitude": 178.2,
+    "longitude": -10.122,
+    "depth": 50,
+    "altitude": 20
+}

examples/ecosub_c2/platform_status.json

 {
-    "message_ID": "b427003c-7bc8-11ed-a1eb-0242ac120002",
+    "message_ID": "b427003c-0000-11aa-a1eb-bvcdfghjgfdd",
     "timestamp": "2022-11-16T00:00:00Z",
-    "version": 1,
-    "source": "hermes_ecosub",
+    "version": 2,
+    "source": "ecosub_c2",
     "destination": "autonomy_engine",
+    "delivery_type": "publish",
     "encoded": false,
-    "type": "platform_status",
-    "payload": {
-        "platform_ID": 2,
-        "active": true,
-        "platform_state": "IDLE",
-        "autonomy_plan_ID": 1,
-        "mission_track_ID": 4,
-        "mission_action_ID": 1,
-        "range_to_go": 12.3,
-        "speed_over_ground": 10.3,
-        "water_current_velocity": 10.3,
-        "thrust_applied": 0.3,
-        "health_status": "Warning",
-        "gps_data": [
-            {
-                "gps_source": "internal",
-                "latitude_type": "",
-                "longitude_type": "",
-                "latitude": -7.432,
-                "longitude": 50.365,
-                "depth": 5.0,
-                "altitude": 20.0
-            }
-        ],
-        "localisation_error": null,
-        "usbl_fix_seconds_ago": null,
-        "battery_remaining_capacity": 80,
-        "sensor_config": {
-            "sensor_ID": 22,
-            "serial": "sidescan-2x",
-            "sensor_status": true,
-            "additional_data": {
-                "whiskers_on": true
-            }
-        }
-    }
-  }
\ No newline at end of file
+    "message_type": "platform_status",
+    "platform_ID": 1,
+    "platform_timestamp": "2022-12-21T00:00:00Z",
+    "status_source": "onboard_platform",
+    "autonomy_engine_plan_ID": 1,
+    "battery_remaining_capacity": 80.2,
+    "active": true,
+    "platform_state": "ABORT",
+    "mission_action_ID": 1,
+    "mission_track_ID": 4,
+    "latitude": 178.2,
+    "longitude": -10.122,
+    "depth": 50,
+    "altitude": 20,
+    "heading": 90.0,
+    "health_status": "Warning",
+    "localisation_error": null,
+    "usbl_fix_seconds_ago": null,
+    "range_to_go": 124.3,
+    "sensor_config": {
+        "sensor_ID": 22,
+        "serial": "sidescan-2x",    
+        "sensor_on": true,
+        "additional_data": {
+            "whiskers_on": true
+        }      
+    },
+    "current_pitch": -4,
+    "speed_over_ground": 4.3,
+    "thrust_applied": 124.3,
+    "water_current_velocity": 124.3
+}

[Trishna Saeharaseelan]

Note: Ignore all the files under examples/*. Main files are:

• README.md
• init.py
• autonomy_configuration.py
• platform_status.py

[izzat]

Another thing to consider is the sensor data for perception or scientific observation, and whether it should go under platform status or a separate status message/topic for on-board sensor payload. Another thing we discussed previously is if this could be made generic or if we should allow customised data/messages for some platforms.

For SoAR, this would be for the pings/targets/feature-of-interests detected by the AUVs.

[Trishna Saeharaseelan]

[Alvaro] Add message format version

[Trishna Saeharaseelan]

To add separate message schema for usbl_position_updates

latitude: <>
longitude: <>
depth: <>
timestamp: <>
comms_quality: <>
ref_platform_ID: <>

[Trishna Saeharaseelan]

@geosal to have a look for USBL positions update example.

[Trishna Saeharaseelan]

@geosal to have a look at observations example. Is anything missing?

[Trishna Saeharaseelan]

@geosal @danjon to check out initial sample based on our idea for emergency behaviour in the planning config

[Trishna Saeharaseelan]

Remove safe_command

[Trishna Saeharaseelan]

[DAN] Useful to add a heading value in the vehicle status updates

[James Kirk]

Should action also be required? I guess the sub can declare its position without that, but I'd think that might be an action in and of itself? declare_pos action or something?

[Trishna Saeharaseelan]

[TO DISCUSS] action is basically from the Autonomy engine's side (i.e. what is the "action" in autonomy engine lingo). Let's discuss?

[Trishna Saeharaseelan]

To be platform-agnostic; remove flightstyle field + make action mandatory

declare_pos > not needed

[James Kirk]

Few comments of little bits I think have been from copy + pasting Only other thing is there's a lot of TODOs, commented out stuff, and just things to come back to later. Are the comments needed in there? And are there issues or notes somewhere of what work is left to do amongst all the TODOs and that?

[Dan Jones]

Can we have a chat about the structure of the messages maybe after stand-up tomorrow? This (I think) is the opposite way round from the schema I've based the prototype on. In the prototype schema every message is an instance of Message and the anyOf defines which payload schema it contains. The idea is that the backbone only deals in messages and passes the message on to the protocol handler which cares about what a message contains.

The model here is there are a bunch of different top-level schemas which each contain the content of the "Message" schema. It may well be fine or even preferable but it doesn't match the thing I've currently got working so it's probably worth a conversation.

[Trishna Saeharaseelan]

[TO DISCUSS] Interesting. Let's discuss this (need to clarify). If we need to change it from this end instead of the core, we should make the changes sooner since the partners will be working with this schemas as a start

[Trishna Saeharaseelan]

• Trishna to follow the structure the adapter is using to read schemas
• Validate all examples

[Dan Jones]

Have you tried validating the examples against this schema? I had a go with npm swagger-model-validator which I've used for the javascript adapter in the prototype. So far I've only tried the ecosub platform status message but it's not successfully validating.

[Trishna Saeharaseelan]

So with openapi-spec-validator it was validated successfully. Let's chat. That being said, I will add a task to build some tests

[Dan Jones]

That's interesting. I'll see if there's a javascript equivalent and have a go at validating it with that.

[Dan Jones]

Oh wait a minute. openapi-spec-validator is validating that the swagger.json conforms to the open api standard. It's not parsing a messaging and validating that the message conforms to the schema definition. What I'm talking about is validating that the example platform_status.json messages validate against the PlatformStatus definition defined in the schema.

[Dan Jones]

I think the python version of what I'm talking about is here: https://github.com/p1c2u/openapi-schema-validator

[Dan Jones]

The swagger.json is also curiously different in structure to all out existing APIs. Normally the models are under definitions but here they're under components/schemas and definitions is there but empty.

[Dan Jones]

The other thing Ashley suggested was that rather than define our own schema for anything geographic we just point to the existing GeoJson feature schema: https://geojson.org/schema/Feature.json

We could explicitly require a Point or a BoundingBox or whatever or just point to the parent feature definition.

Point is not very specific - it just has a min of 2 coordinates so we could supply 3=z or even 4=depth+altitude if it were necessary.

[Dan Jones]

The advantage of using Feature would be that the operating area and exclusion zones could be a BoundingBox or they could be a Polygon or even a MultiPolygon. We might not want to allow that complexity for the MVP but it would be good to have it in future - especially given the ultimate goal of the funders.

[Trishna Saeharaseelan]

Actually geojson is being in the planning_configuration message for exclusion_zones and the optional region_of_interest. Would we want to generate a 4-point polygon for each point then?

[Dan Jones]

What I mean by using geojson is adding a ref to the published geojson schema.

"coordinates": {
  "$ref": "https://geojson.org/schema/Point.json"
},

What the planning configuration message is doing is creating a data structure that matches GeoJSON.

To do that we'd need a validator that supported resolving remote refs which it seems like the one I'm currently using doesn't.

[Dan Jones]

Would we want to generate a 4-point polygon for each point then

No - We use the GeoJSON feature that represents what we want to draw. So Point for a status update/area of interest, Polygon for an operating area / exclusion zone.

[Dan Jones]

This one does it: https://www.npmjs.com/package/json-schema-remote

[Trishna Saeharaseelan]

@allore what's the best terminology to distinguish between the type of status message (i.e. from USBL or from the platform itself) vs. source/rx mode of the message (i.e. from acoustics, iridium, or wifi)?

[Alvaro Lorenzo]

Tricky. Maybe we want a field called something like platform_source: {USBL, Slocum, Ecosub, ALR ...} and other called transmission_mean : {Wifi, acoustics, iridium, starlink ... }

[Trishna Saeharaseelan]

To add warm-up time field in the planning config (e.g. for ecosub warm-up of sidescan payload)

[Dan Jones]

Needs the validator packages adding in here or in a requirements-dev.txt

[Trishna Saeharaseelan]

DONE

[Dan Jones]

Why aren't you validating the whole mock_data as a whole against the MESSAGE schema?

Should the tests be pytest/unittest or something so we can tell if they pass/fail in the CI later?

[Trishna Saeharaseelan]

1. Didn't just validate the FULL mock messages against message_schema because it kept passing even when it should have errors. Decided to break it down in the meantime to ensure it gets validated & does prompt errors (as it should). Added TODO in epic issue card to clean this up for better validation

2. DONE