GitLab

Rather than encoding data in the same form as a GeoJson polygon coordinates linear ring, test referencing the GeoJson schema directly.

We could then test allowing things to be defined as a Feature or FeatureCollection although at least in the short term we probably want to keep things simple by linking things to types (waypoint=point, operating-area=polygon, transect=linestring

We can't use this unless it works in at least python and JS

• JS
  • https://www.npmjs.com/package/@apidevtools/json-schema-ref-parser
• Python
  • https://github.com/purukaushik/ref-resolver - this only resolves relative path local files
  • https://python-jsonschema.readthedocs.io/en/v4.10.2/references/ - this is outdated
  • https://python-jsonschema.readthedocs.io/en/stable/referencing/ - need to read this in detail

Apparently dynamically resolving refs from URIs is discouraged for reasons I don't really understand. You can do it like this: https://python-jsonschema.readthedocs.io/en/stable/referencing/#automatically-retrieving-resources-over-http - I think you could combine this with resolving local refs in the schema but it looks like we have to do that work for some reason.

We could use the refs to load the schemas into a local store and use those for validation but that's effectively the same thing. I suppose the difference is doing it once vs doing it on the fly.

The objection to doing this appears to be that the URI might be an identifier not a link that resolves to the schema.

The JS approach is to crawl the schema and resolve all the remote refs into the local schema registry once. It looks like there used to be an equivalent crawler process in python which has since been deprecated for .. reasons?

I suppose one good reason for not live loading a published schema is change control. There's no version in the geojson url (https://geojson.org/schema/Polygon.json). If geojson publish a non-backward-compatible update we'd immediately start validating against the new schema and failing.

If we do this ref resolver stuff in the CI build and store the results then we can run the tests against the resolved schemas and then if the published version changes the build would fail in the CI.

My problem with this is that would mean serving up geojson feature schemas as part of our built schema to be loaded by the adapters which feels dirty.

Have done a quick experiment with resolving GeoJSON into local refs in our schema at build time and I quite like it.

Means the runtime doesn't change and means we run the tests against any changes to the remote schemas before they become part of our message definitions.

created merge request !31 (merged) to address this issue

mentioned in merge request !31 (merged)

changed milestone to %CB-2024W33

assigned to @danjon

added Status::In Progress label

changed milestone to %CB-2024W35

In the interests of making the changes to the actual message definitions as incremental version bumps I think I'm going to back out of changing any more of message definitions on this branch (lat/long to Points) so this branch just changes the region definition used to define the region_of_interest and exclusion_zones as a proof of concept and those elements will be removed in the refactor to the list of typed primitives.

added Status::In Review label and removed Status::In Progress label

added Status::In Progress label and removed Status::In Review label

I am beset by demons!

I could have sworn I tested the latest openapi-spec-validator package before suggesting that upgrading it would fix the issue but now having upgraded, rebased (and checked with pip freeze it's definitely there) I'm still getting the SAME ERROR.

Balls!

So I can make Feature/FeatureCollection valid (in python) and import the GeoJSON schemas unaltered if I call the schema openapi version v3.1.0 however:

1. The current JS schema validator only works up to v3.0.x
2. There is a JS schema validator that works with v3.1.x but it's ES only and doesn't work in CommonJS (which is how we're using it)
3. The JS validator we're using to validate the messages themselves is also only up to v3.0.x

I'm going to stick all this on another branch and give it some more thought

added Project::MAS-DT Status::In Review Weight::3 labels and removed Status::In Progress label

Where I've got to is it now validates against 3.0.x which is what we were using before.

That should mean existing clients don't need to upgrade dependencies.

There is a bit of an ugly workaround to downgrade the GeoJSON schemas. It doesn't effect the structure of the data or how it validates the GeoJSON content. The only functional change is you can't have a nulls but I don't think we have need of a feature with a null geometry or a feature with null properties. All our GeoJSON content should have both geography and context.

closed