Unverified Commit 0a4f14a6 authored by Dan Jones's avatar Dan Jones
Browse files

docs: corrections and updates to the readme

+ change the org name in the licence
parent e247a1a5
MIT License
Copyright (c) 2022 National Oceanography Centre CLG
Copyright (c) 2022 National Oceanography Centre
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
......
......@@ -2,84 +2,134 @@
Communications Backbone by C2 Team (NOC)
### Infrastructure
The backbone has 3 runtime components:
- RabbitMQ - run as a docker container
- The bus which handles the delivery of messages between queues
- The API which provides an interface to read from and write to the queues
### Data flow
- Client A sends to `client-a-outbox` (or POSTs to API /send - not yet implemented)
- Client A sends to `client-a-outbox` (by POSTing to API /send)
- Messages are forwarded from `client-a-outbox` to `soar-publish`
- Messages are published from `soar-publish` with the given topic read from the message
- Subscribers listen to for messages
- Subscription is delivered to `client-b-inbox`
- Client B reads from `client-b-inbox (or GETs from API /receive)
There is a parallel flow when a client sends to `client-a-notify` in which case the
messages are delivered through the broadcast exchange to all clients `client-x-inbox`.
- Subscription is delivered to `client-b-inbox` (if the client subscription matches the message topic)
- Client B reads from `client-b-inbox (by GETting from API /receive)
### Auth placeholder
There is a parallel flow when a client sends to `client-a-broadcast` (by POSTing to /notify).
In this case the messages are delivered through the broadcast exchange to all clients `client-x-inbox`.
As a proxy for proper authentication, when you post a client a random secret is
returned in the response. To send to / receive from the bus you then call the API
with the client_id and secret and it checks they match. The client_id determines
which queues it reads from.
### Prerequisites
Subsequent requests to the client endpoint return the client_id but not the secret.
- Python >= 3.8
- A virtual environment manager - virtualenv, venv or pipenv
- Docker and docker-compose
### Testing
Current coverage:
- API: yes
- Pika RabbitMQ implementation: no
### Running via docker-compose
```
pytest
run-compose.sh
```
### Running via docker-compose
Using `docker-compose` will mean that everything is setup automatically, this includes the `rabbitmq` container, the backbone API, and the backbone bus. The `run-compose.sh` script has been provided to simplify this even further - all you have to do is set whatever env vars you need in the `.env` file and then run `./run-compose.sh` (the defaults in `.env` are fine for local dev work, but ones not labelled `optional` will need setting in a production setting). The env vars are:
- `DATA_DIR` - Where to mount the volume of the API container on your local system. This defaults to the result of `pwd`, which should be within the `communications-backbone` repo
- `SOAR_TOKEN_LIFETIME` (Optional) - The number of hours until a newly created token expires
- `SOAR_TOKEN_SECRET` (Optional) - A secret key used to encrypt/decrypt token data. If specified the value should be set using TokenModel.getKey()
### Running manually
### Running the bus and API natively (without docker)
#### Setup
In a virtual environment
We recommend using some form of python virtual environment to maintain a consistent
python version and ring-fence the package management.
In your virtual environment:
```
pip install -r requirements-dev.txt
```
This installs both the development and runtime dependencies for the project.
### Testing
Current coverage:
- API: yes
- Pika RabbitMQ implementation: no
In your virtual environment:
```
pytest
```
#### RabbitMQ
`docker run --rm -p 5672:5672 -d --hostname rmq --name rmq rabbitmq:management`
#### API
In your virtual environment:
```
python api.py
```
#### Event bus
In your virtual environment:
```
python soar_bus.py
```
At present the soar bus creates the clients defined in the API when it starts
but does not monitor for new clients so if you create a new client you will
need to restart the bus. This will be fixed in a later release.
### Usage
To use the backbone you have to create a client.
The client id and name should be unique but human
readable. A client secret is returned by the API
and client credential grants using this client_id
and secret are used to authenticate client
application connections.
- Run the backbone
- Create some clients
- Restart the soar_bus service
- `cd docker && docker-compose restart soar_bus`
- Save your client credentials
- Test reading and writing directly to the queues
- Create an adapter
- Test sending and receiving via the adapter
#### Create some clients
`POST` to `http://localhost:3000/clients`
`POST` to `http://localhost:8087/client`
The POST body should be JSON.
```json
{
"client_id": "noc-c2",
"client_name": "NOC C2",
"subscription": "soar.*.noc.#"
}
```
- `client_id` - a project unique human readable name
- `client_name` - how to refer to that client on screen
- `subscription` - the topic pattern identifying the messages you want to receive
(`*` = single word wildcard, `#` = multi-word wildcard)
The response from the post contains your client `secret`. This is only displayed once.
Subsequent GETS to `/client` or `/client/[client-id]` will not return the secret.
You should save the response as `soar-config.json` adding the additional `api` field
to specify the root URL of the API you're connecting to
(eg http://localhost:8087 for local development)
#### Send / Receive directly
......@@ -93,12 +143,40 @@ python client_send.py noc-c2-outbox 'soar.noc.slocum.something' from noc-c2
python client_read.py noc-sfmc-inbox
```
#### Receive via API
These scripts bypass authentication reading and writing directly to rabbitmq.
#### Authentication
Authentication is handled by a client credentials grant which is a GET
request to `/token?client_id=[client-id]&secret=[secret]`
The response includes a token and this token should be included as an
authorization header in requests to the API:
```json
{
"Authorization": "Bearer [token]"
}
```
#### Send and Receive via API
Once you have a bearer token you can make requests to send and receive
messages via the API:
- GET `/receive?max_messages=[X]` - gets all messages in the queue
- the default value of `max_messages` if not specified is 10
- POST `/send` - publish a message to a given topic
- POST `/notify` - broadcast a message to all clients
Supplying the token as an Authorization header as described above.
#### Send and Receive using adapters
As a placeholder for proper authentication you post the `client_id` and
`secret` and it checks that a client with that id exists and that the
secret matches before allowing the request.
If you have implemented one of the adapter templates then authentication
is handled for you.
This should be replaced with a proper auth layer.
You will need to create a set of client credentials as above using the
POST `/client` endpoint and save the response.
`GET http://localhost:5000/receive?client_id=[client_id]&secret=[secret]`
Then you pass your credentials when you create your adapter instance.
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment