README.md

# Communications Backbone

Communications Backbone by C2 Team (NOC)

### Infrastructure 

The backbone has 3 runtime components: 

- RabbitMQ - run as a docker container 
- The bus which handles the delivery of messages between queues 
- The API which provides an interface to read from and write to the queues

### Data flow 

- Client A sends to `client-a-outbox` (by POSTing to API /send)
- Messages are forwarded from `client-a-outbox` to `soar-publish` 
- Messages are published from `soar-publish` with the given topic read from the message 
- Subscribers listen to for messages 
- Subscription is delivered to `client-b-inbox` (if the client subscription matches the message topic) 
- Client B reads from `client-b-inbox` (by GETting from API /receive)

There is a parallel flow when a client sends to `client-a-broadcast` (by POSTing to /notify). 
In this case the messages are delivered through the broadcast exchange to all clients `client-x-inbox`.

### Prerequisites 

- Python >= 3.8
- A virtual environment manager - virtualenv, venv or pipenv 
- Docker and docker-compose 

### Running via docker-compose

```
run-compose.sh
```

Using `docker-compose` will mean that everything is setup automatically, this includes the `rabbitmq` container, the backbone API, and the backbone bus. The `run-compose.sh` script has been provided to simplify this even further - all you have to do is set whatever env vars you need in the `.env` file and then run `./run-compose.sh` (the defaults in `.env` are fine for local dev work, but ones not labelled `optional` will need setting in a production setting). The env vars are:

- `DATA_DIR` - Where to mount the volume of the API container on your local system. This defaults to the result of `pwd`, which should be within the `communications-backbone` repo
- `SOAR_TOKEN_LIFETIME` (Optional) - The number of hours until a newly created token expires
- `SOAR_TOKEN_SECRET` (Optional) - A secret key used to encrypt/decrypt token data. If specified the value should be set using TokenModel.getKey()

### Running the bus and API natively (without docker)

#### Setup 

We recommend using some form of python virtual environment to maintain a consistent 
python version and ring-fence the package management.

In your virtual environment: 

```
pip install -r requirements-dev.txt  
```

This installs both the development and runtime dependencies for the project.

### Testing 

Current coverage: 

- API: yes 
- Pika RabbitMQ implementation: no

In your virtual environment: 

```
pytest
```

#### RabbitMQ 

`docker run --rm -p 5672:5672 -d --hostname rmq --name rmq rabbitmq:management`

#### API 

In your virtual environment:

```
python api.py 
```

#### Event bus 

In your virtual environment: 

``` 
python soar_bus.py
```

At present the soar bus creates the clients defined in the API when it starts 
but does not monitor for new clients so if you create a new client you will 
need to restart the bus. This will be fixed in a later release.

### Client Adapters 

The intended use of the backbone is that multiple clients connect to the backbone 
using adapters. An adapter handles: 

- Authentication 
- Sending and receiving over the backbone API 
- Transforming messages between local formats and the backbone message formats

We have implemented the following template client adapters: 

- [Javascript](https://git.noc.ac.uk/communications-backbone-system/backbone-adapter-javascript)
- [Python](https://git.noc.ac.uk/communications-backbone-system/backbone-adapter-python)

If you need to port the adapter to another language please contact us. 

The adapters can be installed as packages and sub-classed as required for your 
client application.

For install and usage instructions for the adapters see the READMEs. 

### Usage

- Run the backbone 
- Create some clients
- Restart the soar_bus service 
  - `cd docker && docker-compose restart soar_bus` 
- Save your client credentials 
- Test reading and writing directly to the queues
- Create an adapter 
- Test sending and receiving via the adapter    

#### Create some clients

##### With the script 

```bash 
python client_create.py
# will create a default 'admin' client subscribed to all messages (#)
# OR 
python client_create.py --id=[client_id] --name="Your Client Name" --sub="something.something.#"
# will create a client with your preferred name, id and subscription
```

##### Through the API 

`POST` to `http://localhost:8087/client`

The POST body should be JSON. 

```json
{ 
  "client_id": "noc-c2",
  "client_name": "NOC C2", 
  "subscription": "soar.*.noc.#"
}
```

- `client_id` - a project unique human readable name 
- `client_name` - how to refer to that client on screen 
- `subscription` - the topic pattern identifying the messages you want to receive 

(`*` = single word wildcard, `#` = multi-word wildcard)

The response from the post contains your client `secret`. This is only displayed once. 

Subsequent GETS to `/client` or `/client/[client-id]` will not return the secret. 

You should save the response as `soar-config.json` adding the additional `api` field 
to specify the root URL of the API you're connecting to 
(eg http://localhost:8087 for local development)

#### Send / Receive directly 

```
# Send a message 
python client_send.py noc-c2-outbox 'soar.noc.slocum.something' from noc-c2
```

```
# Receive messages  
python client_read.py noc-sfmc-inbox
```

These scripts bypass authentication reading and writing directly to rabbitmq.

#### Authentication 

Authentication is handled by a client credentials grant which is a GET 
request to `/token?client_id=[client-id]&secret=[secret]`

The response includes a token and this token should be included as an 
authorization header in requests to the API: 

```json
{
  "Authorization": "Bearer [token]"
}
```

#### Send and Receive via API

Once you have a bearer token you can make requests to send and receive 
messages via the API: 

- GET `/receive?max_messages=[X]` - gets all messages in the queue 
  - the default value of `max_messages` if not specified is 10
- POST `/send` - publish a message to a given topic 
- POST `/notify` - broadcast a message to all clients 

Supplying the token as an Authorization header as described above.

#### Send and Receive using adapters 

If you have implemented one of the adapter templates then authentication 
is handled for you.

You will need to create a set of client credentials as above using the 
POST `/client` endpoint and save the response. 

Then you pass your credentials when you create your adapter instance.