Communications Backbone

Communications Backbone by C2 Team (NOC)

Infrastructure

The backbone has 3 runtime components:

• RabbitMQ - run as a docker container
• The bus which handles the delivery of messages between queues
• The API which provides an interface to read from and write to the queues

Data flow

• Client A sends to client-a-outbox (by POSTing to API /send)
• Messages are forwarded from client-a-outbox to soar-publish
• Messages are published from soar-publish with the given topic read from the message
• Subscribers listen to for messages
• Subscription is delivered to client-b-inbox (if the client subscription matches the message topic)
• Client B reads from client-b-inbox (by GETting from API /receive)

There is a parallel flow when a client sends to client-a-broadcast (by POSTing to /notify). In this case the messages are delivered through the broadcast exchange to all clients client-x-inbox.

Prerequisites

• Python >= 3.8
• A virtual environment manager - virtualenv, venv or pipenv
• Docker and docker-compose

Running via docker-compose

run-compose.sh

Using docker-compose will mean that everything is setup automatically, this includes the rabbitmq container, the backbone API, and the backbone bus. The run-compose.sh script has been provided to simplify this even further - all you have to do is set whatever env vars you need in the .env file and then run ./run-compose.sh (the defaults in .env are fine for local dev work, but ones not labelled optional will need setting in a production setting). The env vars are:

• DATA_DIR - Where to mount the volume of the API container on your local system. This defaults to the result of pwd, which should be within the communications-backbone repo
• SOAR_TOKEN_LIFETIME (Optional) - The number of hours until a newly created token expires
• SOAR_TOKEN_SECRET (Optional) - A secret key used to encrypt/decrypt token data. If specified the value should be set using TokenModel.getKey()

Running the bus and API natively (without docker)

Setup

We recommend using some form of python virtual environment to maintain a consistent python version and ring-fence the package management.

In your virtual environment:

pip install -r requirements-dev.txt  

This installs both the development and runtime dependencies for the project.

Testing

Current coverage:

• API: yes
• Pika RabbitMQ implementation: no

In your virtual environment:

pytest

RabbitMQ

docker run --rm -p 5672:5672 -d --hostname rmq --name rmq rabbitmq:management

API

In your virtual environment:

python api.py 

Config

The API reads it's config from ./data/api-config.json if present.

The default config for development environments only enables connections from local host on any port http://localhost:*.

If you want to open the API up to connections from anywhere or from specific known client domains you can create an api-config.json to do that.

This can also be used to limit requests to specified endpoints.

The default config is intended for running a local development environment. For production it is expected that the config file will exist with settings like the following:

{
  "cors": {
    "*": {
      "origins": [
        "*"
      ]
    },
    "/client": {
      "origins": [
        "http://localhost:*"
      ]
    }
  }
}

This opens up the all endpoints except for the /client endpoint. For now we expect client administration to be managed centrally.

Event bus

In your virtual environment:

python soar_bus.py

At present the soar bus creates the clients defined in the API when it starts but does not monitor for new clients so if you create a new client you will need to restart the bus. This will be fixed in a later release.

Client Adapters

The intended use of the backbone is that multiple clients connect to the backbone using adapters. An adapter handles:

• Authentication
• Sending and receiving over the backbone API
• Transforming messages between local formats and the backbone message formats

We have implemented the following template client adapters:

• Javascript
• Python

If you need to port the adapter to another language please contact us.

The adapters can be installed as packages and sub-classed as required for your client application.

For install and usage instructions for the adapters see the READMEs.

Usage

• Run the backbone
• Create some clients
• Restart the soar_bus service
  • cd docker && docker-compose restart soar_bus
• Save your client credentials
• Test reading and writing directly to the queues
• Create an adapter
• Test sending and receiving via the adapter

Create some clients

With the script

python client_create.py
# will create a default 'admin' client subscribed to all messages (#)
# OR 
python client_create.py --id=[client_id] --name="Your Client Name" --sub="something.something.#"
# will create a client with your preferred name, id and subscription

Through the API

POST to http://localhost:8087/client

The POST body should be JSON.

{ 
  "client_id": "noc-c2",
  "client_name": "NOC C2", 
  "subscription": "soar.*.noc.#"
}

• client_id - a project unique human readable name
• client_name - how to refer to that client on screen
• subscription - the topic pattern identifying the messages you want to receive

(* = single word wildcard, # = multi-word wildcard)

The response from the post contains your client secret. This is only displayed once.

Subsequent GETS to /client or /client/[client-id] will not return the secret.

You should save the response as soar-config.json adding the additional api field to specify the root URL of the API you're connecting to (eg http://localhost:8087 for local development)

Send / Receive directly

# Send a message 
python client_send.py noc-c2-outbox 'soar.noc.slocum.something' from noc-c2

# Receive messages  
python client_read.py noc-sfmc-inbox

These scripts bypass authentication reading and writing directly to rabbitmq.

Authentication

Authentication is handled by a client credentials grant which is a GET request to /token?client_id=[client-id]&secret=[secret]

The response includes a token and this token should be included as an authorization header in requests to the API:

{
  "Authorization": "Bearer [token]"
}

Send and Receive via API

Once you have a bearer token you can make requests to send and receive messages via the API:

• GET /receive?max_messages=[X] - gets all messages in the queue
  • the default value of max_messages if not specified is 10
• POST /send - publish a message to a given topic
• POST /notify - broadcast a message to all clients

Supplying the token as an Authorization header as described above.

Send and Receive using adapters

If you have implemented one of the adapter templates then authentication is handled for you.

You will need to create a set of client credentials as above using the POST /client endpoint and save the response.

Then you pass your credentials when you create your adapter instance.