goprobe/query repository overview

⁠Introduction

Today, targeted analyses of network traffic patterns have become increasingly difficult due to the sheer amount of traffic encountered. To enable them, traffic needs to be captured and examined and broken down to key descriptors which yield a condensed explanation of the underlying data.

The NetFlow standard was introduced to address this reduction. It uses the concept of flows, which combine packets based on a set of shared packet attributes. NetFlow information is usually captured on one device and collected in a central database on another device. Several software probes are available, implementing NetFlow exporters and collectors.

goProbe deviates from traditional NetFlow as flow capturing and collection is run on the same device and the flow fields reduced. It was designed as a lightweight, standalone system, providing both optimized low-footprint packet capture and a storage backend tailored to the flow data in order to provide lightning-fast analysis queries.

⁠Scope of this image

This Docker image provides the query component (i.e. global-query) in order to perform distributed queries to various instances of the sensor component.

⁠Usage (Docker)

In order to spin up a running instance of a global-query instance, simply run the query image (note that the --network host setting is only required if the sensor⁠ is running on the same host):

docker run --rm --name global_quqery --network host -e CONFIG_PATH=/app/query.yaml \
           -v ./examples/docker/query.example.yaml:/app/query.yaml:ro \
           -v ./examples/docker/querier.example.yaml:/app/goprobe-global-querier.yaml:ro \
           goprobe/query:latest

⁠Usage (Compose)

A full docker-compose.yaml (including build parameters) can be found in the GitHub repository⁠. A minimal example could look like this:

services:
  query:
    image: goprobe/query:latest
    restart: unless-stopped
    container_name: global_query
    environment:
      - SERVER_ADDR=127.0.0.1:8145
      - HOSTS_RESOLVER_TYPE=string
      - QUERIER_TYPE=api
      - QUERIER_MAX_CONCURRENT=32
      - QUERIER_CONFIG=/app/goprobe-global-querier.yaml
    volumes:
      - ./querier.yaml:/app/goprobe-global-querier.yaml:ro

For an example configuration file, take a look here⁠.

⁠API / Querying

The example configuration will provide an API endpoint on http://127.0.0.1:8146/⁠ (including an auto-generated documentation endpoint under http://127.0.0.1:8146/docs⁠).

The interface can be used to run some simple queries, e.g. using the following payload for the query⁠ endpoint (assuming both sensor⁠ and the query containers are running and using the example configuration provided here⁠):

{
  "caller": "goQuery",
  "format": "json",
  "host_id": 123456,
  "hostname": "localhost",
  "ifaces": "lo",
  "first": "-24h",
  "live": true,
  "query": "sip,dip,dport,proto",
  "query_hosts": "localhost"
}