cisagov/gatherer repository overview

⁠Gatherer

⁠Docker Image

This is a Docker container that uses the cisagov fork⁠ of 18F's domain-scan⁠ to gather domains as a precursor to scanning by pshtt⁠, trustymail⁠, and sslyze⁠.

This Docker container is intended to be run via orchestrator⁠.

N.B.: The secrets in the src/secrets directory are only used when testing via the compose.yml composition. Normally this Docker container is run via the Docker composition in cisagov/orchestrator⁠, which expects the secrets in a different location.

⁠Running

⁠Running with Docker

To run the cisagov/gatherer image via Docker:

docker run cisagov/gatherer:2.0.4

⁠Running with Docker Compose

1. Create a compose.yml file similar to the one below to use Docker Compose⁠.

   ---
   name: gatherer
   
   services:
     gatherer:
       image: cisagov/gatherer:2.0.4
       volumes:
         - source: <your_output_dir>
           target: /home/cisa/shared
           type: bind
   

   Copy

2. Start the container and detach:

   docker compose up --detach
   

   Copy

⁠Using secrets with your container

This container also supports passing sensitive values via Docker secrets⁠. Passing sensitive values like your credentials can be more secure using secrets than using environment variables. See the secrets⁠ section below for a table of all supported secret files.

1. To use secrets, create a database_creds.yml file in this format⁠:

   ---
   version: '1'
   
   database:
     name: cyhy
     uri: mongodb://readonly:[email protected]:27017/cyhy
   
   

   Copy

2. Then add the secret to your compose.yml file:

   ---
   name: gatherer
   
   secrets:
     database_creds:
       file: database_creds.yml
   
   services:
     gatherer:
       image: cisagov/gatherer:2.0.4
       secrets:
         - source: database_creds
           target: database_creds.yml
       volumes:
         - source: <your_output_dir>
           target: /home/cisa/shared
           type: bind
   

   Copy

⁠Updating your container

⁠Docker Compose

1. Pull the new image from Docker Hub:

   docker compose pull
   

   Copy

2. Recreate the running container by following the previous instructions⁠:

   docker compose up --detach
   

   Copy

⁠Docker

1. Stop the running container:

   docker stop <container_id>
   

   Copy

2. Pull the new image:

   docker pull cisagov/gatherer:2.0.4
   

   Copy

3. Recreate and run the container by following the previous instructions⁠.

⁠Updating Python dependencies

This image uses Pipenv⁠ to manage Python dependencies using a Pipfile⁠. Both updating dependencies and changing the Pipenv⁠ configuration in src/Pipfile will result in a modified src/Pipfile.lock file that should be committed to the repository.

⁠Updating dependencies

If you want to update existing dependencies you would run the following command in the src/ subdirectory:

pipenv lock

⁠Modifying dependencies

If you want to add or remove dependencies you would update the src/Pipfile file and then update dependencies as you would above.

⁠Image tags

The images of this container are tagged with semantic versions⁠ of the underlying gatherer project that they containerize. It is recommended that most users use a version tag (e.g. :2.0.4).

See the tags tab⁠ on Docker Hub for a list of all the supported tags.

⁠Volumes

⁠Ports

There are no ports exposed by this container.

⁠Environment variables

⁠Required

There are no required environment variables.

⁠Optional

There are no optional environment variables.

⁠Secrets

⁠Building from source

Build the image locally using this git repository as the build context⁠:

docker build \
  --tag cisagov/gatherer:2.0.4 \
  https://github.com/cisagov/gatherer.git#develop

⁠Cross-platform builds

To create images that are compatible with other platforms, you can use the buildx⁠ feature of Docker:

1. Copy the project to your machine using the Code button above or the command line:

   git clone https://github.com/cisagov/gatherer.git
   cd gatherer
   

   Copy

2. Create the Dockerfile-x file with buildx platform support:

   ./buildx-dockerfile.sh
   

   Copy

3. Build the image using buildx:

   docker buildx build \
     --file Dockerfile-x \
     --platform linux/amd64 \
     --output type=docker \
     --tag cisagov/gatherer:2.0.4 .
   

   Copy

⁠Contributing

We welcome contributions! Please see CONTRIBUTING.md⁠ for details.

⁠License

This project is in the worldwide public domain⁠.

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication⁠.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.