Integrate Kubernetes and pfSense
100K+
kubernetes-pfsense-controller (kpc) works hard to keep pfSense and Kubernetes in sync and harmony. The primary focus is to facilitate a first-class Kubernetes cluster by integrating and/or implementing features that generally do not come with bare-metal installation(s).
This is generally achieved using the standard Kubernetes API along with the xmlrpc API for pfSense. Speaking generally
the Kubernetes API is watched and then updates to the pfSense config.xml are sent via xmlrpc calls along with
appropriate reload/restart/update/sync methods to apply changes.
Disclaimer: this is new software bound to have bugs. Please make a backup before using it as it may eat your configuration. Having said that, all known code paths appear to be solid and working without issue. If you find a bug, please report it!
The controller is comprised of several plugins that are enabled/disabled/configured via a Kubernetes ConfigMap. Details about each plugin follows below.
MetalLB implements LoadBalancer type Services in Kubernetes. This is done via any
combination of Layer2 or BGP type configurations. Layer2 requires no integration with pfSense, however, if you want to
leverage the BGP implementation you need a BGP server along with neighbor configuration. kpc dynamically updates
bgp neighbors for you in pfSense by continually monitoring cluster Nodes.
The plugin assumes you've already installed openbgp and configured it as well as created a group to use with MetalLB.
metallb:
enabled: true
nodeLabelSelector:
nodeFieldSelector:
bgp-implementation: openbgp
options:
openbgp:
template:
md5sigkey:
md5sigpass:
groupname: metallb
row:
- parameters: announce all
parmvalue:
haproxy-declarative plugin allows you to mostly statically create HAProxy frontend/backend definitions as
ConfigMap resources in the cluster. When declaring backends however, the pool of servers can (will) be dynamically
created/update based on cluster nodes. See declarative-example.yaml for an
example.
haproxy-declarative:
enabled: true
haproxy-ingress-proxy plugin allows you to mirror cluster ingress rules handled by an ingress controller to HAProxy
running on pfSense. If you run pfSense on the network edge with non-cluster services already running, you now can
dynamically inject new rules to route traffic into your cluster while simultaneously running non-cluster services.
Combined with haproxy-declarative you can create a dynamic backend service (ie: your ingress controller) and
subsequently dynamic frontend services based off of cluster ingresses.
haproxy-ingress-proxy:
enabled: true
ingressLabelSelector:
ingressFieldSelector:
defaultFrontend: http-80
defaultBackend: traefik
#allowedHostRegex: "/.*/"
kpc provides various options to manage DNS entries in pfSense based on cluster state. Note that these options can be
used in place of or in conjunction with external-dns to support
powerful setups/combinations.
pfsense-dns-services watches for services of type LoadBalancer that have the annotation dns.pfsense.org/hostname
with the value of the desired hostname. kpc will create the DNS entry in unbound/dnsmasq. Note that to actually get
an IP on these services you'll likely need MetalLB deployed in the cluster (regardless of the metallb plugin running
or not).
pfsense-dns-services:
enabled: true
serviceLabelSelector:
serviceFieldSelector:
#allowedHostRegex: "/.*/"
dnsBackends:
dnsmasq:
enabled: true
unbound:
enabled: true
pfsense-dns-ingresses watches ingresses and automatically creates DNS entries in unbound/dnsmasq. This requires proper
support from the ingress controller to set IPs on the ingress resources.
pfsense-dns-ingresses:
enabled: true
ingressLabelSelector:
ingressFieldSelector:
#allowedHostRegex: "/.*/"
dnsBackends:
dnsmasq:
enabled: true
unbound:
enabled: true
pfsense-dns-haproxy-ingress-proxy monitors the HAProxy rules created by the haproxy-ingress-proxy plugin and creates
host aliases for each entry. To do so you create an arbitrary host in unbound/dnsmasq (something like
<frontend name>.k8s) and bind that host to the frontend through the config option frontends.<frontend name>. Any
proxy rules created for that frontend will now automatically get added as aliases to the configured hostname. Make
sure the static hostname created in your DNS service of choice points to the/an IP bound to the corresponding
frontend.
pfsense-dns-haproxy-ingress-proxy:
enabled: true
#allowedHostRegex: "/.*/"
dnsBackends:
dnsmasq:
enabled: true
unbound:
enabled: true
frontends:
http-80:
hostname: http-80.k8s
primary_frontend_name2:
hostname: primary_frontend_name2.k8s
regex parameters are passed through php's preg_match() method, you can test your syntax using that. Also note that
if you want to specify a regex ending ($), you must escape it in yaml as 2 $
(ie: #allowedHostRegex: "/.example.com$$/").
kpc stores it's stateful data in the cluster as a ConfigMap (kube-system.pfsense-controller-store by default). You
review the data there to gain understanding into what the controller is managing.
You may need/want to bump up the webConfigurator setting for Max Processes to ensure enough simultaneous connections
can be established. Each kpc instance will only require 1 process (ie: access to the API is serialized by kpc).
XML config structure (note that ha_backends is actually frontends...it's badly named):
haproxy
ha_backends
item
item
...
ha_pools
item
ha_servers
item
item
...
item
...
Content type
Image
Digest
sha256:4d1e2bcf1…
Size
34.5 MB
Last updated
over 3 years ago
docker pull travisghansen/kubernetes-pfsense-controller:v0.5.14