Install the Router in Kubernetes

ziti-router

Host an OpenZiti router in Kubernetes

Add the OpenZiti Charts Repo to Helm

helm repo add openziti https://docs.openziti.io/helm-charts/

Public Router

The default configuration listens for incoming edge connections and router links. Set a public address for this listener (edge.advertisedHost) or disable it (linkListeners.transport.enabled) to avoid routers continually failing to dial into it.

# get a router enrollment token from the controller's management API
ziti edge create edge-router "router1" \
  --tunneler-enabled --jwt-output-file /tmp/router1.jwt

# subscribe to the openziti Helm repo
helm repo add openziti https://openziti.github.io/helm-charts/

# install the router chart with a public address
helm upgrade --install \
  "ziti-router-123456789" \
  openziti/ziti-router \
    --set-file enrollmentJwt=/tmp/router1.jwt \
    --set ctrl.endpoint=ctrl.ziti.example.com:443 \
    --set edge.advertisedHost=router1.ziti.example.com \

Ingress TLS Passthrough

All router TLS listeners must terminate TLS, so it's essential that Ingress resources use TLS passthrough.

This example demonstrates creating TLS pass-through Ingress resources for use with ingress-nginx.

Ensure you have the ingress-nginx chart installed with controller.extraArgs.enable-ssl-passthrough=true. You can verify this feature is enabled by running kubectl describe pods {ingress-nginx-controller pod} and checking the args for --enable-ssl-passthrough=true.

If not enabled, then you must patch the ingress-nginx deployment to enable the SSL passthrough option.

kubectl patch deployment "ingress-nginx-controller" \
    --namespace ingress-nginx \
    --type json \
    --patch '[{"op": "add",
        "path": "/spec/template/spec/containers/0/args/-",
        "value":"--enable-ssl-passthrough"
    }]'

# subscribe to ingress-nginx
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx/

# install ingress-nginx
helm install \
  --namespace ingress-nginx --create-namespace --generate-name \
  ingress-nginx/ingress-nginx \
    --set controller.extraArgs.enable-ssl-passthrough=true

Create a Helm chart values file for this router chart.

# router-values.yml
ctrl:
  endpoint: ziti-controller-ctrl.ziti-controller.svc:1280
advertisedHost: router1.ziti.example.com
edge:
  advertisedPort: 443
  service:
    type: ClusterIP
  ingress:
    enabled: true
    ingressClassName: nginx
    annotations:
      kubernetes.io/ingress.allow-http: "false"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"

Now upgrade your router chart release with the values file.

helm upgrade --install \
  "ziti-router-1" \
  openziti/ziti-router \
    --set-file enrollmentJwt=/tmp/router1.jwt \
    --values /tmp/router-values.yml

Private Router

Disable the link listener if the router does not have a public address set (edge.advertisedHost). Ziti identities inside the cluster can still use the private router's edge listener ClusterIP service by authorizing them with a Ziti edge router policy.

helm upgrade --install \
  "ziti-router-1" \
  openziti/ziti-router \
    --set-file enrollmentJwt=/tmp/router1.jwt \
    --set ctrl.endpoint=ctrl.ziti.example.com:443 \
    --set linkListeners.transport.enabled=false

Tunnel Modes

Host tunnel mode

Default: tunnel.mode=none

Host mode enables a router's identity to reverse proxy Ziti service traffic to a target address on the regular network. Enable this mode by updating the router's identity in the controller to enable tunneling, then set tunnel.mode=host and upgrade the Helm release to start hosting Ziti services.

ziti edge update identity "router1" --tunneler-enabled

Proxy tunnel mode

tunnel.mode=proxy

Proxy mode enables the router to publish Ziti services as Kubernetes services.

Here's an example router values' snippet to merge with your other values:

tunnel:
  mode: proxy
  proxyServices:
    # this will be bound on the "default" proxy Kubernetes service, see below
    - zitiService: my-ziti-service.svc
      containerPort: 10443
      advertisedPort: 10443
    # this will be bound on an additionally configured proxy Kubernetes service, see below
    - zitiService: my-other-service.svc
      containerPort: 10022
      advertisedPort: 10022
  proxyDefaultK8sService:
    enabled: true
    type: ClusterIP
  proxyAdditionalK8sServices:
    - name: myservice
      type: LoadBalancer
      annotations:
        metallb.universe.tf/loadBalancerIPs: 192.168.1.100

Additional Listeners and Volumes

You can configure an additional edge listener by setting edge.additionalListeners. This is useful for making a WebSocket edge listener available for BrowZer clients that require a trusted server certificate.

This example configures a wss listener and requests a certificate from cert-manager. The alternative certificate must have a DNS SAN that is distinct from the public address of the default edge listener (edge.advertisedHost). This cert-manager approach has the advantage of automatically renewing the certificate and ensuring the DNS SAN of the certificate matches an additional listener's advertised host.

edge:
  advertisedHost: router1.ziti.example.com
  advertisedPort: 443
  ingress:
    annotations:
      kubernetes.io/ingress.allow-http: "false"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    enabled: true
    ingressClassName: nginx
  service:
    enabled: true
    type: ClusterIP
  additionalListeners:
    - name: router1-edge-wss
      protocol: wss
      containerPort: 3023                           # must be unique
      advertisedHost: router1-wss.ziti.example.com  # must be distinct from edge.advertisedHost
      advertisedPort: 443
      addHostToSan: false                           # must be false to avoid colliding DNS SANs between listeners
      service:
        enabled: true
        type: ClusterIP
      ingress:
        enabled: true
        annotations:
          kubernetes.io/ingress.allow-http: "false"
          nginx.ingress.kubernetes.io/ssl-passthrough: "true"
        ingressClassName: nginx

identity:
  altServerCerts:
    - name: alt-server-cert-1
      mode: certManager
      secretName: ziti-router1-alt-server-certs1
      additionalListenerName: router1-edge-wss
      mountPath: /etc/ziti/alt-server-cert-1
      issuerRef:
        group: cert-manager.io
        kind: ClusterIssuer
        name: cloudflare-dns01-issuer-staging

You don't have to use cert-manager. If you have a TLS secret named ziti-router1-alt-server-certs1 from some other issuer in the same namespace as the router containing the certificate and key, you can use it by setting values like these. You must also configure the additional listener as in the prior example with an advertisedHost that matches a DNS SAN of the alternative certificate.

# this is an generic approach for mounting configmaps, secrets, csi volumes, etc.
additionalVolumes:
  - name: alt-server-cert-2
    volumeType: secret
    mountPath: /etc/ziti/alt-server-cert-2
    secretName: ziti-router1-alt-server-cert-2

# this looks up a TLS secret's mountpoint to configure the router's identity
identity:
  altServerCerts:
    - mode: secret
      secretName: ziti-router1-alt-server-cert-2

You may also specify matching file paths for an additional volume and alternative certificate if the volume is not a TLS secret.

additionalVolumes:
  - name: alt-server-cert-3
    volumeType: csi
    driverName: csi.bpfd.dev
    attributes: volumeAttributes
    mountPath: /etc/ziti/alt-server-cert-3

identity:
  altServerCerts:
    - mode: localFile
      serverCert: /etc/ziti/alt-server-cert-3/server3.crt
      serverKey: /etc/ziti/alt-server-cert-3/server3.key

Values Reference

Key	Type	Default	Description
additionalVolumes	list	[]	additional volumes to mount to ziti-router container
affinity	object	{}	deployment template spec affinity
configFile	string	"ziti-router.yaml"	filename of router config YAML
configMountDir	string	"/etc/ziti/config"	writeable mountpoint where read-only config file is projected to allow router to write ./endpoints statefile in same dir
csr	object	{"country":"","locality":"","organization":"","organizationalUnit":"","province":"","sans":{"dns":[],"email":[],"ip":[],"noDefaults":false,"uri":[]}}	Certificate signing request distinguished name and subject alternative names
csr.country	string	""	country
csr.locality	string	""	city
csr.organization	string	""	organization
csr.organizationalUnit	string	""	organizational unit
csr.province	string	""	state
csr.sans.dns	list	[]	additional DNS SANs
csr.sans.email	list	[]	additional email SANs
csr.sans.ip	list	[]	additional IP SANs
csr.sans.noDefaults	bool	false	if true, disable computing default SANs from the advertisedHost, etc.
csr.sans.uri	list	[]	additional URI SANs
ctrl.endpoint	string	""	required control plane endpoint, e.g., ctrl.ziti.example.com:443
dnsConfig	object	{}	it allows to override dns options when dnsPolicy is set to None.
dnsPolicy	string	"ClusterFirstWithHostNet"
edge.additionalListeners	list	[]	additional edge listeners have the same shape as the default edge listener, except they're enabled if defined, and you must specify a unique name for each additional edge listener. The name distinguishes their respective cluster services. This is useful for BrowZer clients that require a trusted certificate for the edge WebSocket and advertisedHost must resolve to a public IP that presents a trusted certificate, e.g., an Ingress with TLS termination.
edge.advertisedHost	string	""	Domain name that edge clients will use to reach this router's edge listener. Default is cluster-internal service DNS name:port.
edge.advertisedPort	int	443	cluster service, node port, load balancer, and ingress port
edge.containerPort	int	3022	cluster service target port on the container
edge.enabled	bool	true	enable the edge listener in the router config; usually true because tunnel bindings require the edge which must have at least on listener
edge.ingress.annotations	object	{}	ingress annotations, e.g., to configure ingress-nginx for passthrough TLS
edge.ingress.enabled	bool	false	create an ingress for the cluster service; typically paired with a ClusterIP service type when enabled
edge.ingress.ingressClassName	string	""	ingress class name
edge.ingress.labels	object	{}	ingress labels
edge.options	object	{}	additional common xgress options
edge.protocol	string	"tls"	edge listener protocol: tls, wss; usually tls because additionalListeners can be used to provide a wss listener
edge.service.annotations	object	{}	service annotations
edge.service.enabled	bool	true	create a cluster service for the edge listener; usually true, but you can disable this to effectively un-publish the edge listener
edge.service.labels	object	{}	service labels
edge.service.type	string	"ClusterIP"	expose the service as a ClusterIP, NodePort, or LoadBalancer; default is ClusterIP, but you could use NodePort or LoadBalancer instead of an ingress controller
enrollmentJwt	string	nil	enrollment one time token from the controller's management API
env	object	{}	assign key=value in pod environment
execMountDir	string	"/usr/local/bin"	read-only mountpoint for executables (must be in image's executable search PATH)
fabric.metrics.enabled	bool	false	configure fabric metrics in the router config
forwarder.latencyProbeInterval	int	10
forwarder.linkDialQueueLength	int	1000
forwarder.linkDialWorkerCount	int	32
forwarder.rateLimitedQueueLength	int	5000
forwarder.rateLimitedWorkerCount	int	64
forwarder.xgressDialQueueLength	int	1000
forwarder.xgressDialWorkerCount	int	128
ha.enabled	bool	false	must be enabled if multiple controllers
hostNetwork	bool	false	Host networking requested for a pod if set, i.e. tproxy ports enabled in the host namespace. i.e. egress gateway
identity.altServerCerts	list	[]
identityMountDir	string	"/etc/ziti/identity"	read-only mountpoint for router identity secret specified in deployment for use by router run container
image.additionalArgs	list	[]	additional arguments can be passed directly to the container to modify ziti runtime arguments
image.args	list	["run","{{ .Values.configMountDir }}/{{ .Values.configFile }}"]	deployment container command args and opts
image.command	list	["/entrypoint.bash"]	deployment container command
image.pullPolicy	string	"Always"	deployment image pull policy
image.repository	string	"docker.io/openziti/ziti-router"	container image tag for deployment
image.tag	string	nil	container image tag (default is Chart's appVersion)
linkListeners.transport.advertisedHost	string	"{{ .Values.edge.advertisedHost }}"	DNS name that other routers will use to form mesh transport links to this listener
linkListeners.transport.advertisedPort	string	"{{ .Values.edge.advertisedPort }}"	cluster service, node port, load balancer, and ingress port for this listener; default is edge.advertisedPort
linkListeners.transport.containerPort	string	"{{ .Values.edge.containerPort }}"	cluster service target port on the container; default is to share a listener with the edge, providing ziti-link ALPN
linkListeners.transport.enabled	bool	true	enable the transport listener in the router config; set false for a private router that only connects to other routers and does not accept incoming links
linkListeners.transport.ingress.annotations	object	{}	ingress annotations, e.g., to configure ingress-nginx
linkListeners.transport.ingress.enabled	bool	false	create an ingress for the cluster service
linkListeners.transport.ingress.ingressClassName	string	""	ingress class name
linkListeners.transport.ingress.labels	object	{}	ingress labels
linkListeners.transport.options	object	{}	link listener options
linkListeners.transport.service.annotations	object	{}	service annotations
linkListeners.transport.service.enabled	bool	true	create a cluster service for the router transport link listener; unnecessary if advertisedHost is shared with edge listener (the default)
linkListeners.transport.service.labels	object	{}	service labels
linkListeners.transport.service.type	string	"ClusterIP"	expose the service as a ClusterIP, NodePort, or LoadBalancer
nodeSelector	object	{}	deployment template spec node selector
persistence.accessMode	string	"ReadWriteOnce"	PVC access mode: ReadWriteOnce (concurrent mounts not allowed), ReadWriteMany (concurrent allowed)
persistence.annotations	object	{}	annotations for the PVC
persistence.enabled	bool	true	required: place a storage claim for the ctrl endpoints state file
persistence.existingClaim	string	""	A manually managed Persistent Volume and Claim Requires persistence.enabled: true If defined, PVC must be created manually before volume will be bound
persistence.size	string	"50Mi"	50Mi is plenty for this state file
persistence.storageClass	string	""	Storage class of PV to bind. By default it looks for the default storage class. If the PV uses a different storage class, specify that here.
persistence.volumeName	string	nil	PVC volume name
podAnnotations	object	{}	annotations to apply to all pods deployed by this chart
podSecurityContext	object	{"fsGroup":2171}	deployment template spec security context
podSecurityContext.fsGroup	int	2171	this is the GID of "ziggy" run-as user in the container that has access to any files created by the router process in the emptyDir volume used to persist the list of ctrl endpoints
proxy	object	{}	Explicit proxy setting in the router configuration. Router can be deployed in a site where all egress traffic is forwarded through an explicit proxy. The enrollment will also be forwarded through the proxy.
resources	object	{}	deployment container resources
securityContext	string	nil	deployment container security context
tolerations	list	[]	deployment template spec tolerations
tunnel.diverterPath	string	nil	the tproxy mode can be switched from iptables based interception to bpf interception by passing the user space bpf program path. bpf kernel space program is expected to be loaded prior or during router deployment, e.g. bpfman agent, hostpath, etc
tunnel.dnsSvcIpRange	string	nil	CIDR range for the internal service fqdn to dynamic intercept IP address resolution (default: 100.64.0.0/10)
tunnel.lanIf	string	"lo"	interface device name for setting up INPUT firewall rules if fw enabled. It must be set but not needed in containers. Thus, it is set to lo by default
tunnel.mode	string	"none"	run mode for the router's built-in tunnel component: host, tproxy, proxy, or none
tunnel.proxyAdditionalK8sServices	list	[]	if tunnel mode is "proxy", create a separate cluster service for each Ziti service listed in "proxyServices" which k8sService == name
tunnel.proxyDefaultK8sService	object	{"enabled":true,"type":"ClusterIP"}	if tunnel mode is "proxy", create the a cluster service named {{ release }}-proxy-default listening on each "advertisedPort" defined in "proxyServices"
tunnel.proxyServices	list	[]	list of Ziti services for which K8s services are to be created by this deployment, default is one cluster service port per Ziti service
tunnel.resolver	string	nil	Ziti nameserver listener where OS must be configured to send DNS queries (default: udp://127.0.0.1:53)
websocket.enableCompression	bool	true	enable compression on websocket
websocket.enabled	bool	false	enable the websocket transport. Also requires an appropriate edge.additionalListeners entry.
websocket.handshakeTimeout	int	10	websocket handshake timeout
websocket.idleTimeout	int	5	websocket idle timeout
websocket.pingInterval	int	54	websocket ping timeout
websocket.pongTimeout	int	60	websocket pong timeout
websocket.readBufferSize	int	4096	websocket read buffer size
websocket.readTimeout	int	5	websocket read timeout
websocket.writeBufferSize	int	4096	websocket write buffer size
websocket.writeTimeout	int	10	websocket write timeout