Running Portainer and a edge agent behind a ingress with https

[darkeagle1337]

Hi all,

this is not a bug or a problem, I'd like to share my analysis and solution with you, so other people can find this and adopt it to their solution, also maybe it's something for the portainer agent documentation.

Goal

Portainer UI and API accessible from portainer.example.io and edge agent tunnel working against agent.portainer.example.io.

Step 1: Install Portainer

First I installed portainer as described in the (docs). I am using nginx as the ingress controller.

helm upgrade --install --create-namespace -n portainer portainer portainer/portainer \
    --set service.type=ClusterIP \
    --set tls.force=true \
    --set image.tag=2.21.5 \
    --set ingress.enabled=true \
    --set ingress.ingressClassName=nginx> \
    --set ingress.annotations."nginx\.ingress\.kubernetes\.io/backend-protocol"=HTTPS \
    --set ingress.hosts[0].host=portainer.example.io> \
    --set ingress.hosts[0].paths[0].path="/"

Then I add this ingress for the agent:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: portainer-agent
  namespace: portainer
  annotations:
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/rewrite-target: "/"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"  # 🚀 **Disables automatic HTTPS redirection**
    
    # WebSocket specific annotations
    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-set-header-Upgrade: "$http_upgrade"
    nginx.ingress.kubernetes.io/proxy-set-header-Connection: '"Upgrade"'
    nginx.ingress.kubernetes.io/proxy-cache-bypass: "$http_upgrade"
spec:
  ingressClassName: nginx
  rules:
  - host: agent.portainer.example.io
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: portainer
            port:
              number: 8000
  tls:
  - hosts:
    - agent.portainer.example.io
    secretName: agent-portainer-tls

The connection also allows HTTP connections, so we had to set the nginx.ingress.kubernetes.io/ssl-redirect annotation to false.
The important things are the following annotations. For more information https://nginx.org/en/docs/http/websocket.html

    nginx.ingress.kubernetes.io/proxy-set-header-Upgrade: "$http_upgrade"
    nginx.ingress.kubernetes.io/proxy-set-header-Connection: '"Upgrade"'
    nginx.ingress.kubernetes.io/proxy-cache-bypass: "$http_upgrade"
    nginx.ingress.kubernetes.io/proxy-http-version: "1.1"

I don't use the `nginx.ingress.kubernetes.io/configuration-snippet' annotation because it might crash the ingress controller if the snippet has a syntax error.

Step 2: Add an enviroment

Next, you can create an environment in the Portainer UI. If you select the Docker Edge Agent and create it, you will get a join token that can be used by an Edge Agent to register with a Portainer instance.

The key represents the following data in this particular format
portainer_instance_url|tunnel_server_addr|tunnel_server_fingerprint|endpoint_ID

Example:
Base64 Encoded: aHR0cHM6Ly9wb3J0YWluZXIuZXhhbXBsZS5pb3xhZ2VudC5wb3J0YWluZXIuZXhhbXBsZS5pb3xub2FPb1R3Sk94RHhkZ1pqWXhLWWRsRHFOSmxyVjljaU1sMWZoUWxBaTFZPXw2Njk3
Decoded: https://portainer.example.io|portainer.example.io:8000|noaOoTwJOxDxdgZjYxKYdlDqNJlrV9ciMl1fhQlAi1Y=|6697

By default, tunnel_server_addr is always the host of the portainer and the tunnel port (default: 8000).
https://github.com/portainer/portainer/blob/develop/api/http/handler/endpoints/endpoint_create.go#L378
Have a look at https://github.com/portainer/portainer/blob/develop/api/chisel/key.go#L14

But for our purpose this is not usable, because we don't want to expose port 8000 from our Kubernetes cluster and we want to achieve a secure connection to the websocket.

So we need to modify the decoded token and re-encode it.

https://portainer.example.io|https://agent.portainer.example.io|noaOoTwJOxDxdgZjYxKYdlDqNJlrV9ciMl1fhQlAi1Y=|6697

Tip: wss://<host>:443 look correct but it won't work https://github.com/jpillora/chisel/blob/ab8f06a83048dca0c24dc0b06932dc98df54e8b1/client/client.go#L86

Step 3: Start and connect

Start the Portainer edge agent using the docker command:

docker run -d \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /var/lib/docker/volumes:/var/lib/docker/volumes \
  -v /:/host \
  -v portainer_agent_data:/data \
  --restart always \
  -e EDGE=1 \
  -e EDGE_ID=42a6a9e1-32af-479c-b048-15c2c811c4fc \
  -e EDGE_KEY=aHR0cHM6Ly9wb3J0YWluZXIuZXhhbXBsZS5pb3xodHRwczovL2FnZW50LnBvcnRhaW5lci5leGFtcGxlLmlvfG5vYU9vVHdKT3hEeGRnWmpZeEtZZGxEcU5KbHJWOWNpTWwxZmhRbEFpMVk9fDY2OTc \
  -e EDGE_INSECURE_POLL=0 \
  --name portainer_edge_agent \
  portainer/agent:2.21.5

From the Portainer UI, you can then connect to your environment.

Best regards,

Christian