TLS handshake error

[wolkenschieber]

When connecting an agent to portainer, the agent is constantly reporting TLS handshake errors:

portainer-agent has been recreated
portainer-agent  | 2024/03/03 12:09PM INF github.com/portainer/agent/cmd/agent/main.go:86 > agent running on Docker platform |
portainer-agent  | 2024/03/03 12:09PM INF github.com/portainer/agent/edge/registry/server.go:101 > starting registry credential server |
portainer-agent  | 2024/03/03 12:09PM INF github.com/portainer/agent/http/server.go:99 > starting Agent API server | api_version=2.19.4 server_addr=0.0.0.0 server_port=9001 use_tls=true
portainer-agent  | 2024/03/03 12:10:28 http: TLS handshake error from 192.168.178.47:56368: EOF
portainer-agent  | 2024/03/03 12:10:28 http: TLS handshake error from 192.168.178.47:56382: EOF
portainer-agent  | 2024/03/03 12:10:28 http: TLS handshake error from 192.168.178.47:56380: EOF
portainer-agent  | 2024/03/03 12:10:32 http: TLS handshake error from 192.168.178.47:42072: EOF
portainer-agent  | 2024/03/03 12:10:40 http: TLS handshake error from 192.168.178.47:33828: EOF
portainer-agent  | 2024/03/03 12:10:42 http: TLS handshake error from 192.168.178.47:33838: EOF
portainer-agent  | 2024/03/03 12:10:43 http: TLS handshake error from 192.168.178.47:33850: EOF

The agents image is portainer/agent:alpine, portainer's image is portainer/portainer-ee:latest.

[Wamy-Dev]

same issue for me

[daidaiwood]

I also encountered this problem.

[lessless]

same issue

[dierochade]

I get this error only when i want to open a terminal in the agent container via Portainer-GUI button....

[MNLierman]

Same issue here as well. I started on a new Portainer system and initially the Agent connected with the server fine for a while, and then I'm unsure what happened but it disconnected. Leading me to waste over 2 hours trying to troubleshoot ports and connections. The Portainer UI doesn't even state anything about TLS handshake, it simply says timeout occurred. After testing with netcat and determining that this wasn't a firewall issue, I happened across the logs which states unknown certificate.

[Emilien-Lambert]

same issue up

[lessless]

Unfortunately I don't remember how I solved it but it was something very straightforward and obvious in the hindsight.

[MNLierman]

Same. While I don't recall the specific solution to this original issue, I've compiled some helpful tips and troubleshooting steps based on my experience with Docker and Portainer over the last year. I've posted this somewhere else and received incredibly positive feedback. The solution to this specific is more than likely covered here, and if i remember exactly what it was I will remember to come back and update this.

I currently maintain eight Portainer instances that I use for testing and learning. I’m a DevOps newcomer, with the goal to become hired in this space. This covers common issues I encountered while managing my personal setup (local and cloud-based servers).

Networking Troubleshooting & Commands

• Check if a container is listening on the correct port: netstat -tulpn | grep <port>
• Persist iptables rules across reboots: apt install iptables-persistent. This not only saves and restores your firewall rules but also adds extra functionality. This was crucial for resolving some of my container traffic issues and might have been the key to fixing my Portainer Agent problems.
• Troubleshoot container host firewall and routing: iptables -L | grep <port> and ip6tables -L | grep <port>. These commands show whether a port is listed in your firewall rules. Remember, IPv6 is typically fully routable and often doesn't require specific iptables rules. However, your router's firewall and other network devices could still be dropping IPv6 traffic. Note: Portainer Server doesn't currently support IPv6 routing for the Agent. Docker imports iptables rules on reboot; if this fails, restarting Docker with the systemctl command might be necessary.

Docker Troubleshooting & Commands

• Restart Docker and containers without rebooting the server: systemctl restart docker

Portainer Troubleshooting & Commands

If your Portainer Agent encounters problems, try the following:

1. docker stop portainer-agent
2. docker rm portainer-agent
3. Now redeploy the agent using the command the Portainer Server gives you.

Portainer Does Not Auto-Update: Make sure you do this yourself on occasion.
You must stop, remove and redeploy the Agent and Server Docker containers to update the version you use, and I definitely suggest doing this from time to time, as this resolved occasional bugs I’ve ran into.

About Portainer SSL Certificates
Portainer usually manages its SSL certificates automatically. Unless you're using a custom domain or advanced configuration, you can generally rely on Portainer's built-in handling. Ensure your Portainer Server and Agent versions match, their docs explicity state that the versions must match. Consider using Watchtower to automate updates for specific containers. Note: Currently, only Portainer Business Edition offers automated updates (though a free business account with five licenses is available).

Cloud Provider Firewalls

When using cloud VPS providers, remember to configure their firewalls. I prefer creating whitelist rules with the IPv4 and IPv6 ranges of all my servers and my home network. This is generally more secure than opening world-accessable ports, and you don’t have to keep going back and adding more ports to the firewall. However, just be aware that whitelisting restricts access from outside your defined networks (e.g., no access using 5G). In such cases, you'll need a world-accessible port or deploy a free Cloudflare Tunnel.