docs: update custom TLS certificate usage patch

[7-zete-7]

This will allow you to configure a custom certificate and not get the following error:

Error: adapting config using caddyfile: server listening on [:80] is HTTP, but attempts to configure TLS connection policies

This issue was mentioned in:

• Own SSL cert and key - where to put? #359
• Configuration with Let's Encrypt does not work #579
• Using custom TLS certificates does not work #604
• Update tls.md #738

With this configuration, Caddy will use the configured certificate only for access via a secure protocol.

[7-zete-7]

I could not find information on how Caddy merges configurations of several addresses. I checked it on my projects and it works. If someone attaches links to information on how Caddy does this, I would be grateful.

[maxhelias]

I could not find information on how Caddy merges configurations of several addresses. I checked it on my projects and it works. If someone attaches links to information on how Caddy does this, I would be grateful.

Maybe @dunglas or @mholt have the answer ?

[mholt]

Maybe @dunglas or @mholt have the answer ?

Sorry, I've looked at the linked issues but I'm not sure I understand what the question is exactly ("how Caddy merges configurations of several addresses"?). Is there a Caddy config in particular that is yielding that error?

The error means that the resulting Caddy config -- whatever it is -- has an HTTP server configured to listen on port 80, the HTTP port, but the server configures TLS connection policies, which enables TLS, making it HTTPS. If you intend to serve both HTTP and HTTPS, as opposed to only HTTPS (even with implicit HTTP redirects), then you should create an HTTP server and an HTTPS server in the config.

[7-zete-7]

Thanks for the quick response, @mholt!

The question is rather in the Caddyfile like:

https:// {
  tls /etc/caddy/certs/tls.pem /etc/caddy/certs/tls.key
}

localhost, php:80 {
  # ...
}

Observe that the configurations of both addresses (https:// and localhost, php:80) affect the final configuration of the localhost address. I got this solution by accident. Tried to search for information on how Caddy combines these configurations, but I could not find specific information. I got curious.

Planned to dive into more detailed searching and reading the source codes over the weekend to better understand this mechanics, but I will be glad to know that such information already exists 😀

[mholt]

@7-zete-7 Weird, I cannot reproduce the error with that config. It adapts to this JSON and seems to work fine:

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "localhost"
                  ]
                }
              ],
              "terminal": true
            }
          ],
          "tls_connection_policies": [
            {
              "certificate_selection": {
                "any_tag": [
                  "cert0"
                ]
              }
            }
          ]
        },
        "srv1": {
          "listen": [
            ":80"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "php"
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    },
    "tls": {
      "certificates": {
        "load_files": [
          {
            "certificate": "/etc/caddy/certs/tls.pem",
            "key": "/etc/caddy/certs/tls.key",
            "tags": [
              "cert0"
            ]
          }
        ]
      }
    }
  }
}

Since only the :443 server has TLS connection policies.

[7-zete-7]

@mholt, I probably didn't communicate the information well, sorry about that 😅. This PR proposes a solution to the error, not its existence.

A configuration like #774 (comment) solves the error.

I wanted to know information about how Caddy works when a site in a Caddyfile corresponds to several site blocks. I am curious myself, and I would attach this information to this PR to explain the proposed configuration.

I could only find the following information:

To catch all hosts, omit the host portion of the address, for example, simply https://. This is useful when using On-Demand TLS, when you don't know the domains ahead of time.

Source: https://caddyserver.com/docs/caddyfile/concepts#addresses

This solution was based on this information. However, this information does not make it clear that the configurations will be merged, and this puzzles me.

[maxhelias]

Oh yes, of course, now I remember — the initial problem was the php:80 in the server name here : https://github.com/dunglas/symfony-docker/blob/main/compose.yaml#L6. I think, (not 100% sure), we should suggest to setting auto_https: disable_certs in CADDY_GLOBAL_OPTIONS to keep redirect http-to-https and communication with http inside the docker network, it should work like this.

Your case works because you're asking all HTTPS domains to use the custom TLS, see : https://caddyserver.com/docs/caddyfile/concepts#addresses

So maybe we should just adapt to explain that in this doc — what do you think?

[mholt]

@7-zete-7 Ah, gotcha. Well, kind of:

this information does not make it clear that the configurations will be merged

What do you mean by "the configurations will be merged"? The Caddyfile is adapted into JSON, it's a whole new config document. It's not really "merging" Caddyfile and JSON.

how Caddy works when a site in a Caddyfile corresponds to several site blocks.

A site block does not correspond with other/several site blocks, except in this sense: A site block with a more specific site address will take priority over those with less specific matching site addresses. There is no inheritance or cascading, if that's what you mean?

@maxhelias I'm still not 100% sure I follow, but, if a site block defined as:

localhost, php:80 {
    # ....
}

has a tls directive in it, that will cause problems because you're telling Caddy to serve HTTPS on port 80.

[7-zete-7]

@mholt, you are right. The words inheritance or cascading better fit what I was trying to say. Thanks for the correction.

Today I tried to play with the directives in Caddyfile and see how they affect the configuration in JSON (I used Caddapto for this). As a result, I got two directives that affect only the server, without creating a route. These are log and tls. And this seems quite logical. Working with the JSON configuration, this is even obvious. However, I could not find any mention of such behavior of these directives in the documentation. Although, as it turned out, a good indicator was the absence of a directive in the Directive order section.

As a result, I got a Caddyfile where the site blocks, roughly speaking, extend each other:

The resulting Caddyfile

# creates apps.http.servers.srv0 and apps.http.servers.srv1 items
localhost, awesome-project.local, private.local:80 {
    respond "Hello, World!"
}

# doesn't not creates a new apps.http.servers item
# directives are relates to the localhost, awesome-project.local
https://, :443 {
    log {
        output stderr
    }

    tls public.pem private.pem
}

# doesn't not creates a new apps.http.servers item
# directives are relates to the private.local:80
http://, :80 {
    log {
        output stderr
    }
}

Configuration in JSON

{
  "logging": {
    "logs": {
      "default": {
        "exclude": [
          "http.log.access.log0",
          "http.log.access.log1"
        ]
      },
      "log0": {
        "writer": {
          "output": "stderr"
        },
        "include": [
          "http.log.access.log0"
        ]
      },
      "log1": {
        "writer": {
          "output": "stderr"
        },
        "include": [
          "http.log.access.log1"
        ]
      }
    }
  },
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "localhost",
                    "awesome-project.local"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "body": "Hello, World!",
                          "handler": "static_response"
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ],
          "tls_connection_policies": [
            {
              "certificate_selection": {
                "any_tag": [
                  "cert0"
                ]
              }
            }
          ],
          "logs": {
            "default_logger_name": "log0"
          }
        },
        "srv1": {
          "listen": [
            ":80"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "private.local"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "body": "Hello, World!",
                          "handler": "static_response"
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ],
          "logs": {
            "default_logger_name": "log1"
          }
        }
      }
    },
    "tls": {
      "certificates": {
        "load_files": [
          {
            "certificate": "public.pem",
            "key": "private.pem",
            "tags": [
              "cert0"
            ]
          }
        ]
      }
    }
  }
}

Found a similar question in caddyserver/website#235. I'll move the discussion on this behavior there.

[maxhelias]

I'm still not 100% sure I follow, but, if a site block defined as:
localhost, php:80 {
# ....
}
has a tls directive in it, that will cause problems because you're telling Caddy to serve HTTPS on port 80.

That's what we do, so I guess that's where the problem is.
Two suggest to fix your issue :

• Recommend that the user use auto_https: disable_certs (not 100% sure) if they add a tls directive.
• Move that common part into a snippet, split the site blocks and import it in each block — makes it customizable and prevents the error.

[7-zete-7]

@maxhelias, if I understand correctly, you are proposing a configuration of the following:

Suggested Configuration

# Caddyfile

{
    {$CADDY_GLOBAL_OPTIONS}

    # ...
}

(app) {
    # ...
}

{$CADDY_EXTRA_CONFIG}

{$SERVER_NAME:localhost} {
    import app

    ${CADDY_APP_SERVER_EXTRA_DIRECTIVES}
}

php:80 {
    import app

    ${CADDY_PRIVATE_SERVER_EXTRA_DIRECTIVES}
}

 # compose.yaml

 services:
   php:
     # ...
     environment:
-      SERVER_NAME: ${SERVER_NAME:-localhost}, php:80
+      SERVER_NAME: ${SERVER_NAME:-localhost}
       # ...
+      CADDY_GLOBAL_OPTIONS: auto_https disable_certs
+      CADDY_APP_SERVER_EXTRA_DIRECTIVES: tls public.pem private.pem
       # ...

IMO, the Caddyfile has become somewhat untidy. But it should definitely work. Although I'm not sure that auto_https: disable_certs will somehow change the behavior of Caddy for this configuration.

Perhaps we should stop using php:80 and access the server by its SERVER_NAME? To prevent requests from going beyond the Docker network, add SERVER_NAME to service.php.network.aliases. But then the Mercure publisher will start accessing via HTTPS instead of HTTP. This may worsen performance. And the appearance of the networks attribute in the php service complicates this file.

Suggested Configuration

 # compose.yaml

 services:
   php:
     # ...
+    networks:
+      default:
+        aliases:
+          - ${SERVER_NAME:-localhost}
     # ...
     environment:
-      SERVER_NAME: ${SERVER_NAME:-localhost}, php:80
+      SERVER_NAME: ${SERVER_NAME:-localhost}
     # ...
-    MERCURE_URL: ${CADDY_MERCURE_URL:-http://php/.well-known/mercure}
+    MERCURE_URL: ${CADDY_MERCURE_URL:-https://${SERVER_NAME:-localhost}/.well-known/mercure}

[maxhelias]

No need to have CADDY_GLOBAL_OPTIONS: auto_https disable_certs with the snippet solution. It's either the snippet or the CADDY_GLOBAL_OPTIONS: auto_https disable_certs not both, but i'm not sure that the last solution fix the erreur.

I think we still need to keep http, I'm not convinced to remove it at the moment.

[mholt]

@7-zete-7 To clarify, is there anything wrong with that? The adapted config looks correct at a glance.

[7-zete-7]

Suggested set of configurations

Note

Certificate mount blocks are omitted for simplicity. Mounts will be present in the final solution.

Snippet solution

Sites ${SERVER_NAME:localhost} and php:80 are split into two separate ones. The common part are moved to a snippet.

Suggested Configuration

# Caddyfile

{
    {$CADDY_GLOBAL_OPTIONS}

    # ...
}

(app) {
    # ...
}

{$CADDY_EXTRA_CONFIG}

{$SERVER_NAME:localhost} {
    import app

    ${CADDY_APP_SERVER_EXTRA_DIRECTIVES}
}

php:80 {
    import app

    ${CADDY_PRIVATE_SERVER_EXTRA_DIRECTIVES}
}

 # compose.yaml

 services:
   php:
     # ...
     environment:
-      SERVER_NAME: ${SERVER_NAME:-localhost}, php:80
+      SERVER_NAME: ${SERVER_NAME:-localhost}
       # ...

 # compose.override.yaml

 services:
   php:
     # ...
     environment:
       # ...
+      CADDY_APP_SERVER_EXTRA_DIRECTIVES: tls public.pem private.pem
       # ...

auto_https disable_certs solution

Use the global option auto_https with the value disable_certs.

Suggested Configuration

 # compose.override.yaml

 services:
   php:
     # ...
     environment:
       # ...
+      CADDY_GLOBAL_OPTIONS: auto_https disable_certs
+      CADDY_SERVER_EXTRA_DIRECTIVES: tls public.pem private.pem
       # ...

Still have folowing error:

Error: adapting config using caddyfile: server listening on [:80] is HTTP, but attempts to configure TLS connection policies

Separated Site Block solution

Use a separate site block to set directives for a secure connection.

Suggested Configuration

 # compose.override.yaml

 services:
   php:
     # ...
     environment:
       # ...
+      CADDY_EXTRA_CONFIG: |
+        https:// {
+            tls public.pem private.key
+        }
       # ...

[7-zete-7]

@mholt, I believe this behavior is quite useful and not wrong. However, I think it should be documented for clarity. I'll proceed with discussing this in the relevant issue. Thank you for your help in clarifying the information! 😊

[7-zete-7]

@maxhelias, it looks like I misunderstood the CADDY_GLOBAL_OPTIONS: auto_https disable_certs solution.

[7-zete-7]

There is incorrect formatting here (Caddyfile uses tabs for indentation)

Suggested change

	+ https:// {
	+ tls /etc/caddy/certs/tls.pem /etc/caddy/certs/tls.key
	+ }
	+ https:// {
	+ tls /etc/caddy/certs/tls.pem /etc/caddy/certs/tls.key
	+ }

IMO it looks wrong visually (as if a single space is used for indentation, not a tab).

[7-zete-7]

In suggestion it looks better than it renders in Markdown