Description
Proposed Change
App containers currently have 4 ports exposed to the diego cell's IP and local network:
- Direct access to the app process's :8080
- Direct access to the diego-sshd process's :2222
- Envoy proxied TLS/mTLS that goes to the app process on localhost:8080
- Envoy proxied SSH that goes to the diego-sshd process on localhost:2222
The containers.proxy.enable_unproxied_port_mappings
property on diego-release's rep job will disable the direct access ports from getting NAT rules generated for each container. However this breaks TCP routes, as TCP-router does not know how to configure haproxy for TLS/mTLS traffic.
We would like to add TLS/mTLS support to tcp-router to allow the use of containers.proxy.enable_unproxied_port_mappings
with TCP routes. This has the additional benefit of ensuring that all on-the-wire traffic between TCP router and apps is TLS-encrypted.
We would also like CF to adopt this more secure posture once this is resolved, and disable the unproxied ports by default, while providing an ops file to make those ports available if needed.
This will require a number of changes to routing-release, diego-release + cf-deployment:
- Update
tcp-router
to handle TLS based routes. For cross compatibility with diego-release versions and supporting TCP routes registered rom route-registrar which do not have an envoy proxy available, we will adding thebackend_tls_port
andinstance_id
fields to the route model to determine how the route should behave.backend_tls_port
will determine what port to connect to for TLS traffic - the magic value-1
will be used to indicate that the route is intentionally not using TLS. The magic value0
will be used to indicate that the route is unintentionally not using TLS. Any other value is the port number to use on the backend. The intentional/unintentional TLS route determination will be used to identify whether operators have opted out of TLS for TCP routes, or if there are older route-registrar/route-emitter processes incapable of using TLS routes, resulting in TCP routes not being encrypted as is likely desired.instance_id
should be the container instance GUID, used for route-integrity checking to ensure the tcp-router talks to the correct backend, the same way gorouter ensure handles route-integrity.
- Update
tcp-router
to support TLS route info provided by routing-api to connect to the TLS enabled backends using TLS or mTLS.- New properties for the client cert+key + trusted CA will need to be added
- If certs are present but no CA is present, an error will be thrown during BOSH deployment.
- If neither cert nor CA is present, no TLS routes will be used, it should fall back to the
backend_port
field of every route - If a CA is present and the
backend_tls_port
for a route is 0 or negative, usebackend_port
. Otherwise use thebackend_tls_port
. - if client certs are present, add them to the tcp-router TLS config
- validate the
instance_id
field of the route backend matches what was advertised by the backend TLS data
- Update
routing-api
to add thebackend_tls_port
andinstance_id
fields to the tcp-route model, and ensure they can be set + fetched properly. - Update
route-emitter
to supplybackend_tls_port
andinstance_id
fields to tcp routes, whentcp.enable_tls
is set to true on the route-emitter. This will allow operators to opt out of the TLS overhead on TCP routes if desired. In this caseinstance_id
will be empty, andbackend_tls_port
will be set to-1
. - Update
route-registar
to supply a-1
value forbackend_tls_port
and emptyinstance_id
value, as there is no envoy proxy around to handle the TLS termination. - Update cf-deployment:
- Set rep's
containers.proxy.enable_unproxied_port_mappings
to false by default - Set route-emitter's
tcp.enable_tls
to true by default - Add + plumb the client certs + CA certs for tcp-router + the diego-instance-ca
- Add ops files to disable TLS for TCP routes, and to re-enable unproxied port mappings.
- Set rep's
Backwards Compatibility
tcp-router
will support both TLS + non-TLS TCP routes.- Processes registering TCP routes will supply TLS data when enabled, or
-1
for thebackend_tls_port
if not enabled. routing-ap
i will accept TCP routes with or without TLS data.- If a route was not explicitly configured to be non-TLS, warning log messages will be emitted indicating the lack of encryption that was likely desired.
- During a deployment, the
tcp-router
+routing-api
are updated to add TLS support prior toroute-emitter
processes. Old routes will continue to emit as normal and be treated as unintentinally not using TLS triggering the above log message. When diego cells update, route-registrar will switch to either intentionally not using TLS to suppress the log messages, or to switch over to using TLS. In three all scenarios the tcp route will continue to function. Ifroute-emitter
orroute-registrar
processes are updated beforetcp-router
androuting-api
, these routes will continue to behave as they did prior to the upgrade, using non-TLS routing.
Acceptance criteria
- cf-deployment enables TLS for TCP routes by default.
- cf-deployment disables unproxied container ports by default.
- cats passes for http + tcp routing tests
- ops files exist to disable TLS on tcp routes as well as to re-enable unproxied container ports.
Related links
No response
Metadata
Metadata
Assignees
Labels
Type
Projects
Status