Fix connection broker mode for stat / listings

[bbockelm]

The "connection broker" mode for an origin allows it to be run from behind a firewall and require no host certificate.

However, recent new features have partially broken the mode. I spot two issues:

1. In the new code @haoming29 added to have the director do a HEAD against the origin as part of the origin-selection process.
2. When the client does a PROPFIND against the director for an object listing, the director will redirect it to the origin (which wouldn't work here).

Suggested fixes:

1. In broker/client.go, we already have a function that will return a TCP socket connected to the origin based on the origin name and broker address. Add a new method to the director module that returns a http.Transport object which is aware of the origins that need a connection broker. Internally, this new object would implement the DialContext method. Within DialContext, if the requested hostname is that of an origin which needs a connection broker, invoke the connection broker logic to get a new net.Conn from broker.ConnectToOrigin and return that. Otherwise, perform a "normal" dial to create a TCP socket.
2. Migrate any place where we invoke GetTransport in the director to the director-specific logic from step (1).
3. When handling a PROPFIND request, instead of redirecting to the origin, proxy the request to the origin.
   • This means all clients will go through the director and not contact the origin, allowing the director to reverse the connection.
   • Additionally, we could implement rate-limits for object listing in the future. Golang has powerful libraries for rate limiting -- and we know metadata queries are one potential way to overload the object stores behind an origin. Alternatively, we could just proxy for origins that require connection brokers and otherwise redirect.

[haoming29]

Some thoughts on this while I'm working on #1393
If the director decides to proxy the PROPFIND request, it should:

• Responds with 207 (per WebDAV standard)
• Still includes Pelican-X-Namespace header with collections-url field
  This is how I modify the client code to be upward compatible with this change. The client still expects 307 response as another option and it will take the Location header as the collections url in that case.

@bbockelm any words of wisdom before I commit the changes to #1393 ?

[github-actions]

This issue is stale because it has been open for 60 days with no activity.

[github-actions]

This issue is stale because it has been open for 60 days with no activity.

[jhiemstrawisc]

Closing this because I believe the approach outlined in #2425 supersedes this.