feat(transport/websocket): support SOCKS proxy with wss

[MarcoPolo]

closes #286 (a 7 year old issue??)

This PR allows users to use a SOCKS5 proxy when dialing /wss addresses by using the HTTPS_PROXY environment variable.

Here's an example using vole to dial through a socks proxy:

HTTPS_PROXY="socks5://127.0.0.1:9081" vole libp2p ping /dns4/am6.bootstrap.libp2p.io/tcp/443/wss/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb

[parkan]

lgtm 👍

I guess setEnv doesn't work in CI?

[MarcoPolo]

I didn't really debug, but SetEnv might be surprising if other tests are run in parallel.

[parkan]

I didn't really debug, but SetEnv might be surprising if other tests are run in parallel.

huh I was fairly certain that SetEnv only affects the current process and children, though I guess the tests may run in the same process as separate goroutines?

[sukunrt]

Do you think it's worth it to add a dependency on a socks5 server to test this end to end?

[sukunrt]

I feel parseMultiaddr should handle this.

[sukunrt]

This PR didnt introduce this, so it's fine to leave this as is.

[parkan]

I am going to make a test build of singularity against this for a functional test and report tmw.

[MarcoPolo]

Do you think it's worth it to add a dependency on a socks5 server to test this end to end?

Not really. The fact that it hand shakes with the socks5 should be enough. If it doesn't work from there the issue is in the websocket library, not our setup.

[parkan]

ok, so far I can say that we are able to build latest singularity against this commit and that many tools e.g. vole seem to pick it up correctly

I cannot verify that it's working correctly with singularity dealpusher just yet, however I don't think that should stop merging as any issues are unlikely to be w libp2p at that point

[parkan]

hmm actually can't verify correct behavior w/vole, hold please

[parkan]

ok the behavior I am observing is that using HTTPS_PROXY specifically appears to work against a ws remote, while HTTP_PROXY does not; this doesn't seem quite right to me but I will take it

I don't currently have a wss remote to test against but will work on that

[parkan]

OK I can't say that this 💯 works just yet because we're getting proxy errors, but the fact that proxy use is attempted is a big win and the issue is probably with allowlists on our side:

104.202.252.41:24001: i/o timeout\n  |   * [/ip4/104.202.252.41/tcp/24002/ws] proxy: failed to read greeting from SOCKS5 proxy at http-proxy.us.archive.org:8080: read tcp 172.17.0.9:59326->207.241.238.251:8080: i/o timeout\n      └─ Wraps: (6) failed to dial /ip4/104.202.252.41/tcp/24002/ws\n        └─ Wraps: (7) proxy: failed to read greeting from SOCKS5 proxy at http-proxy.us.archive.org:8080: read tcp 172.17.0.9:59326->207.241.238.251:8080: i/o timeout\n      └─ Wraps: (8) failed to dial /ip4/104.202.252.41/tcp/24001\n        └─ Wraps: (9) dial tcp4 104.202.252.41:24001\n          └─ Wraps: (10) i/o timeout\n      └─ Wraps: (11) all dials failed\nError types: (1) *withstack.withStack (2) *withstack.withStack (3) *errutil.withPrefix (4) *fmt.wrapError (5) *swarm.DialError (6) 

draft PR for singularity here data-preservation-programs/singularity#450

[MarcoPolo]

I can make this work with HTTP_PROXY and ws. Can you give me a ws endpoint to check against?

Just curious, what is the reason you are using /ws without tls? Why can’t you open a tcp stream? Is the client on a browser?

[parkan]

@MarcoPolo client is inside a secure network where all connections to the outside must go through SOCKS and only to allowlisted hosts -- this is actually not entirely uncommon in "enterprise" environments though it is a huge pain

for a ws endpoint you can check /ip4/104.202.252.41/tcp/24002/ws/p2p/12D3KooWQnScgybqBcTv6f3yvgbF8pH7sLnrkrbYVB3FRyZY2NsR, I've confirmed it's healthy just now

[MarcoPolo]

Testing this locally with vole using this branch of go-libp2p I was able to connect to your endpoint using HTTP_PROXY.

HTTP_PROXY="socks5://127.0.0.1:9081" vole libp2p ping /ip4/104.202.252.41/tcp/24002/ws/p2p/12D3KooWQnScgybqBcTv6f3yvgbF8pH7sLnrkrbYVB3FRyZY2NsR             

ws uses HTTP_PROXY and wss uses HTTPS_PROXY.

[MarcoPolo]

This may not be a problem for your use case, but are you aware that the WebSocket protocol will mask all the application data you send it in order to protect against faulty intermediaries? This may have a noticeable performance impact as it introduces another copy of all data and a non-zero overhead for the masking operation. See https://datatracker.ietf.org/doc/html/rfc6455#section-5.1 for more details.

Could you proxy TCP traffic through your SOCKS proxy? If so, we could add something like this to go-libp2p to support a TCP_PROXY env var.

[parkan]

@MarcoPolo we're only sending deal proposals so the payload size is trivial, using websockets from that perspective is fine

separately I am finding that our proxy may not actually even have a limited SOCKS implementation... so 🤦

[parkan]

thank you! I am going to work on getting our proxy to... not suck and ideally have proper SOCKS transports so the normal tcp connections can also traverse it as in your example

[parkan]

ok, upon review it seems that we may be able to get raw tcp through the proxy after all (once underlying blockage is fixed), which will help us avoid all the ws rough edges... is it possible to ship the TCP_PROXY suggestion @MarcoPolo? (the HTTP(S)_PROXY fix will actually matter for us elsewhere so that effort is much appreciated)

EDIT: yes, we have confirmed:

• SOCKS support actually not enabled so must use HTTP CONNECT to create the tunnel, I don't see this being handled in the standard fromUrl dialer chooser: https://cs.opensource.google/go/x/net/+/refs/tags/v0.34.0:proxy/proxy.go;l=81 (my initial confusion was that the proxy was described as SOCKS to me but did not appear to support anything other than http... which in the end turned out to be no socks but yes to http tunneling)
• an overlooked http_access deny CONNECT was blocking my access despite port being listed in acl, we can now open http tunnels

so, if it's reasonably possible to use e.g. https://github.com/mwitkow/go-http-dialer the current use case can be fully supported, otherwise I will have to negotiate to rebuild squid with socks (which, in fairness, was the original problem statement so that could be a reasonable compromise)

[MarcoPolo]

I think what makes the most sense is to allow a user to set a custom net.Dialer. Then they can use whatever custom logic they want. This would let you use https://github.com/mwitkow/go-http-dialer without tying the library to that specific case.

[parkan]

Sounds like a good idea, it should allow us to pass in a http proxy supporting dialer. If we can ship TCP_PROXY support first I can get moving with an interim solution (like using a danted socks->http proxy chain shim)