Client With Proxy

[doroved]

Can you write a client code that will use an external proxy to access blocked and sites and still be able to decrypt traffic?
You can use the standard Basic header as authentication, but in my case the authentication is custom and just uses the x-token: auth_token header.
In the end we will have:

   let proxy_client = create_proxy_client(
        opt.proxy_addr,
        opt.auth_token,
    ).await?
    
    let default_client = DefaultClient::new()?

so that you can send the desired hosts through an external proxy

[doroved]

Reference https://docs.proxyman.io/advanced-features/external-proxy

[doroved]

@hatoo are you planning to implement this feature? Or at least write how to implement it, I'm trying to do it myself. but it doesn't work.

[hatoo]

http_mitm_proxy::default_client::DefaultClient is just a default client, you can use any HTTP client as long as you can obtain hyper::Response.
I added examples/reqwest_proxy.rs to demonstrate how to use reqwest client (which supports proxy).

[doroved]

http_mitm_proxy::default_client::DefaultClient is just a default client, you can use any HTTP client as long as you can obtain hyper::Response. I added examples/reqwest_proxy.rs to demonstrate how to use reqwest client (which supports proxy).

I should have said that I want to use an https proxy with custom authentication.
Here's an example of how I do tunneling requests through the proxy, but I can't figure out how I can do the same + decrypt requests.

async fn tunnel_via_proxy(
    &self,
    mut client: TokioIo<hyper::upgrade::Upgraded>,
    addr: String,
    proxy_config: &ProxyConfig,
) {
    tracing::info!("Using external proxy for addr: {}", addr);

    // Establish a TCP connection with a timeout
    let tcp = match timeout(
        Duration::from_secs(10),
        TcpStream::connect(&proxy_config.addr),
    )
    .await
    {
        Ok(Ok(tcp)) => tcp,
        Ok(Err(e)) => {
            tracing::error!("Failed to connect to proxy: {}", e);
            return;
        }
        Err(_) => {
            tracing::error!("Timeout connecting to proxy");
            return;
        }
    };

    // Establish a TLS connection with a timeout
    let tls = tokio_native_tls::TlsConnector::from(native_tls::TlsConnector::new().unwrap());
    let mut stream = match timeout(
        Duration::from_secs(10),
        tls.connect(proxy_config.addr.split(':').next().unwrap(), tcp),
    )
    .await
    {
        Ok(Ok(stream)) => stream,
        Ok(Err(e)) => {
            tracing::error!("TLS connection failed: {}", e);
            return;
        }
        Err(_) => {
            tracing::error!("TLS connection timeout");
            return;
        }
    };

    // Formulate and send the CONNECT request
    let connect_req = format!(
        "CONNECT {addr} HTTP/1.1\r\n\
        Host: {addr}\r\n\
        Proxy-Connection: Keep-Alive\r\n\
        X-Auth-Token: {}\r\n\r\n",
        &proxy_config.auth_token,
    );

    if let Err(e) = stream.write_all(connect_req.as_bytes()).await {
        tracing::error!("Failed to send CONNECT request: {}", e);
        return;
    }

    // Read the response with a timeout
    let mut response = [0u8; 1024];
    let n = match timeout(Duration::from_secs(5), stream.read(&mut response)).await {
        Ok(Ok(n)) => n,
        Ok(Err(e)) => {
            tracing::error!("Failed to read proxy response: {}", e);
            return;
        }
        Err(_) => {
            tracing::error!("Timeout reading proxy response");
            return;
        }
    };

    // Check the response status
    if !response[..n].windows(3).any(|window| window == b"200") {
        tracing::error!(
            "Proxy connection failed: {}",
            String::from_utf8_lossy(&response[..n])
        );
        return;
    }

    // Copy data between the client and the proxy
    let _ = tokio::io::copy_bidirectional(&mut client, &mut stream).await;
}

This is how I handle requests directly

/// Establishes a direct tunnel to the specified authority using an upgraded HTTP connection.
///
/// # Arguments
///
/// * `client` - The upgraded client connection to communicate with.
/// * `connect_authority` - The target authority to connect to, represented as a URI Authority.
async fn direct_tunnel(
    &self,
    client: hyper::upgrade::Upgraded,
    connect_authority: hyper::http::uri::Authority,
) {
    // Attempt to connect to the specified authority using TcpStream.
    match TcpStream::connect(connect_authority.as_str()).await {
        // If the connection is successful, proceed with bidirectional data transfer.
        Ok(mut server) => {
            // Use Tokio's IO utilities to copy data bidirectionally between the client and server.
            let _ = tokio::io::copy_bidirectional(&mut TokioIo::new(client), &mut server).await;
        }
        // If the connection fails, log the error.
        Err(err) => {
            tracing::error!("Failed to connect to {}: {}", connect_authority, err);
        }
    }
}

And now as an example you can put a condition for CONNECT requests

#[derive(Clone)]
pub struct ProxyConfig {
    pub addr: String,
    pub auth_token: String,
    pub domains: Vec<String>,
}

#[derive(Clone)]
/// The main struct to run proxy server
pub struct MitmProxy<C> {
    /// Root certificate to sign fake certificates. You may need to trust this certificate on client application to use HTTPS.
    ///
    /// If None, proxy will just tunnel HTTPS traffic and will not observe HTTPS traffic.
    pub root_cert: Option<C>,
    /// Cache to store generated certificates. If None, cache will not be used.
    /// If root_cert is None, cache will not be used.
    ///
    /// The key of cache is hostname.
    pub cert_cache: Option<Cache<String, CertifiedKeyDer>>,

    pub proxy_config: Option<ProxyConfig>,
}

//.........
// Check if an external proxy should be used first
if let Some(proxy_config) = &proxy.proxy_config {
    let addr = connect_authority.to_string();
    let host = connect_authority.host().to_string();

    // Check if the host matches any of the specified domains for the proxy
    if proxy_config.domains.iter().any(|domain| host.contains(domain)) {
        // Use the proxy to tunnel the connection for matching domains
        proxy.tunnel_via_proxy(TokioIo::new(client), addr, proxy_config).await;
    } else {
        // Direct connection for any other hosts that do not match the proxy domains
        proxy.direct_tunnel(client, connect_authority).await;
    }
} else {
    // If proxy_config is not provided, use a direct connection
    proxy.direct_tunnel(client, connect_authority).await;
}

Question, how to implement proxying of MITM requests?
Normal tunneling is done by copy_bidirectional, and MITM uses .serve_connection

[hatoo]

Question, how to implement proxying of MITM requests?

Are you asking how to do the same thing in the above codes using hyper::Request?
You can use reqwest with an HTTP proxy. It's the same at the TCP level.

[doroved]

Are you asking how to do the same thing in the above codes using hyper::Request?
You can use reqwest with an HTTP proxy. It's the same at the TCP level.

I have exactly https proxies being used, with authentication happening during CONNECT.

[hatoo]

Ok, if you want to do something on CONNECT, you can create your own hyper server and use MITM things as a component.

Please see https://github.com/hatoo/http-mitm-proxy/blob/master/examples/https.rs

http-mitm-proxy/examples/https.rs

Line 135 in 0efd900

.serve_connection(TokioIo::new(stream), service)

You can create your own hyper service wrapping MITM service.

[doroved]

Ok, if you want to do something on CONNECT, you can create your own hyper server and use MITM things as a component.

Please see https://github.com/hatoo/http-mitm-proxy/blob/master/examples/https.rs

http-mitm-proxy/examples/https.rs

Line 135 in 0efd900

.serve_connection(TokioIo::new(stream), service)

You can create your own hyper service wrapping MITM service.

I understood this solution to have MITM proxy on HTTPS (https://127.0.0.1:3003), but I don't need it).

I just want to proxy requests through my external https proxy with custom header in CONNECT and decrypt them like mitmproxy can do for example.
https://docs.mitmproxy.org/stable/concepts-modes/#upstream-proxy

But I can't figure out how to do it (

[hatoo]

What is the problem with using reqwest to make requests through your proxies?
I think reqwest can do exactly do the right side of mitmproxy(upstream) in the picture.
https://docs.mitmproxy.org/stable/schematics/proxy-modes-upstream.png

[doroved]

What is the problem with using reqwest to make requests through your proxies? I think reqwest can do exactly do the right side of mitmproxy(upstream) in the picture. https://docs.mitmproxy.org/stable/schematics/proxy-modes-upstream.png

I use my own HTTP over TLS proxy server
https://github.com/doroved/proxerver
You cannot via reqwest use such proxy + set custom header in CONNECT request with proxy for authentication.

[doroved]

I gave above an example of tunnel_via_proxy function for tunneling traffic through HTTP over TLS proxy, but how to implement support for these proxies in the process of decrypting traffic through MITM I don't understand, I tried for 3 days and still don't understand.

[doroved]

Maybe I don't quite understand how MITM works.
We just receive a decrypted request and send it using an HTTP client? I.e. there is no tunneling as with a normal proxy server?
So we need to implement an external proxy in the client itself in the default_client.rs file?
I tried to do it in lib.rs

[hatoo]

We just receive a decrypted request and send it using an HTTP client? I.e. there is no tunneling as with a normal proxy server?

Yes, we can see data from a client after CONNECT. We parse HTTP requests after CONNECT and return a response. There is no tunneling.

So we need to implement an external proxy in the client itself in the default_client.rs file?

We have hyper::Request and we need to return hyper::Response. We can do anything to create hyper::Response.
default_client.rs is just an example, I want to keep it simple, I don't want to add proxy feature to it.

[doroved]

We have hyper::Request and we need to return hyper::Response. We can do anything to create hyper::Response.
default_client.rs is just an example, I want to keep it simple, I don't want to add proxy feature to it.

I will try to implement proxy support as it is done in tunnel_via_proxy function in default_client.rs file.
I just didn't immediately understand how MITM works and tried to do proxying in wrap_service.
And you can just create a separate client with an example of external proxy support via hyper, since this is a basic functionality in all such programs, proxyman, charles, mitmproxy, etc. This helps to decrypt traffic from hosts that are blocked from accessing from our IP.

[doroved]

http_mitm_proxy::default_client::DefaultClient is just a default client, you can use any HTTP client as long as you can obtain hyper::Response. I added examples/reqwest_proxy.rs to demonstrate how to use reqwest client (which supports proxy).

Could you please tell me how to modify the request and response when using reqwest? No way, this eternal struggle with moving a variable.

PS.: I realized, it seems to be possible to modify only in hyper, in reqwest it is not clear how, for the sake of interest if you want you can add a modification to the example reqwest_proxy.rs)

let res = client.execute(req).await?;
// You can modify response here
let (parts, body) = from_reqwest(res).into_parts();

Ok::<_, http_mitm_proxy::default_client::Error>(Response::from_parts(
  parts,
  http_body_util::Full::new(body),
))

[doroved]

I added examples/reqwest_proxy.rs to demonstrate how to use reqwest client (which supports proxy).

When using the reqwest client there is a problem with redirects, we get the body of the final url but it loads at the original URL.
Do you have any idea what the problem could be?
For example, https://shorturl.at/MJxnx