Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IAM: Add caching to HTTP client #3148

Merged
merged 12 commits into from
Jun 4, 2024
Merged

IAM: Add caching to HTTP client #3148

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
4cce501
IAM: Add caching to HTTP client
reinkrul May 27, 2024
13ff083
fix
reinkrul May 27, 2024
95f3002
tests
reinkrul May 27, 2024
326fd84
copyright
reinkrul May 28, 2024
ac4f8c7
PR feedback
reinkrul May 31, 2024
a7b0b44
instantiate IAM Client once to allow caching
reinkrul May 31, 2024
609c809
f2
reinkrul May 31, 2024
cff28a1
wip
reinkrul May 31, 2024
ba9974f
move core.StrictHTTPClient to HTTP module to allow global caching
reinkrul Jun 1, 2024
dcd8d49
Merge branch 'master' into iam/httpclient-caching
reinkrul Jun 1, 2024
8a5c8f5
f
reinkrul Jun 2, 2024
183825a
Merge branch 'master' into iam/httpclient-caching
reinkrul Jun 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ The following options can be configured on the server:
verbosity info Log level (trace, debug, info, warn, error)
httpclient.timeout 30s Request time-out for HTTP clients, such as '10s'. Refer to Golang's 'time.Duration' syntax for a more elaborate description of the syntax.
**Crypto**
crypto.storage Storage to use, ''%s' for an external backend (deprecated).' for file system (for development purposes), 'fs' for HashiCorp Vault KV store, 'vaultkv' for Azure Key Vault.%!(EXTRA string=azure-keyvault, string=external)
crypto.storage Storage to use, 'fs' for file system (for development purposes), 'vaultkv' for HashiCorp Vault KV store, 'azure-keyvault' for Azure Key Vault, 'external' for an external backend (deprecated).
crypto.azurekv.hsm false Whether to store the key in a hardware security module (HSM). If true, the Azure Key Vault must be configured for HSM usage. Default: false
crypto.azurekv.timeout 10s Timeout of client calls to Azure Key Vault, in Golang time.Duration string format (e.g. 10s).
crypto.azurekv.url The URL of the Azure Key Vault.
Expand All @@ -194,13 +194,14 @@ The following options can be configured on the server:
discovery.server.ids [] IDs of the Discovery Service for which to act as server. If an ID does not map to a loaded service definition, the node will fail to start.
**HTTP**
http.log metadata What to log about HTTP requests. Options are 'nothing', 'metadata' (log request method, URI, IP and response code), and 'metadata-and-body' (log the request and response body, in addition to the metadata). When debug vebosity is set the authorization headers are also logged when the request is fully logged.
http.cache.maxbytes 10485760 HTTP client maximum size of the response cache in bytes. If 0, the HTTP client does not cache responses.
http.internal.address 127.0.0.1:8081 Address and port the server will be listening to for internal-facing endpoints.
http.internal.auth.audience Expected audience for JWT tokens (default: hostname)
http.internal.auth.authorizedkeyspath Path to an authorized_keys file for trusted JWT signers
http.internal.auth.type Whether to enable authentication for /internal endpoints, specify 'token_v2' for bearer token mode or 'token' for legacy bearer token mode.
http.public.address \:8080 Address and port the server will be listening to for public-facing endpoints.
**JSONLD**
jsonld.contexts.localmapping [https://schema.org=assets/contexts/schema-org-v13.ldjson,https://nuts.nl/credentials/v1=assets/contexts/nuts.ldjson,https://www.w3.org/2018/credentials/v1=assets/contexts/w3c-credentials-v1.ldjson,https://w3id.org/vc/status-list/2021/v1=assets/contexts/w3c-statuslist2021.ldjson,https://w3c-ccg.github.io/lds-jws2020/contexts/lds-jws2020-v1.json=assets/contexts/lds-jws2020-v1.ldjson] This setting allows mapping external URLs to local files for e.g. preventing external dependencies. These mappings have precedence over those in remoteallowlist.
jsonld.contexts.localmapping [https://w3id.org/vc/status-list/2021/v1=assets/contexts/w3c-statuslist2021.ldjson,https://w3c-ccg.github.io/lds-jws2020/contexts/lds-jws2020-v1.json=assets/contexts/lds-jws2020-v1.ldjson,https://schema.org=assets/contexts/schema-org-v13.ldjson,https://nuts.nl/credentials/v1=assets/contexts/nuts.ldjson,https://www.w3.org/2018/credentials/v1=assets/contexts/w3c-credentials-v1.ldjson] This setting allows mapping external URLs to local files for e.g. preventing external dependencies. These mappings have precedence over those in remoteallowlist.
jsonld.contexts.remoteallowlist [https://schema.org,https://www.w3.org/2018/credentials/v1,https://w3c-ccg.github.io/lds-jws2020/contexts/lds-jws2020-v1.json,https://w3id.org/vc/status-list/2021/v1] In strict mode, fetching external JSON-LD contexts is not allowed except for context-URLs listed here.
**PKI**
pki.maxupdatefailhours 4 Maximum number of hours that a denylist update can fail
Expand Down
3 changes: 2 additions & 1 deletion auth/client/iam/openid4vp.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import (
"context"
"encoding/json"
"fmt"
"github.com/nuts-foundation/nuts-node/http/client"
"net/http"
"net/url"
"time"
Expand Down Expand Up @@ -56,7 +57,7 @@ func NewClient(wallet holder.Wallet, keyResolver resolver.KeyResolver, jwtSigner
return &OpenID4VPClient{
httpClient: HTTPClient{
strictMode: strictMode,
httpClient: core.NewStrictHTTPClient(strictMode, httpClientTimeout, nil),
httpClient: client.NewWithCache(httpClientTimeout),
},
keyResolver: keyResolver,
jwtSigner: jwtSigner,
Expand Down
5 changes: 2 additions & 3 deletions auth/client/iam/openid4vp_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,12 @@ import (
"crypto/tls"
"encoding/json"
"fmt"
"github.com/nuts-foundation/nuts-node/http/client"
"net/http"
"net/http/httptest"
"testing"
"time"

"github.com/nuts-foundation/nuts-node/core"

ssi "github.com/nuts-foundation/go-did"
"github.com/nuts-foundation/go-did/did"
"github.com/nuts-foundation/go-did/vc"
Expand Down Expand Up @@ -399,7 +398,7 @@ func createClientTestContext(t *testing.T, tlsConfig *tls.Config) *clientTestCon
wallet: wallet,
httpClient: HTTPClient{
strictMode: false,
httpClient: core.NewStrictHTTPClient(false, 10*time.Second, tlsConfig),
httpClient: client.NewWithTLSConfig(10*time.Second, tlsConfig),
},
jwtSigner: jwtSigner,
keyResolver: keyResolver,
Expand Down
41 changes: 0 additions & 41 deletions core/http_client.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,10 @@ package core

import (
"context"
"crypto/tls"
"errors"
"fmt"
"github.com/sirupsen/logrus"
"io"
"net/http"
"time"
)

// HttpResponseBodyLogClipAt is the maximum length of a response body to log.
Expand Down Expand Up @@ -175,41 +172,3 @@ func newEmptyTokenGenerator() AuthorizationTokenGenerator {
return "", nil
}
}

// NewStrictHTTPClient creates a HTTPRequestDoer that only allows HTTPS calls when strictmode is enabled.
func NewStrictHTTPClient(strictmode bool, timeout time.Duration, tlsConfig *tls.Config) *StrictHTTPClient {
if tlsConfig == nil {
tlsConfig = &tls.Config{
MinVersion: tls.VersionTLS12,
}
}

transport := http.DefaultTransport
// Might not be http.Transport in testing
if httpTransport, ok := transport.(*http.Transport); ok {
// cloning the transport might reduce performance.
httpTransport = httpTransport.Clone()
httpTransport.TLSClientConfig = tlsConfig
transport = httpTransport
}

return &StrictHTTPClient{
client: &http.Client{
Transport: transport,
Timeout: timeout,
},
strictMode: strictmode,
}
}

type StrictHTTPClient struct {
client *http.Client
strictMode bool
}

func (s *StrictHTTPClient) Do(req *http.Request) (*http.Response, error) {
if s.strictMode && req.URL.Scheme != "https" {
return nil, errors.New("strictmode is enabled, but request is not over HTTPS")
}
return s.client.Do(req)
}
12 changes: 0 additions & 12 deletions core/http_client_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,6 @@ import (
"net/url"
"strings"
"testing"
"time"
)

func TestHTTPClient(t *testing.T) {
Expand Down Expand Up @@ -91,17 +90,6 @@ func TestHTTPClient(t *testing.T) {
})
}

func TestStrictHTTPClient_Do(t *testing.T) {
t.Run("error on HTTP call when strictmode is enabled", func(t *testing.T) {
client := NewStrictHTTPClient(true, time.Second, nil)
httpRequest, _ := stdHttp.NewRequest("GET", "http://example.com", nil)

_, err := client.Do(httpRequest)

assert.Error(t, err)
})
}

func TestUserAgentRequestEditor(t *testing.T) {
GitVersion = ""
req := &stdHttp.Request{Header: map[string][]string{}}
Expand Down
3 changes: 2 additions & 1 deletion discovery/api/server/client/http.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ import (
"github.com/nuts-foundation/go-did/vc"
"github.com/nuts-foundation/nuts-node/core"
"github.com/nuts-foundation/nuts-node/discovery/log"
"github.com/nuts-foundation/nuts-node/http/client"
"io"
"net/http"
"net/url"
Expand All @@ -36,7 +37,7 @@ import (
// New creates a new DefaultHTTPClient.
func New(strictMode bool, timeout time.Duration, tlsConfig *tls.Config) *DefaultHTTPClient {
return &DefaultHTTPClient{
client: core.NewStrictHTTPClient(strictMode, timeout, tlsConfig),
client: client.NewWithTLSConfig(timeout, tlsConfig),
}
}

Expand Down
3 changes: 2 additions & 1 deletion docs/pages/deployment/cli-reference.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ The following options apply to the server commands below:
--crypto.azurekv.hsm Whether to store the key in a hardware security module (HSM). If true, the Azure Key Vault must be configured for HSM usage. Default: false
--crypto.azurekv.timeout duration Timeout of client calls to Azure Key Vault, in Golang time.Duration string format (e.g. 10s). (default 10s)
--crypto.azurekv.url string The URL of the Azure Key Vault.
--crypto.storage string Storage to use, ''%s' for an external backend (deprecated).' for file system (for development purposes), 'fs' for HashiCorp Vault KV store, 'vaultkv' for Azure Key Vault.%!(EXTRA string=azure-keyvault, string=external)
--crypto.storage string Storage to use, 'fs' for file system (for development purposes), 'vaultkv' for HashiCorp Vault KV store, 'azure-keyvault' for Azure Key Vault, 'external' for an external backend (deprecated).
--crypto.vault.address string The Vault address. If set it overwrites the VAULT_ADDR env var.
--crypto.vault.pathprefix string The Vault path prefix. (default "kv")
--crypto.vault.timeout duration Timeout of client calls to Vault, in Golang time.Duration string format (e.g. 1s). (default 5s)
Expand All @@ -38,6 +38,7 @@ The following options apply to the server commands below:
--events.nats.timeout int Timeout for NATS server operations (default 30)
--goldenhammer.enabled Whether to enable automatically fixing DID documents with the required endpoints. (default true)
--goldenhammer.interval duration The interval in which to check for DID documents to fix. (default 10m0s)
--http.cache.maxbytes int HTTP client maximum size of the response cache in bytes. If 0, the HTTP client does not cache responses. (default 10485760)
--http.internal.address string Address and port the server will be listening to for internal-facing endpoints. (default "127.0.0.1:8081")
--http.internal.auth.audience string Expected audience for JWT tokens (default: hostname)
--http.internal.auth.authorizedkeyspath string Path to an authorized_keys file for trusted JWT signers
Expand Down
Loading
Loading