Issue with certificate validity check

[radhupr]

I have two cases where the cert validity check fails.

1. SecretType: Opaque and custom Key example below. The metrics exported gives value 0 for validity days check(screenshot)
   • type: Opaque
     key: caCert
   • type: Opaque
     key: clientCert

2. I have added daemonset config to monitor a kubeconf file used by kubelet on the nodes. The file has reference to ca cert and client cert as path. The daemonsets complains "failed to parse \"/mnt/watch/kube-7a917bc4a584e5a4952cd8401e656e1cec0f2cee/data/folder/kubelet/kubelet.conf\", readlink ///kubelet/path/pki/kubelet-client-current.pem: no such file or directory"

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <base64 encoded val>
    server: https://lb-cluster.sub.domain.company.com:443
  name: default-cluster
contexts:
- context:
    cluster: default-cluster
    namespace: default
    user: default-auth
  name: default-context
...
.....
users:
- name: default-auth
  user:
    client-certificate: /kubelet/path/pki/kubelet-client-current.pem
    client-key: /kubelet/path/pki/kubelet-client-current.pem

My config looks as below:

hostPathsExporter:
  daemonSets:
    nodes:
      watchDirectories:
        - /kubelet/path/pki
      watchKubeconfFiles:
        - /data/folder/kubelet/kubelet.conf

[radhupr]

I'm using helm installation on our cluster. v3.18.1

[plaffitt]

Hello,

What metric exactly ? x509_cert_expired, x509_cert_not_after or x509_cert_not_after?

I tried to reproduce your issue without success (x509_cert_expired metrics gives 0 as a result but it is as expected).

Here is my setup, tell me if I missed anything:

values.yaml:

secretsExporter:
  podAnnotations:
    prometheus.io/port: "9793"
    prometheus.io/scrape: "true"
  secretTypes:
  - type: Opaque
    key: caCert
  - type: Opaque
    key: clientCert
service:
  create: false
prometheusServiceMonitor:
  create: false
prometheusRules:
  create: false

cert.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: my-tls-secret
type: Opaque
data:
  caCert: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURhekNDQWxPZ0F3SUJBZ0lVUXNrdHdBZTVoYVlySnMvbWF2V2pBcytUZlJ3d0RRWUpLb1pJaHZjTkFRRUwKQlFBd1JURUxNQWtHQTFVRUJoTUNRVlV4RXpBUkJnTlZCQWdNQ2xOdmJXVXRVM1JoZEdVeElUQWZCZ05WQkFvTQpHRWx1ZEdWeWJtVjBJRmRwWkdkcGRITWdVSFI1SUV4MFpEQWVGdzB5TkRFeU16QXhOekF5TkRGYUZ3MHlOVEV5Ck16QXhOekF5TkRGYU1FVXhDekFKQmdOVkJBWVRBa0ZWTVJNd0VRWURWUVFJREFwVGIyMWxMVk4wWVhSbE1TRXcKSHdZRFZRUUtEQmhKYm5SbGNtNWxkQ0JYYVdSbmFYUnpJRkIwZVNCTWRHUXdnZ0VpTUEwR0NTcUdTSWIzRFFFQgpBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRRHBRZUF4ek1abzdFQitYcWF4YkZHVXErajM2eEJPYllqUytHRWUwYzVQCm5HZFlIb29RZWpxWTdQYXdpdlhhQjdvTzJnNEtueEpDanlyZEhlcEVzUjF6eXBweWJrVkpvZjRYZUY2WmJCbGwKOWF2eXlMUEJkZWIxUnhvR2N3ZmY0cU9lblJCRUJkRHRJakVjcC9Mb1dEYlBQd0lLWjJDOUtUVWZ6eUdHRlRYcgpHZ1FVdTVDWEx5REk3VEZEN3QyZVRZUTFYclBYdlgzZVZEVkV2Rkh3c3Z4eUtiSUlDU0hCZE4rWnBTR3lkcUdXCkliOEZOdTRyZFhkV3BXcDFKZ3B0K3hOaUc1UDBtK1R3TGhjbEo5V0RoQzdXaVVwY0kvYU56Q1FmdnV6enJTZXQKM1dLengwdGNvNm92UkVNbkRsUDQwUDZaZVlqaGVSbERNM0lXWHJvMDYvanJBZ01CQUFHalV6QlJNQjBHQTFVZApEZ1FXQkJUQ3AzZUo5ck92aVh2aS9Ja1Nnb3k0WnBUWkdEQWZCZ05WSFNNRUdEQVdnQlRDcDNlSjlyT3ZpWHZpCi9Ja1Nnb3k0WnBUWkdEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQncKRHVpZWgvZi9PV0JwQkFueVZFV0JDQXc2SXUwWVc0UE92OFY1NkoxZjlhb2hUNWxsOUROejJiVUg0NEEyN0JhTgpOd0N0QUd5cjlmK2txK05hb2tRckZ1VEZ4Yzg2UkYwVlBKc0c0TkdPMlBJaHVJSUY1YkZmSTFGZWJpYTJnT0M0CmN0Wk9NY2ZxSDdoaVdDMkpsRW55YXlzS1RWWTJzOG14RTdoek43YWszUVd0S1A1d2xPR0FVak1lTi94aVduQzMKMUt3V3hxeFkwS09GQmlFVFZZYmJEVStGeHhIeVBndnhGZFFYYVVGSGFVRDdVK1IybmdDaW9qOVJyNHJrS0gzbwpHSGJRVU9vVXRsZjk5SVA1WWtLS3VkVDFtaUdaQ0pTTHBsQ3VIeXZwaXhqK2ZGMEJhYmJ5NjIrRTV1bU9DR3AxClIyZzYyMnh5ckQvUnJYTERTUHJlCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
  clientCert: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURXakNDQWtLZ0F3SUJBZ0lVY2FsOWRhbjlaY0t6Tmh4QWFZUmhGM2psQUZFd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1JURUxNQWtHQTFVRUJoTUNRVlV4RXpBUkJnTlZCQWdNQ2xOdmJXVXRVM1JoZEdVeElUQWZCZ05WQkFvTQpHRWx1ZEdWeWJtVjBJRmRwWkdkcGRITWdVSFI1SUV4MFpEQWVGdzB5TkRFeU16QXhOekF5TlRKYUZ3MHlOVEV5Ck16QXhOekF5TlRKYU1FVXhDekFKQmdOVkJBWVRBa0ZWTVJNd0VRWURWUVFJREFwVGIyMWxMVk4wWVhSbE1TRXcKSHdZRFZRUUtEQmhKYm5SbGNtNWxkQ0JYYVdSbmFYUnpJRkIwZVNCTWRHUXdnZ0VpTUEwR0NTcUdTSWIzRFFFQgpBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRQzdqUkFYM1YrNDRKbWk1bnZvSlhaRjZTb052Q0lmMkU0VUJPTmlpWWdRCnRpY2xvaERGMXluNEtqaUdQcWRjSWtWMHdJdXY4Sit1ZmFPbnFJMnhxZ3hIL2d0eW1XZTZjaUJCbDBJbTBROVUKU0NqU3RCOENkM1JRWE1ISmxyZ2JlOHVycW5reGlEQnhjWlo1M0tSRWtzSFUxMHYwOE9WNjU1Z0NJS1YvWThmYQpNQW1UaWpHK1YvUDVCeEdKM0VncmdsRGJ2ZEI3Wk9kdVhnb3F6b3lZT1k2L0NCRytERTNuVk9IYi9GTVNrVG1MCnZHNyt5RjFNSnlNR0NUT3BNdmladGMrczRoRlp5NHM4MUVhZzE2anJtTzR0L3N4Y2tjL0V5ZUZnZkxBNDJHRGkKK0FEMjM1QnUxSXA1TWluOFVmczhnYVF1NkRsVUcxUXIvOHVuVllmWDBqaVhBZ01CQUFHalFqQkFNQjBHQTFVZApEZ1FXQkJSN0F3OHZVUGRUYUJZdHQ0cG5XMjZZWitpU1d6QWZCZ05WSFNNRUdEQVdnQlRDcDNlSjlyT3ZpWHZpCi9Ja1Nnb3k0WnBUWkdEQU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFTRUUwUm01cWNWbmtzSURYVmhPcTNFTHQKTkhuZTd4QUVlV0xDQ254RWFycGlNWXJiSzFoQmNyWXJMVjVjT0wxM2RLMms0ZjluWDZLVjlEZmd1Y1V4MUtPYgpuYzRaS3BKR1NuZzZkSllyQ1oyZ3FIQ2o5MTU1cnRPTDFZdng4a01VTTRiWGFyS0lObHJPdmRnd2RVTGpQeWV5ClcyaXNQT21TeEpFR1ltbGxYR2RtMXcyL01CN0JtUEloMngwVm5FcGxwTjhDa2JoeFBaSHdVei9WK05TSHJGalgKOGgvckxqbzdLWUk1U0dFMXc5Nm94dEpRcWcwUFpsc0IzZ1pFdExwUXJ5eGhkY2xNRkZpUFFtakpDTnZBRnVvVwpaSERrdE9MOGsxd0JnYmRUdHE0TEk4dkJVZXAwclpXODZmRVpDWmJCRy9WdzA4eXpnbzdaVm8yVUpWRjNMdz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K

And got this output:

# HELP x509_cert_expired Indicates if the certificate is expired
# TYPE x509_cert_expired gauge
x509_cert_expired{issuer_C="AU",issuer_O="Internet Widgits Pty Ltd",secret_key="caCert",secret_name="my-tls-secret",secret_namespace="default",serial_number="381279826056627323497150651996306024695967481116",subject_C="AU",subject_O="Internet Widgits Pty Ltd"} 0
x509_cert_expired{issuer_C="AU",issuer_O="Internet Widgits Pty Ltd",secret_key="clientCert",secret_name="my-tls-secret",secret_namespace="default",serial_number="648895712116379187671852670445008588586732552273",subject_C="AU",subject_O="Internet Widgits Pty Ltd"} 0
# HELP x509_cert_not_after Indicates the certificate's not after timestamp
# TYPE x509_cert_not_after gauge
x509_cert_not_after{issuer_C="AU",issuer_O="Internet Widgits Pty Ltd",secret_key="caCert",secret_name="my-tls-secret",secret_namespace="default",serial_number="381279826056627323497150651996306024695967481116",subject_C="AU",subject_O="Internet Widgits Pty Ltd"} 1.767114161e+09
x509_cert_not_after{issuer_C="AU",issuer_O="Internet Widgits Pty Ltd",secret_key="clientCert",secret_name="my-tls-secret",secret_namespace="default",serial_number="648895712116379187671852670445008588586732552273",subject_C="AU",subject_O="Internet Widgits Pty Ltd"} 1.767114172e+09
# HELP x509_cert_not_before Indicates the certificate's not before timestamp
# TYPE x509_cert_not_before gauge
x509_cert_not_before{issuer_C="AU",issuer_O="Internet Widgits Pty Ltd",secret_key="caCert",secret_name="my-tls-secret",secret_namespace="default",serial_number="381279826056627323497150651996306024695967481116",subject_C="AU",subject_O="Internet Widgits Pty Ltd"} 1.735578161e+09
x509_cert_not_before{issuer_C="AU",issuer_O="Internet Widgits Pty Ltd",secret_key="clientCert",secret_name="my-tls-secret",secret_namespace="default",serial_number="648895712116379187671852670445008588586732552273",subject_C="AU",subject_O="Internet Widgits Pty Ltd"} 1.735578172e+09

[radhupr]

@plaffitt Thanks for getting back to this. The issue was for x509_cert_not_after and x509_cert_not_before metrics. I retested and it seems the issue is not there. May be it was some kind of temporary issue. The first issue can be considered as resolved.

I still see error on node daemonset about the file reference mentioned on second point. Could you help me check that?

Thanks.

[syjabri]

Hey, I'm getting the same warning in the x509-certificate-exporter logs any news regarding that topic level=WARN msg="failed to parse \"/mnt/watch/kube-87dc0192f1ae9905eaf104bc294b632069be55fe/etc/kubernetes/kubelet.conf\", readlink ///var/lib/kubelet/pki/kubelet-client-current.pem: no such file or directory"