Skip to content

Commit 3e8bc8d

Browse files
smirasmira
committed
feat: enable watch retries on Omni side
Enable retry logger in `omnictl`, now if Omni goes down, `omnictl` can correctly restart watches: ``` $ _out/omnictl-linux-amd64 get users -w * NAMESPACE TYPE ID VERSION + default User be9d5331-74b9-4dc8-a4eb-387107faf262 1 + default User cad2763b-1477-4bd5-9bc9-b3a369e39ffd 1 + default User d4a0e14c-5b76-43ad-9b81-08a95cdd6731 1 2024-11-05T20:04:11.309+0400 WARN client/client.go:504 watch retrying {"error": "rpc error: code = Internal desc = server closed the stream without sending trailers", "bookmark": "AAAAAAAAASw=", "backoff": "678.945448ms", "namespace": "default", "type": "Users.omni.sidero.dev"} 2024-11-05T20:04:11.990+0400 WARN client/client.go:504 watch retrying {"error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 100.119.35.2:443: connect: connection refused\"", "bookmark": "AAAAAAAAASw=", "backoff": "979.468482ms", "namespace": "default", "type": "Users.omni.sidero.dev"} 2024-11-05T20:04:12.970+0400 WARN client/client.go:504 watch retrying {"error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 100.119.35.2:443: connect: connection refused\"", "bookmark": "AAAAAAAAASw=", "backoff": "850.913234ms", "namespace": "default", "type": "Users.omni.sidero.dev"} 2024-11-05T20:04:13.821+0400 WARN client/client.go:504 watch retrying {"error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 100.119.35.2:443: connect: connection refused\"", "bookmark": "AAAAAAAAASw=", "backoff": "1.683608302s", "namespace": "default", "type": "Users.omni.sidero.dev"} 2024-11-05T20:04:15.505+0400 WARN client/client.go:504 watch retrying {"error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 100.119.35.2:443: connect: connection refused\"", "bookmark": "AAAAAAAAASw=", "backoff": "3.759619924s", "namespace": "default", "type": "Users.omni.sidero.dev"} 2024-11-05T20:04:19.266+0400 WARN client/client.go:504 watch retrying {"error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 100.119.35.2:443: connect: connection refused\"", "bookmark": "AAAAAAAAASw=", "backoff": "3.985097297s", "namespace": "default", "type": "Users.omni.sidero.dev"} 2024-11-05T20:04:23.253+0400 WARN client/client.go:504 watch retrying {"error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 100.119.35.2:443: connect: connection refused\"", "bookmark": "AAAAAAAAASw=", "backoff": "4.74055025s", "namespace": "default", "type": "Users.omni.sidero.dev"} 2024-11-05T20:04:27.995+0400 WARN client/client.go:504 watch retrying {"error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 100.119.35.2:443: connect: connection refused\"", "bookmark": "AAAAAAAAASw=", "backoff": "7.308402811s", "namespace": "default", "type": "Users.omni.sidero.dev"} 2024-11-05T20:04:35.309+0400 WARN client/client.go:504 watch retrying {"error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 100.119.35.2:443: connect: connection refused\"", "bookmark": "AAAAAAAAASw=", "backoff": "8.806341922s", "namespace": "default", "type": "Users.omni.sidero.dev"} default User cad2763b-1477-4bd5-9bc9-b3a369e39ffd 2 - default User cad2763b-1477-4bd5-9bc9-b3a369e39ffd 2 ``` Signed-off-by: Andrey Smirnov <[email protected]>
1 parent 23ccdb5 commit 3e8bc8d

File tree

7 files changed

+34
-10
lines changed

7 files changed

+34
-10
lines changed

‎client/go.mod

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ require (
1414
github.com/ProtonMail/gopenpgp/v2 v2.7.5
1515
github.com/adrg/xdg v0.5.3
1616
github.com/blang/semver v3.5.1+incompatible
17-
github.com/cosi-project/runtime v0.6.4
17+
github.com/cosi-project/runtime v0.7.1
1818
github.com/dustin/go-humanize v1.0.1
1919
github.com/fatih/color v1.18.0
2020
github.com/gertd/go-pluralize v0.2.1

‎client/go.sum

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,8 +27,8 @@ github.com/containerd/go-cni v1.1.10 h1:c2U73nld7spSWfiJwSh/8W9DK+/qQwYM2rngIhCy
2727
github.com/containerd/go-cni v1.1.10/go.mod h1:/Y/sL8yqYQn1ZG1om1OncJB1W4zN3YmjfP/ShCzG/OY=
2828
github.com/containernetworking/cni v1.2.3 h1:hhOcjNVUQTnzdRJ6alC5XF+wd9mfGIUaj8FuJbEslXM=
2929
github.com/containernetworking/cni v1.2.3/go.mod h1:DuLgF+aPd3DzcTQTtp/Nvl1Kim23oFKdm2okJzBQA5M=
30-
github.com/cosi-project/runtime v0.6.4 h1:roifc5e+Q1+72EI36BYSRT9aXyskU+coiKHeoBBWkMg=
31-
github.com/cosi-project/runtime v0.6.4/go.mod h1:EMLs8a55tJ6zA4UyDbRsTvXBd6UIlNwZfCVGvCyiXK8=
30+
github.com/cosi-project/runtime v0.7.1 h1:cOF2/ljLa0NPQV6/S1RZHcSUXM0UyOlD5F3fwwYSQEg=
31+
github.com/cosi-project/runtime v0.7.1/go.mod h1:EMLs8a55tJ6zA4UyDbRsTvXBd6UIlNwZfCVGvCyiXK8=
3232
github.com/cpuguy83/go-md2man/v2 v2.0.4/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
3333
github.com/cpuguy83/go-md2man/v2 v2.0.5 h1:ZtcqGrnekaHpVLArFSe4HK5DoKx1T0rq2DwVB0alcyc=
3434
github.com/cpuguy83/go-md2man/v2 v2.0.5/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=

‎client/pkg/client/omni/omni.go

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ import (
1111
"github.com/cosi-project/runtime/api/v1alpha1"
1212
"github.com/cosi-project/runtime/pkg/state"
1313
"github.com/cosi-project/runtime/pkg/state/protobuf/client"
14+
"go.uber.org/zap"
1415
"google.golang.org/grpc"
1516
"google.golang.org/grpc/metadata"
1617

@@ -27,6 +28,7 @@ import (
2728

2829
// Options defines additional Omni client options.
2930
type Options struct {
31+
retryLogger *zap.Logger
3032
infraProviderID string
3133
}
3234

@@ -40,6 +42,13 @@ func WithProviderID(id string) Option {
4042
}
4143
}
4244

45+
// WithRetryLogger sets logger for retry operations.
46+
func WithRetryLogger(lg *zap.Logger) Option {
47+
return func(o *Options) {
48+
o.retryLogger = lg
49+
}
50+
}
51+
4352
// Client for Omni resource API (COSI).
4453
type Client struct {
4554
conn *grpc.ClientConn
@@ -51,13 +60,16 @@ type Client struct {
5160
func NewClient(conn *grpc.ClientConn, options ...Option) *Client {
5261
c := &Client{
5362
conn: conn,
63+
options: Options{
64+
retryLogger: zap.NewNop(),
65+
},
5466
}
5567

5668
for _, o := range options {
5769
o(&c.options)
5870
}
5971

60-
c.state = state.WrapCore(client.NewAdapter(v1alpha1.NewStateClient(c)))
72+
c.state = state.WrapCore(client.NewAdapter(v1alpha1.NewStateClient(c), client.WithRetryLogger(c.options.retryLogger)))
6173

6274
return c
6375
}

‎client/pkg/omnictl/internal/access/client.go

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,10 @@ import (
1414
"github.com/cosi-project/runtime/pkg/safe"
1515
"github.com/cosi-project/runtime/pkg/state"
1616
"github.com/siderolabs/go-api-signature/pkg/serviceaccount"
17+
"go.uber.org/zap"
1718

1819
"github.com/siderolabs/omni/client/pkg/client"
20+
"github.com/siderolabs/omni/client/pkg/client/omni"
1921
"github.com/siderolabs/omni/client/pkg/omni/resources"
2022
"github.com/siderolabs/omni/client/pkg/omni/resources/system"
2123
"github.com/siderolabs/omni/client/pkg/omnictl/config"
@@ -112,6 +114,16 @@ func WithClient(f func(ctx context.Context, client *client.Client) error, client
112114
url = endpointEnv
113115
}
114116

117+
loggerCfg := zap.NewDevelopmentConfig()
118+
loggerCfg.Development = false
119+
120+
logger, err := loggerCfg.Build()
121+
if err != nil {
122+
return err
123+
}
124+
125+
opts = append(opts, client.WithOmniClientOptions(omni.WithRetryLogger(logger)))
126+
115127
client, err := client.New(url, opts...)
116128
if err != nil {
117129
return err

‎frontend/bun.lockb

-228 KB
Binary file not shown.

‎go.mod

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,8 +26,8 @@ require (
2626
github.com/blang/semver v3.5.1+incompatible
2727
github.com/blang/semver/v4 v4.0.0
2828
github.com/containers/image/v5 v5.32.2
29-
github.com/cosi-project/runtime v0.6.4
30-
github.com/cosi-project/state-etcd v0.3.2
29+
github.com/cosi-project/runtime v0.7.1
30+
github.com/cosi-project/state-etcd v0.4.0
3131
github.com/crewjam/saml v0.4.14
3232
github.com/dustin/go-humanize v1.0.1
3333
github.com/emicklei/dot v1.6.2

‎go.sum

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -109,10 +109,10 @@ github.com/coreos/go-semver v0.3.1 h1:yi21YpKnrx1gt5R+la8n5WgS0kCrsPp33dmEyHReZr
109109
github.com/coreos/go-semver v0.3.1/go.mod h1:irMmmIw/7yzSRPWryHsK7EYSg09caPQL03VsM8rvUec=
110110
github.com/coreos/go-systemd/v22 v22.5.0 h1:RrqgGjYQKalulkV8NGVIfkXQf6YYmOyiJKk8iXXhfZs=
111111
github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
112-
github.com/cosi-project/runtime v0.6.4 h1:roifc5e+Q1+72EI36BYSRT9aXyskU+coiKHeoBBWkMg=
113-
github.com/cosi-project/runtime v0.6.4/go.mod h1:EMLs8a55tJ6zA4UyDbRsTvXBd6UIlNwZfCVGvCyiXK8=
114-
github.com/cosi-project/state-etcd v0.3.2 h1:8Lnj5dPIUzkv5DYFpKvkP+vX5O9M9sWthXQfoKJrF78=
115-
github.com/cosi-project/state-etcd v0.3.2/go.mod h1:wHp8V3OvGRILYvbJEmhMt0AFuByaGNkxXLDc61/+yFM=
112+
github.com/cosi-project/runtime v0.7.1 h1:cOF2/ljLa0NPQV6/S1RZHcSUXM0UyOlD5F3fwwYSQEg=
113+
github.com/cosi-project/runtime v0.7.1/go.mod h1:EMLs8a55tJ6zA4UyDbRsTvXBd6UIlNwZfCVGvCyiXK8=
114+
github.com/cosi-project/state-etcd v0.4.0 h1:/zf2mbqVHnp32wWDBaFGa5U1+ZwyenBzeZOgrxmNzhY=
115+
github.com/cosi-project/state-etcd v0.4.0/go.mod h1:V3UiPxXs2Nl3kls8TSnk7ABKx68fCjNZDTu2Ux9CGK8=
116116
github.com/cpuguy83/go-md2man/v2 v2.0.4/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
117117
github.com/cpuguy83/go-md2man/v2 v2.0.5 h1:ZtcqGrnekaHpVLArFSe4HK5DoKx1T0rq2DwVB0alcyc=
118118
github.com/cpuguy83/go-md2man/v2 v2.0.5/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=

0 commit comments

Comments
 (0)