bug: using ApiExports and ApiBindings produces strange number of logicalclusters

[SimonTheLeg]

We were testing with a test which creates 10000 workspaces and afterwards creates 500 ApiExports and 9500 ApiBindings. Lastly it creates objects of the particular type in the 9500 consumer workspaces.
After applying the Bindings and Exports the number of logicalclusters jumps from 10000 to 13363, which seems a bit strange and not rooted in any correlation to our test-numbers.

Additionally this spike also only happened for two of three shards (root & shard-2), but not for shard-3. It also doesnot seem to be reported for any of the non-active replicas

/kind bug

[mjudeikis]

ANALYSIS — RAM cost of APIBindings/APIExports

TL;DR

Adding 100 APIExports + 5,000 APIBindings on top of a 1,000-workspace
kcp adds ~4.5 GiB resident heap and ~345k goroutines, and the cost
does not drain over 5 minutes of idle — it is steady-state, not
transient alloc churn.

Per APIBinding (after backing out workspace overhead):

• ~66 goroutines
• ~0.85 MiB HeapInuse
• ~22 informer ring buffers (k8s.io/utils/buffer.TypedRingGrowing[interface{}])

~99% of the new goroutines and ~44% of the new heap trace to a single
controller: kcp-kube-quota, whose QuotaMonitor spins up a fresh
informer (and supporting goroutines + event ring buffer) per
APIBinding-surfaced GVR in every consumer workspace.

Setup

Level	Command	Workspaces	APIExports	APIBindings
L1 (workspaces only)	quickstart --scenario workspaces --tree-depth 10 --tree-count 1000	1,003	0	0
L2 (api-provider)	quickstart --scenario api-provider --providers-count 100 --consumers-count 1000 --bindings-per-consumer 5	1,103	100	5,000

L2 has 100 more workspaces than L1 (100 providers + 1,000 consumers
vs. L1's 1,000-node tree under one org). We back this out using the
L1−L0 per-workspace cost (~152 goroutines, ~3 MiB HeapInuse) before
per-binding attribution.

Snapshots used:

• L1_workspaces/T1_after_create (2026-05-29 08:35 EEST)
• L2_apiprovider/T1_after_create (2026-05-29 10:00 EEST)
• L2_apiprovider/T2_settled_5m (2026-05-29 10:05 EEST)

Headline RAM diff

Metric	L1	L2 (T1)	Δ (L2 − L1)	Per-binding¹
Goroutines	155,905	501,535	+345,630 (+222%)	~66
HeapAlloc	2,116 MiB	5,170 MiB	+3,054 MiB	~0.55 MiB
HeapInuse	3,331 MiB	7,795 MiB	+4,464 MiB	~0.83 MiB
HeapSys	4,149 MiB	10,382 MiB	+6,233 MiB	~1.19 MiB
HeapObjects	15.4 M	25.9 M	+10.5 M	~2,100

¹ Attribution: subtract 100 workspaces × (152 goroutines / 3 MiB)
≈ 15.2k goroutines / 300 MiB, treat 100 APIExports as negligible vs
5,000 bindings, divide remainder by 5,000.

Drain check (T2 settled 5m)

After 5 min of idle:

Metric	T1	T2	Δ
Goroutines	501,535	501,532	−3 (noise)
HeapAlloc	5,170 MiB	5,175 MiB	+5
HeapInuse	7,795 MiB	7,795 MiB	~0
HeapObjects	25.90 M	25.89 M	−9k (noise)
HeapReleased	689 MiB	1,676 MiB	+987 MiB (OS reclaim from idle pool, not freed objects)
MaxRSS	10,651 MiB	11,156 MiB	+505 (lifetime peak, monotonic)

Cost is resident. HeapInuse and live object count are flat across
the 5-min idle window. The +987 MiB on HeapReleased is Go runtime
giving back pages it was already counting as HeapIdle — working
set never included those.

Where the heap went

go tool pprof -top -inuse_space -base L1 L2_T2 (top flat allocators
of the +4,464 MiB delta):

Allocator	Flat	% of delta
k8s.io/utils/buffer.NewTypedRingGrowing[interface{}]	1,944 MiB	~44%
runtime.mallocgc	186 MiB	4%
informers.TypedRingGrowing.WriteOne	134 MiB	3%
apimachinery/v2/client.clientCache.Evict	132 MiB	3%
json.objectInterface (decode)	73 MiB	2%
apimachinery.DeepCopyJSONValue	49 MiB	1%
structToUnstructured (flat)	27 MiB / 95 cum	2% cum

Cum trail for the dominant 50% flat allocator:

wait.BackoffUntilWithContext              57% cum
└─ resourcequota.QuotaMonitor.SyncMonitors        56% cum
   └─ resourcequota.QuotaMonitor.controllerFor    55% cum
      └─ informers.AddEventHandlerWithResyncPeriod 54% cum
         └─ informers.newProcessListener           53% cum
            └─ buffer.NewTypedRingGrowing          51% flat  ← here

Ring buffer object count (Δ L2−L1): 110,547 rings,
1,944 MiB total → ~17.6 KiB per ring on average. Across 5,000
bindings that's ~22 rings per binding — i.e. each APIBinding's
GVR fans out across multiple informer layers (cache hierarchy:
shared informer + scoped + processor listeners), each holding its
own event ring.

Where the goroutines went

Goroutine attribution by controller pprof label:

Controller	L1	L2	Δ	Share of +345,630
kcp-kube-quota	122,366	465,466	+343,100	~99%
KCPKubeResourceQuota	3,018	3,318	+300	0.1%
unlabeled / other	~30,521	~32,748	~+2,200	0.6%

Per-workspace baseline of kcp-kube-quota is 122 goroutines (verified
in L0: 244 quota goroutines for root + system:shard). Solving for
per-binding cost:

L2 kcp-kube-quota = base_per_ws × total_ws + per_binding × total_bindings
465,466          = 122 × 1,103       + X × 5,000
                 = 134,566           + 5,000 X
       →  X      = 66.18 goroutines per APIBinding

Every APIBinding adds ~66 goroutines, all inside kcp-kube-quota.

Mechanism (hypothesis)

Each consumer workspace runs its own QuotaMonitor. The monitor
discovers the workspace's GVRs and starts an informer per GVR for
quota tracking. APIBindings surface new GVRs in the consumer, so:

+1 APIBinding in consumer C  →  +1 GVR visible to C's QuotaMonitor
                             →  +N informer-ring-buffers (informer + processor listeners)
                             →  +M quota-monitor goroutines

Observed N ≈ 22 rings, M ≈ 66 goroutines per binding.

This is a "tracks-everything-by-default" pattern: the QuotaMonitor
sets up the informer regardless of whether any ResourceQuota
object in the workspace actually constrains that GVR. For an
APIBinding to a custom resource with no quota policy, the informer
exists purely to support potential future quota enforcement and
never observes a meaningful event.

Sanity checks / confounds

• Workspace-shape confound: L1 is a deep tree, L2 is flat under
  one org. We back out workspace count, not workspace shape. The
  100-extra-workspace correction (~15k goroutines / ~300 MiB) is
  ~4% of the L2−L1 delta — small enough that shape effects don't
  change the per-binding numbers materially.
• Negative entries in pprof diff: some allocators (e.g.
  etcd/mvcc.KeyValue.Unmarshal at −41 MiB, grpc.simpleBufferPool
  at −19 MiB) show negative deltas — those are transient sites that
  happened to hold more bytes in the L1 capture instant than in L2;
  not load-bearing.
• kcp-kube-quota vs KCPKubeResourceQuota: separate controllers,
  both tagged via pprof labels. Only the former scales with bindings;
  the latter (~3k goroutines) is per-workspace-only and stays flat.

Recommended next steps

1. Verify per-binding vs per-consumer scaling. Re-run L2 with
   --bindings-per-consumer 1 (1,000 bindings total). If quota
   goroutines drop ~5× (343k → ~69k) the per-binding model holds.
   If they drop only marginally, the cost is "QuotaMonitor reacts
   to any binding-induced discovery refresh" and scales with
   consumers, not bindings.
2. Audit whether QuotaMonitor needs to start informers for
   APIBinding-surfaced GVRs at all. Skipping informer setup for
   workspaces that have no ResourceQuota constraining the GVR
   would zero out most of this cost. (Workloads outside the
   ResourceQuota admission path don't need quota tracking.)
3. Consider sharing informers across logical clusters for the
   same exported GVR. With round-robin binding 50 consumers share
   each APIExport — that's 50 separate quota informers tracking
   the same upstream stream. A scoped/shared informer would cut
   the ring buffer count from consumers × bindings_per_consumer
   toward bindings_per_consumer.
4. Cap or right-size the ring buffer initial capacity. ~18 KiB
   per ring × 110k rings = 1.9 GiB. The default TypedRingGrowing
   capacity may be over-provisioned for the quota use case (the
   QuotaMonitor's listener rarely backlogs).

The single biggest lever is #2 (don't track GVRs with no quota policy);
it would eliminate the majority of both the goroutines and the heap.