Fix DRATestDriver readiness passing before drivers and ResourceSlices exist

[alaypatel07]

What type of PR is this?

/kind bug
/kind flake

What this PR does / why we need it:

DRATestDriver readiness could pass while no driver DaemonSet pods or ResourceSlices existed yet (DesiredNumberScheduled == NumberReady == 0, and 0 ResourceSlices matching 0 running plugin pods). FastFill then starts before the DRA driver is usable, which shows up as scheduler "cannot allocate all claims" retries and inflated create_to_schedule / FastFillPodStartup p99.

This change waits until WorkerNodeCount driver pods are Desired/Ready and at least that many ResourceSlices are published, and uses one typed getWorkerCount helper for both manifest templating and readiness checks.

Which issue(s) this PR fixes:

Fixes # kubernetes/kubernetes#141046

Special notes for your reviewer:

Observed on aws-dra-500Nodes-with-workload FastFillPodStartup p99 spikes where scheduler logs repeatedly report "500 cannot allocate all claims" before binding succeeds.

[kubernetes-prow]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: alaypatel07
Once this PR has been reviewed and has the lgtm label, please assign wojtek-t for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• clusterloader2/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[alaypatel07]

From what I recall, this logic was put in place because at larger scale, some nodes might go to NotReady and it will block the test from moving forward. To avoid that, only running pods are counted against resource slices.

However, I see two issues with this:

1. the logic is in-correct in finding terminal state. The resource slice count will be equal to running pods at many different points of the driver daemonset rollout not just the terminal state.
2. If the nodes are NotReady, you do not have enough capacity and the test is going to fail anyway. So user running the tests should find other ways of dealing with NotReady Nodes, like cordoning/tainting them and subtracting them from node count, so it is ignored by the test.

[johnbelamaric]

@alaypatel07 is the failure related?

[alaypatel07]

@johnbelamaric no the specific test doesnt exercise DRA path:

kubetest2: Test expand_less23m1s{  exit status 255} | kubetest2: Test expand_less | 23m1s | {  exit status 255}
kubetest2: Test expand_less | 23m1s
{  exit status 255}

Failure seems to be unrelated.

[alaypatel07]

/test pull-perf-tests-ec2-500-node-dra-with-workload-amazonvpc-using-cl2

[alaypatel07]

I0730 21:23:58.534542   21321 framework.go:285] Applying templates for "manifest/exec_deployment.yaml"
I0730 21:24:08.705996   21321 framework.go:285] Applying templates for "manifests/daemonset.yaml"
I0730 21:25:18.842531   21321 clusterloader.go:494] Test config successfully dumped to: /logs/artifacts/generatedConfig_dra-steady-state.yaml
I0730 21:25:18.842544   21321 clusterloader.go:265] --------------------------------------------------------------------------------
I0730 21:25:18.842547   21321 clusterloader.go:266] Running /home/prow/go/src/k8s.io/perf-tests/clusterloader2/testing/dra/config.yaml
I0730 21:25:18.842548   21321 clusterloader.go:267] --------------------------------------------------------------------------------
I0730 21:25:19.036291   21321 framework.go:285] Applying templates for "manifests/dra-example-driver/*.yaml"
E0730 21:30:34.074327   21321 clusterloader.go:275] --------------------------------------------------------------------------------
E0730 21:30:34.074348   21321 clusterloader.go:276] Test Finished
E0730 21:30:34.074350   21321 clusterloader.go:277]   Test: /home/prow/go/src/k8s.io/perf-tests/clusterloader2/testing/dra/config.yaml
E0730 21:30:34.074352   21321 clusterloader.go:278]   Status: Fail
E0730 21:30:34.074354   21321 clusterloader.go:280]   Errors: [dependency Install dra-example-driver for test setup timed out after 5m0s]
E0730 21:30:34.074355   21321 clusterloader.go:282] --------------------------------------------------------------------------------

https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/perf-tests/4248/pull-perf-tests-ec2-500-node-dra-with-workload-amazonvpc-using-cl2/2082934316282679296

this is relevant failure, but I dont see any logs to debug. I will try it locally if I can reproduce this scale.

[alaypatel07]

/test pull-perf-tests-ec2-500-node-dra-with-workload-amazonvpc-using-cl2