Overwrite provider manager flags

[nikParasyr]

User Story

I would like to be able to overwrite flags on the manager container that are set by default by the provider.

Detailed Description

Each provider sets some defaults flags. For example (for cluster-api core provider 1.8.5):

        - --diagnostics-address=:8443
        - --insecure-diagnostics=false
        - --use-deprecated-infra-machine-naming=false

I would like to be able to overwrite some of them.
When trying to use spec.manager.additionalArgs I get an error for duplicate args(in this case for the openstack infra provider, but im able to replicate on core providers as well):

E0625 14:43:25.629951 1 controller.go:347] "Reconciler error" err="failed to customize manager container: arg "--diagnostics-address" already exists" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capo-system/openstack" namespace="capo-system" name="openstack" reconcileID="b0de390a-2830-4575-b5b9-6cea2e555796"

Anything else you would like to add:

cluster-api-operator version: v0.20.0
Simple core provider definition to replicate this:

apiVersion: operator.cluster.x-k8s.io/v1alpha2
kind: CoreProvider
metadata:
  namespace: capi-system
  name: cluster-api
spec:
  version: v1.8.5
  manager:
    verbosity: 2
    featureGates:
      MachinePool: true
      ClusterTopology: true
      RuntimeSDK: true
      MachineSetPreflightChecks: false
    additionalArgs:
      --diagnostics-address=0.0.0.0:8443

Note: We specifically hit this issue with diagnostics-address, I see there is a spec.manager.metrics object but this writes the --metrics-bind-addr flag ( which has been deprecated in many providers ) and not the diagnostics one. A similar field for the diagnostics would work for our current use case but I think the problem is a bit more generic with not being able to overwrite default flags of a provider

/kind feature

[k8s-ci-robot]

This issue is currently awaiting triage.

If CAPI Operator contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[nikParasyr]

I was able to achieve what i want using this:

apiVersion: operator.cluster.x-k8s.io/v1alpha2
kind: CoreProvider
metadata:
  namespace: capi-system
  name: cluster-api
spec:
  version: v1.8.5
  manager:
    verbosity: 2
    featureGates:
      MachinePool: true
      ClusterTopology: true
      RuntimeSDK: true
      MachineSetPreflightChecks: false
  deployment:
    containers:
    - name: manager
      args: 
        --diagnostics-address: 0.0.0.0:8443

After some testing, there are 3 places were i can write manager args:

1. spec.manager.XYZ: these take precedence from the other and cannot be overwritten
2. spec.deployment.containers.0.args: these cannot overwrite args from 1, but can overwrite default flags of the provider
3. spec.manager.additionalArgs: these cannot overwrite args from 1. they cannot overwrite default flags of the provider

this is not so intuitive, i think it should at least be documented

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[fariaass]

HI @nikParasyr! These metrics fields were added to the CRDs in this PR #871