bgp_best_path_policies - Fails with ACI 614

[camrossi]

ACI 614 introduces an API level check to block applying the ignore_igp_metric flag unless the L3OUT is a Floating SVI.

However the provider runs the node_profiles module before the interface_profiles and that make sense but this breaks the terraform deployment as it tries to apply the bgp.as_path_policy before the FSVI are created.

For example

l3outs:
  - name: cilium-custer-mesh-c1
    node_profiles:
      - name: anchors
        nodes:
          - node_id: 101
            router_id: 101.1.0.3
        bgp:
          as_path_policy: cilium-bp
        interface_profiles:     
          - name: anchors              # this creates a single interface profile for the vpc, with 2x anchor nodes
            bfd_policy: cilium-bfd
            interfaces:
              - node_id: 101
                mode: regular
                vlan: 11
                floating_svi: true
                paths:
                  - floating_ip: 192.168.1.13/28
                    physical_domain: proxmox-cluster
                ip: 192.168.1.11/28
                ip_shared: 192.168.1.14/28
                mtu: 9000
                bgp_peers:
                  - ip: 192.168.1.0/28
                    remote_as: 65100
                    bfd: true
                    peer_prefix_policy: cilium-cni.
policies:
  bgp_best_path_policies:
    - name: cilium-bp
      ignore_igp_metric: true

Fails with:

│ Error: The post rest request failed
│ 
│   with module.aci.module.aci_l3out_node_profile_manual["cilium-custer-mesh-c1/cilium-custer-mesh-c1/anchors"].aci_rest_managed.bgpRsBestPathCtrlPol[0],
│   on .terraform/modules/aci/modules/terraform-aci-l3out-node-profile/main.tf line 290, in resource "aci_rest_managed" "bgpRsBestPathCtrlPol":
│  290: resource "aci_rest_managed" "bgpRsBestPathCtrlPol" {
│ 
│ Response Status Code: 400, Error Code: 1, Error Message: Invalid Configuration  - VRF Validation failed for VRF = uni/tn-cilium-custer-mesh-c1/ctx-vrf:  BGP Best Path Ctrl Policy:
│ uni/tn-cilium-custer-mesh-c1/bestpath-cilium-bp with Ignore IGP Metric is not supported with regular L3Out: uni/tn-cilium-custer-mesh-c1/out-dci configured on node 101. If this was an attempt to
│ modify, consider deletion followed by addition..

As a work around you can comment out the ignore_igp_metric: true run the plan un comment ignore_igp_metric: true and run the plan again.

[robvand]

Confirmed to be still working in 6.1(3).

There is however another issue with this module. Setting ignore_igp_metric: false results in an error.

  # module.aci.module.aci_bgp_best_path_policy["ts-ocp-1/cilium-bp"].aci_rest_managed.bgpBestPathCtrlPol will be updated in-place
  ~ resource "aci_rest_managed" "bgpBestPathCtrlPol" {
      ~ content     = {
          ~ "ctrl"  = "ignoreIgpMetric" -> null
            # (2 unchanged elements hidden)
        }
        id          = "uni/tn-ts-ocp-1/bestpath-cilium-bp"
        # (4 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.aci.module.aci_bgp_best_path_policy["ts-ocp-1/cilium-bp"].aci_rest_managed.bgpBestPathCtrlPol to include new values learned so far during apply, provider
│ "registry.terraform.io/ciscodevnet/aci" produced an invalid new value for .content["ctrl"]: was cty.StringVal(""), but now cty.StringVal("ignoreIgpMetric").
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Workaround is removing ignore_igp_metric: true so it sets ctrl to null.

[juchowan]

@camrossi running this config snippet (with vrf and domain) on 6.1.5 fabric and newest code works fine for me, I can't reproduce this problem. I don't have 6.1.4 to test but could this be some extra validation that was removed for 6.1.5? Are you able to verify on your side that this works fine now on 6.1.5 ?
@robvand I also cannot reproduce this error when changing ignore_igp_metric from true to false, or commenting out etc.