feat: Allow tagging on per-subnet basis

[ghost]

Description

New variables public_subnet_tags_per_subnet, private_subnet_tags_per_subnet, etc. Optional, no effect if not specified. If specified, they are additional tags to apply to each respective public and private subnet.

Extend the ability to allow naming and tagging route tables, EIPs, and NAT Gateways on per-subnet basis as well, add variables for that.

Motivation and Context

Previously, it was only possible to customize arbitrary tags on a per-AZ basis, not a per-subnet basis. You could customize the Name tag on a per-subnet basis but not any other tag.

The VPCs in our environment have several different types of subnets. We would like to tag them differently, e.g. for cost attribution and reference in external Terraform code. Currently this is not possible: subnets can only be disambiguated by VPC, AZ, and visibility.

Breaking Changes

No breaking changes

How Has This Been Tested?

• I have updated at least one of the examples/* to demonstrate and validate my change(s)
• I have tested and validated these changes using one or more of the provided examples/* projects
• I have executed pre-commit run -a on my pull request

This is currently running in production

[ghost]

Testing in our infrastructure with:

Generating expected Terraform plan:

[ghost]

This is now live in our infrastructure. I also added the ability to customize route table tagging.

[ghost]

I've completed the rollout of the new tagging system throughout our infrastructure, using the code from this pull request in production. No problems uncovered.

[kahirokunn]

Awesome!
LGTM!

[jseiser]

I think this behavior is what we are looking for.

We wanted to be able to have 2 sets of private subnets.

1 for our EKS cluster's worker nodes, and 1 for some other random servers that still survive post EKS migration. So we could just target the server private subnets when deploying those specific EC2 Instances

[davepattie]

Ahh, I see now that tags per az is no use when you have more than one private subnet per az

[hpschry]

I'd love to see this merged, this would address our use case as well: We'll need individual tags per sub-net, e.g. to control the assignment of load balancers to specific sub-nets in an EKS scenario, which is controlled by sub-net tags.

[reoring]

Once this Pull Request is merged, I will use it right away

[KevinBrooke]

We have been using this branch in production for 3 weeks without any issues.

[davepattie]

@antonbabenko would you mind taking a look at this PR please, it's a really helpful and thorough addition, thanks

[github-actions]

This PR has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this PR will be closed in 10 days

[kahirokunn]

keep

[jseiser]

@raxod502-plaid

Any chance you can fix the conflicts, id really love to get this one merged in.

[ghost]

There's not much point in fixing merge conflicts until we get confirmation from a maintainer that they are interested in merging the PR.

[github-actions]

This PR has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this PR will be closed in 10 days

[kahirokunn]

i need this PR.

[rarescosma]

We're hitting the same needs for our EKS setup at the moment: need to tag secondary subnets differently to tell karpenter to slowly start moving the nodes towards these subnets.

What are the plans for moving this PR forward?

[andrewmiskell]

I'm running into some of the same issues in our environment as well, needing to tag specific subnets differently due to different uses.

[netomin]

Hi, hitting the same needs here, it would be great to have this PR to move forward ;)

[kahirokunn]

I think so!

[ghost]

I personally do not see the use of this - it seems like an anti-pattern to me. What is the scenario where one subnet needs a particular tag, or set of tags, that is different from its neighboring subnet?

Why would it be assumed that all subnets within a VPC will be identical? There are many obvious use cases where that is not true, for example if different subnets are used for different workload types, or have different security properties, and so on.

Also, this PR is doing more than what its stating

Not sure where that claim comes from, the PR desc pretty clearly says: "Extend the ability to allow naming and tagging route tables, EIPs, and NAT Gateways on per-subnet basis as well, add variables for that."

I'm not sure what the route table changes are intended to do or why they are valid here

Can you clarify? Seems pretty clear to me that the route table changes are intended to allow you to add tags to your route tables. What is your query about whether they are "valid"?

you can also provide the subnet IDs to the AWS LBC controller as well [...] this is what data sources are created to solve

This is what tags are created to solve. Sure, you can shoehorn every possible way you might want to filter subnets in a data source by stuffing them all into the Name tag - or you can manually list out subnet IDs in config files - but it is way better to use tags the way they are intended to be used.

For example, we have tags that can be used to identify the subset of subnets that run Kubernetes workloads, the subset that are used for data-plane versus specialized control-plane redundancy, tags that can be used to identify subnets used for different subsets of Kubernetes clusters appropriate to different use cases, etc etc.

It's really unclear to me why it would not be recommended to use tags to manage your resources, in any context.

[bryantbiggs]

I personally do not see the use of this - it seems like an anti-pattern to me. What is the scenario where one subnet needs a particular tag, or set of tags, that is different from its neighboring subnet?

Why would it be assumed that all subnets within a VPC will be identical? There are many obvious use cases where that is not true, for example if different subnets are used for different workload types, or have different security properties, and so on.

Also, this PR is doing more than what its stating

Not sure where that claim comes from, the PR desc pretty clearly says: "Extend the ability to allow naming and tagging route tables, EIPs, and NAT Gateways on per-subnet basis as well, add variables for that."

I'm not sure what the route table changes are intended to do or why they are valid here

Can you clarify? Seems pretty clear to me that the route table changes are intended to allow you to add tags to your route tables. What is your query about whether they are "valid"?

you can also provide the subnet IDs to the AWS LBC controller as well [...] this is what data sources are created to solve

This is what tags are created to solve. Sure, you can shoehorn every possible way you might want to filter subnets in a data source by stuffing them all into the Name tag - or you can manually list out subnet IDs in config files - but it is way better to use tags the way they are intended to be used.

For example, we have tags that can be used to identify the subset of subnets that run Kubernetes workloads, the subset that are used for data-plane versus specialized control-plane redundancy, tags that can be used to identify subnets used for different subsets of Kubernetes clusters appropriate to different use cases, etc etc.

It's really unclear to me why it would not be recommended to use tags to manage your resources, in any context.

Always new this was going to be a contentious one, but oof 😅

If I look at the title of your PR and compare it with the code changes, they are quite a bit different. So the first couple of comments that you replied to, that is what I am referring to. PR body's are not captured in the changelog or release notes, only the title

Why would it be assumed that all subnets within a VPC will be identical? There are many obvious use cases where that is not true, for example if different subnets are used for different workload types, or have different security properties, and so on.

I don't follow this one. This module has always worked off the notion of subnet *groups - where there are different subnet group types (public, private, intra, database, etc.), and you can have n-number of subnets within each group. Their integration and configuration is pretty much consistent across all subnets within a subnet group - so how can one treat subnets differently within the same subnet group *within this module? or better yet, what is the motivation for doing this (or trying to do this)?

This is what tags are created to solve. Sure, you can shoehorn every possible way you might want to filter subnets in a data source by stuffing them all into the Name tag - or you can manually list out subnet IDs in config files - but it is way better to use tags the way they are intended to be used.

This one is hard to debate since its subjective. Tags are great - I don't think anyone will debate that. And we have tags for the subnet *groups (i.e. - public, private, intra, etc.), so thats great too and aligns with the statement above about consistent integration and configuration across those things created/associated within a subnet *group. But here, it sounds like you are wanting to name each individual subnet differently - which sounds like we are treating them as pets, sort of. And that is the use case I am struggling to find validity in supporting. In some instances above, I think folks are trying to take a subnet group and further segment that - so lets say we take the subnets created within the private subnet group and break those up further into n-number of sub-groups within that private subnet group. And I suspect this pattern is due to the fact that we currently do not support a generic - "create as many different subnet groups of your choosing" - its only those pre-defined subnet group types that we support in this module today. This is something @antonbabenko and I have discussed in the past; whether or not we should add a generic "subnets" or "subnet_group" (naming TBD) that would allow users to stamp out as many additional subnet groups as they would like, and we'd just need to figure out how that integrates into the main module we have today (so that it feels like a nautral extension of the main module)

Personally, I do not believe this is the intended use case for tags. If you look around within AWS, you will find many concepts that relate back to a container, and the things that go into that container. We have subnet groups for groups of subnets, we have node groups for groups of nodes (EKS), we even have a generic resource group which is more of a meta object within AWS. You create a new group for a particular use case - but you don't create a generic group and try to treat them differently within that group. Not at least when there is an existing notion of a group that allows you to create this explicit segregation of resources. For example - you wouldn't create a bunch of IAM users and then try to mange which users get which permissions by looking up their tags (metadata) - you would add them into the appropriate, respective group. Likewise for Kubernetes nodes - there are numerous ways of managing a collection of compute resources as a collective amongst the much larger whole (cluster).

So that said - if there were a generic sub-module where users could create various different groups of subnets, does this issue become a moot point?

[bryantbiggs]

But here, it sounds like you are wanting to name each individual subnet differently

*not just name, but more generically - treat them differently via unique tags

[ghost]

this pattern is due to the fact that we currently do not support a generic - "create as many different subnet groups of your choosing" - its only those pre-defined subnet group types that we support in this module today

Yes, that is the requirement. If you provided a way to create an arbitrary set of subnet groups then it would not be necessary to parameterize tags within a subnet group.

We aren't treating subnets as pets, we simply have more types of subnets than just "private" and "public" (plus the few other special cases in this module).

So that said - if there were a generic sub-module where users could create various different groups of subnets, does this issue become a moot point?

Yes, exactly.

[andrewmiskell]

I think a generic submodule to stamp out as many private/public sets of subnets I want each with it's own set of tags would make this whole thing moot, at least in my use-case.

In my case, I need to different sets of public subnets that have different tags. One tagged with Subnet_Type "public" and another tagged with Subnet_Type "sending" (our use case is very particular and somewhat edge case due to the custom software we use). Currently I'm using the redshift subnets feature with enable_public_redshift enabled in order to do this. Having a more generic way to create subnets without have to "repurpose" the pre-defined subnet types would be very helpful.

[erpel]

So this "what you're trying to do is covered by groups" argument is the first time I understood the resistance. To me the groups always seemed really type based rather than the main differentiation. This has me rethinking how we use the module in some way.

My use case mainly comes from a situation like this:
We have for example 5 subnets that are "private" in type. That's more than one per AZ - which is problematic in some services where only one subnet per AZ is accepted.

Using tags would be how I'd like to set which subnets of the type "private" should be used for EKS control plane (don't hold me to the exact example, I'm going off memory here).
The subnets are otherwise quite the same, the fact that they are separate subnets comes from some organizational history...

[bryantbiggs]

If I look at the recent issues/PRs, I think most of them can be traced back to - more subnet flexibility. Such as:

• Custom amount of NAT gateways #1065
• Using create_multiple_public_route_tables = true only creates an internet gateway route on one route table. #1087
• "Error in function call" when using neither private subnets nor NAT #944
• Pass already created VPC id  #1082
• Error when creating VPC without any private subnets and has NAT gateways #1068
• This module, doesnt integrate with the Network Firewall Module #978
• Enable all Az specific resources to have a {resource_type}_tags_per_az map #984
• Confusing DNS64 behaviour with public subnets #972
• fix: NAT gw route table associations shouldn't be created when enable_nat_gateway=false #1090
• feat: Custom NAT gateways #1066
• feat: Enable the creation of private NAT gateways #1072

(not including those that have since been closed - this should suffice to show the intent)

[github-actions]

This PR has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this PR will be closed in 10 days

[PLeS207]

up

[pseudomorph]

What's the state of this? Is there a path forward here?

[bryantbiggs]

it requires time and resources - nothing planned at the moment

[Chili-Man]

@bryantbiggs @antonbabenko We have a use case for this particular feature at work as well. As other folks have pointed out here in the comments already, we would like to have several private subnets, with different roles. In particular, we want to have our VPC look like this:

Public Subnets
3 public subnets for AWS load balancers and nat gateways.
The AWS EKS load balancer controller will manage the LBs, so it needs to be tagged with kubernetes.io/role/elb

Private Subnets
3 private subnets specifically for EKS. These need to be tagged with kubernetes.io/role/internal-elb for the LBC.
3 additional general purpose private subnets for workloads running outside of EKS. These must not be tagged with kubernetes.io/role/internal-elb , since we don't want the LBC to deploy in those subnets.

Although we could technically tell LBC which subnets to specifically use at via annotations, coordination and passing of this information from Terraform to the applications we deploy on the EKS cluster is burdensome and additional work, especially when that particular configuration information can be abstracted away from the applications via the subnet discovery tags. If we could have passed the specific default subnets we want the LBC to deploy in via the controller command line flags, then I would have less qualms about this, but unfortunately that is not the case.

From reading through the issue here, I understand that there are potential plans to have a generic subnet submodule that would address all of the concerns here, but since there are no concrete plans for that, I'd like to suggest that during this interim, we reconsider implementing this feature. Specifically, I'd like to propose the following:

Scope the change of adding per subnet tags to only the private and public subnet groups that are created from this module. Looking through all of the comments here, the primary concern is with these two types of subnets, since the other subnet types (e.g. redshift, elasticache, etc) are already specific with a use case in mind. This minimizes the surface area of change, so its less likely to cause issues down the road and easier to maintain. We already have the "private_subnet_names" (and similar) variable that allows to customize each subnet name , which is in a similar vein as this feature request, so I'm having trouble understanding why this change would be rejected. EDIT: Looks like someone already implemented it here as described in the following PR: #1139

I'm happy to open up a new PR for this, but I'd like to gauge appetite on this from the maintainers. Looking at the this repos issues and pr history, its been a feature request for years now, and I don't see why we need to continue waiting for the perfect solution for it to happen.

[ghost]

PR has not seen movement for 2 years. Closing as unlikely to be merged.

[ghost]

For what it's worth, this is still running successfully as a load-bearing component of our production infrastructure.

[Chili-Man]

@raxod502-plaid can you please keep it open? Even though there hasn't been much movement on it, doesn't necessarily mean the community doesn't want this feature. If you close the issue, the maintainers are more likely to forget about this

[bogatuadrian]

@bryantbiggs @antonbabenko This functionality would be very useful to us as well.

Our use case is that we require different types of private subnets for different types of EKS nodes (due to security considerations), and we need to tag those subnets accordingly, so we can configure Karpenter to scale certain nodes that use their respective subnets.

I also agree with what @Chili-Man said in a previous post. You could consider this change only for the public/private subnets. Some work in that direction has been done here #1139.

Also to address some previous points in this thread:

1. There are definitely reasons to have different types of private subnets, subnets that wouldn't fall in any of the other categories (intra, database etc.), our use-case, separate security constrains for each subnet, being one (of probably many)
2. Tags are essential for Karpenter discoverability, but also incredibly useful in a IaC & GitOps workflow where it's easier to reference a resource by tags, rather than by (a generated) ID.

EDIT to add: having a way to add custom subnets to the module could be an improvement, but the main appeal of this module is the out-of-the-box configuration for how a "private" subnet is configured. I just want that "private" configuration, but for 2 groups of subnets that will end up being used in different ways down the line, hence needing custom tagging.

[github-actions]

This PR has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this PR will be closed in 10 days

[github-actions]

This PR was automatically closed because of stale in 10 days