Providing evidence in compliance findings

[darrendao]

Our company is using Cloud Custodian for compliance checks. We have a “security hub” similar to AWS security hub. The code for doing that is very similar to the code for c7n securityhub. What is different about our security hub is that we want to show evidence of why the checks PASSED or FAILED. For example, we have a policy that checks for whether or not ALBs are using weak TLS listeners by using the listener filter. We noticed that Cloud Custodian would include the matched listeners in the finding, which is good for evidence, but when there is no match, the listeners info is not included in the finding. We need to have evidence for both cases and we're not sure how to support such functionality. I know some folks have suggested writing 2 policies for each compliance check, one for PASSED and one for FAILED. This approach is rather cumbersome and not the long term state we desire. Furthermore, that approach might also not work because certain filters don’t have matched annotation set to true and so the matched values won’t be included in the finding.

Looking at the code, I believe the problem here is due to the fact that the evidence we’re looking for is not inherently a part of the targeted resource (i.e. listener is its own AWS resource and is not a property of the ALB resource. It just happens that the ALB resource has a filter for listeners). I can think of a two ways we can solve the problem:

1. Update the Describe functionality of each resource to include all of the necessary information via the “augment” method when loading the resource. I notice we already have code like this for resources like s3. So for my example of ALB and weak TLS listeners, we would need to augment the ALB Describe to include the listeners information. The problem I see with this approach is that we would have to go through and do that for any resources with such issue.

2. We were thinking if there’s a way to change how filtering works at the base level so that we always annotate the properties being evaluated so that in either case of match or not match, we would have the evidence of what was being evaluated. I imagine this would be a big change to how filtering works but it might be the most practical route.

Do you have any suggestions on which approach is better? Or is there something else we can do? Is this even something we want to support in Cloud Custodian?

[kapilt]

hi @darrendao, a couple of thoughts.

1. if your using aws config, then you have a historical revision of resources already (cloud asset inventory gcp or graph in azure can provide similar).

2. generally filters do annotations on resources when their looking up information via separate apis, this is distinct from the matched annotation, however there isn't any hard and fast rule on this. its generally done to cache lookups if the same filter is going to be used multiple times in a policy .. but there are trade offs in enlarging the resources when the annotation gets large. That trade off is most evident in notify transports as they have size limitations for each provider (256kb aws, 64kb azure, 10mb gcp) that vary by transport (sqs, sns, storage queue, pub/sub, etc).

3. augments are generally something we only do universally for tag information. we don't do general augmentation anymore, as its extraneous api calls if a policy doesn't evaluate that information, vs having a dedicated filter for it. s3 is both historical and a special case in that regard.

4. in this context the right thing would be to have the filter annotate the load balancer with the listener information. subject to consideration about over inflating the size of the resource, ie there isn't per se a constraint on number of listeners. currently we do static size windows, its a backlog item to dynamic size them based on actual size of the resource, which would also potentially alleviate this.

5. one other option for listeners, would be to expose them as a child resource, which would allow collecting them directly.

6. in the general case, I don't think you need an assertion/negation policy pair, as an alternative you could use a per resource type policy that can serve as an evidence collector. in some cases that will need to include additional filters to get informational annotations to serve as evidence.

are there other examples you have beyond alb?

[darrendao]

Thanks @kapilt for the detail reply. My responses are below

1. The configuration changes timeline in AWS Config is great for auditing but requires you to know what you're looking for. Further more, we want a central place for all compliance check (AWS and non AWS) that can just show compliance STATUS and evidence for PASS or FAILED
2. So to confirm, you're saying that outside of matched annotation, there's also the concept of filter annotations? And that those annotations are there regardless of whether or not the filter matches? Can you point me to where we do that in the code?
3. My idea going into this was to use augments but it sounds like you're not recommending that here, and that augments is more for tags?
4. Can you elaborate more on what you mean by "have filter annotate the load balancer"? I'm guessing this is related to what you mentioned in 2?
5. This was the other option that I was thinking of too. Do you think option 4 is better though?

I'll have to go through the list of policies we have but here are two other resources/filters that have similar issue with ALB and listeners

• hostedzone resource and shield-enabled filter
• network-addr resource and shield-enabled filter

[darrendao]

@kapilt Looking through the code, I think I understand what you mean by filter annotation now. However, correct me if I'm wrong, those annotations are saved in the resource manager cache so if I'm not using cache, then they won't be available in subsequent "get resources" calls. Furthermore, for certain policy mode like ConfigRuleMode, resources are load via the source class (ConfigSource) rather than the resource manager, which then bypasses the cache so the filter annotations won't be available either.

[kapilt]

1. if you've got policies for compliant (ie a resource type policy), and non compliant, then you know what your looking for in config (resource id, type, time). alternatively you also now have that data in your custodian output bucket. ie doing an evidence collector as single policy per resource type is one option.
2. yes filter annotations are basically ways of caching additional lookups directly on the resource,
3. augments are for tags or resources whose enumeration result is just strings. they are not appropriate for general query, as they impose a cost for every policy using that resource regardless if they use the additional data or not. augments are part of the cache
4. yes its per 2
5. this feel like its worth a talk/conversation, one other option is also making a separate resource for app/net elb listeners.

re followup, the filter annotation has nothing todo with the cache, it has to do with the output of resources into blob store, annotations are not in cache. wrt to config mode and cache, cache happens regardless of source. I'm not sure how this part is relevant re cache, it feel like this is getting lost in the weeds instead of focusing on the use case.