Python SDK Documentation

[wjdingman1]

Hey, most of the documentation that I have seen for c7 is all for the CLI tool but I am struggling to find anything for the Python SDK.

Any help would be appreciated.

[ajkerrigan]

Custodian is intended and documented for CLI use, so there is no stable/documented library usage (see also #191).

Depending on the "why" behind your question, invoking the CLI from Python might still be useful. For example:

import c7n.cli

c7n.cli.main(['--help'])
c7n.cli.main(['run', '-s', '/my/output/dir/', '-vd', '/path/to/my/policy.yml'])

...but that's only useful if your goal is "running from Python". For the higher level discussion of library usage, #191 is the best context I'm aware of.

[wjdingman1]

We've figured out a way to get it working but we are still looking to optimize

from c7n.config import Config
from c7n.policy import Policy
from c7n_org import cli


cli.load_available()

config=Config.empty(region="us-east-1")
policy = Policy({ "name" : "foo", "resource" : "aws.ec2"})
res = policy.run()

We don't have complex policies, 99% are just collection, so this works for us, but we still have some things to figure out e.g., getting Config working for multiple region and suppressing file output.

[ajkerrigan]

Ah, fair enough you're already in the hole! If you're going down to that level you'll probably find it useful to look at tests and other internals then. For example, you can follow breadcrumbs from here to see how the AWS provider does some magic behind the scenes to expand regions and reuse sessions.

As for suppressing output, Custodian does have a null output but as far as I know it's only used/intended for internal testing. You could give that a shot following examples in the tests (you'll notice most tests call load_policy() which sets a null output path), but I'd probably recommend using Python's own temp path features instead. If you want to be careful to avoid leaving garbage around, setting up temp directories in a context manager and running your policies inside those blocks seems reasonable (example).

[wjdingman1]

That first snippet of code is exactly what I needed, thank you!

[wjdingman1]

Another shot in the dark here.

We are querying a number of resources for a handful of different accounts. EC2, S3, IAM stuff etc.

99% of the time everything is working as expected but for some random resources (auto scaling group, ecs task definition) they are not being returned even though we are confirmed to have them existing in the account

Any thoughts on this? Some weird caching maybe?

I'm sure this is a user error but wanted to just double check

[wjdingman1]

Also thank you for help, if there is any other way to thank you e.g. buying you a coffee or contributing to something please let me know

[wjdingman1]

hey @ajkerrigan back again, quick question if you had any insight into the error below.

This is occurring when running a collection against aws.elb. The policy is able to validate fine and when the region does not have any load balancers in it then it will succeed. However when the region does have resources in it then it will fail with the error below

An error occurred (InvalidParameterException) when calling the GetResources operation: arn:aws:elasticloadbalancing:us-west-1::loadbalancer/west-load-balancer is not a valid AmazonResourceName (ARN)

[ajkerrigan]

Hmm that looks like a classic load balancer ARN that's missing the account ID piece... do you see the same error running the policy from the CLI?

Not sure if somewhere along the line you're either not seeing an account ID or if something else is going on that's tripping up generate_arn (which gets called because GetResources needs ARNs and the classic ELB describe call doesn't return them directly).

[kapilt]

please note, custodian does not have a stable python interface, so this may break at any time. the cli and policy language is what we focus on for compatibility.

we have at least one major upheaval coming with the move to c7n_core that will likely break the code samples in this discussion.