Lambda deployment (mode: periodic) and --output-dir s3://my-custodian-bucket

[sanjaypatel11]

Running policy in poll mode at the terminal I can specify --output-dir with s3:// to save resource json and run log to the S3 bucket.
How could I generate similar S3 data in the Lambda mode?

[kapilt]

There's an ability to specify cli options specific to the lambda execution via the execution-options mapping on the lambda mode. ie.

mode:
    execution-options:
         output_dir: s3://xyz

[sanjaypatel11]

@kapilt Thx for the quick response.

1. When I run via CLI as: "custodian run --verbose --output-dir s3://ptsn-custodian-bucket1 test5b.yml", I see "2018-06-04 ...: custodian.output:DEBUG Storing output with <S3Output to bucket:ptsn-custodian-bucket1 prefix:/test5b_policy/2018/06/04/10>" in the output log. I also see custodian run log & resources.json in the bucket.

custodian run --verbose --output-dir s3://ptsn-custodian-bucket1 test5b.yml

policies:
  - name: test5b_policy
     resource: ec2
     filters:
      - "tag:MyCustodianTagKey" : present
    actions:
      - type: tag
        value: 'my-val'

2. Added following mode to generate Lambda. Executed lambda, but see "Storing output with <DirectoryOutput to dir:/tmp/..." in the Lambda log output. No data in the S3 bucket.

    mode:
      type: periodic
      schedule: 'cron(0/5 * * * ? *)'
      role: arn:aws:iam::{account_id}:role/spCustodianRole
      execution-options:
        output_dir: s3://ptsn-custodian-bucket1

Lambda's config.json looks ok:

{
  "policies": [
    {
      "name": "test5b_policy",
      "resource": "ec2",
      "mode": {
        "type": "periodic",
        "schedule": "cron(0/5 * * * ? *)",
        "role": "arn:aws:iam::1234567890:role/spCustodianRole",
        "execution-options": {
          "output_dir": "s3://ptsn-custodian-bucket1"
        }
      },
      "filters": [
        {
          "tag:MyCustodianTagKey": "present"
        }
      ],
      "actions": [
        {
          "type": "tag",
          "value": "my-val"
        }
      ]
    }
  ]
}

Following code in dispatch_event() of handler.py does show correct data
options_overrides = policy_config[ 'policies'][0].get('mode', {}).get('execution-options', {})
options_overrides --> {u'output_dir': u's3://ptsn-custodian-bucket1'}
Looks like it is overwritten back to /tmp by: options_overrides['output_dir'] = output_dir

What am I missing?

[kapilt]

which version of custodian are you using?

[sanjaypatel11]

Version: 0.8.28.2

[techsami]

Hi @sp5963 , is your problem is resolved? I am also facing similar problem. Thanks for your response.

[velmuruganvelayutham]

execution-options work for me !.

policies:
- name: eb-env-termination
 resource: elasticbeanstalk-environment
 mode:
   type: periodic
   schedule: "rate(10 minutes)"
   role: arn:aws:iam::xxxxxxxx:role/d-it-instance-role
   execution-options:
     output_dir: s3://xxx-dev-it-yyy/ci/beanstalk/custodian/
 filters:
   - "tag:custodian-terminate": "true"
 actions:
   - terminate