Add local module support to the cloudformation package command

[ericzbeard]

This PR adds basic client-side multi-file support to CloudFormation by changing the behavior of the aws cloudformation package command.

Modules are imported into the parent template or parent module via a new Modules section. This is a departure from registry modules, which are configured as resources. Since local modules can not only emit multiple Resources, in addition to unresolved Conditions and Mappings, configuring them as a “resource” feels unintuitive.

Modules:
    Content:
     Source: ./module.yaml

Modules are a superset of CloudFormation templates. They support Parameters, Resources, Conditions, and Outputs.

A sample module:

Parameters:
  Name:
    Type: Scalar
Resources:
  Bucket:
    Type: AWS::S3::Bucket
    Metadata:
      OverrideMe: abc
    Properties:
      BucketName: !Ref Name
Outputs:
   BucketArn: 
    Value: !GetAtt Bucket.Arn

An example of using the module above:

Modules:
  Content:
    Source: ./module.yaml
    Properties:
      Name: foo
    Overrides:
      Bucket:
        Metadata:
          OverrideMe: def
Outputs:
  TheArn:
    Value: !GetAtt Content.BucketArn

The packaged output of the above template:

Resources:
  ContentBucket:
    Type: AWS::S3::Bucket
    Metadata:
      OverrideMe: def
    Properties:
      BucketName: foo

Parameters and ParameterSchema

Parameters are configured with the Properties attribute in the parent, and act much like normal parameters, except it is possible to pass in objects and lists to a module. The new ParameterSchema section provides a way to define the structure and validation rules for parameters, especially for complex types like objects and arrays.

Parameters:
  UserConfig:
    Type: Object
    Description: "Configuration for user settings"
    Default: {}

ParameterSchema:
  UserConfig:
    Type: Object
    Required: ["Username", "Roles"]
    Properties:
      Username:
        Type: String
        MinLength: 3
        MaxLength: 64
        Pattern: "^[a-zA-Z0-9_-]+$"
      Email:
        Type: String
        Pattern: "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$"
      Roles:
        Type: Array
        MinItems: 1
        Items:
          Type: String
          Enum: ["Admin", "Developer", "Reader"]
      Settings:
        Type: Object
        Properties:
          Theme:
            Type: String
            Default: "Light"
            Enum: ["Light", "Dark", "System"]
          Notifications:
            Type: Boolean
            Default: true

The parameter schema type system is based on JSON Schema with CloudFormation-specific extensions:

• String: Text values
• Number: Numeric values
• Boolean: True/false values
• Object: Key-value maps
• Array: Ordered lists

Keyword	Applies To	Description
Required	Object	List of required property names
MinLength	String	Minimum string length
MaxLength	String	Maximum string length
Pattern	String	Regular expression pattern
Enum	Any	List of allowed values
Minimum	Number	Minimum value (inclusive)
Maximum	Number	Maximum value (inclusive)
ExclusiveMinimum	Number	Minimum value (exclusive)
ExclusiveMaximum	Number	Maximum value (exclusive)
MinItems	Array	Minimum array length
MaxItems	Array	Maximum array length
Items	Array	Schema for array items
Properties	Object	Schema for object properties
Default	Any	Default value if not provided

Outputs

The Outputs of a module are reference-able in the parent by using GetAtt or Sub in the parent template, with the combination of the module name and Output name. Module outputs can be scalars, lists, or objects.

Overrides

The Overrides attribute allows the consumer to override content from the module that is included in the parent template, if they know the internal structure of the module. Instead of forcing a module author to anticipate every possible use case with numerous parameters, the author can focus on basic use cases and allow the consumer to get creative if they want to change certain elements of the output. Overrides only apply to the Resources emitted by the module. Using Overrides carries a risk of breaking changes when a module author changes things, so it will generally be safer to rely on Parameters and References, but the overrides provide a flexible and easy-to-use escape hatch that can solve some tricky design challenges associated with a declarative language like YAML. We do not plan to advertise this as a best practice, but we do think that the escape hatch is necessary, just as it is for CDK.

Complex objects are merged when an override is specified, so it’s possible to do things like add statements to a policy without actually overwriting the entire thing. We could consider replacing the Override keyword with more specific words like Replace, Append, Merge, or Delete, since currently, it’s not possible to remove elements from a list.

Constants

The new Constants section is a simple key-value list of strings or objects that are referred to later with Fn::Sub or Ref. This feature reduces copy-paste of repeated elements within a module.

Constants:
  `S3Arn``:`` ``"arn:${AWS::Partition}:s3:::"`
`  ``BucketName``:`` ``"${Name}-${AWS::Region}-${AWS::AccountId}"
  BucketArn: "${Const::S3Arn}${Const::BucketName}"
  AnObject:
    Foo: bar
`

Before any processing of the template happens, all instances of constants in !Sub strings are replaced, including in subsequent constant declarations. Constant references are prefixed by Const``::. Constants are supported not just in modules, but in the parent template as well. For constants that are objects, they are referenced with !Ref Const::name.

Module Sources

After constants are processed, the Modules section is processed. Module source files are specified using a path that is relative to the parent template or module. The path is not relative to where the package command is being run. So, if a module is in the same directory as the parent, it is simply referred to as module.yaml or ./module.yaml. Modules can also reference an HTTPS URL or an S3 URI as the source location.

Modules can contain other modules, with no enforced maximum limit on nesting. Modules are not allowed to refer to themselves directly or in cycles. Module A can’t import Module A, and it can’t import Module B if that module imports Module A.

Transforms

The Transform section of the module, if there is one, is emitted into the parent, merging with existing transforms if necessary. Modules don’t do anything special with transforms, since these need to run on the server.

ForEach and Fn::ForEach

Modules support a basic form of looping/foreach by either using the familiar Fn::ForEach syntax, or with a shorthand by adding a ForEach attribute to the module configuration. Special variables $Identifier and $Index can be used to refer to the value and list index. With the shorthand, or if you don't put the Identifier in the logical id, logical ids are auto-incremented by adding an integer starting at zero. Since this is a client-side-only feature, list values must be fully resolved scalars, not values that need to be resolved at deploy time.

Parameters:
  List:
    Type: CommaDelimitedList
    Default: A,B,C

Modules:
  Content:
    Source: ./map-module.yaml
    ForEach: !Ref List
    Properties:
      Name: !Sub my-bucket-$Identifier
      
# OR

Modules:
  Fn::ForEach::Content:
  - Identifier
  - !Ref List
  - Content:
      Source: ./map-module.yaml
      Properties:
        Name: !Sub my-bucket-$Identifier

Assuming the module itself simply creates a bucket, the output would be:

Parameters:
  List: 
    Type: CommaDelimitedList  
    Default: A,B,C
Resources:  
  Content0Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: my-bucket-A
  Content1Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: my-bucket-B
  Content2Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: my-bucket-C

It’s also possible to refer to elements within a ForEach using syntax like !GetAtt Content[0].Arn for a single element, or !GetAtt Content[*].Arn, which resolves to a list of all of the Arn outputs from that module. Single elements can also be referenced by the key: Content[A].Arn. If using Fn::ForEach, in this case, Content refers to the loop name, not the OutputKey.

ForEach with Complex Objects

When ForEach receives a list of objects (such as from Fn::Flatten, described later in the doc), each object must have an Identifier property. In this case:

• $Identifier refers to the value of the Identifier property
• $Value provides access to the entire object

This allows direct property access without requiring additional lookups:

Modules:
  Service:
    Source: ./service-module.yaml
    ForEach:
      Fn::Flatten:
        Source: !Ref ServiceConfig
        Transform:
          Template:
            Identifier: "$item.name-$env"
            ServiceName: "$item.name"
            Environment: "$env"
            Type: "service"
          Variables:
            env: "$item.environments[*]"
    Properties:
      Name: !Sub "${Identifier}"
      ServiceName: !Sub "${Value.ServiceName}"
      ServiceType: !Sub "${Value.Type}"
      InstanceNumber: "1"
      Environment: !Sub "${Value.Environment}"

ForEach with Resources

The ForEach attribute can also be applied to a Resource within a module. It works in an almost identical way to modules and can be considered a shorthand for the Fn::ForEach intrinsic function. It adds the ability to reference maps in addition to simple CSV lists, can be combined with Fn::Flatten, and gives you access to the enhanced GetAtt functionality to refer to resource properties.

Resources:
 Bucket:
   Type: AWS::S3::Bucket
   ForEach: !Ref Environments
   Properties:
     BucketName: !Sub "my-bucket-${Identifier}"
     Tags:
        - Key: Environment
          Value: !Sub ${Identifier}
Outputs:
 BucketArns:
   Value: !GetAtt Bucket[*].Arn
 DevBucketArn:
   Value: !GetAtt Bucket[dev].Arn

Conditions

When a module is processed, the first thing that happens is parsing of the Conditions within the module. Any Resources, Modules, or Outputs marked with a false condition are removed, and any property nodes with conditions are processed. Any values of !Ref AWS::NoValue are removed. Any unresolved conditions (for example, a condition that references a parameter in the parent template, or something like AWS::Region) are emitted into the parent template, prefixed with the module name.

Mappings

The Mappings section of a module is processed, and if possible, any Fn::FindInMap functions are fully resolved. If they cannot be fully resolved, the mapping is emitted into the parent template with the module name as a prefix.

References

Much of the value of module output is in the smart handling of Ref, Fn::GetAtt, and Fn::Sub. For the most part, we want these to “just work” in the way that an author would expect them to. Since the logical IDs in the module result in a concatenation of the module name and the ID specified inside the module, we have to modify self-references within the module. And we have to fix up any references to parameter values, so that in the final rendered output, they match up with actual IDs and property names.

From the parent, the author has two options for referring to module properties. If they know the structure of the module, they can use the predicted final name for resources, in which case we leave the strings alone. For example, in a module that creates a bucket, if the logical id within the module is Bucket and the name of the module in the parent is Content, the author could write !Ref ContentBucket or !GetAtt ContentBucket.BucketName. The second, safer way is for the module author to specify an output reference called Name that refs the bucket name, so the parent author could write !GetAtt Content.Name. Module authors are encouraged to provide Outputs that provide access to all needed values, but there will be cases when they cannot predict everything a consumer needs. For Sub strings, if we can fully resolve the value, we get rid of the Fn::Sub and simply write the string to the output.

Intrinsics

As a part of this design, we are adding support for intrinsic functions that do not exist for normal CloudFormation templates, and can only be processed by the package command. We also augment the behavior of some intrinsics. These functions are fully resolved locally and won’t need server-side support. (If we decide to add server-side support via S3 module Sources, the depth of support is an open question. We could still limit these intrinsics to the “packaging phase” in the API and not allow them after the packaged template is submitted.)

Fn::GetAtt (Enhanced)

Fn::GetAtt has added capabilities in modules. It is possible to refer to elements of a module that has been replicated with Fn::ForEach. For example, !GetAtt Content[0].Arn references the Arn Output value from the first element of a ForEach called Content. You can also use the Identifier keys for the foreach, for example !GetAtt Content[A].Arn. Content[*].Arn returns a list of all of the Arn Output values.

Fn::GetAtt can also reference Parameters that hold Mappings. This is very similar to Fn::FindInMap functionality.

!GetAtt MapName[*] - Returns a list of keys in the map
!GetAtt MapName[Key] - Returns the entire object at that key
!GetAtt MapName[Key].Attribute - Returns the attribute of the object at that key

An alternative valid syntax is to use dots instead of brackets.

!GetAtt MapName.*
!GetAtt MapName.Key
!GetAtt MapName.Key.Attribute

Fn::Merge (New)

Fn::Merge merges the contents of multiple objects. This is useful for things like tagging, where each module might want to add a set of tags, in addition to any tags defined by the parent module. This pattern is ubiquitous in Terraform modules.

  # Typical Terraform resource tag configuration
  tags = merge(
    { "Name" = var.name },
    var.tags,
    var.vpc_tags,
  )

Example merge usage:

# In the parent template

Modules:
  Network:
    Source: ./vpc.yaml
    Properties:
      Tags:
        Name: foo
        
# In the module:

Resources:
  VPC:
    Properties:
      Tags:
        Fn::Merge:
          - !Ref Tags
          - X:Y

Fn::InsertFile (New)

Fn::InsertFile inserts the contents of a local file directly into the template as a string. This is a convenience that makes it easier to embed code such as lambda functions into a template. This function only makes sense client-side (or for the server-side use case where a user uploads an entire zip file and we package it for them).

Fn::Invoke (New)

Fn::Invoke allows modules to be treated sort of like functions. A module can be created with only Parameters and Outputs, and by using Invoke later, you can get different outputs. An example use case is a module that creates standard resource names, accepting things like Env and App and Name as parameters, and returning a concatenated string. The utility of this function is in treating snippets of code that are smaller than resources, like modules. An invoke-able module basically just has Parameters and Outputs, and the output conditionally differs depending on the inputs. How Fn::Invoke works with modules

Fn::Flatten (New)

The Fn::Flatten function transforms complex nested data structures into simpler, flattened collections that can be processed with ForEach. It accepts a source map or list and provides options for data manipulation. It is very common in Terraform modules, especially with the foreach function, to flatten structures into a list of properties to iterate over when creating multiple resources of the same type.

Fn::Flatten:
  Source: <List or Map>
  
  # Optional path expression to extract nested values
  Pattern: <String> 
  
  # Optional attribute to group results by 
  GroupBy: <String>  
  
  # Optional transformation to apply to each item
  Transform:        
    Template: <Object>
    Variables:
      <Variable>: <Path Expression>

Key features include:

• Deep flattening of nested structures
• Pattern matching using JSONPath-like expressions
• Transformation of items using templates
• Cross-product generation from multiple arrays
• Grouping results by specific attributes

This capability simplifies module structure by eliminating the need for cascading module references to handle nested iterations.

Fn::Flatten:
    Source:
        users:
            - name: "user1"
            roles: ["admin", "developer"]
            - name: "user2"
            roles: ["reader"]
        Pattern: "$.users[*].roles[*]"

# ->

["admin", "developer", "reader"]

Fn::Flatten:
    Source:
    - name: "app"
        regions: ["us-east-1", "us-west-2"]
        environments: ["dev", "prod"]
    Transform:
        Template:
            AppName: "$item.name"
            Region: "$region"
            Environment: "$env"
            ResourceName: "$item.name-$env-$region"
        Variables:
            region: "$item.regions[*]"
            env: "$item.environments[*]"
        
# ->

-  AppName: "app"
    Environment: "dev"
    Region: "us-east-1"
    ResourceName: "app-dev-us-east-1"
-  AppName: "app"
    Environment: "prod"
    Region: "us-east-1"
    ResourceName: "app-prod-us-east-1"
-  AppName: "app"
    Environment: "dev"
    Region: "us-west-2"
    ResourceName: "app-dev-us-west-2"
-  AppName: "app"
    Environment: "prod"
    Region: "us-west-2"
    ResourceName: "app-prod-us-west-2"

Fully resolving intrinsics

Fn::Select is collapsed if we can fully resolve the return value as a string. We do the same for Fn::Join and Fn::FindInMap.

Metadata

When processing modules, the package command adds metadata to the template for metrics gathering, so that we know how many customers are using the feature. (NOTE: We will add more relevant data to this section)

Metadata:
  AWSToolsMetrics:
    CloudFormationPackage:
      Modules: true

Each rendered resource has an added Metadata property to indicate where it originated. This can be useful for tooling such as IDE integration, and for troubleshooting in the console. (NOTE: The format is being designed separately, to be compatible with CDK)

Resources:
  ContentBucket:
    Metadata:
      SourceMap: "./modules/bucket.yaml:Bucket:35"

Packages

In order to reference a collection of modules, you can add a Packages section to the template. Packages are zip files containing modules. This allows a group of related modules to be packaged up and versioned as a unit. Modules within the package can refer to other modules in the package with a local path. This functionality will serve as the basis for a more robust dependency management system in the future.

Packages:
  abc:
    Source: ./package.zip
  def:
    Source: https://example.com/packages/package.zip

Packages are referenced by using an alias to the package name that starts with $.

Modules:
  Foo:
    Source: $abc/foo.yaml
  Bar:
    Source: $def/a/b/bar.yaml

---

Appendix A - Module Spec

# Changes to parent templates that are only understood by the package command

# Constants is a new section evaluated at packaging time.
# It contains a map of names to values that are substituted later in the template.
Constants {
  <String>: <Scalar|List|Object>
}

# Packages is a new section that configures aliases to collections of modules
Packages {
  <String> { 
    Source: <Scalar> # Local or remote URI
  }
}

# Modules is a new section that holds a map of modules to include
Modules {
  <String> {
    Source: <Scalar> # Local or remote URI (HTTPS or S3)
    Properties: <Object> # Configures module parameters
    Overrides: <Object>
    Map: <!Ref to a CommaSeparatedList> # Repeat for each element in the list
  }
}

# Module file

# Similar to template Parameters, but allow lists and objects.
# The Type is just a hint, since module packaging is dynamic and only cares
# if the value passed in is a string, list, or dictionary.
Parameters {
  <String>: 
    Type: <String>
    Default: <Any>
    Description: String
}

ParameterSchema {
  <String>:
    Type: <String>
    Required: <Boolean>
    Properties: <Object>
}

# Conditions are evaluated client-side and can't require any deploy-time values.
# They behave just like template conditions, omitting content based on parameters.
Conditions {
  <String>: <Any>
}
    
# Same as above for the parent template
Constants {
  <String>: <Scalar|List|Object>
}

# Same as above for the parent template.
Packages {}

# Same as above for the parent template. Modules can include other modules.
Modules {}

# Resources behave the same as template Resources.
# See CloudFormation template anatomy
Resources {}

# Similar to template Outputs, but they are only used
# to facilitate !Ref, !Sub, and !GetAtt from the parent.
# If Exports is specified, it is ignored.
Outputs {
  <String>: 
    Value: <Scalar|Ref|Sub|GetAtt>
    Description: String
}

How to test this PR

Check out my branch and install the aws cli into a local Python environment.

git clone [email protected]:ericzbeard/aws-cli.git
git fetch origin cfn-modules
git checkout cfn-modules
python3 -m venv .env # Or whatever you use for python environments
source .env/bin/activate
pip install -e .
aws cloudformation package \
    --template-file tests/unit/customizations/cloudformation/modules/type-template.yaml

Examples

See the test templates in this PR at tests/unit/customizations/cloudformation/modules/, a draft PR with a full serverless webapp here: aws-cloudformation/aws-cloudformation-templates#457, and a draft PR with a CodePipeline and VPC: aws-cloudformation/aws-cloudformation-templates#463

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 83.14465% with 402 lines in your changes missing coverage. Please review.

Project coverage is 92.13%. Comparing base (2935fc0) to head (da1abd8).
Report is 252 commits behind head on develop.

Files with missing lines	Patch %	Lines
...scli/customizations/cloudformation/modules/read.py	58.75%	66 Missing ⚠️
...i/customizations/cloudformation/modules/process.py	82.72%	62 Missing ⚠️
...i/customizations/cloudformation/modules/foreach.py	83.08%	56 Missing ⚠️
...i/customizations/cloudformation/modules/flatten.py	82.62%	49 Missing ⚠️
...ustomizations/cloudformation/modules/validation.py	82.69%	36 Missing ⚠️
...ustomizations/cloudformation/modules/conditions.py	81.54%	31 Missing ⚠️
...i/customizations/cloudformation/modules/outputs.py	89.13%	25 Missing ⚠️
...i/customizations/cloudformation/modules/resolve.py	88.43%	20 Missing ⚠️
...customizations/cloudformation/artifact_exporter.py	57.14%	15 Missing ⚠️
.../customizations/cloudformation/modules/mappings.py	78.57%	15 Missing ⚠️
... and 6 more

Additional details and impacted files

@@             Coverage Diff              @@
##           develop    #9124       +/-   ##
============================================
+ Coverage     0.08%   92.13%   +92.05%     
============================================
  Files          210      228       +18     
  Lines        16984    19365     +2381     
============================================
+ Hits            14    17842    +17828     
+ Misses       16970     1523    -15447     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[kddejong]

Solid start. I have some corrections and some questions.

[benbridts]

I was thinking about how to refer to values from a template and noticed that that is a place where the leaky abstraction might be a downside (most of it is true for the overrides too, but it feels more expected there):

If you want to reference a resource in a module you would use e.g. !GetAtt ContentBucket.Arn, which means:

• the module author can't change any resource name in the template
• (not a problem now, but possibly when extending to allow conditions) there is no way to have to different resources cover the same generic module. Imagine e.g a ApiGateway module that creates an HTTP or REST api, depending on a parameter/condition.
• Even if users are not using any overrides, they would need to know the resources (and names) created by the template - I thing the same argument as why this is not a problem with overrides can be applied here, but if you see overrides as a more advanced use case, it might not.

Having an "outputs" or "outputs" equivalent could solve this.

e.g

Parent template

Modules:
  Content:
    Source: ./basic-module.yaml
    Properties:
      Name: foo
Outputs:
  ExampleOutput:
    Value: !GetAtt Content.BucketArn

basic-module.yaml

Parameters:
  Name:
    Type: String
Resources:
  Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Ref Name
Outputs:
  BucketArn: !GetAtt Bucket.Arn

Output from the package command

Resources:
  ContentBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: foo
Outputs:
  ExampleOutput:
    Value: !GetAtt ContentBucket.Arn

[kddejong]

Agreed with Ben on this one. You could still allow for an override capability that allows for a missing output that may be needed but having to understand the module resource names can get frustrating and prevents flexible in changing them. There are going to be limitations in this approach that are similar to modules. Adding mappings/conditions into a module will limit its usage because it cannot rely on another resource from the parent template or other module.

[ericzbeard]

Having an "outputs" or "outputs" equivalent could solve this.

I don't see any reason not to add Outputs. It's definitely a more predictable way to reference resources in the module. I still want the Overrides section and the ability to reference things in the module directly if you know the name, but that can be considered a more advanced use case. If you're consuming a 3rd party module, the risk is higher than if you're referencing your own local modules.

[ericzbeard]

Having an "outputs" or "outputs" equivalent could solve this.

Implemented here: 160be7c

[kddejong]

This will fail on JSON files. While YAML can parse JSON there are some limitations that require the JSON parsers.

[ericzbeard]

Just for compose, or for yaml_parse as well?

[kddejong]

I might be missing the point of having a Parameter with a Default for defining a list. Package doesn't support a parameter so it isn't changeable with that process? Is this for sub-moduling?

Parameters:
  List:
    Type: CommaDelimitedList
    Default: A,B,C

Modules:
  Content:
    Source: ./foreach-module.yaml
    ForEach: !Ref List
    Properties:
      Name: !Sub my-bucket-$Identifier

If I can override it from a parent module or template then it would create some caveats on how the Outputs work. As in we can no longer guarantee that a particular value exists?

[ericzbeard]

For the first question, since this is all resolved client-side, we have to have either a default value for the parameter, or the parent template/module needs to specify it with a Property.

For the second, are you talking about conditional resources? Not sure how that will work with outputs that depend on a resource that is not actually emitted. Need to test it.