Skip to content

False negative if 'schema' args reversed in stateful check #60

@trstephen-amazon

Description

@trstephen-amazon

The --stateful check assumes the --schema arguments are ordered previous then current versions. If the order is flipped then the check will succeed when it should fail.

Reproduction

  1. Download a schema
    aws cloudformation describe-type \
         --type RESOURCE \
         --type-name AWS::QLDB::Stream \
         | jq -r '.Schema | fromjson' > qldb-stream-schema.json
    
  2. Copy and make a breaking change. For example, change .definitions[].Tag.properties.Value.maxLength 255 -> 2
  3. Run a stateful resource check with original/modified (should fail ENSURE_MAXLENGTH_NOT_CONTRACTED)
     guard-rail \
         --schema file://path-to-file/qldb-stream-schema.json \
         --schema file://path-to-file/modified-qldb-stream-schema.json \
         --stateful --format
    
  4. Run a stateful resource check with modified/original
     guard-rail \
         --schema file://path-to-file/modified-qldb-stream-schema.json \
         --schema file://path-to-file/qldb-stream-schema.json \
         --stateful --format
    

What happens: Passing the schema in modified/original order passes despite the breaking change

➜ guard-rail \
    --schema file://path-to-file/modified-qldb-stream-schema.json \
    --schema file://path-to-file/qldb-stream-schema.json \
    --stateful --format
{"maxLength": {"changed": [{"property": "/properties/Tags/*/Value", "old_value": 2, "new_value": 255}]}}

---------
[SKIPPED]:

ENSURE_OLD_PROPERTY_NOT_TURNED_IMMUTABLE
ENSURE_PROPERTY_TYPE_NOT_CHANGED
ENSURE_MAXIMUM_NOT_CONTRACTED
ENSURE_PROPERTY_STRING_PATTERN_NOT_CHANGED
ENSURE_PRIMARY_IDENTIFIER_NOT_CHANGED
ENSURE_OLD_PROPERTY_NOT_REMOVED_FROM_READONLY
ENSURE_NO_MORE_REQUIRED_PROPERTIES
ENSURE_MAXITEMS_NOT_CONTRACTED
ENSURE_MINLENGTH_NOT_CONTRACTED
ENSURE_ENUM_NOT_CHANGED
ENSURE_MINITEMS_NOT_CONTRACTED
ENSURE_MINIMUM_NOT_CONTRACTED
ENSURE_OLD_PROPERTY_NOT_TURNED_WRITEONLY
ENSURE_OLD_PROPERTY_NOT_REMOVED

[PASSED]:

ENSURE_MAXLENGTH_NOT_CONTRACTED

[WARNING]:

What should happen: Check output should be the same (failing) regardless of order of args


I think you may have to scrap the --schema flags and go with args that don't consider order e.g. --current-schema, --original-schema

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions