Serverless Data Lake Framework 2.6.0

This is a new release for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• Mark Octagon Configuration, Events, Artifacts, Metrics as legacy by @cnfait in #400
• mark Octagon-DataSchemas as legacy by @cnfait in #401
• update version to 2.6.0 by @cnfait in #402

Full Changelog: 2.5.0...2.6.0

Serverless Data Lake Framework 2.5.0

This is a new release for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• new stage: sdlf-stage-lambda by @cnfait in #338
• force use of cfn-lint 0.87.7 since release 1.3.0 break cloudformation-cli by @cnfait in #341
• new stage: sdlf-stage-ecsfargate by @cnfait in #342
• new stage: emrserverless by @cnfait in #344
• new stage: sdlf-stage-glue by @cnfait in #345
• [sdlf-stage-*] generic get_transform_details in routing lambda by @cnfait in #346
• pin cfn-lint to <1 instead of 0.87.7 by @cnfait in #349
• [sdlf-pipeline] support for pipelines with no source event by @cnfait in #350
• [sdlf-stage-*] put metadata in transform field, handle decimal metadata by @cnfait in #351
• [vpc] support shared vpc subnets by @cnfait in #348
• Preview: CDK Python by @cnfait in #353
• build stage-lambda and stage-glue by default by @cnfait in #355
• [sdlf-team][cdk] lakeformation data_location_access permissions by @cnfait in #356
• [sdlf-dataset][cdk] lake formation resources by @cnfait in #357
• avoid code duplication in domain-cicd by @cnfait in #359
• [sdlf-team][cdk] lakeformation tag and tag permissions by @cnfait in #360
• [sdlf-stage-emrserverless/ecsfargate] CDK Python by @cnfait in #361
• [sdlf-foundations/team][cdk] kms removal policy by @cnfait in #362
• [sdlf-*][cdk] set removal policy on stateful resources by @cnfait in #363
• [sdlf-*][cdk] extends Construct instead of Stack, add requirements.txt by @cnfait in #364
• SeedFarmer Support by @cnfait in #365
• [sdlf-foundations] generic deployspec.yaml by @cnfait in #366
• [sdlf-team] seedfarmer support by @cnfait in #367
• [sdlf-dataset] seedfarmer support by @cnfait in #369
• pin cfn-lint to <1 in github workflow by @cnfait in #370
• disable w12 check on modulestack where exception is documented by @cnfait in #371
• remove hash from glue job cfn reference & script name by @cnfait in #372
• [sdlf-cicd] glue job deployer: support for custom templates by @cnfait in #373
• [sdlf-cicd] github support as an alternative to codecommit by @cnfait in #374
• poetry/pyproject.toml for sdlf cdk modules by @cnfait in #378
• [sdlf-monitoring] logs account policy for subscription filters by @cnfait in #379
• [sdlf-monitoring] log group for data firehose lambda by @cnfait in #380
• upgrade to seedfarmer 5.0.0 (from 4.0.4) by @cnfait in #381
• [sdlf-monitoring] firehose lambda: use logging, do not merge log events by @cnfait in #382
• [sdlf-stage-*] peh_id is under transform now by @cnfait in #383
• [sdlf-datalakeLibrary] seedfarmer support by @cnfait in #384
• skip poetry install if cfn module by @cnfait in #385
• workshop seedfarmer manifest examples by @cnfait in #386
• update version to 2.5.0-beta.1 by @cnfait in #387
• seedfarmer - make sure setuptools is available by @cnfait in #388
• delete cfn stack properly with seedfarmer destroy by @cnfait in #389
• avoid hardcoding SDLF deployment role name in LakeFormation admin list by @cnfait in #390
• use globbing for cfn-lint, set ignore_templates by @cnfait in #391
• update dependencies by @cnfait in #392
• update version to 2.5.0-beta.2 by @cnfait in #393
• avoid substitution issue on macosx by @cnfait in #394
• avoid buildspec yaml syntax issues by @cnfait in #395
• [sdlf-cicd] lambda:TagResource for event-source-mapping by @cnfait in #396
• [sdlf-cicd] lambda:TagResource for event-source-mapping in team role by @cnfait in #397
• (sdlf-stage-*) don't set default dataset by @cnfait in #398
• update version to 2.5.0 by @cnfait in #399

Full Changelog: 2.4.0...2.5.0

Serverless Data Lake Framework 2.5.0-beta.2

Work is ongoing on version 2.5.0 of the Serverless Data Lake Framework.

This is a pre-release, not ready for production workloads. New features may not work properly, may be rewritten significantly or even removed before 2.5.0 is officially available.

This beta release is for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• seedfarmer - make sure setuptools is available by @cnfait in #388
• delete cfn stack properly with seedfarmer destroy by @cnfait in #389
• avoid hardcoding SDLF deployment role name in LakeFormation admin list by @cnfait in #390
• use globbing for cfn-lint, set ignore_templates by @cnfait in #391
• update dependencies by @cnfait in #392
• update version to 2.5.0-beta.2 by @cnfait in #393
• avoid substitution issue on macosx by @cnfait in #394
• avoid buildspec yaml syntax issues by @cnfait in #395
• [sdlf-cicd] lambda:TagResource for event-source-mapping by @cnfait in #396
• [sdlf-cicd] lambda:TagResource for event-source-mapping in team role by @cnfait in #397

Full Changelog: 2.5.0-beta.1...2.5.0-beta.2

Serverless Data Lake Framework 2.5.0-beta.1

Work is ongoing on version 2.5.0 of the Serverless Data Lake Framework.

This is a pre-release, not ready for production workloads. New features may not work properly, may be rewritten significantly or even removed before 2.5.0 is officially available.

This beta release is for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• [sdlf-monitoring] logs account policy for subscription filters by @cnfait in #379
• [sdlf-monitoring] log group for data firehose lambda by @cnfait in #380
• upgrade to seedfarmer 5.0.0 (from 4.0.4) by @cnfait in #381
• [sdlf-monitoring] firehose lambda: use logging, do not merge log events by @cnfait in #382
• [sdlf-stage-*] peh_id is under transform now by @cnfait in #383
• [sdlf-datalakeLibrary] seedfarmer support by @cnfait in #384
• skip poetry install if cfn module by @cnfait in #385
• workshop seedfarmer manifest examples by @cnfait in #386
• update version to 2.5.0-beta.1 by @cnfait in #387

Full Changelog: 2.5.0-beta.0...2.5.0-beta.1

Serverless Data Lake Framework 2.5.0-beta.0

Work is ongoing on version 2.5.0 of the Serverless Data Lake Framework.

This is a pre-release, not ready for production workloads. New features may not work properly, may be rewritten significantly or even removed before 2.5.0 is officially available.

This alpha release is for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• poetry/pyproject.toml for sdlf cdk modules by @cnfait in #378

Full Changelog: 2.5.0-alpha.2...2.5.0-beta.0

Serverless Data Lake Framework 2.5.0-alpha.2

Work is ongoing on version 2.5.0 of the Serverless Data Lake Framework.

This is a pre-release, not ready for production workloads. New features may not work properly, may be rewritten significantly or even removed before 2.5.0 is officially available.

This alpha release is for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• [sdlf-foundations] generic deployspec.yaml by @cnfait in #366
• [sdlf-team] seedfarmer support by @cnfait in #367
• [sdlf-dataset] seedfarmer support by @cnfait in #369
• pin cfn-lint to <1 in github workflow by @cnfait in #370
• disable w12 check on modulestack where exception is documented by @cnfait in #371
• remove hash from glue job cfn reference & script name by @cnfait in #372
• [sdlf-cicd] glue job deployer: support for custom templates by @cnfait in #373
• [sdlf-cicd] github support as an alternative to codecommit by @cnfait in #374

Full Changelog: 2.5.0-alpha.1...2.5.0-alpha.2

Serverless Data Lake Framework 2.5.0-alpha.1

Work is ongoing on version 2.5.0 of the Serverless Data Lake Framework.

This is a pre-release, not ready for production workloads. New features may not work properly, may be rewritten significantly or even removed before 2.5.0 is officially available.

This alpha release is for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• SeedFarmer Support by @cnfait in #365

Full Changelog: 2.5.0-alpha.0...2.5.0-alpha.1

Serverless Data Lake Framework 2.5.0-alpha.0

Work is ongoing on version 2.5.0 of the Serverless Data Lake Framework.

This is a pre-release, not ready for production workloads. New features may not work properly, may be rewritten significantly or even removed before 2.5.0 is officially available.

This alpha release is for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• new stage: sdlf-stage-lambda by @cnfait in #338
• force use of cfn-lint 0.87.7 since release 1.3.0 break cloudformation-cli by @cnfait in #341
• new stage: sdlf-stage-ecsfargate by @cnfait in #342
• new stage: emrserverless by @cnfait in #344
• new stage: sdlf-stage-glue by @cnfait in #345
• [sdlf-stage-*] generic get_transform_details in routing lambda by @cnfait in #346
• pin cfn-lint to <1 instead of 0.87.7 by @cnfait in #349
• [sdlf-pipeline] support for pipelines with no source event by @cnfait in #350
• [sdlf-stage-*] put metadata in transform field, handle decimal metadata by @cnfait in #351
• [vpc] support shared vpc subnets by @cnfait in #348
• Preview: CDK Python by @cnfait in #353
• build stage-lambda and stage-glue by default by @cnfait in #355
• [sdlf-team][cdk] lakeformation data_location_access permissions by @cnfait in #356
• [sdlf-dataset][cdk] lake formation resources by @cnfait in #357
• avoid code duplication in domain-cicd by @cnfait in #359
• [sdlf-team][cdk] lakeformation tag and tag permissions by @cnfait in #360
• [sdlf-stage-emrserverless/ecsfargate] CDK Python by @cnfait in #361
• [sdlf-foundations/team][cdk] kms removal policy by @cnfait in #362
• [sdlf-*][cdk] set removal policy on stateful resources by @cnfait in #363
• [sdlf-*][cdk] extends Construct instead of Stack, add requirements.txt by @cnfait in #364

Full Changelog: 2.4.0...2.5.0-alpha.0

Serverless Data Lake Framework 2.4.1

This is a bugfix release for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• bugfix force use of cfn-lint 0.87.7 since release 1.3.0 breaks cloudformation-cli by @cnfait 4badb63 and #341

Full Changelog: 2.4.0...2.4.1

Serverless Data Lake Framework 2.4.0

This is a new release for SDLF 2.x. See the release notes for SDLF 2.0.0 to learn about all the changes in this major version.

For users of SDLF 1.x, version 1 is still available on the master branch. Development of newer versions of SDLF (2.x) happens on branch main. The workshop still contains sections for version 1 as well.

What's Changed

• Support for disabling stages by @cnfait in #326
• sdlf-cicd: support for gitlab as an alternative to codecommit by @cnfait in #331
• use mkdocs instead of sphinx for documentation website by @cnfait in #332
• general documentation update by @cnfait in #333
• diagrams for sdlf stages and sdlf-monitoring by @cnfait in #335
• Fix #272: Replace logs:TagLogGroup permission with logs:TagResource by @IAMDAVID0920 in #330
• replace uses of /SDLF/S3/CentralBucket with /SDLF/S3/RawBucket by @cnfait in #336
• add /SDLF/Dynamo/Datasets SSM parameter for datasets dynamodb table by @cnfait in 0f2c16c
• doc update: sdlf in-a-nutshell diagram by @cnfait in #337

New Contributors

• @IAMDAVID0920 made their first contribution in #330

Full Changelog: 2.3.0...2.4.0