feat: personalization layer (Usage Analysis + My Stuff) — v1.1.0

README.md

@@ -28,6 +28,7 @@ The dashboard covers:
 - **Services and features** — availability status, expected launch dates, and expansion plans per region
 - **API operations** — individual API action availability per region for each AWS service
 - **CloudFormation resource types** — which resource types are supported in each region
+- **Personalized usage (opt-in)** — a "My Stuff" view that filters everything down to services, APIs, and resources actually used in your account, derived from CloudTrail and CloudFormation
 
 ![Dashboard overview](docs/images/dashboard-overview.png)
 
@@ -37,7 +38,7 @@ The solution deploys entirely within your VPC so that all data remains within yo
 
 ![High-level architecture](docs/images/high-level-architecture.png)
 
-The solution deploys a static website, REST API, and Lambda functions into your VPC. For a detailed breakdown of all resources, see [Architecture](#architecture).
+The solution deploys a static website, REST API, and Lambda functions into your VPC. Personalization is provided by an opt-in second stack that adds a Step Functions state machine and analyzer Lambdas that read your account's CloudTrail logs and CloudFormation stacks. For a detailed breakdown of all resources, see [Architecture](#architecture).
 
 ## Installation
 
@@ -112,7 +113,8 @@ npm run deploy -- \
   --api-access-subnet-id subnet-0def456 \
   --deployment-assets-bucket-name my-deploy-bucket \
   --source-access-point-arn arn:aws:s3:us-east-1:123456789012:accesspoint/my-access-point \
-  --source-folders aws-cn,public
+  --source-folders aws-cn,public \
+  --enable-usage-analysis
 ```
 
 | Flag                              | Description                                                                                                             |
@@ -123,6 +125,8 @@ npm run deploy -- \
 | `--deployment-assets-bucket-name` | S3 bucket where deployment assets (Lambda code zip) are stored.                                                         |
 | `--source-access-point-arn`       | S3 access point ARN for the capability data source (provided during onboarding).                                        |
 | `--source-folders`                | Comma-separated list of data sources to pull from (default: `public`). Include additional partitions if granted access. |
+| `--enable-usage-analysis`         | Deploy the opt-in Usage Analysis stack to enable personalization.                                                       |
+| `--cloudtrail-bucket`             | CloudTrail logs bucket used by the analyzer (only with `--enable-usage-analysis`). Auto-discovered if omitted.          |
 
 #### Teardown
 
@@ -238,6 +242,12 @@ Open the side navigation to switch between the Capability by Region dashboard an
 
 ![Settings](docs/images/user-guide-settings.png)
 
+### Personalizing the dashboard (opt-in)
+
+If you deployed with `--enable-usage-analysis`, the dashboard offers a **My Stuff** toggle that filters services, APIs, and CloudFormation resources to only what's actually used in your account. The data is produced by analyzers that read your CloudTrail logs and active CloudFormation stacks, then written back to the website bucket as a personalized data set.
+
+The analysis runs on a daily schedule. You can also trigger it on demand from the Settings page using the **Run usage analysis** button. The page shows progress and the result counts when the run completes; refresh the dashboard afterwards to see the updated personalization.
+
 ## Architecture
 
 This repository provides two CloudFormation stacks:
@@ -271,6 +281,20 @@ A development stack that mimics a customer environment for testing. Customers de
 | EC2 Instance (Linux)      | Amazon Linux 2023 instance for testing            |
 | IAM Role                  | Instance role with SSM and S3 access              |
 
+### Usage Analysis Stack (opt-in)
+
+Adds personalization. Deployed when you pass `--enable-usage-analysis` to `npm run deploy`. Reads CloudTrail logs (via Athena) and active CloudFormation stacks to build a per-account view of services, APIs, and CloudFormation resources that are actually in use, then writes a personalized data set back to the website bucket. The dashboard's "My Stuff" toggle reads that data set.
+
+| Resource                       | Description                                                                                |
+| ------------------------------ | ------------------------------------------------------------------------------------------ |
+| Step Functions State Machine   | Orchestrates the analyzers in parallel and runs the decorator                              |
+| CloudTrail Analyzer Lambda     | Queries CloudTrail logs in Athena and emits per-service usage records                      |
+| CloudFormation Analyzer Lambda | Lists active CloudFormation stacks and emits per-resource records                          |
+| Usage Decorator Lambda         | Joins analyzer output with the master capability catalog and writes the personalized files |
+| Glue Database / Table          | Schema over the CloudTrail bucket so the analyzer can run Athena queries                   |
+| Lake Formation Permissions     | Grants the analyzer role read access to the Glue database                                  |
+| EventBridge Rule               | Schedules the state machine to run daily (configurable via `AnalysisSchedule` parameter)   |
+
 ### Package Structure
 
 This project uses [npm workspaces](https://docs.npmjs.com/cli/using-npm/workspaces) to manage four packages under `source/`.
@@ -297,6 +321,7 @@ CDK application that defines the two CloudFormation stacks. We use CDK as a deve
 
 - **API Lambda** (`api-lambda-main.ts`): Backs the API Gateway and routes requests from the website.
 - **DataFetch Lambda** (`data-fetch-lambda-main.ts`): Reads capability data from the source S3 access point, merges data across multiple source folders, and writes the results to the website bucket in both JSON and CSV formats.
+- **CloudTrail Analyzer** (`cloudtrail-analyzer.ts`), **CloudFormation Analyzer** (`cloudformation-analyzer.ts`), **Usage Decorator** (`usage-decorator.ts`): Power the opt-in Usage Analysis stack. Triggered by a Step Functions state machine that runs the two analyzers in parallel, then the decorator merges their output with the master catalog into personalized files for the dashboard's "My Stuff" view.
 
 #### `source/website`
 
@@ -326,8 +351,13 @@ npm run dev:setup -- --ec2-key-pair ci-key
 
 # Deploy Capability Insights using the CapabilityInsightsSampleEnvironment outputs
 npm run dev:deploy
+
+# Or, to also deploy the opt-in Usage Analysis stack for personalization:
+npm run dev:deploy -- --enable-usage-analysis
 ```
 
+`--enable-usage-analysis` auto-discovers the CloudTrail bucket from your account. Pass `--cloudtrail-bucket <name>` to override.
+
 ### Available Scripts
 
 | Command                | Description                                                     |

deployment/deploy.sh

@@ -19,6 +19,9 @@ Deploy options (pass as flags or omit to be prompted):
   --deployment-assets-bucket-name <name> S3 bucket for deployment assets
   --source-access-point-arn <arn>        S3 access point ARN for capability data source
   --source-folders <folders>             Comma-separated list of source folders (default: public)
+  --enable-usage-analysis                Deploy the Usage Analysis stack
+  --cloudtrail-bucket <name>             S3 bucket containing CloudTrail logs (for usage analysis,
+                                         auto-discovered from your account's CloudTrail trails if omitted)
   -y, --yes                              Skip confirmation prompts
 
 Examples:
@@ -56,7 +59,7 @@ prompt_if_empty() {
 }
 
 cmd_deploy() {
-  local private_vpc_id="" backend_subnet_id="" api_access_subnet_id="" deployment_assets_bucket_name="" source_access_point_arn="" source_folders="" auto_approve=""
+  local private_vpc_id="" backend_subnet_id="" api_access_subnet_id="" deployment_assets_bucket_name="" source_access_point_arn="" source_folders="" cloudtrail_bucket="" enable_usage_analysis="" auto_approve=""
 
   while [[ $# -gt 0 ]]; do
     case $1 in
@@ -66,6 +69,8 @@ cmd_deploy() {
       --deployment-assets-bucket-name)   deployment_assets_bucket_name="$2"; shift 2 ;;
       --source-access-point-arn)         source_access_point_arn="$2"; shift 2 ;;
       --source-folders)                  source_folders="$2"; shift 2 ;;
+      --cloudtrail-bucket)               cloudtrail_bucket="$2"; shift 2 ;;
+      --enable-usage-analysis)           enable_usage_analysis="true"; shift ;;
       -y|--yes)                          auto_approve="true"; shift ;;
       *) echo "Unknown option: $1"; usage ;;
     esac
@@ -91,6 +96,28 @@ cmd_deploy() {
     fi
   done
 
+  # Auto-discover a CloudTrail bucket when --enable-usage-analysis is set
+  # but --cloudtrail-bucket was not. Queries CloudTrail directly for the
+  # account's configured trails and pulls the S3 bucket from the matching
+  # one. If multiple match, list and prompt; if none, prompt with a hint.
+  if [[ "$enable_usage_analysis" == "true" && -z "$cloudtrail_bucket" ]]; then
+    local discovered_buckets
+    discovered_buckets=$(aws cloudtrail describe-trails --query 'trailList[].S3BucketName' --output text 2>/dev/null | tr '\t' '\n' | grep -v '^$' | sort -u || true)
+    local discovered_count
+    discovered_count=$(printf '%s\n' "$discovered_buckets" | grep -c . || true)
+    if [[ "$discovered_count" == "1" ]]; then
+      cloudtrail_bucket="$discovered_buckets"
+      echo "Auto-discovered CloudTrail bucket: $cloudtrail_bucket"
+    elif [[ "$discovered_count" -gt 1 ]]; then
+      echo "Multiple CloudTrail buckets found:"
+      printf '  %s\n' $discovered_buckets
+      prompt_if_empty cloudtrail_bucket "CloudTrailBucket (paste one of the above)"
+    else
+      echo "No CloudTrail trails found in this account/region. Configure CloudTrail to log to S3, or pass --cloudtrail-bucket explicitly."
+      prompt_if_empty cloudtrail_bucket "CloudTrailBucket"
+    fi
+  fi
+
   echo ""
   echo "Deploying to account $AWS_ACCOUNT in $AWS_REGION"
   if [[ "$auto_approve" != "true" ]]; then
@@ -150,8 +177,84 @@ cmd_deploy() {
   echo "── Uploading website assets ──"
   get_account_and_region
   local website_bucket="capability-insights-website-${ACCOUNT_ID}-${REGION}"
+  local website_bucket_arn="arn:aws:s3:::${website_bucket}"
   aws s3 sync "$SCRIPT_DIR/dist/website/" "s3://$website_bucket/"
 
+  echo "── Deploying Usage Analysis stack ──"
+  if [[ "$enable_usage_analysis" == "true" ]]; then
+    aws cloudformation deploy \
+      --template-file "$SCRIPT_DIR/dist/template/usage-analysis.template.json" \
+      --stack-name CapabilityInsightsUsageAnalysis \
+      --parameter-overrides \
+        WebsiteBucketName="$website_bucket" \
+        WebsiteBucketArn="$website_bucket_arn" \
+        DeploymentAssetsBucketName="$deployment_assets_bucket_name" \
+        LambdaCodeZipPath="$lambda_key" \
+        CloudTrailBucketName="${cloudtrail_bucket:-}" \
+      --capabilities CAPABILITY_NAMED_IAM \
+      --no-cli-pager || true
+    echo "  ✓ Usage Analysis stack deployed."
+
+    # Get outputs from Usage Analysis stack
+    local analysis_state_machine_arn cloudtrail_analyzer_lambda_name cloudformation_analyzer_lambda_name usage_decorator_lambda_name
+    analysis_state_machine_arn=$(aws cloudformation describe-stacks \
+      --stack-name CapabilityInsightsUsageAnalysis \
+      --query "Stacks[0].Outputs[?OutputKey=='AnalysisStateMachineArn'].OutputValue" --output text 2>/dev/null || echo "")
+    cloudtrail_analyzer_lambda_name=$(aws cloudformation describe-stacks \
+      --stack-name CapabilityInsightsUsageAnalysis \
+      --query "Stacks[0].Outputs[?OutputKey=='CloudTrailAnalyzerLambdaName'].OutputValue" --output text 2>/dev/null || echo "")
+    cloudformation_analyzer_lambda_name=$(aws cloudformation describe-stacks \
+      --stack-name CapabilityInsightsUsageAnalysis \
+      --query "Stacks[0].Outputs[?OutputKey=='CloudFormationAnalyzerLambdaName'].OutputValue" --output text 2>/dev/null || echo "")
+    usage_decorator_lambda_name=$(aws cloudformation describe-stacks \
+      --stack-name CapabilityInsightsUsageAnalysis \
+      --query "Stacks[0].Outputs[?OutputKey=='UsageDecoratorLambdaName'].OutputValue" --output text 2>/dev/null || echo "")
+
+    # Force Lambda code update (CloudFormation may skip if template is unchanged)
+    if [[ -n "$cloudtrail_analyzer_lambda_name" ]]; then
+      aws lambda update-function-code \
+        --function-name "$cloudtrail_analyzer_lambda_name" \
+        --s3-bucket "$deployment_assets_bucket_name" \
+        --s3-key "$lambda_key" > /dev/null 2>&1 || true
+    fi
+    if [[ -n "$cloudformation_analyzer_lambda_name" ]]; then
+      aws lambda update-function-code \
+        --function-name "$cloudformation_analyzer_lambda_name" \
+        --s3-bucket "$deployment_assets_bucket_name" \
+        --s3-key "$lambda_key" > /dev/null 2>&1 || true
+    fi
+    if [[ -n "$usage_decorator_lambda_name" ]]; then
+      aws lambda update-function-code \
+        --function-name "$usage_decorator_lambda_name" \
+        --s3-bucket "$deployment_assets_bucket_name" \
+        --s3-key "$lambda_key" > /dev/null 2>&1 || true
+    fi
+
+    if [[ -n "$analysis_state_machine_arn" && -n "$cloudtrail_analyzer_lambda_name" && -n "$cloudformation_analyzer_lambda_name" ]]; then
+      echo "── Updating Insights stack with Usage Analysis outputs ──"
+      aws cloudformation deploy \
+        --template-file "$SCRIPT_DIR/dist/template/capability-insights.template.json" \
+        --stack-name CapabilityInsightsForAWS \
+        --parameter-overrides \
+          PrivateVpcId="$private_vpc_id" \
+          BackendSubnetId="$backend_subnet_id" \
+          ApiAccessSubnetId="$api_access_subnet_id" \
+          DeploymentAssetsBucketName="$deployment_assets_bucket_name" \
+          DeploymentAssetsBucketApiLambdaFunctionCodeZipPath="$lambda_key" \
+          SourceAccessPointArn="$source_access_point_arn" \
+          SourceFolders="$source_folders" \
+          AnalysisStateMachineArn="$analysis_state_machine_arn" \
+          CloudTrailAnalyzerLambdaName="$cloudtrail_analyzer_lambda_name" \
+          CloudFormationAnalyzerLambdaName="$cloudformation_analyzer_lambda_name" \
+          ConfiguredCloudTrailBucketName="$cloudtrail_bucket" \
+        --capabilities CAPABILITY_NAMED_IAM \
+        --no-cli-pager || true
+      echo "  ✓ Insights stack updated with analysis integration."
+    fi
+  else
+    echo "  Skipped (pass --enable-usage-analysis to deploy)."
+  fi
+
   echo "── Syncing capability data ──"
   aws lambda invoke --function-name CapabilityInsightsDataFetchLambda --invocation-type Event /dev/null > /dev/null 2>&1
 
@@ -169,14 +272,19 @@ cmd_teardown() {
   local website_bucket="capability-insights-website-${ACCOUNT_ID}-${REGION}"
 
   if [[ "$AUTO_APPROVE" != "true" ]]; then
-    echo "This will delete the CapabilityInsightsForAWS stack and empty the website bucket."
+    echo "This will delete the CapabilityInsightsForAWS and CapabilityInsightsUsageAnalysis stacks and empty the website bucket."
     read -rp "Continue? (y/N): " confirm
     [[ "$confirm" =~ ^[Yy]$ ]] || exit 0
   fi
 
   echo "── Emptying website bucket ──"
   aws s3 rm "s3://$website_bucket" --recursive || true
 
+  echo "── Destroying Usage Analysis stack ──"
+  aws cloudformation delete-stack --stack-name CapabilityInsightsUsageAnalysis 2>/dev/null || true
+  aws cloudformation wait stack-delete-complete --stack-name CapabilityInsightsUsageAnalysis 2>/dev/null || true
+  echo "  ✓ Usage Analysis stack deleted."
+
   echo "── Destroying stack (this will likely take ~15 minutes) ──"
   aws cloudformation delete-stack --stack-name CapabilityInsightsForAWS
   local elapsed=0

deployment/dev.sh

@@ -21,6 +21,9 @@ Setup options:
 Deploy options:
   --source-access-point-arn <arn>  S3 access point ARN for capability data source
   --source-folders <folders>       Comma-separated list of source folders (default: public)
+  --enable-usage-analysis          Also deploy the opt-in Usage Analysis stack
+  --cloudtrail-bucket <name>       CloudTrail logs bucket (only with --enable-usage-analysis,
+                                   auto-discovered from your account if omitted)
 
 Global options:
   -y, --yes                        Skip confirmation prompts
@@ -60,11 +63,13 @@ cmd_setup() {
 }
 
 cmd_deploy() {
-  local source_access_point_arn="" source_folders=""
+  local source_access_point_arn="" source_folders="" enable_usage_analysis="" cloudtrail_bucket=""
   while [[ $# -gt 0 ]]; do
     case $1 in
       --source-access-point-arn) source_access_point_arn="$2"; shift 2 ;;
       --source-folders) source_folders="$2"; shift 2 ;;
+      --enable-usage-analysis) enable_usage_analysis="true"; shift ;;
+      --cloudtrail-bucket) cloudtrail_bucket="$2"; shift 2 ;;
       -y|--yes) shift ;;
       *) echo "Unknown option: $1"; usage ;;
     esac
@@ -96,12 +101,21 @@ cmd_deploy() {
     access_point_args+=(--source-folders "$source_folders")
   fi
 
+  local usage_analysis_args=()
+  if [[ "$enable_usage_analysis" == "true" ]]; then
+    usage_analysis_args+=(--enable-usage-analysis)
+    if [[ -n "$cloudtrail_bucket" ]]; then
+      usage_analysis_args+=(--cloudtrail-bucket "$cloudtrail_bucket")
+    fi
+  fi
+
   local deploy_args=(
     --private-vpc-id "$private_vpc_id"
     --backend-subnet-id "$backend_subnet_id"
     --api-access-subnet-id "$api_access_subnet_id"
     --deployment-assets-bucket-name "$deployment_assets_bucket_name"
     "${access_point_args[@]}"
+    "${usage_analysis_args[@]}"
   )
   [[ -n "$AUTO_APPROVE" ]] && deploy_args+=(--yes)