Python: Catalog scanner to create service-level example catalog files.

.github/workflows/catalog.yml

@@ -0,0 +1,35 @@
+name: Validate Catalogs
+
+# Controls when the action will run.
+on:  # yamllint disable-line rule:truthy
+  # Triggers the workflow on push or pull request events but only for the main branch
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+  # Allows you to run this workflow manually from the Actions tab
+  workflow_dispatch:
+
+# A workflow run is made up of one or more jobs that can run sequentially or in parallel
+jobs:
+  # This workflow contains a single job called "build"
+  build:
+    name: Validate Catalogs
+    # The type of runner that the job will run on
+    runs-on: ubuntu-latest
+
+    # Steps represent a sequence of tasks that will be executed as part of the job
+    steps:
+      - name: Checkout repo content
+        uses: actions/checkout@v4  # checkout the repository content to github runner.
+      - name: Setup Python
+        uses: actions/setup-python@v3
+        with:
+          python-version: 3.11  # install the python needed
+      - name: Install dependencies
+        run: >-
+          python3 -m pip install -r .tools/scanners/requirements.txt
+      - name: Check Catalogs
+        run: >-
+          python3 .tools/scanners/catalog.py --check --diff

.github/workflows/writeme.yml

@@ -29,7 +29,7 @@ jobs:
           python-version: 3.11  # install the python needed
       - name: Install dependencies
         run: >-
-          python3 -m pip install -r .tools/readmes/requirements.txt
+          python3 -m pip install -r .tools/scanners/requirements.txt
       - name: Check WRITEMEs
         run: >-
-          python3 .tools/readmes/writeme.py --check --diff
+          python3 .tools/scanners/writeme.py --check --diff

.tools/readmes/README.md → .tools/scanners/README.md

@@ -2,17 +2,19 @@
 
 ## Overview
 
-Describes WRITEME, a tool to automatically generate service-level READMEs from
-metadata and Jinja templates.
+Describes a set of scanners to generate documents from the example metadata in this repository.
 
-This is an internal tool intended for use only by the AWS code examples team.
+- [writeme.py](#WRITEME) - generates READMEs from metadata. Enabled for all SDK languages.
+- [catalog.py](#Catalog) - generates catalog files with example listings. Currently enabled for Python only.
+
+### General Information
 
 ## Prerequisites
 
-We recommend a virtual environment to run this tool.
+We recommend a virtual environment to run these tools.
 
 ```
-cd .tools/readmes
+cd .tools/scanners
 python -m venv .venv
 
 # Windows
@@ -26,30 +28,6 @@ Depending on how you have Python installed and on your operating system,
 the commands might vary slightly. For example, on Windows, use `py` in place of
 `python` and uses backslashes in the `venv` path.
 
-## Generate a README
-
-> These instructions assume you're running the commands from the `.tools/readmes`
-> directory, using the venv installed there.
-
-WRITEME creates content primarily from metadata you have already
-authored for the SOS project. After you have authored metadata and snippet tags
-for your examples, run the following command in the `.tools/readmes` folder:
-
-```
-python -m writeme --languages <language>:<version> --services <service>
-```
-
-WRITEME reads metadata and config data and generates READMEs in the service
-folder for the specified languages, versions, and services.
-
-For example, to generate an S3 README for Python:
-
-```
-python -m writeme --languages Python:3 --services s3
-```
-
-This creates a README.md file in the `python/example_code/s3` folder.
-
 ### Parameters
 
 - `--languages` must match a top-level language:version in sdks.yaml.
@@ -58,8 +36,8 @@ This creates a README.md file in the `python/example_code/s3` folder.
   `saved_readme` value in config.py (such as README.old.md).
 - `--verbose` When set, output verbose debugging info.
 - `--dry-run`, `--no-dry-run` In dry run, compare current vs generated and exit with failure if they do not match.
-- `--check` Verifies whether the existing README.md matches the proposed new README.md
-  (but does not write a new README.md). This is the same check that is run by the GitHub action.
+- `--check` Verifies whether the existing generated file matches the generated content.
+  (but does not write a new file). This is the same check that is run by the GitHub action.
 
 You can get inline usage info by using the `-h` flag:
 
@@ -73,6 +51,7 @@ Additional configuration is kept in `config.py`.
 
 - `entities` is a dictionary of entities that are not otherwise defined in
   services.yaml.
+- `catalog_filename` is the base name for generated catalog files.
 - `language` is a dictionary of language and version for each SDK. Fields are:
   - `base_folder` the root folder for the SDK version.
   - `service_folder` a Jinja template of the service folder for the SDK version.
@@ -83,6 +62,38 @@ Additional configuration is kept in `config.py`.
     and if the generated link is wrong, you can update it manually. On subsequent runs
     of WRITEME, the existing link is kept.
 
+
+### WRITEME
+WRITEME is a tool to automatically generate service-level READMEs from
+metadata and Jinja templates.
+
+This is an internal tool intended for use only by the AWS code examples team.
+
+## Generate a README
+
+> These instructions assume you're running the commands from the `.tools/scanners`
+> directory, using the venv installed there.
+
+WRITEME creates content primarily from metadata you have already
+authored for the SOS project. After you have authored metadata and snippet tags
+for your examples, run the following command in the `.tools/scanners` folder:
+
+```
+python -m writeme --languages <language>:<version> --services <service>
+```
+
+WRITEME reads metadata and config data and generates READMEs in the service
+folder for the specified languages, versions, and services.
+
+For example, to generate an S3 README for Python:
+
+```
+python -m writeme --languages Python:3 --services s3
+```
+
+This creates a README.md file in the `python/example_code/s3` folder.
+
+
 ### Custom content
 
 Custom content can be per-SDK or per-README.
@@ -114,13 +125,13 @@ empty if you don't need custom content.
 versions, and services.
 
 ```
-python .tools/readmes/writeme.py --languages <language1>:<version> <language2>:<version> --service <service1> <service2>
+python .tools/scanners/writeme.py --languages <language1>:<version> <language2>:<version> --service <service1> <service2>
 ```
 
 For example, to generate S3 and STS READMEs for Python sdk version 3 and Go sdk version 2:
 
 ```
-python .tools/readmes/writeme.py --languages Python:3 Go:2 --services s3 sts
+python .tools/scanners/writeme.py --languages Python:3 Go:2 --services s3 sts
 ```
 
 This creates the README.md files in `python/example_code/s3` and other folders.
@@ -147,3 +158,32 @@ And yes, building all readmes for all languages after changing metadata or templ
 ```
 python -m writeme
 ```
+
+### Catalog
+Catalog.py is a tool to automatically generate service-level example catalogs (examples_catalog.json) from metadata using shared document generation tools.
+
+This is an internal tool intended for use only by the AWS code examples team.
+
+## Generate a Catalog
+
+> These instructions assume you're running the commands from the `.tools/scanners`
+> directory, using the venv installed there.
+
+WRITEME creates content primarily from metadata you have already
+authored for the SOS project. After you have authored metadata and snippet tags
+for your examples, run the following command in the `.tools/scanners` folder:
+
+```
+python -m catalog --languages <language>:<version> --services <service>
+```
+
+Catalog.py reads metadata and config data and generates listings of examples in the service
+folder for the specified languages and versions. The file will not be generated if no examples exist.
+
+For example, to generate an S3 README for Python:
+
+```
+python -m catalog --languages Python:3 --services s3
+```
+
+This creates an `examples_catalog.json` file in the `python/example_code/s3` folder.

.tools/scanners/catalog.py

@@ -0,0 +1,26 @@
+#!/usr/bin/env python3
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+import sys
+
+from update import update
+
+# For debugging, engineers can skip update with the --no-update flag. Yes, it's
+# a double negative, but it's quick and early in the startup because of the
+# reliance on the side-effect imports from `update` and needing them to happen
+# before importing runner, which means before importing the runner argparser.
+NO_UPDATE_FLAG = "--no-update"
+
+
+if __name__ == "__main__":
+    if NO_UPDATE_FLAG not in sys.argv:
+        update()
+    else:
+        sys.argv.remove(NO_UPDATE_FLAG)
+
+    # This import must remain in the main, after the update, as it depends on
+    # importing the things that got changed during update.
+    from catalog_runner import main
+
+    sys.exit(main())