Refactor for migration

.gitignore

@@ -5,3 +5,4 @@ raw_data/
 .coverage
 .vscode/settings.json
 http_cache.sqlite
+sdk.csv

README.md

@@ -1,57 +1,252 @@
-# hub-utils
+# `hub-utils`
 
-## Add
+[MeltanoHub](https://hub.meltano.com/) Utilities - A utility CLI intended
+to streamline the work needed to maintain MeltanoHub.
 
-A utility CLI intended to streamline the steps needed to add Singer taps/targets to [MeltanoHub](https://hub.meltano.com/).
+**Installation**
 
 ```
-export HUB_ROOT_PATH='/Users/pnadolny/Documents/Git/GitHub/meltano/hub'
+export HUB_ROOT_PATH='/Users/meltano/hub'
 
 poetry install
-poetry run hub_utils --help
+poetry run hub-utils --help
+```
+
+**Tests**
+
+```bash
+poetry run pytest -v
 
-poetry run hub_utils add https://github.com/MeltanoLabs/tap-google-analytics
-poetry run hub_utils add https://github.com/MeltanoLabs/tap-google-analytics --auto-accept
+poetry run pytest -v tests/test_core.py::test_sdk_about_parsing_1
 ```
 
-## Bulk Add
+**Refreshing This README**
 
-This will iterate all reo_urls in a CSV file and prompt as it goes through.
-After writing output it excludes the repo_url from an output CSV so you can pause iterating and restart. If skipped or added, it shouldnt prompt again.
+The CLI is auto documenting so put all content in the CLI modules
+and this will use [typer-cli](https://typer.tiangolo.com/typer-cli/) utilities
+to render them as markdown.
 
+Run the following command:
 
+```bash
+poetry run typer hub_utils/main.py utils docs --name hub-utils --output README.md
 ```
-poetry run hub_utils add-bulk /path/to/csv_file
+
+**Usage**:
+
+```console
+$ hub-utils [OPTIONS] COMMAND [ARGS]...
 ```
 
-## Add Airbyte
+**Options**:
+
+* `--install-completion`: Install completion for the current shell.
+* `--show-completion`: Show completion for the current shell, to copy it or customize the installation.
+* `--help`: Show this message and exit.
 
+**Commands**:
+
+* `add`: Add a new tap or target to the hub.
+* `download-metadata`: NOTE: USED FOR...
+* `extract-sdk-metadata-to-s3`: NOTE: USED FOR...
+* `get-variant-names`: NOTE: USED FOR...
+* `merge-metadata`: NOTE: USED FOR...
+* `sdk-variants-as-csv`: Generate a `sdk.csv` CSV file in the...
+* `update-definition`: Update the definition of a tap or target...
+* `update-quality`: Update the quality of all taps and targets...
+* `upload-airbyte`: NOTE: USED FOR...
+
+## `hub-utils add`
+
+Add a new tap or target to the hub.
+It will prompt you for any attributes that need input.
+
+If the plugin is SDK based it will do its best to install the plugin and scrape
+the settings for you. It will prompt you for any missing attributes. If its not
+SDK based it will prompt you for settings 1 at a time and help you by suggesting
+defaults that you can accept or override.
+
+**Usage**:
+
+```console
+$ hub-utils add [OPTIONS]
 ```
-poetry run hub_utils add-airbyte
+
+**Options**:
+
+* `--repo-url TEXT`
+* `--auto-accept / --no-auto-accept`: [default: no-auto-accept]
+* `--help`: Show this message and exit.
+
+## `hub-utils download-metadata`
+
+NOTE: USED FOR
+[AUTOMATION](https://github.com/meltano/hub/tree/main/.github/workflows) ONLY
+Download the latest metadata for the given variants from S3.
+
+**Usage**:
+
+```console
+$ hub-utils download-metadata [OPTIONS] LOCAL_PATH
 ```
 
+**Arguments**:
+
+* `LOCAL_PATH`: [required]
+
+**Options**:
+
+* `--variant-path-list TEXT`
+* `--help`: Show this message and exit.
 
-### Update SDK
+## `hub-utils extract-sdk-metadata-to-s3`
 
+NOTE: USED FOR
+[AUTOMATION](https://github.com/meltano/hub/tree/main/.github/workflows) ONLY
+
+Extract the SDK metadata for the given variants and upload them to S3.
+
+**Usage**:
+
+```console
+$ hub-utils extract-sdk-metadata-to-s3 [OPTIONS] VARIANT_PATH_LIST OUTPUT_DIR
 ```
-poetry run hub_utils update-sdk --repo-url https://github.com/MeltanoLabs/tap-snowflake --auto-accept
+
+**Arguments**:
+
+* `VARIANT_PATH_LIST`: [required]
+* `OUTPUT_DIR`: [required]
+
+**Options**:
+
+* `--help`: Show this message and exit.
+
+## `hub-utils get-variant-names`
+
+NOTE: USED FOR
+[AUTOMATION](https://github.com/meltano/hub/tree/main/.github/workflows) ONLY
+
+Generate a list of variant names for a given set of filters.
+The list will be formatted as escaped JSON to be used by Github Actions.
+
+**Usage**:
+
+```console
+$ hub-utils get-variant-names [OPTIONS] HUB_ROOT
 ```
 
-### Refresh All SDK-based Variants
+**Arguments**:
+
+* `HUB_ROOT`: [required]
+
+**Options**:
+
+* `--metadata-type TEXT`: [default: sdk]
+* `--plugin-type TEXT`
+* `--help`: Show this message and exit.
 
+## `hub-utils merge-metadata`
+
+NOTE: USED FOR
+[AUTOMATION](https://github.com/meltano/hub/tree/main/.github/workflows) ONLY
+
+Merge the latest SDK metadata from S3 with the existing hub
+
+**Usage**:
+
+```console
+$ hub-utils merge-metadata [OPTIONS] HUB_ROOT LOCAL_PATH
 ```
-poetry run hub_utils refresh-sdk-variants --starting-yaml="/_data/meltano/extractors/tap-zendesk-sell/leag.yml"
+
+**Arguments**:
+
+* `HUB_ROOT`: [required]
+* `LOCAL_PATH`: [required]
+
+**Options**:
+
+* `--variant-path-list TEXT`
+* `--help`: Show this message and exit.
+
+## `hub-utils sdk-variants-as-csv`
+
+Generate a `sdk.csv` CSV file in the current directory containing the following
+columns:
+plugin_type, name, variant, sdk
+
+**Usage**:
+
+```console
+$ hub-utils sdk-variants-as-csv [OPTIONS]
 ```
 
-### Export All SDK-based Metadata
+**Options**:
+
+* `--help`: Show this message and exit.
+
+## `hub-utils update-definition`
+
+Update the definition of a tap or target in the hub.
 
+Similar to the `add` command it will try to auto update using SDK settings or prompt
+you for input. When merging it will use the following rules:
+- if SDK setting description is empty it prefers the existing description
+- if the existing description longer than the scraped setting and has new lines
+    then its likely manually overridden on the hub so prefer that one.
+
+**Usage**:
+
+```console
+$ hub-utils update-definition [OPTIONS]
 ```
-poetry run hub_utils extract-metadata --output-dir=/Users/pnadolny/Documents/Git/GitHub/pnadolny/hub-utils/data
+
+**Options**:
+
+* `--repo-url TEXT`
+* `--plugin-name TEXT`
+* `--auto-accept / --no-auto-accept`: [default: no-auto-accept]
+* `--help`: Show this message and exit.
+
+## `hub-utils update-quality`
+
+Update the quality of all taps and targets on the hub.
+
+This command accepts a path to the [variant_metrics.yml]
+(https://github.com/meltano/hub/blob/main/_data/variant_metrics.yml)
+yaml file, make sure its the most up to date version likely sourced from S3.
+
+**Usage**:
+
+```console
+$ hub-utils update-quality [OPTIONS] METRICS_FILE_PATH
 ```
-## Tests
 
-```bash
-poetry run pytest -v
+**Arguments**:
 
-poetry run pytest -v tests/test_core.py::test_sdk_about_parsing_1
-```
+* `METRICS_FILE_PATH`: [required]
+
+**Options**:
+
+* `--help`: Show this message and exit.
+
+## `hub-utils upload-airbyte`
+
+NOTE: USED FOR
+[AUTOMATION](https://github.com/meltano/hub/tree/main/.github/workflows) ONLY
+
+Upload the given Airbyte artifacts to S3.
+
+**Usage**:
+
+```console
+$ hub-utils upload-airbyte [OPTIONS] VARIANT_PATH_LIST ARTIFACT_NAME
+```
+
+**Arguments**:
+
+* `VARIANT_PATH_LIST`: [required]
+* `ARTIFACT_NAME`: [required]
+
+**Options**:
+
+* `--help`: Show this message and exit.