improve output

Author: JoshuaFox

README.md

@@ -17,9 +17,8 @@ Resources of all supported types in all or some of the projects in the GCP organ
 ## Note: Organization focus
 
 Note that Iris is designed to serve the organization. 
-* It is not designed around serving a single project (though you can configure that). 
-* Only one instance of Iris runs at any one time in an organization. 
-* The organization focus was chosen because labels are used for billing analysis which is typically done on the organization level (even though projects can be associated arbitrarily with billing accounts). 
+* It is designed to label all projects in the organization (though you can configure that).
+* The organization focus was chosen because labels are used for billing analysis which is typically done on the organization level (even though projects can in fact be associated arbitrarily with billing accounts). 
 
 ## Iris doesn't add new information
 
@@ -90,7 +89,7 @@ names start `_gcp_`. The part of the function name after `_gcp_` is used for the
   [new project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#creating_a_project).
 
 ### Needed roles for deployment
-#### Organization-leve roles
+#### Organization-level roles
 
 * Here are the required organization-level roles for you, the deployer, to allow the deploy script to set up roles and  log sink. (Note that *Organization Owner* is not enough).
     * *Organization Role Administrator*  so the deployment script can create a custom IAM role for Iris that allows it to  get and set labels.
@@ -122,17 +121,14 @@ names start `_gcp_`. The part of the function name after `_gcp_` is used for the
     * Optionally configure by editing the config file. ([See more documentation below](#configuration).)
 
 * Now, run `./deploy.sh <PROJECT_ID> `.
-* For options, run `deploy.sh -h` for documentation on usage of command-line options.
+  * For documentation on usage of command-line options, run `deploy.sh -h` 
 
-* When the labeling occurs
+* Choosing when the labeling occurs
   * By default, labeling occurs on resource-creation, and also using Cloud Scheduler ("cron"). Each of these types works on certain resource types and not others (see [Supported Google Cloud Labels](#Supported Google Cloud Labels) above), depending on configuration (`config.yaml`).
   * Use `-c` switch on `deploy.sh` to label using Cloud Scheduler only.
   * Use `-e` switch on `deploy.sh` to label on-event only.
   * If you use **both** `-c` and `-e` or **neither**, both types of labeling occur.
-  * If you change from having Cloud Scheduler labeling to not having it, or vice versa, be sure to deploy both org-level and project-level elements , not just project elements, since this involves the org-level sink.
-* Organization-level and project-level elements
-  * First, note that Iris is an organization-level application. A single instance of Iris labels all projects in an org (unless you limit it by configuration). Iris has architecture elements which are deployed to both the org and the project.
-  * If you want a person with all permissions to deploy the org-level elements and a person without org-level permissions to redeploy only the project-level elements (e.g. when you change configuration), you can do this with flags on the script. Run   `deploy.sh -h`.
+
 
 
 # Configuration
@@ -156,6 +152,11 @@ names start `_gcp_`. The part of the function name after `_gcp_` is used for the
 # Architecture
 
 * Iris runs in Google App Engine Standard Environment (Python 3).
+* Organization-level and project-level elements
+  * First, note that Iris is an organization-level application.
+  * A single instance of Iris in one project labels all projects in an org (unless you limit it by configuration). 
+  * Iris has architecture elements which are deployed to both the org and the project.
+  * If you want a person with all permissions to deploy the org-level elements and a person without org-level permissions to redeploy only the project-level elements (e.g. when you change configuration), you can do this with flags on the script. Run `deploy.sh -h`.
 * The Cloud Scheduler "cron" job triggers Iris at configured intervals. See `cron_full.yaml` for config. The GCP Console view for this is in a  separate App Engine tab in the Cloud Scheduler view.
 * For newly created resources, a Log Sink on the organization level sends all logs for resource creation to a PubSub  topic.
     * The Log Sink is filtered to include only supported resource types
@@ -169,74 +170,7 @@ names start `_gcp_`. The part of the function name after `_gcp_` is used for the
     * For security, these two PubSub subscriptions [use JWT auth](https://cloud.google.com/pubsub/docs/authenticate-push-subscriptions), where the JWT token is verified in the Iris webapp.
     * A dead-letter subscription. This is a pull subscription. By default, it just accumulates the messages. You can use it to see statistics, or you can pull messages from it.
 
-# Local Development
-
-## Development tools
-
-* Prerequisites for developing and building.
-    * See [Installation](#installation)
-    * Also, for development, set up a virtual env and run `pip3 install -r requirements.txt`
-* Run the server locally
-    * Run `main.py` as an ordinary Flask application as follows:
-        * To use the command-line,
-          use `export FLASK_ENV=development;export FLASK_RUN_PORT=8000;export FLASK_DEBUG=1;FLASK_APP=main.py python -m flask run`
-        * In an interactive development environment, run `main.py`, first setting these environment variables.
-* For hands-on debugging
-    * Use `test_do_label` and `test_label_one` and `test_schedule` to trigger against your localhost dev-server; this should   label actual Cloud resources that you have pre-deployed.
-        * See the `test_...` files for instructions.
-
-## Adding new kinds of labels
-
-Iris adds about twenty kinds of labels. More can be added, but don't add too many: Billing analytics work best when not swamped by excess labels. This is why Iris does not implement all possible labeling, say by automatically copying all fields from each resource into labels.
-
-### Developing new labels for an existing resource type
-
-To add a new label key to an existing resource type, add `_gcp_<LABEL_NAME>` methods (like `_gcp_zone()`) in the relevant file in `/plugins`, following the example of the existing ones. Labels will be added with a key from the function name (`zone` in that example), and a value returned by the function (in our example, the zone identifier).
-
-For example, you might want to add a label identifying the creator of a resource, or add the name of the topic to its subscriptions.
-
-### Supporting new resource types
-
-Iris is easily extensible with plugins, to support labeling of other GCP resources. Use existing files in `/plugins` as
-examples.
-
-1. Create a Python file in the `/plugins` directory, holding a subclass of `Plugin`.
-
-   a. The filename and class name take the form: `cloudsql.py` and `Cloudsql`. That's lowercase and Titlecase. (Only the
-   first character is capitalized, even in multiword names.) The two names should be the same except for case.
-
-   b. Implement abstract methods from the `Plugin` class.
-
-   c. Add `_gcp_<LABEL_NAME>` methods (like `_gcp_zone()`). Labels will be added with a key from the function
-   name (`zone` in that example), and a value returned by the function
-   (in our example, the zone identifier).
-
-   d. For resources that cannot be labeled on creation (like CloudSQL, which takes too long to initialize), you should
-   override `is_labeled_on_creation()` and return `False`  (though if you don't, the only bad-side effect will be errors
-   in the logs).
-
-   e. For resources with mutable labels  (like Disks, for which attachment state may have changed), override `relabel_on_cron()` and return `True`. This will allow Cloud Scheduler to relabel them. (This is because after a resource is created and possibly labeled, so Cloud Scheduler is the way to relabel mutated state.)
-
-2. Add your API to the `required_svcs` in `deploy.sh`.
-
-3. Add your Google Cloud API "methods" to `log_filter` in `deploy.sh`.
-    * `methodName` is part of the logs generated on creation.
-    * See examples of such logs in `sample_data` directory.
-        * E.g., you can see a log sample for bucket creation, in
-          file `sample_data/storage.buckets.create.log_message.json`. (Or create a bucket and look at the log.)
-        * In that file you see `"methodName": "storage.buckets.create"`.
-
-4. Add permissions in `iris-custom-role.yaml` to be granted to the Iris custom role. 
-   * This role allows Iris, for each resource type, to list, get, and update. 
-     * ("Update" requires permission `setLabels` where available or permission `update`  otherwise.) 
-     * The name of this custom role is `iris3`by default, but may be set by passing env variable `IRIS_CUSTOM_ROLE` in calling `deploy.sh` or `uninstall.sh`
-
-# Testing
-
-## Integration test
-
-`./test_scripts/integration_test.py` creates a Google App Engine app and cloud resources, and tests against them. Run it without parameters for usage instructions.
-
-# Next steps
+# Development and Testing
 
-See `TODO.md` and GitHub issues for potential future improvements.
+Please see [docs_for_dev_and_testing](./docs_for_dev_and_testing)
+ 

deploy.sh

@@ -34,7 +34,6 @@ deploy_org=
 export LABEL_ON_CRON=
 export LABEL_ON_CREATION_EVENT=
 
-
 while getopts 'cepoh' opt; do
   case $opt in
   c)
@@ -91,7 +90,7 @@ fi
 
 export PROJECT_ID=$1
 
-pip3 install -r requirements.txt >/dev/null
+pip3 install -r requirements.txt > /dev/null
 
 # If both -c and -e are not given, then actually act as if both are there.
 if [[ "$LABEL_ON_CRON" != "true" ]] && [[ "$LABEL_ON_CREATION_EVENT" != "true" ]]; then
@@ -106,14 +105,13 @@ if [[ "$deploy_org" != "true" ]] && [[ "$deploy_proj" != "true" ]]; then
   deploy_proj=true
 fi
 
-
-gcloud projects describe "$PROJECT_ID" >/dev/null|| {
+gcloud projects describe "$PROJECT_ID" >/dev/null || {
   echo "Project $PROJECT_ID not found"
   exit 1
 }
 
-#echo "Project ID $PROJECT_ID"
-gcloud config set project "$PROJECT_ID"
+gcloud auth application-default set-quota-project $PROJECT_ID > /dev/null 2>&1
+gcloud config set project "$PROJECT_ID" > /dev/null  2>&1
 
 
 if [[ "$deploy_org" == "true" ]]; then

docs_for_dev_and_testing.md

@@ -0,0 +1,71 @@
+# Iris: Dev and Testing
+ 
+ 
+## Development tools
+
+* Prerequisites for developing and building.
+    * See [Installation](#installation)
+    * Also, for development, set up a virtual env and run `pip3 install -r requirements.txt`
+* Run the server locally
+    * Run `main.py` as an ordinary Flask application as follows:
+        * To use the command-line,
+          use `export FLASK_ENV=development;export FLASK_RUN_PORT=8000;export FLASK_DEBUG=1;FLASK_APP=main.py python -m flask run`
+        * In an interactive development environment, run `main.py`, first setting these environment variables.
+* For hands-on debugging
+    * Use `test_do_label` and `test_label_one` and `test_schedule` to trigger against your localhost dev-server; this should   label actual Cloud resources that you have pre-deployed.
+        * See the `test_...` files for instructions.
+
+## Adding new kinds of labels
+
+Iris adds about twenty kinds of labels. More can be added, but don't add too many: Billing analytics work best when not swamped by excess labels. This is why Iris does not implement all possible labeling, say by automatically copying all fields from each resource into labels.
+
+## Developing new labels for an existing resource type
+
+To add a new label key to an existing resource type, add `_gcp_<LABEL_NAME>` methods (like `_gcp_zone()`) in the relevant file in `/plugins`, following the example of the existing ones. Labels will be added with a key from the function name (`zone` in that example), and a value returned by the function (in our example, the zone identifier).
+
+For example, you might want to add a label identifying the creator of a resource, or add the name of the topic to its subscriptions.
+
+## Supporting new resource types
+
+Iris is easily extensible with plugins, to support labeling of other GCP resources. Use existing files in `/plugins` as examples.
+
+1. Create a Python file in the `/plugins` directory, holding a subclass of `Plugin`.
+
+   a. The filename and class name take the form: `cloudsql.py` and `Cloudsql`. That's lowercase and Titlecase. (Only the
+   first character is capitalized, even in multiword names.) The two names should be the same except for case.
+
+   b. Implement abstract methods from the `Plugin` class.
+
+   c. Add `_gcp_<LABEL_NAME>` methods (like `_gcp_zone()`). Labels will be added with a key from the function
+   name (`zone` in that example), and a value returned by the function
+   (in our example, the zone identifier).
+
+   d. For resources that cannot be labeled on creation (like CloudSQL, which takes too long to initialize), you should
+   override `is_labeled_on_creation()` and return `False`  (though if you don't, the only bad-side effect will be errors
+   in the logs).
+
+   e. For resources with mutable labels  (like Disks, for which attachment state may have changed), override `relabel_on_cron()` and return `True`. This will allow Cloud Scheduler to relabel them. (This is because after a resource is created and possibly labeled, so Cloud Scheduler is the way to relabel mutated state.)
+
+2. Add the relevant Google Cloud API to the `required_svcs` in `deploy.sh`.
+
+3. Add your Google Cloud API "methods" to `log_filter` in `deploy.sh`.
+    * `methodName` is part of the logs generated on creation.
+    * See examples of such logs in `sample_data` directory.
+        * E.g., you can see a log sample for bucket creation, in
+          file `sample_data/storage.buckets.create.log_message.json`. (Or create a bucket and look at the log.)
+        * In that file you see `"methodName": "storage.buckets.create"`.
+
+4. Add permissions in `iris-custom-role.yaml` to be granted to the Iris custom role. 
+   * This role allows Iris, for each resource type, to list, get, and update. 
+     * ("Update" requires permission `setLabels` where available or permission `update`  otherwise.) 
+     * The name of this custom role is `iris3`by default, but may be set by passing env variable `IRIS_CUSTOM_ROLE` in calling `deploy.sh` or `uninstall.sh`
+
+# Testing
+
+## Integration test
+
+`./test_scripts/integration_test.py` creates a Google App Engine app and cloud resources, and tests against them. Run it without parameters for usage instructions.
+
+# Next steps
+
+See `TODO.md` and GitHub issues for potential future improvements.

scripts/_deploy-org.sh

@@ -7,14 +7,14 @@
 #set -x
 # The following lines must come before set -u
 if [[ -z "$IRIS_CUSTOM_ROLE" ]]; then IRIS_CUSTOM_ROLE=iris3; fi
-if [[ -z "$SKIP_ADDING_IAM_BINDINGS" ]] ; then SKIP_ADDING_IAM_BINDINGS=""; fi
+if [[ -z "$SKIP_ADDING_IAM_BINDINGS" ]]; then SKIP_ADDING_IAM_BINDINGS=""; fi
 set -u
 set -e
 
 LOG_SINK=iris_log
 
 # Get organization id for this project
-ORGID=$(gcloud projects get-ancestors $PROJECT_ID --format='value(TYPE,ID)' | awk '/org/ {print $2}')
+ORGID=$(gcloud projects get-ancestors "$PROJECT_ID" --format='value(TYPE,ID)' | awk '/org/ {print $2}')
 
 set +e
 # Create custom role to run iris
@@ -25,19 +25,17 @@ existing_role=$(gcloud iam roles describe --organization "$ORGID" $IRIS_CUSTOM_R
 # 3. For non-existing role, empty-string
 if [ -n "$existing_role" ]; then
   if [[ "$existing_role" == *"True"* ]]; then # It's a soft-deleted role
-    gcloud iam roles undelete -q "$IRIS_CUSTOM_ROLE"  --organization "$ORGID"  >/dev/null
+    gcloud iam roles undelete -q "$IRIS_CUSTOM_ROLE" --organization "$ORGID" >/dev/null
   fi
 
   gcloud iam roles update -q "$IRIS_CUSTOM_ROLE" --organization "$ORGID" --file iris-custom-role.yaml >/dev/null
   role_error=$?
-
 else
-    gcloud iam roles create -q "$IRIS_CUSTOM_ROLE"  --organization "$ORGID" --file iris-custom-role.yaml  >/dev/null
-    role_error=$?
+  gcloud iam roles create -q "$IRIS_CUSTOM_ROLE" --organization "$ORGID" --file iris-custom-role.yaml >/dev/null
+  role_error=$?
 fi
 
 set -e
-
 if [[ "$role_error" != "0" ]]; then
   echo "Error in accessing organization.
    If you just want to redeploy to the same project,
@@ -52,8 +50,7 @@ fi
 gcloud organizations add-iam-policy-binding "$ORGID" \
   --member "serviceAccount:$PROJECT_ID@appspot.gserviceaccount.com" \
   --role "organizations/$ORGID/roles/$IRIS_CUSTOM_ROLE" \
-  --condition=None >/dev/null
-
+  --condition=None >/dev/null 2>&1
 
 if [[ "$LABEL_ON_CREATION_EVENT" != "true" ]]; then
   echo >&2 "Will not label on creation event."
@@ -95,30 +92,25 @@ else
 
   # Create or update a sink at org level
   if ! gcloud logging sinks describe --organization="$ORGID" "$LOG_SINK" >&/dev/null; then
-    #echo >&2 "Creating Log Sink/Router at Organization level."
     gcloud logging sinks create "$LOG_SINK" \
       pubsub.googleapis.com/projects/"$PROJECT_ID"/topics/"$LOGS_TOPIC" \
       --organization="$ORGID" --include-children \
-      --log-filter="${log_filter[*]}" --quiet
+      --log-filter="${log_filter[*]}" --quiet >/dev/null 2>&1
   else
-    #echo >&2 "Updating Log Sink/Router at Organization level."
     gcloud logging sinks update "$LOG_SINK" \
       pubsub.googleapis.com/projects/"$PROJECT_ID"/topics/"$LOGS_TOPIC" \
       --organization="$ORGID" \
-      --log-filter="${log_filter[*]}" --quiet
+      --log-filter="${log_filter[*]}" --quiet >/dev/null 2>&1
   fi
 
   # Extract service account from sink configuration.
   # This is the service account that publishes to PubSub.
-  svcaccount=$(gcloud logging sinks describe --organization="$ORGID" "$LOG_SINK"  |
+  svcaccount=$(gcloud logging sinks describe --organization="$ORGID" "$LOG_SINK" |
     grep writerIdentity | awk '{print $2}')
 
   if [[ "$SKIP_ADDING_IAM_BINDINGS" != "true" ]]; then
-      echo >&2 "Adding IAM bindings in _deploy-org"
     # Assign a publisher role to the extracted service account.
     gcloud projects add-iam-policy-binding "$PROJECT_ID" \
-      --member="$svcaccount" --role=roles/pubsub.publisher --quiet > /dev/null
-  else
-    echo >&2 "Not adding IAM bindings in _deploy-org"fi
+      --member="$svcaccount" --role=roles/pubsub.publisher --quiet >/dev/null 2>&1
   fi
 fi