support folders

Author: JoshuaFox

README.md

@@ -1,7 +1,7 @@
+
 # Iris
 
-In Greek mythology, Iris (/ˈaɪrɪs/; Greek: Ἶρις) is the personification of the rainbow and messenger of the gods. She
-was the handmaiden to Hera.
+In Greek mythology, Iris(Ἶρις) is the personification of the rainbow and messenger of the gods. She was the handmaiden to Hera.
 
 # Blog post
 
@@ -26,9 +26,9 @@ Note that Iris is designed to serve the org. It is not designed around serving a
 Iris does not *add* information, only *copy* values that already exist. For example, it can label a VM instance with its zone, since this information is known; but it cannot add a "business unit" label because it does not know what business
 unit a resource should be attributed to. For that, you should label all resources when creating them, e.g., in your Terraform scripts.
 
-## Iris doesn't by default label all existing resources.
+## Iris **doesn't** label all existing resources in the default setup.
 
-To do it, see section [Labeling existing resources](#Labeling existing resources) below.
+To do this, see section "[Labeling existing resources](#labeling-existing-resources)" below.
 
 # Open source
 Iris is open-source: Feel free to add functionality and add new types of labels. See the `TODO.md` file and Github issues for features and fixes you might do.
@@ -47,8 +47,8 @@ Or you can disable the schedule labeling.
 
 ## Labeling existing resources
 
-* When you first use Iris, you may want to label all existing resources. This is not Iris' preferred flow for adding labels, but you can do it.
-* To do this, deploy it with `label_all_on_cron: True` and wait for the next scheduled run, or manually trigger a run.
+* When you first use Iris, you may want to label all existing resources. This is not Iris' default flow for adding labels, but you can do it.
+* To do this, deploy it with `label_all_on_cron: True` and wait for the next scheduled run, or manually trigger a run through  Cloud Scheduler.
 * Thenּ, you may want to then redeploy Iris with `label_all_on_cron: False` to avoid the  resource consumption of relabeling all resources with the same label every day forever.
 
 # Supported Google Cloud resources
@@ -85,13 +85,13 @@ The part of the function name after `_gcp_` is used for the label key.
 * You can deploy Iris in any project within your Google Cloud organization, but we recommend using a
   [new project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#creating_a_project).
 
-* You need to have certain permissions to run the deployment script to deploy Iris on the org level--the deploy script sets up  roles and log sink. You will need to have these roles on the *organization* where Iris is deployed. You do not need to have these roles if you are doing further re-deployments of new versions of Iris on the project level (where you use the `-p` switch on `deploy.sh`.
+* Here are the required organization-level roles for you, the deployer, to allow the  deploy script to set up  roles and log sink.  You do not need to have these roles if you are doing further re-deployments of new versions of Iris on the project level (where you use the `-p` switch on `deploy.sh`.
     * *Organization Role Administrator*  so the deployment script can create a custom IAM role for Iris that allows to get and set labels.
     * (Note that *Organization Owner* is not enough).
     * *Security Admin* **or** *Organization Administrator* so the deployment script can allow the Iris app engine service account to have the needed permissions.
     * *Logs Configuration Writer* so the deployment script can  create an organization log sink that sends logs to PubSub.
 
-* You need to have certain permissions to run the deployment script to deploy Iris on the project level.
+* Here are the required  project-level roles that you need on the Iris deployment project.
   * One option: You can have *Project Owner* on the project where Iris is deployed
   * Another option: You can have these roles.
       * *Project IAM Admin* to let the deployment script set up the bindings.

TODO.md

@@ -1,10 +1,12 @@
 # Improvements and fixes
 ## Note: see also Github Issues
 
-* Instead of just the -c switch on install.sh, create consistency by adding an -e (for "event-driven") switch to mean "event-driven labeling". Running install.sh with neither -c or -e means the equivalent of having both -c -e.
- In doing this, validate that any installation with only -c or only -e is NOT -p (project-only) or -o (org-only) and fail if so. (i.e., proceed with installation only if neither or both switches are used.)
+* P2 Redo the whole thing with Cloud Asset Inventory including feeds.
+ Cloud Asset Inventory only recognizes assets for listing with a delay; and it is not clear whether the feeds are "real-time" or have a delay.
+* P2 Instead of just the -c switch on install.sh, create consistency by adding an -e (for "event-driven") switch to mean "event-driven labeling". Running install.sh with neither -c or -e should mean the same as having both -c -e.
+ In doing this, validate that any installation with only -c or only -e is NOT -p (project-only) or -o (org-only) and fail if so. (i.e., proceed with installation if and only if neither or both switches are used.)
 
-* P2 Even an empty AppEngine app (not Iris, just a Hello World with 3 lines of code in total) crashes on out-of-memory for the smalled AppEngine instance. Google has confirmed this. See if there is a workaround. This will same money.
+* P2 Memory consumption: Even an empty AppEngine app (not Iris, just a Hello World with 3 lines of code in total) crashes on out-of-memory for the smalled AppEngine instance. Google has confirmed this. See if there is a workaround. This will same money.
 
 * P2 PubSub push endpoint security:
   Note: The token by itself is not very secure, though  
@@ -49,8 +51,6 @@
 
 * P3 Rethink the need for title case in class names. This is clumsy for `Cloudsql`.
 
-* P3 Concurrent execution
-    * Init of multiple plugins is among the biggest slowdowns. They could be initialized concurrently.
 
 * P4 Implement new labels, for example using ideas from
   the [GCP Auto Tag project](https://github.com/doitintl/gcp-auto-tag/)

deploy.sh

@@ -3,7 +3,7 @@
 # See usage (deploy.sh -h).
 # Deploys Iris to Google App Engine, setting up Roles, Sinks, Topics, and Subscriptions as needed.
 
-#set -x
+set -x
 set -u
 set -e
 

gce_base/gce_base.py

@@ -4,7 +4,7 @@
 import proto
 
 from plugin import Plugin
-from util.gcp_utils import (
+from util.gcp.gcp_utils import (
     cloudclient_pb_obj_to_dict,
     cloudclient_pb_objects_to_list_of_dicts,
 )

gce_base/gce_zonal_base.py

@@ -6,8 +6,8 @@
 from typing import Dict, Optional
 
 from gce_base.gce_base import GceBase
-from util import gcp_utils
-from util.gcp_utils import add_loaded_lib
+from util.gcp import gcp_utils
+from util.gcp.gcp_utils import add_loaded_lib
 from util.utils import timing
 
 

main.py

@@ -2,16 +2,15 @@
 
 import sys
 
-
 from flask import Response
 
 print("Initializing ", file=sys.stderr)
 import flask
-from util.gcp_utils import (
+from util.gcp.gcp_utils import (
     increment_invocation_count,
     count_invocations_by_path,
 )
-from util.detect_gae import detect_gae
+from util.gcp.detect_gae import detect_gae
 
 app = flask.Flask(__name__)
 if detect_gae():
@@ -37,8 +36,9 @@
 import os
 
 from plugin import Plugin, PluginHolder
-from util import pubsub_utils, gcp_utils, utils, config_utils
-from util.gcp_utils import (
+from util import pubsub_utils, utils, config_utils
+from util.gcp import gcp_utils
+from util.gcp.gcp_utils import (
     detect_gae,
     is_appscript_project,
     all_projects,
@@ -51,7 +51,7 @@
     is_project_enabled,
     pubsub_token,
     is_in_test_or_dev_project,
-    is_test_or_dev_configuration,
+    is_test_configuration,
     iris_homepage_text,
 )
 from util.utils import log_time, timing
@@ -73,11 +73,10 @@
 
 @app.route("/")
 def index():
-
     increment_invocation_count("index")
     with gae_memory_logging("index"):
         msg = iris_homepage_text()
-        if config_utils.is_test_or_dev_configuration():
+        if config_utils.is_test_configuration():
             msg += "\nI'm running in test or dev mode."
 
         logging.info(
@@ -88,7 +87,6 @@ def index():
 
 @app.route("/_ah/warmup")
 def warmup():
-
     increment_invocation_count("warmup")
     with gae_memory_logging("warmup"):
         logging.info("warmup() called")
@@ -129,32 +127,41 @@ def __get_enabled_projects():
         enabled_projs = configured_as_enabled
     else:
         all_proj = all_projects()
-        # In my testing, we do NOT get appscript projects in the list.
-        # There is a small chance that with other permissions, these appscript projects would appear.
-        # so here we filter them out.
 
+        # In some cases, lots of appscript projects would appear.
+        # so here we filter them out. However, In my testing, we do NOT get appscript projects in the list.
         nonappscript_projects = (p for p in all_proj if not is_appscript_project(p))
-
-        enabled_only = (
-            p for p in nonappscript_projects if config_utils.is_project_enabled(p)
-        )
+        # The following filtering is probably useless since the else-clause that
+        # that we are in is reached only if there are no explicitly enabled projects.
+        enabled_only = (p for p in nonappscript_projects if config_utils.is_project_enabled(p))
         enabled_projs = list(enabled_only)
     enabled_projs.sort()
     if not enabled_projs:
         raise Exception("No projects enabled at all")
-
+    explain = []
     if (  # These are three indications that we are running in dev/test
-        not detect_gae()
-        or is_test_or_dev_configuration()
-        or is_in_test_or_dev_project(current_project_id())
+            not detect_gae()
+            or is_test_configuration()
+            or is_in_test_or_dev_project(current_project_id())
     ):
+        if not detect_gae():
+            explain.append("Not running in App Engine")
+        if is_test_configuration():
+            explain.append("Using a test configuration file " + config_utils.config_test_file())
+        if is_in_test_or_dev_project(current_project_id()):
+            explain.append("Running a project " + current_project_id() +
+                           " which has one of the markers of a test project" +
+                           " configured under key test_or_dev_project_markers")
         max_proj_in_dev = 3
         if len(enabled_projs) > max_proj_in_dev:
+            assert explain, "If we got here, one of the conditioned should have been true"
             raise Exception(
-                """In development or testing, we support no more than {max_proj_in_dev} projects, \
+                f"""When running Iris in development/testing mode, \
+                we support no more than {max_proj_in_dev} projects, \
                 to avoid accidentally flooding the system.
                 {len(enabled_projs)} projects are available, which exceeds that. 
-                See README.md for explanation.
+                We are in development/testing mode because:
+                {"; ".join(explain)}
                 """
             )
     return enabled_projs
@@ -165,9 +172,9 @@ def __send_pubsub_per_projectplugin(configured_projects):
     for project_id in configured_projects:
         for plugin_cls in PluginHolder.plugins:
             if (
-                not plugin_cls.is_labeled_on_creation()
-                or plugin_cls.relabel_on_cron()
-                or config_utils.label_all_on_cron()
+                    not plugin_cls.is_labeled_on_creation()
+                    or plugin_cls.relabel_on_cron()
+                    or config_utils.label_all_on_cron()
             ):
                 pubsub_utils.publish(
                     msg=json.dumps(
@@ -191,7 +198,6 @@ def __send_pubsub_per_projectplugin(configured_projects):
 
 @app.route("/label_one", methods=["POST"])
 def label_one():
-
     increment_invocation_count("label_one")
     with gae_memory_logging("label_one"):
 

plugin.py

@@ -9,7 +9,8 @@
 from googleapiclient import discovery
 from googleapiclient import errors
 
-from util import gcp_utils, config_utils
+from util import config_utils
+from util.gcp import gcp_utils
 from util.config_utils import (
     is_copying_labels_from_project,
     iris_prefix,
@@ -222,6 +223,9 @@ def plugin_cls_by_name(cls, name) -> Type[Plugin]:
     @classmethod
     @log_time
     def init(cls):
+        """Loads the classes, but does not create instances, which happens
+        in get_plugin_instance"""
+
         def load_plugin_class(name) -> Type:
             module_name = PLUGINS_MODULE + "." + name
             __import__(module_name)
@@ -231,6 +235,7 @@ def load_plugin_class(name) -> Type:
         loaded = []
         for _, module, _ in pkgutil.iter_modules([PLUGINS_MODULE]):
             if config_utils.is_plugin_enabled(module):
+
                 plugin_class = load_plugin_class(module)
                 cls.plugins[
                     plugin_class
@@ -241,6 +246,7 @@ def load_plugin_class(name) -> Type:
 
     @classmethod
     def get_plugin_instance(cls, plugin_cls):
+        """Lazy-initialize  the instance. The classes are loaded in init()"""
         with cls.__lock:
             assert plugin_cls in cls.plugins, plugin_cls + " " + cls.plugins
             plugin_instance = cls.plugins[plugin_cls]

plugins/bigquery.py

@@ -9,8 +9,8 @@
 from ratelimit import limits, sleep_and_retry
 
 from plugin import Plugin
-from util import gcp_utils
-from util.gcp_utils import add_loaded_lib
+from util.gcp import gcp_utils
+from util.gcp.gcp_utils import add_loaded_lib
 from util.utils import log_time, timing, dict_to_camelcase
 
 

plugins/buckets.py

@@ -2,8 +2,8 @@
 from functools import lru_cache
 
 from plugin import Plugin
-from util import gcp_utils
-from util.gcp_utils import add_loaded_lib
+from util.gcp import gcp_utils
+from util.gcp.gcp_utils import add_loaded_lib
 from util.utils import log_time, timing, dict_to_camelcase
 
 

plugins/disks.py

@@ -6,8 +6,8 @@
 from googleapiclient import errors
 
 from gce_base.gce_zonal_base import GceZonalBase
-from util import gcp_utils
-from util.gcp_utils import add_loaded_lib
+from util.gcp import gcp_utils
+from util.gcp.gcp_utils import add_loaded_lib
 from util.utils import log_time
 
 

plugins/instances.py

@@ -6,8 +6,8 @@
 from googleapiclient import errors
 
 from gce_base.gce_zonal_base import GceZonalBase
-from util import gcp_utils
-from util.gcp_utils import add_loaded_lib
+from util.gcp import gcp_utils
+from util.gcp.gcp_utils import add_loaded_lib
 from util.utils import log_time
 
 

plugins/snapshots.py

@@ -4,8 +4,8 @@
 from googleapiclient import errors
 
 from gce_base.gce_base import GceBase
-from util import gcp_utils
-from util.gcp_utils import add_loaded_lib
+from util.gcp import gcp_utils
+from util.gcp.gcp_utils import add_loaded_lib
 from util.utils import log_time, timing
 
 

plugins/subscriptions.py

@@ -5,7 +5,7 @@
 from googleapiclient import errors
 
 from plugin import Plugin
-from util.gcp_utils import (
+from util.gcp.gcp_utils import (
     cloudclient_pb_obj_to_dict,
     cloudclient_pb_objects_to_list_of_dicts,
     add_loaded_lib,

plugins/topics.py

@@ -5,7 +5,7 @@
 from googleapiclient import errors
 
 from plugin import Plugin
-from util.gcp_utils import (
+from util.gcp.gcp_utils import (
     cloudclient_pb_obj_to_dict,
     cloudclient_pb_objects_to_list_of_dicts,
     add_loaded_lib,

scripts/_deploy-project.sh

@@ -28,13 +28,14 @@ if [[ ! -f "config-test.yaml" ]]  && [[ ! -f "config.yaml" ]]; then
 fi
 
 ABBREV__=$(gcloud app describe --project $PROJECT_ID | grep defaultHostname |egrep -o "\.[a-z][a-z]\.")
-GAE_REGION_ABBREV=${ABBREV__:1:2}
+GAE_MULTIREGION_ABBREV=${ABBREV__:1:2} #  Something like uc (us-central) or sa (southeast-asia)
 
 GAE_SVC=$(grep "service:" app.yaml | awk '{print $2}')
+
 # The following line depends on the  the export PYTHON_PATH="." above.
 PUBSUB_VERIFICATION_TOKEN=$(python3 ./util/print_pubsub_token.py)
-LABEL_ONE_SUBSCRIPTION_ENDPOINT="https://${GAE_SVC}-dot-${PROJECT_ID}.${GAE_REGION_ABBREV}.r.appspot.com/label_one?token=${PUBSUB_VERIFICATION_TOKEN}"
-DO_LABEL_SUBSCRIPTION_ENDPOINT="https://${GAE_SVC}-dot-${PROJECT_ID}.${GAE_REGION_ABBREV}.r.appspot.com/do_label?token=${PUBSUB_VERIFICATION_TOKEN}"
+LABEL_ONE_SUBSCRIPTION_ENDPOINT="https://${GAE_SVC}-dot-${PROJECT_ID}.${GAE_MULTIREGION_ABBREV}.r.appspot.com/label_one?token=${PUBSUB_VERIFICATION_TOKEN}"
+DO_LABEL_SUBSCRIPTION_ENDPOINT="https://${GAE_SVC}-dot-${PROJECT_ID}.${GAE_MULTIREGION_ABBREV}.r.appspot.com/do_label?token=${PUBSUB_VERIFICATION_TOKEN}"
 
 declare -A enabled_services
 while read -r svc _; do