Allow labeling all existing resources; useful on first deployment

Author: JoshuaFox

README.md

@@ -25,9 +25,9 @@ Note that Iris is designed to serve the organization. It is not designed around
 
 Iris does not *add* information, only *copy* values that already exist. For example, it can label a VM instance with its zone; but it cannot add a "business unit" label because it does not know a resource's business unit. For that, you should label all resources when creating them, e.g., in your Terraform scripts. (Indeed, iris can be made extraneous in this way.)
 
-## Existing resources are not all labeled (by default)
+## Labeling resources that existing resources when you deploy Iris
 
-If you want to label lots of virtual machines,PubSub topics etc. that *already exist* when you install Iris, see section "[Labeling existing resources](#labeling-existing-resources)" below.
+If you want to label lots of virtual machines,PubSub topics etc. that *already exist* when you deploy Iris, see section "[Labeling existing resources](#labeling-existing-resources)" below.
 
 # Open source
 
@@ -51,8 +51,7 @@ You can also  disable the scheduled labeling. See Deployment below or run `./dep
 ## Labeling existing resources
 
 * When you first use Iris, you may want to label all existing resources. Iris does not do this by default.
-* To do this, deploy Iris with `label_all_on_cron: True` and wait for the next scheduled run, or manually trigger a run   through Cloud Scheduler.
-* Thenּ, you may want to then **redeploy** Iris with `label_all_on_cron: False`, to avoid the resource consumption of  relabeling all resources with the same label every day forever.
+* To do this, publish a PubSub message (the content doesn't matter) to `iris_label_all_topic`, for example with `gcloud pubsub topics publish iris_label_all_topic --message=labelall --project $PROJECT_ID` (substituting the project ID where Iris is deployed.) Of course, you will need to have permissions to publish that message.
 
 # Supported Google Cloud resources
 

TODO.md

@@ -4,8 +4,6 @@
 
 * P2 Memory consumption: Even an empty AppEngine app (not Iris, just a Hello World with 3 lines of code in total) crashes on out-of-memory for the smalled AppEngine instance. Google has confirmed this. See if there is a workaround.  This will save money.
 
- 
-
 * P3 Label immediately after an event in certain cases, as opposed to using a daily cron as is now done.
     * Cloud SQL Instances
         * See above re Cloud Tasks

main.py

@@ -28,7 +28,7 @@
 
 from functools import lru_cache
 
-from typing import Dict, Type
+from typing import Dict, Type, List
 
 import time
 
@@ -91,34 +91,53 @@ def warmup():
     return "", 200, {}
 
 
+@app.route("/label_all", methods=["POST"])
+@log_time
+def label_all():
+    with gae_memory_logging("label_all"):
+        logging.info("Start label_all() invocation")
+        increment_invocation_count("label_all")
+        __check_pubsub_jwt()
+        return __label_multiple_types(True)
+
+
 @app.route("/schedule", methods=["GET"])
 @log_time
 def schedule():
-    """
-    Send out a message per-plugin per-project to label all objects of that type and project.
-    """
-    # Not validated with JWT because validated with cron header (see below)
-    increment_invocation_count("schedule")
     with gae_memory_logging("schedule"):
-        try:
-            logging.info("Start schedule() invocation")
+        logging.info("Start schedule() invocation")
+        increment_invocation_count("schedule")
+        # Not validated with JWT because validated with cron header
+        is_cron = flask.request.headers.get("X-Appengine-Cron")
+        if not is_cron:
+            return "Access Denied: No Cron header found", 403
+        label_all_types = config_utils.label_all_on_cron()
+        return __label_multiple_types(label_all_types)
 
-            is_cron = flask.request.headers.get("X-Appengine-Cron")
-            if not is_cron:
-                return "Access Denied: No Cron header found", 403
 
-            enabled_projects = __get_enabled_projects()
-            __send_pubsub_per_projectplugin(enabled_projects)
-            # All errors are actually caught before this point,
-            # since most errors are unrecoverable.
-            return "OK", 200
-        except Exception:
-            logging.exception("In schedule()")
-            return "Error", 500
+def __label_multiple_types(label_all_types: bool):
+    """
+    Send out a message per-plugin per-project; each msg labels objects of a given type and project.
+    :param label_all_types: if false, only label those that are
+    not labeled on creation (CloudSQL) or those that must be relabeled on cron (Disks),
+    but if true, label all types;
+    """
+    assert label_all_types is not None
+    try:
+        enabled_projects = __get_enabled_projects()
+        __send_pubsub_per_projectplugin(
+            enabled_projects, label_all_types=label_all_types
+        )
+        # All errors are actually caught, logged and ignored *before* this point,
+        # since most errors are unrecoverable.
+        return "OK", 200
+    except Exception:
+        logging.exception("In schedule()")
+        return "Error", 500
 
 
 @lru_cache(maxsize=1)
-def __get_enabled_projects():
+def __get_enabled_projects() -> List:
     configured_as_enabled = config_utils.enabled_projects()
     if configured_as_enabled:
         enabled_projs = configured_as_enabled
@@ -142,14 +161,15 @@ def __get_enabled_projects():
     return enabled_projs
 
 
-def __send_pubsub_per_projectplugin(configured_projects):
+def __send_pubsub_per_projectplugin(configured_projects: List, label_all_types: bool):
     msg_count = 0
     for project_id in configured_projects:
         for plugin_cls in PluginHolder.plugins:
+
             if (
                 not plugin_cls.is_labeled_on_creation()
                 or plugin_cls.relabel_on_cron()
-                or config_utils.label_all_on_cron()
+                or label_all_types
             ):
                 pubsub_utils.publish(
                     msg=json.dumps(
@@ -205,6 +225,11 @@ def label_one():
                 for supported_method in method_names:
                     if supported_method.lower() in method_from_log.lower():
                         if plugin_cls.is_labeled_on_creation():
+                            logging.info(
+                                "plugin_cls %s, with method %s",
+                                plugin_cls.__name__,
+                                method_from_log,
+                            )
                             __label_one_0(data, plugin_cls)
 
                         plugins_found.append(
@@ -254,7 +279,7 @@ def __check_pubsub_jwt():
             logging.error(f"Email verified was {is_email_verif}")
             return False
 
-        logging.info("Claims: " + str(claim))
+        # logging.info("Claims: " + str(claim))
         if (
             email := claim.get("email")
         ) != f"iris-msg-sender@{current_project_id()}.iam.gserviceaccount.com":
@@ -263,7 +288,7 @@ def __check_pubsub_jwt():
     except Exception as e:
         logging.exception(f"Invalid JWT token: {e}")
         return False
-    logging.info("JWT Passed")
+    # logging.info("JWT Passed")
 
     return True
 

plugin.py

@@ -26,9 +26,9 @@
 PLUGINS_MODULE = "plugins"
 
 
-# TODO Since subclasses are already singletons, and we are already using
-# a lot of classmethods and staticmethods, , could convert this to
-# never use instance methods
+# Since subclasses are already singletons, and we are already using
+# a lot of classmethods and staticmethods, could convert this code to
+# never use instance methods, maybe  only staticmethods
 class Plugin(metaclass=ABCMeta):
     # Underlying API  max is 1000; avoid off-by-one errors
     # We send a batch when _BATCH_SIZE or more tasks are in it, or at the end of a label_all
@@ -244,12 +244,12 @@ def load_plugin_class(name) -> Type:
         assert cls.plugins, "No plugins defined"
 
     @classmethod
-    def get_plugin_instance(cls, plugin_cls):
+    def get_plugin_instance(cls, plugin_cls: Type[Plugin]):
         """Lazy-initialize  the instance. The classes are loaded in init()"""
         with cls.__lock:
-            assert plugin_cls in cls.plugins, plugin_cls + " " + cls.plugins
-            plugin_instance = cls.plugins[plugin_cls]
-
+            plugin_instance: Plugin = cls.plugins[plugin_cls]
+            # Note: We initialized all keys in cls.plugins
+            # with None values.
             assert not plugin_instance or isinstance(
                 plugin_instance, (Plugin, plugin_cls)
             )

scripts/_deploy-project.sh

@@ -7,15 +7,18 @@
 # - Pass the project as the first command line argument.
 #
 
+
 #set -x
 set -u
 set -e
 
 SCHEDULELABELING_TOPIC=iris_schedulelabeling_topic
+LABEL_ALL_TOPIC=iris_label_all_topic
 DEADLETTER_TOPIC=iris_deadletter_topic
 DEADLETTER_SUB=iris_deadletter
 DO_LABEL_SUBSCRIPTION=do_label
 LABEL_ONE_SUBSCRIPTION=label_one
+LABEL_ALL_SUBSCRIPTION=label_all
 
 ACK_DEADLINE=60
 MAX_DELIVERY_ATTEMPTS=10
@@ -28,11 +31,12 @@ if [[ ! -f "config-test.yaml" ]]  && [[ ! -f "config.yaml" ]]; then
        exit 1
 fi
 
+#Next line duplicate of our Python func gae_url_with_multiregion_abbrev
 appengineHostname=$(gcloud app describe --project $PROJECT_ID | grep defaultHostname |cut -d":" -f2 | awk '{$1=$1};1' )
 if [[ -z "$appengineHostname" ]]; then
    echo >&2 "App Engine is not enabled in $PROJECT_ID.
-   To do this, please deploy a simple \"Hello World\" default service to enable App Engine.
-   In doing so, select the App Engine region that you prefer. It is immutable."
+   To do this, please enable it with \"gcloud app create [--region=REGION]\",
+   and then deploy a simple \"Hello World\" default service to enable App Engine."
 
    exit 1
 fi
@@ -41,7 +45,8 @@ gae_svc=$(grep "service:" app.yaml | awk '{print $2}')
 
 
 LABEL_ONE_SUBSCRIPTION_ENDPOINT="https://${gae_svc}-dot-${appengineHostname}/label_one"
-DO_LABEL_SUBSCRIPTION_ENDPOINT="https://${gae_svc}-dot-${appengineHostname}/do_label?token"
+DO_LABEL_SUBSCRIPTION_ENDPOINT="https://${gae_svc}-dot-${appengineHostname}/do_label"
+LABEL_ALL_SUBSCRIPTION_ENDPOINT="https://${gae_svc}-dot-${appengineHostname}/label_all"
 
 declare -A enabled_services
 while read -r svc _; do
@@ -59,6 +64,7 @@ required_svcs=(
   storage-component.googleapis.com
   sql-component.googleapis.com
   sqladmin.googleapis.com
+  bigquery.googleapis.com
 )
 for svc in "${required_svcs[@]}"; do
   if ! [ ${enabled_services["$svc"]+_} ]; then
@@ -100,6 +106,9 @@ fi
 
 project_number=$(gcloud projects describe $PROJECT_ID --format json|jq -r '.projectNumber')
 PUBSUB_SERVICE_ACCOUNT="service-${project_number}@gcp-sa-pubsub.iam.gserviceaccount.com"
+# The following line is only needed on first deployment, and so slows things
+# down unnecessarily otherwise. THe same is true for enabling the services, above.
+gcloud beta services identity create --project $PROJECT_ID --service pubsub
 
 msg_sender_sa_name=iris-msg-sender
 
@@ -113,30 +122,18 @@ set -e
 
 MSGSENDER_SERVICE_ACCOUNT=${msg_sender_sa_name}@${PROJECT_ID}.iam.gserviceaccount.com
 
-gcloud projects add-iam-policy-binding ${PROJECT_ID} \
- --member="serviceAccount:${PUBSUB_SERVICE_ACCOUNT}"\
- --role='roles/iam.serviceAccountTokenCreator'
 
-set +e
+
+
 # Allow Pubsub to publish into the deadletter topic
-BINDING_ERR_OUTPUT=$(gcloud pubsub topics add-iam-policy-binding $DEADLETTER_TOPIC \
+gcloud pubsub topics add-iam-policy-binding $DEADLETTER_TOPIC \
         --member="serviceAccount:$PUBSUB_SERVICE_ACCOUNT"\
-         --role="roles/pubsub.publisher" --project $PROJECT_ID 2>&1 )
-if [[ $? -ne 0  && $BINDING_ERR_OUTPUT == *"gcp-sa-pubsub.iam.gserviceaccount.com does not exist."* ]]; then
-  # Sometimes the PubSub svc account ([email protected])
-  # does not exist on new projects. I don't know why.
-  # Note that this svc account is NOT in the user project;
-  # it can be thought of as existing in a hidden Google-controlled project.
-  gcloud beta  services identity create --project $PROJECT_ID --service pubsub
+         --role="roles/pubsub.publisher" --project $PROJECT_ID 2>&1
 
-  set -e
-  #Redo the above binding command
-  gcloud pubsub topics add-iam-policy-binding $DEADLETTER_TOPIC \
-          --member="serviceAccount:$PUBSUB_SERVICE_ACCOUNT"\
-           --role="roles/pubsub.publisher" --project $PROJECT_ID >/dev/null
 
-fi
-set -e
+gcloud projects add-iam-policy-binding ${PROJECT_ID} \
+ --member="serviceAccount:${PUBSUB_SERVICE_ACCOUNT}"\
+ --role='roles/iam.serviceAccountTokenCreator'
 
 # Create PubSub subscription receiving commands from the /schedule handler that is triggered from cron
 # If the subscription exists, it will not be changed.
@@ -220,7 +217,41 @@ else
       --role="roles/pubsub.subscriber" --project $PROJECT_ID >/dev/null
 fi
 
+gcloud pubsub topics describe "$LABEL_ALL_TOPIC" --project="$PROJECT_ID" &>/dev/null ||
+    gcloud pubsub topics create $LABEL_ALL_TOPIC --project="$PROJECT_ID" --quiet >/dev/null
+
+set +e
+gcloud pubsub subscriptions describe "$LABEL_ALL_SUBSCRIPTION" --project="$PROJECT_ID" &>/dev/null
+label_all_subsc_exists=$?
+set -e
+
+if [[ $label_all_subsc_exists -eq 0 ]]; then
+    gcloud pubsub subscriptions update "$LABEL_ALL_SUBSCRIPTION" \
+    --project="$PROJECT_ID" \
+    --push-endpoint "$LABEL_ALL_SUBSCRIPTION_ENDPOINT" \
+    --push-auth-service-account $MSGSENDER_SERVICE_ACCOUNT  \
+    --ack-deadline=$ACK_DEADLINE \
+    --max-delivery-attempts=$MAX_DELIVERY_ATTEMPTS \
+    --dead-letter-topic=$DEADLETTER_TOPIC \
+    --min-retry-delay=$MIN_RETRY \
+    --max-retry-delay=$MAX_RETRY \
+    --quiet >/dev/null
+else
+  gcloud pubsub subscriptions create "$LABEL_ALL_SUBSCRIPTION" \
+    --topic "$LABEL_ALL_TOPIC" --project="$PROJECT_ID" \
+    --push-endpoint "$LABEL_ALL_SUBSCRIPTION_ENDPOINT" \
+    --push-auth-service-account $MSGSENDER_SERVICE_ACCOUNT  \
+    --ack-deadline=$ACK_DEADLINE \
+    --max-delivery-attempts=$MAX_DELIVERY_ATTEMPTS \
+    --dead-letter-topic=$DEADLETTER_TOPIC \
+    --min-retry-delay=$MIN_RETRY \
+    --max-retry-delay=$MAX_RETRY \
+    --quiet >/dev/null
+fi
 
+gcloud pubsub subscriptions add-iam-policy-binding $LABEL_ALL_SUBSCRIPTION \
+    --member="serviceAccount:$PUBSUB_SERVICE_ACCOUNT" \
+    --role="roles/pubsub.subscriber" --project $PROJECT_ID >/dev/null
 
 if [[ "$LABEL_ON_CRON" == "true" ]]; then
     cp cron_full.yaml cron.yaml
@@ -229,6 +260,6 @@ else
    cp cron_empty.yaml cron.yaml
 fi
 
-gcloud app deploy --project $PROJECT_ID --quiet app.yaml cron.yaml
+gcloud app deploy --project "$PROJECT_ID" --quiet app.yaml cron.yaml
 
 rm cron.yaml # In this script, cron.yaml is a temp file, a copy of  cron_full.yaml or cron_empty.yaml