Can now tag a GCE instance

Author: JoshuaFox

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2018 DoiT International
+Copyright (c) 2021 DoiT International
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -6,105 +6,51 @@ In Greek mythology, Iris (/ˈaɪrɪs/; Greek: Ἶρις) is the personification
 
 Iris helps to automatically assign labels to Google Cloud resources for better manageability and billing reporting. Each resource in Google Cloud will get an automatically generated label in a form of [iris_name:name], [iris_region:region] and finally [iris_zone:zone]. For example if you have a Google Compute Engine instance named `nginx`, Iris will automatically label this instance with [iris_name:nginx], [iris_region:us-central1] and [iris_zone:us-central1-a].
 
-Iris will also label short lived Google Compute Engine instances such as preemtible instances or instances managed by Instance Group Manager by listening to Stackdriver Logs and putting required labels on-demand.
+Iris will also label short-lived Google Compute Engine instances such as preemptible instances or instances managed by an Instance Group Manager by listening to Operations (Stackdriver) Logs and adding required labels on-demand. 
 
-**NOTE**: Iris will try tagging resources in _all_ project across your GCP organization. Not just the project it will be deployed into.
+**Supported Google Cloud Products**
 
-## Supported Google Cloud Products
-
-Iris is extensible through plugins and new Google Cloud products may be supported via simply adding a plugin. Right now, there are plugins for the following products:
+Iris is extensible through plugins. New Google Cloud products may be supported via simply adding a plugin. Right now, there are plugins for the following products:
 
 * Google Compute Engine (including disks and snapshots)
 * Google Cloud Storage
+* Google CloudSQL
 * Google BigQuery
 * Google Bigtable
 
-## Installation
-
-We recommend to deploy Iris in a [separate](https://cloud.google.com/resource-manager/docs/creating-managing-projects#creating_a_project) project within your Google Cloud organization.
-To deploy, you will need to have *Owner* role on Iris project and the following roles in your *GCP Organization*:
-
- * _Organization Role Administrator_ - to create a custom IAM role for Iris that allows setting labels on the services
-   (note this is different from _Organization Administrator_, which is in turn not related to Organization-level _Owner_)
- * _Security Admin_ OR _Organization Administrator_ - to allow Iris app engine service account to use the above role
- * _Logs Configuration Writer_ OR _Logs Configuration Writer_ - to configure log events stream on Organization level to watch for new instances, databases, etc.
+**Installation**
 
-### Install dependencies
+We recommend deploying Iris in a [new project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#creating_a_project) within your Google Cloud organization. You will need the following IAM permissions on your Google Cloud organization to complete the deployment: 
 
-```
-pip2.7 install -r requirements.txt -t lib
-```
+ * App Engine Admin
+ * Logs Configuration Writer
+ * Pub/Sub Admin
 
-Yes, we still use Python2.7. Yes, [we know](https://pythonclock.org/).
+##### Install dependencies
 
-#### Deploy
+`pip install -r requirements.txt -t lib`
 
-```
-./deploy.sh <project-id>
-```
+##### Deploy
+`./deploy.sh project-id`
 
-#### Configuration
+##### Configuration
 
-Configuration is stored in the config.json file. The file contains two arrays.
+Configuration is stored in the `config.json` file. The file contains two arrays.
 
-1. tags - A list of tags that will be applied to the resources (if the corresponding plugin implemented a function `_get_<TAGNAME>()`)
-2. on_demand - A List of plugins that will tag whenever a new object of their type is created
+1. `tags` - A list of tags that will be applied to the resources (if the plugin impliments a function by the name _get_TAGNAME)
+2. `on_demand` - A list of plugins that will tag whenever there is a new object of their type (as opposed to tagging as part of a batch command). 
+Note: There is no support for CloudSQL for now)
 
-```json
-{
-  "tags": [
-    "name",
-    "zone",
-    "region",
-    "location",
-    "instance_type"
-  ],
-  "on_demand": [
-    "Gce",
-    "BigQuery",
-    "Gcs",
-    "BigTable",
-    "GceDisks",
-    "GceSnapshots"
-  ]
-}
-```
 
 ### Local Development
 For local development run:
 
  `dev_appserver.py --log_level=debug app.yaml`
 
-Iris is easily extendable to support tagging of other GCP services. You will need to create a Python file in the /plugin directory with `register_signals`,   `def api_name`  and `methodsNames` functions as following:
-
-```python
-     def register_signals(self):
-
-        """
-          Register with the plugin manager.
-        """
-
-        logging.debug("BigQuery class created and registering signals")
-```
-
-```python
- def api_name(self):
-        return "compute.googleapis.com"
-```
-
-```python
-	// a list of log methods to listen on
-    def methodsNames(self):
-        return ["storage.buckets.create"]
-```
-
-All plugins are derived form `Plugin` class and needs to implement the following functions:
-
-1. `do_tag(self, project_id)`
-1. `get_gcp_object(self, data)`
-1. `tag_one(self, gcp_object, project_id)`
-1. `api_name(self)`
-1. `methodsNames(self)`
+## Extension
+Iris is easily extendable to support tagging of other GCP services. You will need to create a Python file in the `/plugin` directory, implementing `register_signals`,   `def api_name`  and `methodsNames` functions as following:
 
+All plugins are derived from `Plugin` class and need to implement the following functions:
 
-Each plugin will execute `gen_labels()` which will loop over all the tags that are defined in the config file and will execute `_get_<TAGNAME>()` function
+ 
+Each plugin will execute will loop over all the tags that are defined in the config file and will execute the `_get_TAGNAME` function

plugins/bigquery.py alt_plugins/bigquery.py

@@ -1,13 +1,13 @@
 """ Taging BQ tabels and datasets."""
 import logging
 import traceback
+import uuid
 
 from google.auth import app_engine
 from googleapiclient import discovery, errors
 from ratelimit import limits, sleep_and_retry
 
 from pluginbase import Plugin
-from utils import utils
 
 SCOPES = ['https://www.googleapis.com/auth/bigquery']
 
@@ -35,7 +35,7 @@ def api_name(self):
         return "bigquery-json.googleapis.com"
 
 
-    def methodsNames(self):
+    def method_names(self):
         return ["datasetservice.insert", "tableservice.insert"]
 
 
@@ -114,12 +114,12 @@ def get_gcp_object(self, data):
             return table
         except Exception as e:
             logging.error(e)
-            print(traceback.format_exc())
+            print((traceback.format_exc()))
 
         return None
 
 
-    def do_tag(self, project_id):
+    def do_label(self, project_id):
         """
         tag tables and data sets
         :param project_id: project id
@@ -162,7 +162,7 @@ def do_tag(self, project_id):
     @limits(calls=35, period=60)
     def tag_one_dataset(self, gcp_object):
         labels = dict()
-        labels['labels'] = self.gen_labels(gcp_object)
+        labels['labels'] = self._gen_labels(gcp_object)
         try:
             self.bigquery.datasets().patch(
                 projectId=gcp_object['datasetReference']['projectId'],
@@ -177,14 +177,15 @@ def tag_one_dataset(self, gcp_object):
     @limits(calls=35, period=60)
     def tag_one_table(self, gcp_object):
         labels = dict()
-        labels['labels'] = self.gen_labels(gcp_object)
+        labels['labels'] = self._gen_labels(gcp_object)
         try:
+
             self.batch.add(self.bigquery.tables().patch(
                 projectId=gcp_object['tableReference']['projectId'],
                 body=labels,
                 datasetId=gcp_object['tableReference']['datasetId'],
                 tableId=gcp_object['tableReference'][
-                    'tableId']), request_id=utils.get_uuid())
+                    'tableId']), request_id=uuid.uuid4())
             self.counter = self.counter + 1
             if self.counter == 1000:
                 self.do_batch()
@@ -193,7 +194,7 @@ def tag_one_table(self, gcp_object):
         if self.counter > 0:
             self.do_batch()
 
-    def tag_one(self, gcp_object, project_id):
+    def label_one(self, gcp_object, project_id):
         try:
             if gcp_object['kind'] == "bigquery#dataset":
                 self.tag_one_dataset(gcp_object)

plugins/bigtable.py alt_plugins/bigtable.py

@@ -1,10 +1,13 @@
 import logging
+import uuid
 
 from google.auth import app_engine
 from googleapiclient import discovery, errors
 
+import utils.gcp_utils
+import utils.gcp_utils
 from pluginbase import Plugin
-from utils import gcp, utils
+from utils import gcp
 
 SCOPES = ['https://www.googleapis.com/auth/bigtable.admin']
 
@@ -32,7 +35,7 @@ def api_name(self):
         return "bigtableadmin.googleapis.com"
 
 
-    def methodsNames(self):
+    def method_names(self):
         return [
             "google.bigtable.admin.v2.BigtableInstanceAdmin.CreateInstance"]
 
@@ -61,7 +64,7 @@ def _get_zone(self, gcp_object):
     def _get_region(self, gcp_object):
         try:
             zone = self.get_location(gcp_object, gcp_object['project_id'])
-            region = gcp.region_from_zone(zone).lower()
+            region = utils.gcp_utils.region_from_zone(zone).lower()
         except KeyError as e:
             logging.error(e)
             return None
@@ -108,7 +111,7 @@ def get_gcp_object(self, data):
             return None
 
 
-    def do_tag(self, project_id):
+    def do_label(self, project_id):
         page_token = None
         more_results = True
         while more_results:
@@ -121,7 +124,7 @@ def do_tag(self, project_id):
                 return
             if 'instances' in result:
                 for inst in result['instances']:
-                    self.tag_one(inst, project_id)
+                    self.label_one(inst, project_id)
             if 'nextPageToken' in result:
                 page_token = result['nextPageToken']
             else:
@@ -130,24 +133,25 @@ def do_tag(self, project_id):
                 self.do_batch()
 
 
-    def tag_one(self, gcp_object, project_id):
+    def label_one(self, gcp_object, project_id):
         labels = dict()
         gcp_object['project_id'] = project_id
-        labels['labels'] = self.gen_labels(gcp_object)
+        labels['labels'] = self._gen_labels(gcp_object)
         gcp_object.pop('project_id', None)
         if 'labels' in gcp_object:
-            for key, val in labels['labels'].items():
+            for key, val in list(labels['labels'].items()):
                 gcp_object['labels'][key] = val
         else:
             gcp_object['labels'] = {}
-            for key, val in labels['labels'].items():
+            for key, val in list(labels['labels'].items()):
                 gcp_object['labels'][key] = val
 
         try:
-            self.batch.add(self.bigtable.projects().instances(
-            ).partialUpdateInstance(
-                name=gcp_object['name'], body=gcp_object,
-                updateMask='labels'), request_id=utils.get_uuid())
+
+            self.batch.add(
+                self.bigtable.projects().instances().partialUpdateInstance(
+                 name=gcp_object['name'], body=gcp_object,
+                 updateMask='labels'), request_id= uuid.uuid4())
             self.counter = self.counter + 1
             if self.counter == 1000:
                 self.do_batch()

plugins/cloudsql.py alt_plugins/cloudsql.py

@@ -4,7 +4,6 @@
 from googleapiclient import discovery, errors
 
 from pluginbase import Plugin
-from utils import utils
 
 SCOPES = ['https://www.googleapis.com/auth/sqlservice.admin']
 
@@ -49,7 +48,7 @@ def api_name(self):
         return "sqladmin.googleapis.com"
 
 
-    def methodsNames(self):
+    def method_names(self):
         return ["cloudsql.instances.create"]
 
 
@@ -79,7 +78,7 @@ def get_gcp_object(self, data):
             return None
 
 
-    def do_tag(self, project_id):
+    def do_label(self, project_id):
         page_token = None
         more_results = True
         while more_results:
@@ -92,16 +91,16 @@ def do_tag(self, project_id):
             if 'items' not in response:
                 return
             for database_instance in response['items']:
-                self.tag_one(database_instance, project_id)
+                self.label_one(database_instance, project_id)
             if 'nextPageToken' in response:
                 page_token = response['nextPageToken']
             else:
                 more_results = False
 
 
-    def tag_one(self, gcp_object, project_id):
+    def label_one(self, gcp_object, project_id):
         labels = dict()
-        labels['labels'] = self.gen_labels(gcp_object)
+        labels['labels'] = self._gen_labels(gcp_object)
         try:
             database_instance_body = dict()
             database_instance_body['settings'] = {}