Skip to content

Commit 2f87a73

Browse files
JoshuaFoxJoshuaFox
committed
fix up label_all_types
1 parent ce00e29 commit 2f87a73

17 files changed

+216
-190
lines changed

‎INSTALL.md

+74
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
# How to Install
2+
3+
(For the main documentation, see [README](./README.md).)
4+
5+
## Before deploying
6+
7+
### In which Project
8+
9+
* You can deploy Iris in any one project within your Google Cloud organization, but we recommend using a [new project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#creating_a_project) just for this.
10+
* The script will deploy Iris (an App Engine service) into this project. That does not mean that Iris is focused on labeling this project. (See [README](./README.md).)
11+
12+
### Needed roles for deployment
13+
#### Organization-level roles
14+
15+
* Here are the required organization-level roles for you, the deployer, to allow the deploy script to set up roles and log sink. (Note that *Organization Owner* is not enough).
16+
* *Organization Role Administrator* so the deployment script can create a custom IAM role for Iris that allows it to get and set labels.
17+
* *Security Admin* so the deployment script can grant the needed role bindings, e.g., to the App Engine service account.
18+
* *Logs Configuration Writer* so the deployment script can create an organization log sink that sends logs to
19+
PubSub.
20+
21+
#### Project-level roles
22+
* The required project-level roles: *Project Owner* or *Project Editor* on the project where Iris is deployed, so that the deployment script can
23+
* create role bindings, topics and subscriptions
24+
* deploy App Engine.
25+
* `actAs` the serivice account `iris-msg-sender` for deploying it to allow JWT auth.
26+
27+
* You cannot replace *Project Owner* or *Project Editor* with fine-granted "predefined roles" are not possible because deploying Cloud Scheduler cron requires at least Editor or Owner, per GCP docs.
28+
29+
## Deployment
30+
31+
* Get the code with `git clone https://github.com/doitintl/iris3.git`
32+
* Have Python 3.9+ as your default `python3`.
33+
* Make sure you have these tools
34+
* `envsubst`
35+
* `jq`
36+
* The command-line `pip3`
37+
* `gcloud`. Make sure it is logged-in using an account with the [above-mentioned](#before-deploying) roles.
38+
* Set up the configuration
39+
* Copy `config.yaml.original` to `config.yaml`.
40+
* Optionally configure by editing the config file. ([See more documentation below](#configuration).)
41+
* As always with App Engine, a default service must exist before any other exists. So if you are working with a new project:
42+
* Initialize App Engine with `gcloud app create [--region=REGION]`
43+
* Deploy a simple "Hello World" default App Engine service. (See `create_project.sh` for an example (not a runnable part of the installation) of creating a project and creating the default App Engine service.)
44+
* Now, run `./deploy.sh <PROJECT_ID> `
45+
* For documentation on usage of command-line options, run `deploy.sh -h`
46+
* Choosing when the labeling occurs
47+
* By default, labeling occurs on resource-creation, and also using Cloud Scheduler ("cron").
48+
* Each of these works on certain resource types and not others (see [Supported Google Cloud resources](README#Supported Google Cloud resources) in the main README), depending on configuration (`config.yaml`).
49+
* Use `-c` switch on `deploy.sh` to label using Cloud Scheduler only.
50+
* Use `-e` switch on `deploy.sh` to label on-event only.
51+
* If you use **both** `-c` and `-e` or **neither**, then both types of labeling occur.
52+
* To label all resources, including those not configured for being labeled on Cloud Scheduler, see section "[Labeling existing resources](README.md#labeling-existing-resources)" in the main README.
53+
54+
55+
# Configuration
56+
57+
* Iris' config file is `config*.yaml`.
58+
* All values are optional.
59+
* `config.yaml.orig` has detailed documentation of the fields.
60+
* Alternatively, you can have `config-test.yaml`. It takes priority if both it and `config-test.yaml` are present.
61+
* `app.yaml` lets you configure App Engine, for example, to set a maximum number of instances. See App Engine documentation.
62+
* Editing `cron_full.yaml` lets you optionally change the timing for the Cloud Scheduler scheduled labelings, e.g. to do it more frequently. See Google App Engine documentation.
63+
64+
# Uninstalling
65+
66+
* Run script `uninstall.sh`
67+
* Use `uninstall.sh -h` for help.
68+
69+
# Architecture
70+
Please see [README_architecture](README_architecture.md)
71+
72+
# Development and Testing
73+
Please see [HACKING](./HACKING.md)
74+

‎README.md

+6-94
Original file line numberDiff line numberDiff line change
@@ -50,9 +50,7 @@ You can also disable the scheduled labeling. See Deployment below or run `./dep
5050
## Labeling existing resources
5151

5252
* When you first use Iris, you may want to label all existing resources. Iris does not do this by default.
53-
* You have two choices:
54-
* Set configuration `label_all_on_cron: true` before deploying. Then, on the next daily Cloud Scheduler run, all resources will be labeled. However, this will increase cost, since all resources will be rescanned every day.
55-
* Alternatively, after deploying publish a PubSub message (the content doesn't matter) to `iris_label_all_topic`, for example with `gcloud pubsub topics publish iris_label_all_topic --message=does_not_matter --project $PROJECT_ID` and a full labeling will be triggered.
53+
* Publish a PubSub message (the content doesn't matter) to `iris_label_all_types_topic`, for example with `gcloud pubsub topics publish iris_label_all_types_topic --message=does_not_matter --project $PROJECT_ID` and a full labeling will be triggered.
5654

5755
# Supported Google Cloud resources
5856

@@ -79,99 +77,13 @@ names start `_gcp_`. The part of the function name after `_gcp_` is used for the
7977
* In addition to these, any labels on a project may be copied into the resourcs that are in the project, if you have enabled this in the
8078
configuration file.
8179

82-
# Installation/Deployment
83-
84-
## Before deploying
85-
86-
### In which Project
87-
88-
* You can deploy Iris in any project within your Google Cloud organization, but we recommend using a
89-
[new project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#creating_a_project).
90-
91-
92-
### Needed roles for deployment
93-
#### Organization-level roles
94-
95-
* Here are the required organization-level roles for you, the deployer, to allow the deploy script to set up roles and log sink. (Note that *Organization Owner* is not enough).
96-
* *Organization Role Administrator* so the deployment script can create a custom IAM role for Iris that allows it to get and set labels.
97-
* *Security Admin* so the deployment script can grant the needed role bindings, e.g., to the App Engine service account.
98-
* *Logs Configuration Writer* so the deployment script can create an organization log sink that sends logs to
99-
PubSub.
100-
101-
#### Project-level roles
102-
* The required project-level roles: *Project Owner* or *Project Editor* on the project where Iris is deployed, so that the deployment script can
103-
* create role bindings, topics and subscriptions
104-
* deploy App Engine.
105-
* `actAs` the serivice account `iris-msg-sender` for deploying it to allow JWT auth.
106-
107-
* Fine-granted "predefined roles" are not possible because deploying Cloud Scheduler cron requires at least Editor or Owner, per GCP docs.
108-
109-
110-
111-
## Deployment
112-
113-
* Get the code with `git clone https://github.com/doitintl/iris3.git`
114-
* Have Python 3.9+ as your default `python3`.
115-
* Make sure you have these tools
116-
* `envsubst`
117-
* `jq`
118-
* The command-line `pip3`
119-
* `gcloud`. Make sure it is logged-in using an account with the [above-mentioned](#before-deploying) roles.
120-
* Set up the configuration
121-
* Copy `config.yaml.original` to `config.yaml`.
122-
* Optionally configure by editing the config file. ([See more documentation below](#configuration).)
123-
* As always with App Engine, a default service must exist before any other exists. So if you are working with a new project, create a simple Hello World App Engine service to fulfil that App Engine requirement.
124-
* Now, run `./deploy.sh <PROJECT_ID> `.
125-
* For documentation on usage of command-line options, run `deploy.sh -h`
12680

127-
* Choosing when the labeling occurs
128-
* By default, labeling occurs on resource-creation, and also using Cloud Scheduler ("cron"). Each of these types works on certain resource types and not others (see [Supported Google Cloud Labels](#Supported Google Cloud Labels) above), depending on configuration (`config.yaml`).
129-
* Use `-c` switch on `deploy.sh` to label using Cloud Scheduler only.
130-
* Use `-e` switch on `deploy.sh` to label on-event only.
131-
* If you use **both** `-c` and `-e` or **neither**, both types of labeling occur.
132-
133-
134-
135-
# Configuration
136-
137-
* Iris' config file is `config*.yaml`.
138-
* All values are optional.
139-
* `config.yaml.orig` has detailed documentation of the fields.
140-
* `config.yaml` is read as the production configuration file
141-
* Alternatively, you can have `config-test.yaml`. It takes priority if both it and `config-test.yaml` are present.
142-
143-
* `app.yaml` lets you configure App Engine, for example, to set a maximum number of instances. See App Engine documentation.
144-
* Editing `cron_full.yaml` lets you optionally change the timing for the Cloud Scheduler scheduled labelings, e.g. to do it more frequently. See Google App Engine documentation.
145-
146-
# Uninstalling
147-
148-
* Run script `uninstall.sh`
149-
* * See a comment in `uninstall.sh` that describes what elements are uninstalled. The default is to uninstall everything, both the org-level components and the project-level components.
150-
* Run `uninstall.sh -h` for help.
151-
* A full uninstall will also delete the custom role. This will then block you from re-deploying unless you first either undelete the custom role or else give a new custom-role name at the top of `custom-iris-role.yaml`.
152-
81+
# Installing
82+
Please see [INSTALL](./INSTALL.md)
15383
# Architecture
154-
155-
* Iris runs in Google App Engine Standard Environment (Python 3).
156-
* Organization-level and project-level elements
157-
* First, note that Iris is an organization-level application.
158-
* A single instance of Iris in one project labels all projects in an org (unless you limit it by configuration).
159-
* Iris has architecture elements which are deployed to both the org and the project.
160-
* If you want a person with all permissions to deploy the org-level elements and a person without org-level permissions to redeploy only the project-level elements (e.g. when you change configuration), you can do this with flags on the script. Run `deploy.sh -h`.
161-
* The Cloud Scheduler "cron" job triggers Iris at configured intervals. See `cron_full.yaml` for config. The GCP Console view for this is in a separate App Engine tab in the Cloud Scheduler view.
162-
* For newly created resources, a Log Sink on the organization level sends all logs for resource creation to a PubSub topic.
163-
* The Log Sink is filtered to include only supported resource types
164-
* If you edit the configuration file so that only specific projects are to be labeled, the Log Sink only captures these.
165-
* PubSub topics:
166-
* One topic receives the resource-creation logs from the Log Sink
167-
* Another topic receives messages sent by the `/schedule` Cloud Scheduler handler in `main.py`. (The `/schedule` path is triggered by the Cloud Scheduler). This then sends out messages, where each one triggers the labeling of a given resource tyope in a given project.
168-
* Another topic is a dead-letter topic.
169-
* PubSub subscriptions
170-
* There is one for each topic: These direct the messages to `/label_one` and `/do_label` in `main.py`, respectively.
171-
* For security, these two PubSub subscriptions [use JWT auth](https://cloud.google.com/pubsub/docs/authenticate-push-subscriptions), where the JWT token is verified in the Iris webapp.
172-
* A dead-letter subscription. This is a pull subscription. By default, it just accumulates the messages. You can use it to see statistics, or you can pull messages from it.
84+
Please see [README_architecture](./README_architecture.md)
17385

17486
# Development and Testing
17587

176-
Please see [README_for_dev_and_testing](./README_for_dev_and_testing.md)
177-
88+
Please see [HACKING](./HACKING.md)
89+

‎README_architecture.md

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Architecture
2+
3+
## For the main documentation
4+
See [README](./README.md).
5+
6+
## Iris' architecture
7+
* Iris runs in Google App Engine Standard Environment (Python 3).
8+
* Organization-level and project-level elements
9+
* First, note that Iris is an organization-level application.
10+
* A single instance of Iris in one project labels all projects in an org (unless you limit it by configuration).
11+
* Iris has architecture elements which are deployed to both the org and the project.
12+
* If you want a person with all permissions to deploy the org-level elements and a person without org-level permissions to redeploy only the project-level elements (e.g. when you change configuration), you can do this with flags on the script. Run `deploy.sh -h`.
13+
* The Cloud Scheduler "cron" job triggers Iris at configured intervals. See `cron_full.yaml` for config. The GCP Console view for this is in a separate App Engine tab in the Cloud Scheduler view.
14+
* For newly created resources, a Log Sink on the organization level sends all logs for resource creation to a PubSub topic.
15+
* The Log Sink is filtered to include only supported resource types
16+
* If you edit the configuration file so that only specific projects are to be labeled, the Log Sink only captures these.
17+
* PubSub topics:
18+
* One topic receives the resource-creation logs from the Log Sink
19+
* Another topic receives messages sent by the `/schedule` Cloud Scheduler handler in `main.py`. (The `/schedule` path is triggered by the Cloud Scheduler). This then sends out messages, where each one triggers the labeling of a given resource tyope in a given project.
20+
* Another topic is a dead-letter topic.
21+
* PubSub subscriptions
22+
* There is one for each topic: These direct the messages to `/label_one` and `/do_label` in `main.py`, respectively.
23+
* For security, these two PubSub subscriptions [use JWT auth](https://cloud.google.com/pubsub/docs/authenticate-push-subscriptions), where the JWT token is verified in the Iris webapp.
24+
* A dead-letter subscription. This is a pull subscription. By default, it just accumulates the messages. You can use it to see statistics, or you can pull messages from it.
25+
26+
# Development and Testing
27+
Please see [HACKING](./HACKING.md)
28+

‎README_for_dev_and_testing.md

+4-4
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
1-
# Iris: Dev and Testing
2-
3-
# For the main documentation
1+
# Iris: How to develop and test
2+
3+
## For the main documentation
44
See [README](./README.md).
55

66
## Development tools
@@ -14,7 +14,7 @@ See [README](./README.md).
1414
use `export FLASK_ENV=development;export FLASK_RUN_PORT=8000;export FLASK_DEBUG=1;FLASK_APP=main.py python -m flask run`
1515
* In an interactive development environment, run `main.py`, first setting these environment variables.
1616
* For hands-on debugging
17-
* Use `test_do_label` and `test_label_one` and `test_schedule` to trigger against your localhost dev-server; this should label actual Cloud resources that you have pre-deployed.
17+
* Use `test_do_label` and `test_label_one` and `test_schedule` to trigger against your localhost dev-server; this will label actual Cloud resources that you have pre-deployed.
1818
* See the `test_...` files for instructions.
1919

2020
## Adding new kinds of labels

‎TODO.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
## Note: see also Github Issues
44

5-
* P2 Memory consumption: Even an empty AppEngine app (not Iris, just a Hello World with 3 lines of code in total) crashes on out-of-memory for the smalled AppEngine instance. Google has confirmed this. See if there is a workaround. This will save money.
5+
* P2 Memory consumption: Even an empty App Engine app (not Iris, just a Hello World with 3 lines of code in total) crashes on out-of-memory for the smalled App Engine instance. Google has confirmed this. See if there is a workaround. This will save money.
66

77
* P3 Label immediately after an event in certain cases, as opposed to using a daily cron as is now done.
88
* Cloud SQL Instances

‎deploy.sh

+9-10
Original file line numberDiff line numberDiff line change
@@ -53,20 +53,19 @@ while getopts 'cepoh' opt; do
5353
Usage deploy.sh PROJECT_ID
5454
Argument:
5555
The project to which Iris will be deployed
56-
Options, to be given before project ID.
57-
If neither -p nor -o is given, the default behavior is used:
58-
to do both; this is equivalent to -p -o
59-
Flags:
56+
Advancd options, to be given before project ID. These are not to be used except in advanced schenarios.
57+
A. Labeling only on-creation or on-schedule:
58+
-c: Label on Cloud Scheduler cron only
59+
-e: Label on-creation-event only
60+
The default, if neither -c or -e are given, is to enable both; equivalent to -c -e
61+
62+
B. Project/Org level with -p or -o. If neither -p nor -o is given,
63+
the default behavior is to do both; equivalent to -p -o
6064
-o: Deploy org-level elements like Log Sink
61-
-p: Deploy project-level elements.
62-
Org-level elements are a pre-requisite.
65+
-p: Deploy project-level elements. Org-level elements are a pre-requisite.
6366
This is useful for redeploying to the same project, e.g., to change config.
6467
If you want to deploy to a *different* project,
6568
then you have to deploy the org-level elements.
66-
The default, if neither -o or -p are given, is to enable both.
67-
-c: Label on Cloud Scheduler cron to add labels
68-
-e: Label on-creation-event.
69-
The default if neither -c or -e are given is to enable both.
7069
7170
Environment variable:
7271
IRIS_CUSTOM_ROLE (Optional, default is iris3) An identifier for the Iris custom role,

0 commit comments

Comments
 (0)