Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Main review mods 2023 12 06 #55

Open
wants to merge 151 commits into
base: main-review-2023-11-17
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 148 commits
Commits
Show all changes
151 commits
Select commit Hold shift + click to select a range
afe0783
branch:HPCC-27615-original-folder-structure. Constructed the original…
Sep 21, 2023
5a51f03
branch:HPCC-27615-fixes-deploy-without-external-storage. Now can depl…
Sep 21, 2023
35b1a43
branch:HPCC-27615-my-local-has-flat-structure
Sep 21, 2023
f31f1c7
branch:HPCC-27615-easy-deploy. Get is a merge of HPCC-27615 latest an…
Sep 22, 2023
c11ef09
branch: HPCC-27615-easy-deploy. This is a merge of HPCC-27615 and bra…
Sep 22, 2023
977e107
branch:new-variable-enable_thor. Now one must set 'enable_thor=true' …
Sep 26, 2023
1b54ae7
branch:fix-roxie-so-port-18002-is-used
Sep 27, 2023
eba6f63
branch:add-htpasswd-support
Sep 28, 2023
58268fa
branch:add-ecl-code-security. Plus, , setup 'storage_data_lz'
Oct 4, 2023
fbafd5d
branch:add-ecl-code-security
Oct 4, 2023
32b918b
Merge pull request #1 from tlhumphrey2/add-ecl-code-security
Oct 5, 2023
226139f
branch:add-terraform-to-deploy-everything
Oct 6, 2023
982e692
Merge pull request #2 from tlhumphrey2/add-terraform-to-deploy-everyt…
Oct 6, 2023
347ceef
branch:add-terraform-to-deploy-everything
Oct 9, 2023
1e0e18f
Merge pull request #3 from tlhumphrey2/add-terraform-to-deploy-everyt…
Oct 9, 2023
e5f5853
branch:aks-is-now-using-easy-deploy-variables
Oct 12, 2023
9f93f16
Merge pull request #4 from tlhumphrey2/aks-is-now-using-easy-deploy-v…
Oct 12, 2023
7fb2c89
branch:aks-is-now-using-easy-deploy-variables
Oct 12, 2023
cde9f4f
Merge pull request #5 from tlhumphrey2/aks-is-now-using-easy-deploy-v…
Oct 12, 2023
0f56039
branch:aks-is-now-using-easy-deploy-variables. Changed scripts/deploy…
Oct 13, 2023
1bf7dc9
Merge pull request #6 from tlhumphrey2/aks-is-now-using-easy-deploy-v…
Oct 13, 2023
e14e9eb
branch:aks-is-now-using-easy-deploy-variables
Oct 17, 2023
3de0cc1
Merge pull request #7 from tlhumphrey2/aks-is-now-using-easy-deploy-v…
Oct 17, 2023
dbe5cef
branch:aks-is-now-using-easy-deploy-variables
Oct 18, 2023
ff58b6d
Merge pull request #8 from tlhumphrey2/aks-is-now-using-easy-deploy-v…
Oct 18, 2023
2203f42
branch:few-changes-20231020
Oct 20, 2023
71434db
Merge pull request #9 from tlhumphrey2/few-changes-20231020
Oct 20, 2023
c54dad8
branch:HPCC-27615-easy-deploy-bryan1
Oct 28, 2023
f077f15
branch:HPCC-27615-easy-deploy-bryan1-w-oss
Oct 28, 2023
1868452
branch:HPCC-27615-easy-deploy-bryan1-w-oss-zones
Oct 28, 2023
1523b64
branch:HPCC-27615-easy-deploy-bryan1-w-oss-zones. Now can optionally …
Oct 30, 2023
8e0e2f1
branch:HPCC-27615-easy-deploy-bryan1-w-oss-zones. Now can optionally …
Oct 30, 2023
d2ee650
branch: HPCC-27615-easy-deploy-bryan1-w-oss-zones
Oct 30, 2023
873a9f7
branch:HPCC-27615-easy-deploy-bryan1-w-oss-zones. merged with github …
Oct 30, 2023
50f7050
branch:HPCC-27615-easy-deploy-bryan2-root-sto-applied-initials-added.…
Oct 31, 2023
108641a
branch:HPCC-27615-easy-deploy-bryan2-root-sto-applied-initials-added.…
Oct 31, 2023
19e114a
branch:HPCC-27615-easy-deploy-bryan2-root-sto-applied-initials-added2
Nov 1, 2023
5e3a1ec
branch:HPCC-27615-easy-deploy-bryan3-roxiepool-optional
Nov 1, 2023
7065e89
branch:HPCC-27615-easy-deploy-bryan4-placing-auto.tfvars-files-aks-st…
Nov 2, 2023
ad678e4
branch:HPCC-27615-easy-deploy-bryan4-placing-auto.tfvars-files-aks-st…
Nov 2, 2023
aca0ced
branch:HPCC-27615-easy-deploy-bryan4-placing-auto.tfvars-files-aks-st…
Nov 2, 2023
1c346ee
branch:HPCC-27615-easy-deploy-bryan4-placing-auto.tfvars-files-aks-st…
Nov 2, 2023
e796355
branch:HPCC-27615-easy-deploy-bryan5-miscellaneous-changes
Nov 4, 2023
3588eec
branch:HPCC-27615-easy-deploy-bryan5-miscellaneous-changes. Added doc…
Nov 5, 2023
8ec2aab
branch:HPCC-27615-easy-deploy-bryan5-miscellaneous-changes. Updated d…
Nov 5, 2023
498d908
branch:HPCC-27615-easy-deploy-bryan6-restrict-hpcc-access
Nov 6, 2023
81afffd
branch:HPCC-27615-easy-deploy-bryan6-restrict-hpcc-access. Updated us…
Nov 6, 2023
902ec17
branch:HPCC-27615-easy-deploy-bryan6-restrict-hpcc-access. Updating U…
Nov 6, 2023
682612e
branch:HPCC-27615-easy-deploy-bryan6-restrict-hpcc-access. Updated Us…
Nov 6, 2023
2417b9a
branch:HPCC-27615-easy-deploy-bryan7-developers-documentation
Nov 7, 2023
d8aa06c
branch:HPCC-27615-easy-deploy-bryan7-developers-documentation
Nov 7, 2023
bf1a64d
branch:HPCC-27615-easy-deploy-bryan7-developers-documentation
Nov 7, 2023
41be7fa
branch:HPCC-27615-easy-deploy-bryan7-developers-documentation
Nov 7, 2023
02fe05b
branch:HPCC-27615-easy-deploy-bryan7-developers-documentation. Update…
Nov 7, 2023
5bc6502
branch:HPCC-27615-easy-deploy-bryan7-developers-documentation. Update…
Nov 7, 2023
a8a8170
branch:HPCC-27615-easy-deploy-bryan7-developers-documentation. Update…
Nov 7, 2023
882a390
branch:HPCC-27615-easy-deploy-bryan7-developers-documentation. Update…
Nov 7, 2023
345453e
branch:HPCC-27615-easy-deploy-bryan7-developers-documentation. Update…
Nov 7, 2023
986595a
branch:HPCC-27615-easy-deploy-bryan8-cleanup-and-make-clearer
Nov 9, 2023
746eb3e
Merge pull request #36 from hpccsystems-solutions-lab/HPCC-27615-easy…
tlhumphrey2 Nov 9, 2023
fc879fe
branch:HPCC-27615-easy-deploy-bryan8-pods-assigned-nodepools
Nov 9, 2023
6ffcf2d
Merge pull request #37 from hpccsystems-solutions-lab/HPCC-27615-easy…
tlhumphrey2 Nov 9, 2023
9f522e2
branch:HPCC-27615-easy-deploy-bryan8-pods-assigned-nodepools. hpcc.tf…
Nov 10, 2023
409e1a5
Merge pull request #38 from hpccsystems-solutions-lab/HPCC-27615-easy…
tlhumphrey2 Nov 10, 2023
ba53432
branch:no-ephemeral-storage-when-external-used
Nov 10, 2023
dc46a6f
Merge pull request #39 from hpccsystems-solutions-lab/no-ephemeral-st…
tlhumphrey2 Nov 10, 2023
cf68e81
branch:no-ephemeral-storage-when-external-used. Fixed paths in script…
Nov 10, 2023
d7ea956
Merge pull request #40 from hpccsystems-solutions-lab/no-ephemeral-st…
tlhumphrey2 Nov 10, 2023
31396de
branch:HPCC-27615-easy-deploy-bryan7-developers-documentation. Update…
Nov 12, 2023
d42c883
Merge pull request #41 from hpccsystems-solutions-lab/HPCC-27615-easy…
tlhumphrey2 Nov 12, 2023
d17572e
branch:HPCC-27615-easy-deploy-bryan9-variable-eclwatch-a-record
Nov 16, 2023
b05a798
Merge pull request #42 from hpccsystems-solutions-lab/HPCC-27615-easy…
tlhumphrey2 Nov 16, 2023
fcc5365
Update hpcc.tf
tlhumphrey2 Nov 16, 2023
a76e524
branch:HPCC-27615-easy-deploy-bryan10-added-hpcc_version
Nov 16, 2023
c175fdb
Merge pull request #43 from hpccsystems-solutions-lab/HPCC-27615-easy…
tlhumphrey2 Nov 16, 2023
1dce33d
Tims' Modifications
dcamper Nov 17, 2023
b44fbcb
branch:HPCC-27615-easy-deploy-bryan10-added-hpcc_version_and_misc. Ad…
Nov 20, 2023
4d284cb
Merge branch 'main' into HPCC-27615-easy-deploy-bryan10-added-hpcc_ve…
tlhumphrey2 Nov 20, 2023
d89b49e
Merge pull request #45 from hpccsystems-solutions-lab/HPCC-27615-easy…
tlhumphrey2 Nov 20, 2023
8747f16
Update destroy
tlhumphrey2 Nov 20, 2023
822acd9
branch:main-review-mods-2023-11-17. The easy fixes
Nov 21, 2023
73f5982
main-review-mods-2023-11-17. Resolved conflicts in scripts/destroy
Nov 21, 2023
339bf98
Merge branch 'main-review-mods-2023-11-17-tlh-changes' into main-revi…
Nov 21, 2023
4d5526d
branch:main-review-mods-2023-11-17-activate-aks_node_size
Nov 21, 2023
5d71764
Merge pull request #46 from hpccsystems-solutions-lab/main-review-mod…
tlhumphrey2 Nov 21, 2023
3aab1f8
branch:main-review-mods-2023-11-17-activate-aks_node_size
Nov 21, 2023
1518d47
Merge pull request #48 from hpccsystems-solutions-lab/main-review-mod…
tlhumphrey2 Nov 21, 2023
2a6805b
branch:main-review-mods-2023-11-17-activate-aks_node_size. aks_node_s…
Nov 22, 2023
a97cb81
Merge pull request #50 from hpccsystems-solutions-lab/main-review-mod…
tlhumphrey2 Nov 22, 2023
6809ae2
branch:main-review-mods-2023-11-17-deploy-hpcc-depends-on-storage
Nov 22, 2023
a646d62
Merge pull request #52 from hpccsystems-solutions-lab/main-review-mod…
tlhumphrey2 Nov 22, 2023
2e5e4f4
branch:main-review-mods-2023-11-17-aks_node-sizes-now-object
Nov 24, 2023
f8fa470
Merge pull request #54 from hpccsystems-solutions-lab/main-review-mod…
tlhumphrey2 Nov 24, 2023
290d023
Remove input variables not used
Dec 5, 2023
801e279
Added column 'updatable' to table of options in README.md
Dec 5, 2023
075d3aa
Removed automation.tf from aks
Dec 5, 2023
fb61631
Increased values of 'managerResources'. NOTE: Dan's 'managerResources…
Dec 5, 2023
c5d78de
Increased values of 'workerResources' to match Dan's
Dec 5, 2023
99d7fcf
Increased cpu of hthor resources to 2. This is higher than what Dan has.
Dec 5, 2023
416b4e4
Eliminated metadata variables not used
Dec 5, 2023
0274223
Removed metadata from aks, hpcc, and vnet (not in storage). Now it is…
Dec 5, 2023
bfd5142
Removed metadata from aks, hpcc, storage, and vnet (not in storage). …
Dec 5, 2023
5617f98
Removed metadata from aks, hpcc, storage, and vnet. Now it is copied …
Dec 5, 2023
6471329
Added new variable, 'aks_capacity' which defines the minimum and maxi…
Dec 6, 2023
ab72960
In lite-variables.tf, I changed the descriptions of 'aks_node_sizes' …
Dec 6, 2023
96bbe2c
In all bash scripts, replaced with
Dec 6, 2023
4d88cc9
Removed referenced branch from all source statements, since the branc…
Dec 6, 2023
91266e9
In lite-variables.tf, line 54 added '\' before each " in description.
Dec 7, 2023
f45720c
In hpcc/outputs.tf, prefixed eclwatch url with 'https://'. In lite-va…
Dec 7, 2023
3405610
In README.md, documented outpuss of hpcc, aks, vnet. There are no out…
Dec 7, 2023
6e6b6c3
Change default node size for spray pool from 2xlarge to large.
Dec 8, 2023
6ece380
Reduced the size of nodes in each node pool.
Dec 8, 2023
eae050f
Fixed value of output 'advisor_recommendations' for both aks/outputs.…
Dec 8, 2023
11554af
Calculates max capacity of thorpool. Set thor cpu and ram.
Dec 11, 2023
5c33a80
Calculates max capacity of thorpool. Set thor cpu and ram. Added to r…
Dec 12, 2023
8dd9ebf
In both hpcc/hpcc.tf and aks/aks.tf, changed source statements so val…
Dec 12, 2023
a00e254
In both hpcc/hpcc.tf and aks/aks.tf, removed prefix 'git@' from sourc…
Dec 12, 2023
9f48ece
Changed variable 'aks_node_sizes' to individual string variables: rox…
Dec 12, 2023
f4847f5
Changed 'source' in aks/aks.tf. Now it points to github repo
Dec 12, 2023
01f09ed
To redone lite-locals.tf, added workerResources cpu and memory.
Dec 13, 2023
b30f1a5
Removed commented-out code from hpcc/main.tf
Dec 13, 2023
b83083c
Removed variable 'hpcc_namespace'
Dec 13, 2023
185c760
Removed 18010 from output of eclwatch URL. Also, changed opinionated …
Dec 13, 2023
b772483
Capitalized Kubernetes everywhere in README.md
Dec 13, 2023
3eb4e6e
Deleted paragraph 'This repo is a fork of the excellent work performe…
Dec 13, 2023
76386ea
Make sure all these are capitalized in README.md when used as product…
Dec 13, 2023
4f37c43
Throughout README.md changed 'terraform' to 'terraform code'
Dec 13, 2023
85b2bef
All fixes for Dan's comments about hpcc-tf-for-developers.md are in t…
Dec 13, 2023
e6ff10c
In lite-locals.tf, deleted all terraform code that was commented-out.
Dec 13, 2023
38ef877
All fixes for all Dan's review in 1:15pm email today.
Dec 13, 2023
6e366a7
In hpcc-tf-for-developers.md, capitalizes all Azure
Dec 13, 2023
e74c3b6
Dan's review fixes in email dated 12/14/2023 7:38am
Dec 14, 2023
44cedc4
Removed azuread_group.subscription_owner from aks/aks.tf and aks/data…
Dec 15, 2023
97f7eef
Make 1 or 4 nodepools optional. Added aks_4nodepools
Dec 18, 2023
4ba65e2
Removed all error messages in 'thorpool_max_capacity' calculations an…
Dec 19, 2023
a8cef3e
Removed all occurrences of region restriction.
Dec 19, 2023
d899526
In lite-variables.tf, changed 8002 to 18002
Dec 19, 2023
2255093
In lite.auto.tfvars.example, changed 8002 to 18002
Dec 19, 2023
a5e3dc5
Removed all error messages in 'thorpool_max_capacity' calculations an…
Dec 19, 2023
abbb3bf
To metadata.tf of storage, added 'additional_tags'. Plus, removed 'de…
Dec 20, 2023
9cf1765
'workerResources' memory. Added 'G'. Caused thor container error.
Dec 27, 2023
f58b8b7
Added scripts/extract-aks-tfvars to properly extra 'aks_' variables f…
Jan 3, 2024
9cdfc88
In lite-variables.tf, no longer says REQUIRED for aks_enable_roxie.
Jan 3, 2024
81eca56
Changed workerMemory.query to same value as workerResources.memory.
Jan 3, 2024
ed598d8
Created 'aks_nodepools_max_capacity'. max_capacity of all hpcc nodepo…
Jan 4, 2024
05f46ff
Corrected 'aks_nodepools_max_capacity' code of lite-locals.tf and lit…
Jan 5, 2024
09cad2b
In README.md, minimum vCPU requirements given. In aks/locals.tf, min_…
Jan 5, 2024
4d1b78f
Added in README.md: 1) info about the directory, and what is in it, …
Jan 8, 2024
e4a8cc4
In scripts/needed-auto-tfvars-files/aks/aks.auto.tfvars.example, chan…
Jan 10, 2024
431fad0
In lite-locals.tf, increased 'helm_chart_timeout' from 300 to 600. Wh…
Jan 10, 2024
3c7c641
In README.md, said that 'jq' and 'kubelogin' are required (i.e. they …
Jan 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 10 additions & 9 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,34 +2,35 @@
**/.terraform/*

# .tfstate files
*.tfstate
*.tfstate.*
**/*.tfstate
**/*.tfstate.*

# Crash log files
crash.log
**/crash.log

# Ignore any .tfvars files that are generated automatically for each Terraform run. Most
# .tfvars files are managed as part of configuration and so should be included in
# version control.
#
# example.tfvars
*.tfvars
*.json
*/*.tfvars
**/*.json

# Ignore data files
**/data

# Ignore override files as they are usually used to override resources locally and so
# are not checked in
override.tf
override.tf.json
*_override.tf
*_override.tf.json
**/override.tf
**/override.tf.json
**/*_override.tf
**/*_override.tf.json

# Include override files you do wish to add to version control using negated pattern
#
# !example_override.tf
.terraform.*.hcl
**/.terraform.*.hcl

# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
# example: *tfplan*
177 changes: 159 additions & 18 deletions README.md

Large diffs are not rendered by default.

127 changes: 0 additions & 127 deletions aks/aks.auto.tfvars.example

This file was deleted.

54 changes: 16 additions & 38 deletions aks/aks.tf
Original file line number Diff line number Diff line change
@@ -1,35 +1,16 @@
# resource "kubernetes_secret" "private_docker_registry" {
# count = can(var.registry.server) && can(var.registry.username) && can(var.registry.password) ? 1 : 0
# metadata {
# name = "docker-cfg"
# }
# type = "kubernetes.io/dockerconfigjson"
# data = {
# ".dockerconfigjson" = jsonencode({
# auths = {
# "${var.registry.server}" = {
# "username" = var.registry.username
# "password" = var.registry.password
# "email" = var.admin.email
# "auth" = base64encode("${var.registry.username}:${var.registry.password}")
# }
# }
# })
# }
# }

module "aks" {
depends_on = [random_string.string]
source = "github.com/gfortil/terraform-azurerm-aks.git?ref=HPCC-27615"
# source = "../../../terraform-azurerm-aks"
#source = "[email protected]:hpccsystems-solutions-lab/tlh-oss-terraform-azurerm-aks.git"
source = "github.com/hpccsystems-solutions-lab/tlh-oss-terraform-azurerm-aks.git"
#source = "/home/azureuser/temp/OSS/terraform-azurerm-aks"

providers = {
kubernetes = kubernetes.default
helm = helm.default
kubectl = kubectl.default
}

location = var.metadata.location
location = local.metadata.location
resource_group_name = module.resource_groups["azure_kubernetes_service"].name

cluster_name = local.cluster_name
Expand All @@ -38,32 +19,34 @@ module "aks" {
# for v1.6.2 aks: sku_tier_paid = false
sku_tier = var.sku_tier

logging_monitoring_enabled = var.aks_logging_monitoring_enabled

cluster_endpoint_access_cidrs = var.cluster_endpoint_access_cidrs

virtual_network_resource_group_name = try(var.use_existing_vnet.resource_group_name, local.get_vnet_config.resource_group_name)
virtual_network_name = try(var.use_existing_vnet.name, local.get_vnet_config.name)
subnet_name = try(var.use_existing_vnet.subnets.aks.name, "aks-hpcc-private")
route_table_name = try(var.use_existing_vnet.route_table_name, local.get_vnet_config.route_table_name)

dns_resource_group_lookup = { "${var.internal_domain}" = var.dns_resource_group }
dns_resource_group_lookup = { "${local.internal_domain}" = local.dns_resource_group }

admin_group_object_ids = [data.azuread_group.subscription_owner.object_id]
admin_group_object_ids = null

rbac_bindings = var.rbac_bindings

availability_zones = var.availability_zones
node_groups = var.node_groups
node_groups = local.node_groups

core_services_config = {
alertmanager = var.core_services_config.alertmanager
coredns = var.core_services_config.coredns
external_dns = var.core_services_config.external_dns
cert_manager = var.core_services_config.cert_manager
alertmanager = local.core_services_config.alertmanager
coredns = local.core_services_config.coredns
external_dns = local.core_services_config.external_dns
cert_manager = local.core_services_config.cert_manager

ingress_internal_core = {
domain = var.core_services_config.ingress_internal_core.domain
subdomain_suffix = "${var.core_services_config.ingress_internal_core.subdomain_suffix}${trimspace(var.owner.name)}" // dns record suffix
public_dns = var.core_services_config.ingress_internal_core.public_dns
domain = local.core_services_config.ingress_internal_core.domain
subdomain_suffix = "${local.core_services_config.ingress_internal_core.subdomain_suffix}${trimspace(local.owner_name_initials)}" // dns record suffix
public_dns = local.core_services_config.ingress_internal_core.public_dns
}
}

Expand All @@ -75,9 +58,4 @@ module "aks" {
}

logging = var.logging

experimental = {
oms_agent = var.hpcc_log_analytics_enabled || var.experimental.oms_agent
oms_agent_log_analytics_workspace_id = fileexists("../logging/data/workspace_resource_id.txt") ? file("../logging/data/workspace_resource_id.txt") : var.experimental.oms_agent_log_analytics_workspace_id != null ? var.experimental.oms_agent_log_analytics_workspace_id : null
}
}
65 changes: 0 additions & 65 deletions aks/automation.tf

This file was deleted.

4 changes: 0 additions & 4 deletions aks/data.tf
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,5 @@ data "http" "host_ip" {
data "azurerm_subscription" "current" {
}

data "azuread_group" "subscription_owner" {
display_name = "ris-azr-group-${data.azurerm_subscription.current.display_name}-owner"
}

data "azurerm_client_config" "current" {
}
35 changes: 35 additions & 0 deletions aks/lite-locals.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
locals {
internal_domain = var.aks_dns_zone_name

dns_resource_group = var.aks_dns_zone_resource_group_name

owner = {
name = var.aks_admin_name
email = var.aks_admin_email
}

owner_name_initials = lower(join("",[for x in split(" ",local.owner.name): substr(x,0,1)]))

core_services_config = {
alertmanager = {
smtp_host = "smtp-hostname.ds:25"
smtp_from = var.aks_admin_email
routes = []
receivers = []
}

coredns = {}

external_dns = {
public_domain_filters = [var.aks_dns_zone_name]
}

cert_manager = {}

ingress_internal_core = {
domain = var.aks_dns_zone_name
subdomain_suffix = "hpcc" // dns record suffix //must be unique accross subscription
public_dns = true
}
}
}
Loading