Skip to content

Commit 7f23994

Browse files
AleksCallebatAleksCallebat
authored
Adding GCP support (#41)
* WIP * WIP * WIP * working sample not documented * non documented * terraform no default value * updated and separated the functions * terraform with gcp support * terraform with gcp support * corrected path * refactored * formatting * outputs to separate file * outputs to separate file * added gcp on new branch * restructuring example folder, adjusting links in the README
1 parent 501a4ce commit 7f23994

File tree

46 files changed

+1116
-73
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

46 files changed

+1116
-73
lines changed

‎README.md

+11-9
Original file line numberDiff line numberDiff line change
@@ -28,16 +28,16 @@ Code in the repository is organized into following folders:
2828

2929
The folder `examples` contains the following Terraform implementation examples :
3030

31-
| Cloud | Example | Description |
32-
| ----- | ---------------------------------------------------------------------------------- |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
33-
| Azure | [adb-lakehouse](examples/adb-lakehouse/) | Lakehouse terraform blueprints |
31+
| Cloud | Example | Description |
32+
| ----- | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
33+
|
34+
|Azure | [adb-lakehouse](examples/adb-lakehouse/) | Lakehouse terraform blueprints|
3435
| Azure | [adb-with-private-link-standard](examples/adb-with-private-link-standard/) | Provisioning Databricks on Azure with Private Link - [Standard deployment](https://learn.microsoft.com/en-us/azure/databricks/administration-guide/cloud-configurations/azure/private-link-standard) |
3536
| Azure | [adb-vnet-injection](examples/adb-vnet-injection/) | A basic example of VNet injected Azure Databricks workspace |
3637
| Azure | [adb-exfiltration-protection](examples/adb-exfiltration-protection/) | A sample implementation of [Data Exfiltration Protection](https://www.databricks.com/blog/2020/03/27/data-exfiltration-protection-with-azure-databricks.html) |
3738
| Azure | [adb-external-hive-metastore](examples/adb-external-hive-metastore/) | Example template to implement [external hive metastore](https://learn.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastore) |
3839
| Azure | [adb-kafka](examples/adb-kafka/) | ADB - single node kafka template |
3940
| Azure | [adb-private-links](examples/adb-private-links/) | Azure Databricks Private Links |
40-
| Azure | [adb-private-links-general](examples/adb-private-links-general/) | Provisioning Databricks on Azure with Private Link and [Data Exfiltration Protection](https://www.databricks.com/blog/2020/03/27/data-exfiltration-protection-with-azure-databricks.html) |
4141
| Azure | [adb-splunk](examples/adb-splunk/) | ADB workspace with single VM splunk integration |
4242
| Azure | [adb-squid-proxy](examples/adb-squid-proxy/) | ADB clusters with HTTP proxy |
4343
| Azure | [adb-teradata](examples/adb-teradata/) | ADB with single VM Teradata integration |
@@ -52,9 +52,10 @@ The folder `examples` contains the following Terraform implementation examples :
5252
| AWS | [aws-databricks-uc](examples/aws-databricks-uc/) | AWS UC |
5353
| AWS | [aws-databricks-uc-bootstrap](examples/aws-databricks-uc-bootstrap/) | AWS UC |
5454
| AWS | [aws-remote-backend-infra](examples/aws-remote-backend-infra/) | Simple example on remote backend |
55-
| AWS | [aws-workspace-config](examples/aws-workspace-config/) | Configure workspace objects |
56-
| GCP | Coming soon | |
57-
55+
| AWS | [aws-workspace-config](examples/aws-workspace-config/) | Configure workspace objects
56+
| GCP | [gcp-sa-provisionning](examples/gcp-sa-provisionning/) | Provisionning of the identity with the permissions required to deploy on GCP. |
57+
| GCP | [gcp-basic](examples/gcp-basic/) | Workspace Deployment with managed vpc |
58+
| GCP | [gcp-byovpc](examples/gcp-byovpc/) | Workspace Deployment with managed vpc |
5859
### Modules
5960

6061
The folder `modules` contains the following Terraform modules :
@@ -71,8 +72,9 @@ The folder `modules` contains the following Terraform modules :
7172
| AWS | [aws-workspace-with-firewall](modules/aws-workspace-with-firewall/) | Provisioning AWS Databricks E2 with an AWS Firewall |
7273
| AWS | [aws-exfiltration-protection](modules/aws-exfiltration-protection/) | An implementation of [Data Exfiltration Protection on AWS](https://www.databricks.com/blog/2021/02/02/data-exfiltration-protection-with-databricks-on-aws.html) |
7374
| AWS | aws-workspace-with-private-link | Coming soon |
74-
| GCP | Coming soon | |
75-
75+
| GCP | [gcp-sa-provisionning](modules/gcp-sa-provisionning/) | Provisions the identity (SA) with the correct permissions |
76+
| GCP | [gcp-workspace-basic](modules/gcp-workspace-basic/) | Provisions a workspace with managed VPC |
77+
| GCP | [gcp-workspace-byovpc](modules/gcp-workspace-byovpc/) | Workspace with customer-managed VPC. |
7678
## CICD pipelines
7779

7880
The folder `cicd-pipelines` contains the following implementation examples of pipeline:

‎examples/adb-basic/terraform.tfvars

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
dbfs_prefix = "anotherdbfsname"

‎examples/adb-kafka/terraform.tfvars

+5-5
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
1-
spokecidr="10.179.0.0/20"
2-
no_public_ip=true
3-
rglocation="southeastasia"
4-
dbfs_prefix="dbfs"
5-
workspace_prefix="adb-kafka"
1+
spokecidr = "10.179.0.0/20"
2+
no_public_ip = true
3+
rglocation = "southeastasia"
4+
dbfs_prefix = "dbfs"
5+
workspace_prefix = "adb-kafka"

‎examples/adb-private-links/versions.tf

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
terraform {
33
required_providers {
44
databricks = {
5-
source = "databricks/databricks"
5+
source = "databricks/databricks"
66
version = ">=0.5.1"
77
}
88

‎examples/adb-squid-proxy/main/versions.tf

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
terraform {
33
required_providers {
44
databricks = {
5-
source = "databricks/databricks"
5+
source = "databricks/databricks"
66
version = "0.3.10"
77
}
88

‎examples/adb-squid-proxy/modules/storage/main.tf

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
resource "azurerm_storage_account" "storage" {
2-
name = var.storagename
3-
resource_group_name = var.resource_group_name
2+
name = var.storagename
3+
resource_group_name = var.resource_group_name
44
location = var.locationtest
55
account_tier = "Standard"
66
account_replication_type = "LRS"

‎examples/adb-squid-proxy/modules/vm/main.tf

+30-30
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ terraform {
44

55
required_providers {
66
azurerm = {
7-
source = "hashicorp/azurerm"
7+
source = "hashicorp/azurerm"
88
version = "~>2.0"
99
}
1010
}
@@ -17,38 +17,38 @@ provider "azurerm" {
1717
resource "azurerm_resource_group" "vmss" {
1818
name = var.resource_group_name
1919
location = var.location
20-
tags = var.tags
20+
tags = var.tags
2121
}
2222

2323
resource "random_string" "fqdn" {
24-
length = 6
25-
special = false
26-
upper = false
27-
number = false
24+
length = 6
25+
special = false
26+
upper = false
27+
number = false
2828
}
2929

3030
resource "azurerm_virtual_network" "vmss" {
3131
name = "vmss-vnet"
3232
address_space = ["10.0.0.0/16"]
3333
location = var.location
3434
resource_group_name = azurerm_resource_group.vmss.name
35-
tags = var.tags
35+
tags = var.tags
3636
}
3737

3838
resource "azurerm_subnet" "vmss" {
3939
name = "vmss-subnet"
4040
resource_group_name = azurerm_resource_group.vmss.name
4141
virtual_network_name = azurerm_virtual_network.vmss.name
42-
address_prefixes = ["10.0.2.0/24"]
42+
address_prefixes = ["10.0.2.0/24"]
4343
}
4444

4545
resource "azurerm_public_ip" "vmss" {
46-
name = "vmss-public-ip"
47-
location = var.location
48-
resource_group_name = azurerm_resource_group.vmss.name
49-
allocation_method = "Static"
50-
domain_name_label = random_string.fqdn.result
51-
tags = var.tags
46+
name = "vmss-public-ip"
47+
location = var.location
48+
resource_group_name = azurerm_resource_group.vmss.name
49+
allocation_method = "Static"
50+
domain_name_label = random_string.fqdn.result
51+
tags = var.tags
5252
}
5353

5454
resource "azurerm_lb" "vmss" {
@@ -65,8 +65,8 @@ resource "azurerm_lb" "vmss" {
6565
}
6666

6767
resource "azurerm_lb_backend_address_pool" "bpepool" {
68-
loadbalancer_id = azurerm_lb.vmss.id
69-
name = "BackEndAddressPool"
68+
loadbalancer_id = azurerm_lb.vmss.id
69+
name = "BackEndAddressPool"
7070
}
7171

7272
resource "azurerm_lb_probe" "vmss" {
@@ -89,7 +89,7 @@ resource "azurerm_lb_rule" "lbnatrule" {
8989
}
9090

9191
data "azurerm_resource_group" "image" {
92-
name = var.packer_resource_group_name
92+
name = var.packer_resource_group_name
9393
}
9494

9595
data "azurerm_image" "image" {
@@ -110,7 +110,7 @@ resource "azurerm_virtual_machine_scale_set" "vmss" {
110110
}
111111

112112
storage_profile_image_reference {
113-
id=data.azurerm_image.image.id
113+
id = data.azurerm_image.image.id
114114
}
115115

116116
storage_profile_os_disk {
@@ -121,10 +121,10 @@ resource "azurerm_virtual_machine_scale_set" "vmss" {
121121
}
122122

123123
storage_profile_data_disk {
124-
lun = 0
125-
caching = "ReadWrite"
126-
create_option = "Empty"
127-
disk_size_gb = 10
124+
lun = 0
125+
caching = "ReadWrite"
126+
create_option = "Empty"
127+
disk_size_gb = 10
128128
}
129129

130130
os_profile {
@@ -150,20 +150,20 @@ resource "azurerm_virtual_machine_scale_set" "vmss" {
150150
name = "IPConfiguration"
151151
subnet_id = azurerm_subnet.vmss.id
152152
load_balancer_backend_address_pool_ids = [azurerm_lb_backend_address_pool.bpepool.id]
153-
primary = true
153+
primary = true
154154
}
155155
}
156-
156+
157157
tags = var.tags
158158
}
159159

160160
resource "azurerm_public_ip" "jumpbox" {
161-
name = "jumpbox-public-ip"
162-
location = var.location
163-
resource_group_name = azurerm_resource_group.vmss.name
164-
allocation_method = "Static"
165-
domain_name_label = "${random_string.fqdn.result}-ssh"
166-
tags = var.tags
161+
name = "jumpbox-public-ip"
162+
location = var.location
163+
resource_group_name = azurerm_resource_group.vmss.name
164+
allocation_method = "Static"
165+
domain_name_label = "${random_string.fqdn.result}-ssh"
166+
tags = var.tags
167167
}
168168

169169
resource "azurerm_network_interface" "jumpbox" {

‎examples/adb-squid-proxy/modules/vm/variables.tf

+2-2
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,6 @@ variable "vnetcidr" {
88
}
99

1010
variable "loc" {
11-
type = string
12-
default = "southeastasia"
11+
type = string
12+
default = "southeastasia"
1313
}

‎examples/adb-teradata/terraform.tfvars

+5-5
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
1-
spokecidr="10.179.0.0/20"
2-
no_public_ip=true
3-
rglocation="southeastasia"
4-
dbfs_prefix="dbfs"
5-
workspace_prefix="adb"
1+
spokecidr = "10.179.0.0/20"
2+
no_public_ip = true
3+
rglocation = "southeastasia"
4+
dbfs_prefix = "dbfs"
5+
workspace_prefix = "adb"

‎examples/aws-databricks-modular-privatelink/privatelink.tf

+12-12
Original file line numberDiff line numberDiff line change
@@ -80,26 +80,26 @@ resource "aws_security_group" "privatelink" {
8080
}
8181

8282
resource "aws_vpc_endpoint" "backend_rest" {
83-
vpc_id = aws_vpc.mainvpc.id
84-
service_name = var.workspace_vpce_service
85-
vpc_endpoint_type = "Interface"
86-
security_group_ids = [aws_security_group.privatelink.id]
87-
subnet_ids = aws_subnet.privatelink[*].id
83+
vpc_id = aws_vpc.mainvpc.id
84+
service_name = var.workspace_vpce_service
85+
vpc_endpoint_type = "Interface"
86+
security_group_ids = [aws_security_group.privatelink.id]
87+
subnet_ids = aws_subnet.privatelink[*].id
8888
private_dns_enabled = true // try to directly set this to true in the first apply
89-
depends_on = [aws_subnet.privatelink]
89+
depends_on = [aws_subnet.privatelink]
9090
tags = {
9191
Name = "${local.prefix}-databricks-backend-rest"
9292
}
9393
}
9494

9595
resource "aws_vpc_endpoint" "backend_relay" {
96-
vpc_id = aws_vpc.mainvpc.id
97-
service_name = var.relay_vpce_service
98-
vpc_endpoint_type = "Interface"
99-
security_group_ids = [aws_security_group.privatelink.id]
100-
subnet_ids = aws_subnet.privatelink[*].id
96+
vpc_id = aws_vpc.mainvpc.id
97+
service_name = var.relay_vpce_service
98+
vpc_endpoint_type = "Interface"
99+
security_group_ids = [aws_security_group.privatelink.id]
100+
subnet_ids = aws_subnet.privatelink[*].id
101101
private_dns_enabled = true
102-
depends_on = [aws_subnet.privatelink]
102+
depends_on = [aws_subnet.privatelink]
103103
tags = {
104104
Name = "${local.prefix}-databricks-backend-relay"
105105
}

‎examples/aws-databricks-uc/providers.tf

+1-1
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ provider "aws" {
1818
provider "databricks" {
1919
alias = "mws"
2020
host = "https://accounts.cloud.databricks.com"
21-
account_id = var.databricks_account_id // like a shared account? HA from multiple email accounts
21+
account_id = var.databricks_account_id // like a shared account? HA from multiple email accounts
2222
username = var.databricks_account_username
2323
password = var.databricks_account_password
2424
auth_type = "basic"

‎examples/aws-databricks-uc/workspace_metastore.tf

+3-3
Original file line numberDiff line numberDiff line change
@@ -81,23 +81,23 @@ resource "aws_iam_role" "external_data_access" {
8181

8282
resource "databricks_storage_credential" "external" {
8383
provider = databricks.ws1
84-
name = aws_iam_role.external_data_access.name
84+
name = aws_iam_role.external_data_access.name
8585
aws_iam_role {
8686
role_arn = aws_iam_role.external_data_access.arn
8787
}
8888
comment = "Managed by TF"
8989
}
9090

9191
resource "databricks_external_location" "some" {
92-
provider = databricks.ws1
92+
provider = databricks.ws1
9393
name = "external"
9494
url = "s3://${aws_s3_bucket.external.id}/some"
9595
credential_name = databricks_storage_credential.external.id
9696
comment = "Managed by TF"
9797
}
9898

9999
resource "databricks_grants" "some" {
100-
provider = databricks.ws1
100+
provider = databricks.ws1
101101
external_location = databricks_external_location.some.id
102102
grant {
103103
principal = "admin group A"

‎examples/gcp-basic/README.md

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Provisioning Databricks workspace on GCP with managed VPC
2+
=========================
3+
4+
In this template, we show how to deploy a workspace with managed VPC.
5+
6+
7+
## Requirements
8+
9+
- You need to have run gcp-sa-provisionning and have a service account to fill in the variables.
10+
- If you want to deploy to a new project, you will need to grant the custom role generated in that template to the service acount in the new project.
11+
- The Service Account needs to be added as Databricks Admin in the account console
12+
13+
## Run as an SA
14+
15+
You can do the same thing by provisionning a service account that will have the same permissions - and associate the key associated to it.
16+
17+
18+
## Run the tempalte
19+
20+
- You need to fill in the variables.tf
21+
- run `terraform init`
22+
- run `teraform apply`

‎examples/gcp-basic/init.tf

+42
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
terraform {
2+
required_providers {
3+
databricks = {
4+
source = "databricks/databricks"
5+
}
6+
google = {
7+
source = "hashicorp/google"
8+
}
9+
}
10+
}
11+
12+
provider "google" {
13+
project = var.google_project
14+
region = var.google_region
15+
zone = var.google_zone
16+
17+
}
18+
19+
20+
// initialize provider in "accounts" mode to provision new workspace
21+
22+
provider "databricks" {
23+
alias = "accounts"
24+
host = "https://accounts.gcp.databricks.com"
25+
google_service_account = var.databricks_google_service_account
26+
account_id = var.databricks_account_id
27+
28+
}
29+
30+
data "google_client_openid_userinfo" "me" {
31+
}
32+
33+
34+
data "google_client_config" "current" {
35+
}
36+
37+
38+
resource "random_string" "suffix" {
39+
special = false
40+
upper = false
41+
length = 6
42+
}

0 commit comments

Comments
 (0)