Skip to content

Commit b07bada

Browse files
r3stl355r3stl355
authored
Update adb-exfiltration-protection to use azurerm v4 (#150)
* Update adb-exfiltration-protection to use azurerm 4 Signed-off-by: Niko <[email protected]> * restore previous outputs, add descriptions, marked as deprecated Signed-off-by: Niko <[email protected]> --------- Signed-off-by: Niko <[email protected]>
1 parent ac23fa7 commit b07bada

File tree

12 files changed

+111
-134
lines changed

12 files changed

+111
-134
lines changed

‎examples/adb-exfiltration-protection/README.md

Lines changed: 12 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -22,13 +22,15 @@ Resources to be created:
2222

2323
## How to use
2424

25-
1. Update `terraform.tfvars` file and provide values to each defined variable
25+
1. Update `terraform.tfvars` file and provide values to each defined variable.
2626
2. (Optional) Configure your [remote backend](https://developer.hashicorp.com/terraform/language/settings/backends/azurerm)
2727
3. Run `terraform init` to initialize terraform and get provider ready.
2828
4. Run `terraform apply` to create the resources.
2929

3030
## How to fill in variable values
3131

32+
Some variables have no default value and will require one, e.g. `subscription_id`
33+
3234
Most of the values are to be found at: https://docs.microsoft.com/en-us/azure/databricks/administration-guide/cloud-configurations/azure/udr
3335

3436
In `variables.tfvars`, set these variables:
@@ -47,16 +49,17 @@ firewallfqdn = ["dbartifactsprodseap.blob.core.windows.net","dbartifactsprodeap.
4749

4850
| Name | Version |
4951
| ---------------------------------------------------------------------------- | ------- |
50-
| <a name="requirement_azurerm"></a> [azurerm](#requirement\_azurerm) | =2.83.0 |
51-
| <a name="requirement_databricks"></a> [databricks](#requirement\_databricks) | 0.3.10 |
52+
| <a name="requirement_azurerm"></a> [azurerm](#requirement\_azurerm) | >=4.0.0 |
53+
| <a name="requirement_databricks"></a> [databricks](#requirement\_databricks) | >=1.52.0|
5254

5355
## Providers
5456

5557
| Name | Version |
5658
| ---------------------------------------------------------------- | ------- |
57-
| <a name="provider_azurerm"></a> [azurerm](#provider\_azurerm) | 2.83.0 |
58-
| <a name="provider_external"></a> [external](#provider\_external) | 2.2.0 |
59-
| <a name="provider_random"></a> [random](#provider\_random) | 3.1.0 |
59+
| <a name="provider_azurerm"></a> [azurerm](#provider\_azurerm) | 4.9.0 |
60+
| <a name="provider_external"></a> [external](#provider\_external) | 1.58.0 |
61+
| <a name="provider_random"></a> [random](#provider\_random) | 3.6.3 |
62+
| <a name="provider_dns"></a> [dns](#provider\_dns) | 3.4.2 |
6063

6164
## Modules
6265

@@ -95,11 +98,11 @@ No modules.
9598

9699
| Name | Description | Type | Default | Required |
97100
| -------------------------------------------------------------------------------------------------------------- | ----------- | ----------- | ----------------- | :------: |
101+
| <a name="input_subscription_id"></a> [subscription\_id](#input\_subscription\_id) | n/a | `string` | n/a | yes |
98102
| <a name="input_dbfs_prefix"></a> [dbfs\_prefix](#input\_dbfs\_prefix) | n/a | `string` | `"dbfs"` | no |
99103
| <a name="input_firewallfqdn"></a> [firewallfqdn](#input\_firewallfqdn) | n/a | `list(any)` | n/a | yes |
100104
| <a name="input_hubcidr"></a> [hubcidr](#input\_hubcidr) | n/a | `string` | `"10.178.0.0/20"` | no |
101105
| <a name="input_metastoreip"></a> [metastoreip](#input\_metastoreip) | n/a | `string` | n/a | yes |
102-
| <a name="input_no_public_ip"></a> [no\_public\_ip](#input\_no\_public\_ip) | n/a | `bool` | `true` | no |
103106
| <a name="input_private_subnet_endpoints"></a> [private\_subnet\_endpoints](#input\_private\_subnet\_endpoints) | n/a | `list` | `[]` | no |
104107
| <a name="input_rglocation"></a> [rglocation](#input\_rglocation) | n/a | `string` | `"southeastasia"` | no |
105108
| <a name="input_sccip"></a> [sccip](#input\_sccip) | n/a | `string` | n/a | yes |
@@ -111,11 +114,7 @@ No modules.
111114

112115
| Name | Description |
113116
| -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
114-
| <a name="output_arm_client_id"></a> [arm\_client\_id](#output\_arm\_client\_id) | n/a |
115-
| <a name="output_arm_subscription_id"></a> [arm\_subscription\_id](#output\_arm\_subscription\_id) | n/a |
116-
| <a name="output_arm_tenant_id"></a> [arm\_tenant\_id](#output\_arm\_tenant\_id) | n/a |
117-
| <a name="output_azure_region"></a> [azure\_region](#output\_azure\_region) | n/a |
118-
| <a name="output_databricks_azure_workspace_resource_id"></a> [databricks\_azure\_workspace\_resource\_id](#output\_databricks\_azure\_workspace\_resource\_id) | n/a |
119-
| <a name="output_resource_group"></a> [resource\_group](#output\_resource\_group) | n/a |
117+
| <a name="output_azure_resource_group_id"></a> [azure\_resource\_group\_id](#output\_azure\_resource\_group\_id) | n/a |
118+
| <a name="output_workspace_id"></a> [workspace\_id](#output\_workspace\_id) | n/a |
120119
| <a name="output_workspace_url"></a> [workspace\_url](#output\_workspace\_url) | n/a |
121120
<!-- END_TF_DOCS -->

‎examples/adb-exfiltration-protection/main.tf

Lines changed: 0 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,7 @@
1-
/**
2-
* Azure Databricks workspace in custom VNet with traffic routed via firewall in the Hub VNet
3-
*
4-
* Module creates:
5-
* * Resource group with random prefix
6-
* * Tags, including `Owner`, which is taken from `az account show --query user`
7-
* * VNet with public and private subnet for Databricks
8-
* * VNet with subnet for deployment of Azure Firewall
9-
* * Azure Firewall with access enabled to Databricks-related resources
10-
* * Databricks workspace
11-
*/
12-
131
module "adb-exfiltration-protection" {
142
source = "../../modules/adb-exfiltration-protection"
153
hubcidr = var.hubcidr
164
spokecidr = var.spokecidr
17-
no_public_ip = var.no_public_ip
185
rglocation = var.rglocation
196
metastore = var.metastore
207
scc_relay = var.scc_relay

‎examples/adb-exfiltration-protection/outputs.tf

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
output "azure_resource_group_id" {
2+
description = "ID of the created Azure resource group"
3+
value = module.adb-exfiltration-protection.azure_resource_group_id
4+
}
5+
6+
output "workspace_id" {
7+
description = "The Databricks workspace ID"
8+
value = module.adb-exfiltration-protection.workspace_id
9+
}
10+
11+
output "workspace_url" {
12+
description = "The Databricks workspace URL"
13+
value = module.adb-exfiltration-protection.workspace_url
14+
}

‎modules/adb-exfiltration-protection/versions.tf renamed to ‎examples/adb-exfiltration-protection/providers.tf

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,12 @@
1-
# versions.tf
21
terraform {
32
required_providers {
4-
databricks = {
5-
source = "databricks/databricks"
6-
version = ">=1.20.0"
7-
}
83
azurerm = {
94
source = "hashicorp/azurerm"
10-
version = ">=2.83.0"
5+
version = ">=4.0.0"
6+
}
7+
databricks = {
8+
source = "databricks/databricks"
9+
version = ">=1.52.0"
1110
}
1211
random = {
1312
source = "hashicorp/random"
@@ -17,3 +16,8 @@ terraform {
1716
}
1817
}
1918
}
19+
20+
provider "azurerm" {
21+
subscription_id = var.subscription_id
22+
features {}
23+
}

‎examples/adb-exfiltration-protection/terraform.tfvars

Lines changed: 13 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,14 @@
1-
hubcidr = "10.178.0.0/20"
2-
spokecidr = "10.179.0.0/20"
3-
no_public_ip = true
4-
rglocation = "westeurope"
1+
subscription_id = "<your Azure Subscription ID here>"
2+
dbfs_prefix = "dbfs"
3+
workspace_prefix = "adb"
4+
hubcidr = "10.178.0.0/20"
5+
spokecidr = "10.179.0.0/20"
6+
rglocation = "westeurope"
7+
58
# We can pull this information automatically, i.e. from
69
# https://github.com/microsoft/AzureTRE/blob/main/templates/workspace_services/databricks/terraform/databricks-udr.json
710
# that is maintained by Microsoft team (although it may not be updated immediately).
8-
metastore = [
11+
metastore = [
912
"consolidated-westeurope-prod-metastore.mysql.database.azure.com",
1013
"consolidated-westeurope-prod-metastore-addl-1.mysql.database.azure.com",
1114
"consolidated-westeurope-prod-metastore-addl-2.mysql.database.azure.com",
@@ -15,24 +18,23 @@ metastore = [
1518
"consolidated-westeuropec2-prod-metastore-2.mysql.database.azure.com",
1619
"consolidated-westeuropec2-prod-metastore-3.mysql.database.azure.com",
1720
]
21+
1822
// get from https://learn.microsoft.com/en-us/azure/databricks/resources/supported-regions#--metastore-artifact-blob-storage-system-tables-blob-storage-log-blob-storage-and-event-hub-endpoint-ip-addresses
19-
scc_relay = [
23+
scc_relay = [
2024
"tunnel.westeurope.azuredatabricks.net",
2125
"tunnel.westeuropec2.azuredatabricks.net"
2226
]
23-
webapp_ips = [
27+
webapp_ips = [
2428
"52.232.19.246/32",
2529
"40.74.30.80/32",
2630
"20.103.219.240/28",
2731
"4.150.168.160/28",
2832
]
29-
eventhubs = [
33+
eventhubs = [
3034
"prod-westeurope-observabilityeventhubs.servicebus.windows.net",
3135
"prod-westeuc2-observabilityeventhubs.servicebus.windows.net",
3236
]
33-
dbfs_prefix = "dbfs"
34-
workspace_prefix = "adb"
35-
firewallfqdn = [ // dbfs rule will be added - depends on dbfs storage name
37+
firewallfqdn = [ // dbfs rule will be added - depends on dbfs storage name
3638
"dbartifactsprodwesteu.blob.core.windows.net", //databricks artifacts
3739
"arprodwesteua1.blob.core.windows.net",
3840
"arprodwesteua2.blob.core.windows.net",

‎examples/adb-exfiltration-protection/variables.tf

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,8 @@
1+
variable "subscription_id" {
2+
type = string
3+
description = "Azure Subscription ID to deploy the workspace into"
4+
}
5+
16
variable "hubcidr" {
27
description = "IP range for creaiton of the Spoke VNet"
38
type = string
@@ -10,12 +15,6 @@ variable "spokecidr" {
1015
default = "10.179.0.0/20"
1116
}
1217

13-
variable "no_public_ip" {
14-
description = "If workspace should be created with No-Public-IP"
15-
type = bool
16-
default = true
17-
}
18-
1918
variable "rglocation" {
2019
description = "Location of resource group"
2120
type = string

‎modules/adb-exfiltration-protection/README.md

Lines changed: 11 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,13 @@
11
# Provisioning Azure Databricks workspace with a Hub & Spoke firewall for data exfiltration protection
22

3-
This template provides an example deployment of: Hub-Spoke networking with egress firewall to control all outbound traffic from Databricks subnets. Details are described in: https://databricks.com/blog/2020/03/27/data-exfiltration-protection-with-azure-databricks.html
3+
This module will create Azure Databricks workspace with a Hub & Spoke firewall for data exfiltration protection.
44

5-
With this setup, you can setup firewall rules to block / allow egress traffic from your Databricks clusters. You can also use firewall to block all access to storage accounts, and use private endpoint connection to bypass this firewall, such that you allow access only to specific storage accounts.
5+
## Module content
66

7-
8-
To find IP and FQDN for your deployment, go to: https://docs.microsoft.com/en-us/azure/databricks/administration-guide/cloud-configurations/azure/udr
9-
10-
## Overall Architecture
7+
This module can be used to deploy the following:
118

129
![alt text](https://raw.githubusercontent.com/databricks/terraform-databricks-examples/main/modules/adb-exfiltration-protection/images/adb-exfiltration-classic.png?raw=true)
1310

14-
Resources to be created:
1511
* Resource group with random prefix
1612
* Tags, including `Owner`, which is taken from `az account show --query user`
1713
* Hub-Spoke topology, with hub firewall in hub vnet's subnet.
@@ -32,22 +28,6 @@ Resources to be created:
3228
6. Run `terraform init` to initialize terraform and get provider ready.
3329
7. Run `terraform apply` to create the resources.
3430

35-
36-
## How to fill in variable values
37-
38-
Most of the values are to be found at: https://learn.microsoft.com/en-us/azure/databricks/resources/supported-regions and https://docs.microsoft.com/en-us/azure/databricks/administration-guide/cloud-configurations/azure/udr
39-
40-
In `variables.tfvars`, set these variables (bigger regions have multiple instances of each service):
41-
42-
```hcl
43-
metastore = ["consolidated-westeurope-prod-metastore.mysql.database.azure.com"]
44-
scc_relay = ["tunnel.westeurope.azuredatabricks.net"]
45-
webapp_ips = ["52.230.27.216/32"] # given at UDR page
46-
eventhubs = ["prod-westeurope-observabilityeventhubs.servicebus.windows.net"]
47-
# find these for your region, follow Databricks blog tutorial.
48-
firewallfqdn = ["dbartifactsprodseap.blob.core.windows.net","dbartifactsprodeap.blob.core.windows.net","dblogprodseasia.blob.core.windows.net","cdnjs.com"]
49-
```
50-
5131
<!-- BEGIN_TF_DOCS -->
5232
## Requirements
5333

@@ -121,11 +101,13 @@ No modules.
121101

122102
| Name | Description |
123103
| -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
124-
| <a name="output_arm_client_id"></a> [arm\_client\_id](#output\_arm\_client\_id) | n/a |
125-
| <a name="output_arm_subscription_id"></a> [arm\_subscription\_id](#output\_arm\_subscription\_id) | n/a |
126-
| <a name="output_arm_tenant_id"></a> [arm\_tenant\_id](#output\_arm\_tenant\_id) | n/a |
127-
| <a name="output_azure_region"></a> [azure\_region](#output\_azure\_region) | n/a |
128-
| <a name="output_databricks_azure_workspace_resource_id"></a> [databricks\_azure\_workspace\_resource\_id](#output\_databricks\_azure\_workspace\_resource\_id) | n/a |
129-
| <a name="output_resource_group"></a> [resource\_group](#output\_resource\_group) | n/a |
104+
| <a name="output_arm_client_id"></a> [arm\_client\_id](#output\_arm\_client\_id) | Deprecated |
105+
| <a name="output_arm_subscription_id"></a> [arm\_subscription\_id](#output\_arm\_subscription\_id) | Deprecated |
106+
| <a name="output_arm_tenant_id"></a> [arm\_tenant\_id](#output\_arm\_tenant\_id) | Deprecated |
107+
| <a name="output_azure_region"></a> [azure\_region](#output\_azure\_region) | Deprecated |
108+
| <a name="output_databricks_azure_workspace_resource_id"></a> [databricks\_azure\_workspace\_resource\_id](#output\_databricks\_azure\_workspace\_resource\_id) | Deprecated |
109+
| <a name="output_resource_group"></a> [resource\_group](#output\_resource\_group) | Deprecated |
130110
| <a name="output_workspace_url"></a> [workspace\_url](#output\_workspace\_url) | n/a |
111+
| <a name="output_resource_group_id"></a> [resource\_group\_id](#output\_resource\_group\_id) | n/a |
112+
| <a name="output_workspace_id"></a> [resource\_workspace\_id](#output\_resource\_workspace\_id) | n/a |
131113
<!-- END_TF_DOCS -->

‎modules/adb-exfiltration-protection/main.tf

Lines changed: 0 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,3 @@
1-
/**
2-
* Azure Databricks workspace in custom VNet
3-
*
4-
* Module creates:
5-
* * Resource group with random prefix
6-
* * Tags, including `Owner`, which is taken from `az account show --query user`
7-
* * VNet with public and private subnet
8-
* * Databricks workspace
9-
*/
10-
provider "azurerm" {
11-
features {}
12-
}
13-
141
resource "random_string" "naming" {
152
special = false
163
upper = false
@@ -44,23 +31,3 @@ resource "azurerm_resource_group" "this" {
4431
location = local.location
4532
tags = local.tags
4633
}
47-
48-
output "arm_client_id" {
49-
value = data.azurerm_client_config.current.client_id
50-
}
51-
52-
output "arm_subscription_id" {
53-
value = data.azurerm_client_config.current.subscription_id
54-
}
55-
56-
output "arm_tenant_id" {
57-
value = data.azurerm_client_config.current.tenant_id
58-
}
59-
60-
output "azure_region" {
61-
value = local.location
62-
}
63-
64-
output "resource_group" {
65-
value = azurerm_resource_group.this.name
66-
}

‎modules/adb-exfiltration-protection/outputs.tf

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
output "databricks_azure_workspace_resource_id" {
2+
description = "**Deprecated** The ID of the Databricks Workspace in the Azure management plane"
3+
value = azurerm_databricks_workspace.this.id
4+
}
5+
6+
output "arm_client_id" {
7+
description = "**Deprecated**"
8+
value = data.azurerm_client_config.current.client_id
9+
}
10+
11+
output "arm_subscription_id" {
12+
description = "**Deprecated**"
13+
value = data.azurerm_client_config.current.subscription_id
14+
}
15+
16+
output "arm_tenant_id" {
17+
description = "**Deprecated**"
18+
value = data.azurerm_client_config.current.tenant_id
19+
}
20+
21+
output "azure_region" {
22+
description = "**Deprecated**"
23+
value = local.location
24+
}
25+
26+
output "resource_group" {
27+
description = "**Deprecated**"
28+
value = azurerm_resource_group.this.name
29+
}
30+
31+
output "workspace_url" {
32+
description = "The Databricks workspace URL"
33+
value = "https://${azurerm_databricks_workspace.this.workspace_url}/"
34+
}
35+
36+
output "azure_resource_group_id" {
37+
description = "ID of the created Azure resource group"
38+
value = azurerm_resource_group.this.id
39+
}
40+
41+
output "workspace_id" {
42+
description = "The Databricks workspace ID"
43+
value = azurerm_databricks_workspace.this.workspace_id
44+
}

‎examples/adb-exfiltration-protection/versions.tf renamed to ‎modules/adb-exfiltration-protection/providers.tf

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,12 @@
1-
# versions.tf
21
terraform {
32
required_providers {
43
databricks = {
54
source = "databricks/databricks"
6-
version = ">=1.20.0"
5+
version = ">=1.52.0"
76
}
87
azurerm = {
98
source = "hashicorp/azurerm"
10-
version = ">=2.83.0"
9+
version = ">=4.0.0"
1110
}
1211
random = {
1312
source = "hashicorp/random"

‎modules/adb-exfiltration-protection/variables.tf

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -10,12 +10,6 @@ variable "spokecidr" {
1010
default = "10.179.0.0/20"
1111
}
1212

13-
variable "no_public_ip" {
14-
description = "If workspace should be created with No-Public-IP"
15-
type = bool
16-
default = true
17-
}
18-
1913
variable "rglocation" {
2014
description = "Location of resource group"
2115
type = string

0 commit comments

Comments
 (0)