Skip to content

Commit edcb69a

Browse files
yessawabyessawab
authored
Feat: Adding new AWS modules to provision E2 workspaces using different configurations. And examples using these module (#35)
* docs:Updating main Readme * docs:Updating main Readme * docs:Updating main Readme * docs:Updating main Readme * feat:Push the terraform plan as a Pull request comment in ADO * feat:Push the terraform plan as a Pull request comment in ADO * feat:Push the terraform plan as a Pull request comment in ADO * feat:Push the terraform plan as a Pull request comment in ADO * feat:Push the terraform plan as a Pull request comment in ADO * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Adding an Lakehouse platform module on Azure * Feat: Adding a new module to provision Databricks on Azure with Private Link - Standard deployment. And an example using this module * Fix: Removing route_table_id variable as it is not used * Feat: Adding a new module for exfiltration protection. And an example using this module * Feat: Adding a new module for exfiltration protection. And an example using this module * Feat: Adding a new module for exfiltration protection. And an example using this module * Feat: Adding a new module for exfiltration protection with Azure private link. And an example using this module * Feat: Adding a new module for exfiltration protection with Azure private link. And an example using this module * Feat: Adding new AWS modules to provision E2 workspaces using different configurations. And examples using these module * Feat: Adding new AWS modules to provision E2 workspaces using different configurations. And examples using these module
1 parent 689b442 commit edcb69a

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

76 files changed

+1842
-35
lines changed

‎README.md

+8-1
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,10 @@ The folder `examples` contains the following Terraform implementation examples :
4242
| Azure | [adb-squid-proxy](examples/adb-squid-proxy/) | ADB clusters with HTTP proxy |
4343
| Azure | [adb-teradata](examples/adb-teradata/) | ADB with single VM Teradata integration |
4444
| Azure | [adb-uc](examples/adb-uc/) | ADB Unity Catalog Process |
45+
| AWS | [aws-workspace-basic](examples/aws-workspace-basic/) | Provisioning AWS Databricks E2 |
46+
| AWS | [aws-workspace-with-firewall](examples/aws-workspace-with-firewall/) | Provisioning AWS Databricks E2 with an AWS Firewall |
47+
| AWS | [aws-exfiltration-protection](examples/aws-exfiltration-protection/) | An implementation of [Data Exfiltration Protection on AWS](https://www.databricks.com/blog/2021/02/02/data-exfiltration-protection-with-databricks-on-aws.html) |
48+
| AWS | aws-workspace-with-private-link | Coming soon |
4549
| AWS | [aws-databricks-flat](examples/aws-databricks-flat/) | AWS Databricks simple example |
4650
| AWS | [aws-databricks-modular-privatelink](examples/aws-databricks-modular-privatelink/) | Deploy multiple AWS Databricks workspaces |
4751
| AWS | [aws-databricks-uc](examples/aws-databricks-uc/) | AWS UC |
@@ -61,7 +65,10 @@ The folder `modules` contains the following Terraform modules :
6165
| Azure | [adb-with-private-link-standard](modules/adb-with-private-link-standard/) | Provisioning Databricks on Azure with Private Link - Standard deployment |
6266
| Azure | [adb-exfiltration-protection](modules/adb-exfiltration-protection/) | A sample implementation of [Data Exfiltration Protection](https://www.databricks.com/blog/2020/03/27/data-exfiltration-protection-with-azure-databricks.html) |
6367
| Azure | [adb-with-private-links-exfiltration-protection](modules/adb-with-private-links-exfiltration-protection/) | Provisioning Databricks on Azure with Private Link and [Data Exfiltration Protection](https://www.databricks.com/blog/2020/03/27/data-exfiltration-protection-with-azure-databricks.html) |
64-
| AWS | Coming soon | |
68+
| AWS | [aws-workspace-basic](modules/aws-workspace-basic/) | Provisioning AWS Databricks E2 |
69+
| AWS | [aws-workspace-with-firewall](modules/aws-workspace-with-firewall/) | Provisioning AWS Databricks E2 with an AWS Firewall |
70+
| AWS | [aws-exfiltration-protection](modules/aws-exfiltration-protection/) | An implementation of [Data Exfiltration Protection on AWS](https://www.databricks.com/blog/2021/02/02/data-exfiltration-protection-with-databricks-on-aws.html) |
71+
| AWS | aws-workspace-with-private-link | Coming soon |
6572
| GCP | Coming soon | |
6673

6774
## CICD pipelines

‎examples/adb-exfiltration-protection/README.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ To find IP and FQDN for your deployment, go to: https://docs.microsoft.com/en-us
1111

1212
## Overall Architecture
1313

14-
![alt text](https://raw.githubusercontent.com/databricks/terraform-databricks-examples/main/examples/adb-exfiltration-protection/images/adb-exfiltration-classic.png?raw=true)
14+
![alt text](https://raw.githubusercontent.com/databricks/terraform-databricks-examples/main/modules/adb-exfiltration-protection/images/adb-exfiltration-classic.png?raw=true)
1515

1616
Resources to be created:
1717
* Resource group with random prefix
@@ -24,7 +24,7 @@ Resources to be created:
2424

2525
1. Update `terraform.tfvars` file and provide values to each defined variable
2626
2. (Optional) Configure your [remote backend](https://developer.hashicorp.com/terraform/language/settings/backends/azurerm)
27-
4. Run `terraform init` to initialize terraform and get provider ready.
27+
3. Run `terraform init` to initialize terraform and get provider ready.
2828
4. Run `terraform apply` to create the resources.
2929

3030
## How to fill in variable values

‎examples/adb-exfiltration-protection/images/adb-exfiltration-classic.png

-287 KB
Binary file not shown.

‎examples/adb-lakehouse/README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -24,5 +24,5 @@ This example can be used to deploy the following:
2424

2525
1. Update `terraform.tfvars` file and provide values to each defined variable
2626
2. (Optional) Configure your [remote backend](https://developer.hashicorp.com/terraform/language/settings/backends/azurerm)
27-
4. Run `terraform init` to initialize terraform and get provider ready.
27+
3. Run `terraform init` to initialize terraform and get provider ready.
2828
4. Run `terraform apply` to create the resources.

‎examples/adb-lakehouse/data.tf

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
data "azurerm_client_config" "current" {
2+
}

‎examples/adb-lakehouse/providers.tf

-4
This file was deleted.

‎examples/adb-with-private-link-standard/README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -22,5 +22,5 @@ This example can be used to deploy the following:
2222

2323
1. Update `terraform.tfvars` file and provide values to each defined variable
2424
2. (Optional) Configure your [remote backend](https://developer.hashicorp.com/terraform/language/settings/backends/azurerm)
25-
4. Run `terraform init` to initialize terraform and get provider ready.
25+
3. Run `terraform init` to initialize terraform and get provider ready.
2626
4. Run `terraform apply` to create the resources.

‎examples/adb-with-private-link-standard/providers.tf

-4
This file was deleted.

‎examples/adb-with-private-links-exfiltration-protection/README.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ Include:
1010
5. Private Endpoint for web-auth traffic.
1111

1212
Overall Architecture:
13-
![alt text](https://raw.githubusercontent.com/databricks/terraform-databricks-examples/main/examples/adb-with-private-links-exfiltration-protection/images/adb-private-links-general.png?raw=true)
13+
![alt text](https://raw.githubusercontent.com/databricks/terraform-databricks-examples/main/modules/adb-with-private-links-exfiltration-protection/images/adb-private-links-general.png?raw=true)
1414

1515
With this deployment, traffic from user client to webapp (notebook UI), backend traffic from data plane to control plane will be through private endpoints. This terraform sample will create:
1616
* Resource group with random prefix
@@ -23,5 +23,5 @@ With this deployment, traffic from user client to webapp (notebook UI), backend
2323

2424
1. Update `terraform.tfvars` file and provide values to each defined variable
2525
2. (Optional) Configure your [remote backend](https://developer.hashicorp.com/terraform/language/settings/backends/azurerm)
26-
4. Run `terraform init` to initialize terraform and get provider ready.
26+
3. Run `terraform init` to initialize terraform and get provider ready.
2727
4. Run `terraform apply` to create the resources.

‎examples/adb-with-private-links-exfiltration-protection/images/adb-private-links-general.png

-504 KB
Binary file not shown.

‎examples/aws-exfiltration-protection/README.md

+26
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# Provisioning Azure Databricks workspace with a Hub & Spoke firewall for data exfiltration protection
2+
3+
This example is using the [aws-exfiltration-protection](../../modules/aws-exfiltration-protection) module.
4+
5+
This template provides an example deployment of AWS Databricks E2 workspace with a Hub & Spoke firewall for data exfiltration protection. Details are described in [Data Exfiltration Protection With Databricks on AWS](https://www.databricks.com/blog/2021/02/02/data-exfiltration-protection-with-databricks-on-aws.html).
6+
7+
## Overall Architecture
8+
9+
![alt text](https://raw.githubusercontent.com/databricks/terraform-databricks-examples/main/modules/aws-exfiltration-protection/images/aws-exfiltration-classic.png?raw=true)
10+
11+
## How to use
12+
13+
> **Note**
14+
> If you are using AWS Firewall to block most traffic but allow the URLs that Databricks needs to connect to, please update the configuration based on your region. You can get the configuration details for your region from [Firewall Appliance](https://docs.databricks.com/administration-guide/cloud-configurations/aws/customer-managed-vpc.html#firewall-appliance-infrastructure) document.
15+
16+
1. Reference this module using one of the different [module source types](https://developer.hashicorp.com/terraform/language/modules/sources)
17+
2. Add a `variables.tf` with the same content in [variables.tf](variables.tf)
18+
3. Add a `terraform.tfvars` file and provide values to each defined variable
19+
4. Configure the following environment variables:
20+
* TF_VAR_databricks_account_username, set to the value of your Databricks account-level admin username.
21+
* TF_VAR_databricks_account_password, set to the value of the password for your Databricks account-level admin user.
22+
* TF_VAR_databricks_account_id, set to the value of the ID of your Databricks account. You can find this value in the corner of your Databricks account console.
23+
5. Add a `output.tf` file.
24+
6. (Optional) Configure your [remote backend](https://developer.hashicorp.com/terraform/language/settings/backends/s3)
25+
7. Run `terraform init` to initialize terraform and get provider ready.
26+
8. Run `terraform apply` to create the resources.

‎examples/aws-exfiltration-protection/main.tf

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
module "aws-exfiltration-protection" {
2+
source = "github.com/databricks/terraform-databricks-examples/modules/aws-exfiltration-protection"
3+
databricks_account_id = var.databricks_account_id
4+
databricks_account_username = var.databricks_account_username
5+
databricks_account_password = var.databricks_account_password
6+
}

‎examples/aws-exfiltration-protection/terraform.tfvars

Whitespace-only changes.

‎examples/aws-exfiltration-protection/variables.tf

+46
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
variable "databricks_account_username" {}
2+
variable "databricks_account_password" {}
3+
variable "databricks_account_id" {}
4+
5+
variable "tags" {
6+
default = {}
7+
}
8+
9+
variable "spoke_cidr_block" {
10+
default = "10.173.0.0/16"
11+
}
12+
variable "hub_cidr_block" {
13+
default = "10.10.0.0/16"
14+
}
15+
variable "region" {
16+
default = "eu-central-1"
17+
}
18+
19+
resource "random_string" "naming" {
20+
special = false
21+
upper = false
22+
length = 6
23+
}
24+
variable "whitelisted_urls" {
25+
default = [".pypi.org", ".pythonhosted.org", ".cran.r-project.org"]
26+
}
27+
28+
variable "db_web_app" {
29+
default = "frankfurt.cloud.databricks.com"
30+
}
31+
32+
variable "db_tunnel" {
33+
default = "tunnel.eu-central-1.cloud.databricks.com"
34+
}
35+
36+
variable "db_rds" {
37+
default = "mdv2llxgl8lou0.ceptxxgorjrc.eu-central-1.rds.amazonaws.com"
38+
}
39+
40+
variable "db_control_plane" {
41+
default = "18.159.44.32/28"
42+
}
43+
44+
variable "prefix" {
45+
default = "demo"
46+
}

‎examples/aws-exfiltration-protection/versions.tf

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# versions.tf
2+
terraform {
3+
required_providers {
4+
databricks = {
5+
source = "databricks/databricks"
6+
version = ">=1.13.0"
7+
}
8+
9+
aws = {
10+
source = "hashicorp/aws"
11+
version = "~> 4.58.0"
12+
}
13+
}
14+
}

‎examples/aws-workspace-basic/README.md

+23
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Provisioning AWS Databricks E2
2+
3+
This example is using the [aws-workspace-basic](../../modules/aws-workspace-basic) module.
4+
5+
This template provides an example of a simple deployment of AWS Databricks E2 workspace.
6+
7+
## Overall Architecture
8+
9+
![alt text](https://raw.githubusercontent.com/databricks/terraform-databricks-examples/main/modules/aws-workspace-basic/images/aws-workspace-basic.png?raw=true)
10+
11+
## How to use
12+
13+
1. Reference this module using one of the different [module source types](https://developer.hashicorp.com/terraform/language/modules/sources)
14+
2. Add a `variables.tf` with the same content in [variables.tf](variables.tf)
15+
3. Add a `terraform.tfvars` file and provide values to each defined variable
16+
4. Configure the following environment variables:
17+
* TF_VAR_databricks_account_username, set to the value of your Databricks account-level admin username.
18+
* TF_VAR_databricks_account_password, set to the value of the password for your Databricks account-level admin user.
19+
* TF_VAR_databricks_account_id, set to the value of the ID of your Databricks account. You can find this value in the corner of your Databricks account console.
20+
5. Add a `output.tf` file.
21+
6. (Optional) Configure your [remote backend](https://developer.hashicorp.com/terraform/language/settings/backends/s3)
22+
7. Run `terraform init` to initialize terraform and get provider ready.
23+
8. Run `terraform apply` to create the resources.

‎examples/aws-workspace-basic/main.tf

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
module "aws-workspace-basic" {
2+
source = "github.com/databricks/terraform-databricks-examples/modules/aws-workspace-basic"
3+
databricks_account_id = var.databricks_account_id
4+
databricks_account_username = var.databricks_account_username
5+
databricks_account_password = var.databricks_account_password
6+
}

‎examples/aws-workspace-basic/terraform.tfvars

Whitespace-only changes.

‎examples/aws-workspace-basic/variables.tf

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
variable "databricks_account_username" {}
2+
variable "databricks_account_password" {}
3+
variable "databricks_account_id" {}
4+
5+
variable "tags" {
6+
default = {}
7+
}
8+
9+
variable "cidr_block" {
10+
default = "10.4.0.0/16"
11+
}
12+
13+
variable "region" {
14+
default = "eu-west-1"
15+
}

‎examples/aws-workspace-basic/versions.tf

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# versions.tf
2+
terraform {
3+
required_providers {
4+
databricks = {
5+
source = "databricks/databricks"
6+
version = ">=1.13.0"
7+
}
8+
9+
aws = {
10+
source = "hashicorp/aws"
11+
version = "~> 4.58.0"
12+
}
13+
}
14+
}

‎examples/aws-workspace-with-firewall/README.md

+23
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Provisioning AWS Databricks E2 with an AWS Firewall
2+
3+
This example is using the [aws-workspace-with-firewall](../../modules/aws-workspace-with-firewall) module.
4+
5+
This template provides an example of a simple deployment of AWS Databricks E2 workspace with an AWS Firewall.
6+
7+
## Overall Architecture
8+
9+
![alt text](https://raw.githubusercontent.com/databricks/terraform-databricks-examples/main/modules/aws-workspace-with-firewall/images/aws-workspace-with-firewall.png?raw=true)
10+
11+
## How to use
12+
13+
1. Reference this module using one of the different [module source types](https://developer.hashicorp.com/terraform/language/modules/sources)
14+
2. Add a `variables.tf` with the same content in [variables.tf](variables.tf)
15+
3. Add a `terraform.tfvars` file and provide values to each defined variable
16+
4. Configure the following environment variables:
17+
* TF_VAR_databricks_account_username, set to the value of your Databricks account-level admin username.
18+
* TF_VAR_databricks_account_password, set to the value of the password for your Databricks account-level admin user.
19+
* TF_VAR_databricks_account_id, set to the value of the ID of your Databricks account. You can find this value in the corner of your Databricks account console.
20+
5. Add a `output.tf` file.
21+
6. (Optional) Configure your [remote backend](https://developer.hashicorp.com/terraform/language/settings/backends/s3)
22+
7. Run `terraform init` to initialize terraform and get provider ready.
23+
8. Run `terraform apply` to create the resources.

‎examples/aws-workspace-with-firewall/main.tf

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
module "aws-workspace-with-firewall" {
2+
source = "github.com/databricks/terraform-databricks-examples/modules/aws-workspace-with-firewall"
3+
databricks_account_id = var.databricks_account_id
4+
databricks_account_username = var.databricks_account_username
5+
databricks_account_password = var.databricks_account_password
6+
}

‎examples/aws-workspace-with-firewall/terraform.tfvars

Whitespace-only changes.

‎examples/aws-workspace-with-firewall/variables.tf

+45
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
variable "databricks_account_username" {}
2+
variable "databricks_account_password" {}
3+
variable "databricks_account_id" {}
4+
5+
variable "tags" {
6+
default = {}
7+
}
8+
9+
variable "cidr_block" {
10+
default = "10.4.0.0/16"
11+
}
12+
13+
variable "region" {
14+
default = "eu-west-2"
15+
}
16+
17+
resource "random_string" "naming" {
18+
special = false
19+
upper = false
20+
length = 6
21+
}
22+
23+
variable "whitelisted_urls" {
24+
default = [".pypi.org", ".pythonhosted.org", ".cran.r-project.org"]
25+
}
26+
27+
variable "db_web_app" {
28+
default = "london.cloud.databricks.com"
29+
}
30+
31+
variable "db_tunnel" {
32+
default = "tunnel.eu-west-2.cloud.databricks.com"
33+
}
34+
35+
variable "db_rds" {
36+
default = "mdio2468d9025m.c6fvhwk6cqca.eu-west-2.rds.amazonaws.com"
37+
}
38+
39+
variable "db_control_plane" {
40+
default = "18.134.65.240/28"
41+
}
42+
43+
variable "prefix" {
44+
default = "demo"
45+
}

‎examples/aws-workspace-with-firewall/versions.tf

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# versions.tf
2+
terraform {
3+
required_providers {
4+
databricks = {
5+
source = "databricks/databricks"
6+
version = ">=1.13.0"
7+
}
8+
9+
aws = {
10+
source = "hashicorp/aws"
11+
version = "~> 4.58.0"
12+
}
13+
}
14+
}

‎modules/adb-exfiltration-protection/README.md

+5-5
Original file line numberDiff line numberDiff line change
@@ -26,11 +26,11 @@ Resources to be created:
2626
2727
1. Reference this module using one of the different [module source types](https://developer.hashicorp.com/terraform/language/modules/sources)
2828
2. Add a `variables.tf` with the same content in [variables.tf](variables.tf)
29-
2. Add a `terraform.tfvars` file and provide values to each defined variable
30-
3. Add a `output.tf` file.
31-
4. (Optional) Configure your [remote backend](https://developer.hashicorp.com/terraform/language/settings/backends/azurerm)
32-
4. Run `terraform init` to initialize terraform and get provider ready.
33-
4. Run `terraform apply` to create the resources.
29+
3. Add a `terraform.tfvars` file and provide values to each defined variable
30+
4. Add a `output.tf` file.
31+
5. (Optional) Configure your [remote backend](https://developer.hashicorp.com/terraform/language/settings/backends/azurerm)
32+
6. Run `terraform init` to initialize terraform and get provider ready.
33+
7. Run `terraform apply` to create the resources.
3434

3535

3636
## How to fill in variable values

‎modules/adb-lakehouse/README.md

+5-5
Original file line numberDiff line numberDiff line change
@@ -28,8 +28,8 @@ This module can be used to deploy the following:
2828
2929
1. Reference this module using one of the different [module source types](https://developer.hashicorp.com/terraform/language/modules/sources)
3030
2. Add a `variables.tf` with the same content in [variables.tf](variables.tf)
31-
2. Add a `terraform.tfvars` file and provide values to each defined variable
32-
3. Add a `output.tf` file.
33-
4. (Optional) Configure your [remote backend](https://developer.hashicorp.com/terraform/language/settings/backends/azurerm)
34-
4. Run `terraform init` to initialize terraform and get provider ready.
35-
4. Run `terraform apply` to create the resources.
31+
3. Add a `terraform.tfvars` file and provide values to each defined variable
32+
4. Add a `output.tf` file.
33+
5. (Optional) Configure your [remote backend](https://developer.hashicorp.com/terraform/language/settings/backends/azurerm)
34+
6. Run `terraform init` to initialize terraform and get provider ready.
35+
7. Run `terraform apply` to create the resources.

‎modules/adb-lakehouse/main.tf

+2
Original file line numberDiff line numberDiff line change
@@ -4,3 +4,5 @@ resource "azurerm_resource_group" "this" {
44
tags = var.tags
55
}
66

7+
data "azurerm_client_config" "current" {
8+
}

0 commit comments

Comments
 (0)