Skip to content

Commit c57346b

Browse files
sfc-gh-jmichalaksfc-gh-jmichalak
authored
chore: Add basic performance tests (#3349)
<!-- Feel free to delete comments as you fill this in --> - Add basic performance tests in the manual tests package. - Include a tf lock file in gitignore. - Update the state of resource rework. <!-- summary of changes --> ## Test Plan <!-- detail ways in which this PR has been tested or needs to be tested --> * [ ] acceptance tests <!-- add more below if you think they are relevant --> * [ ] … ## References #3118 #3169 ## TODO <!-- issues documentation links, etc --> - Publish the summary document.
1 parent beb01c7 commit c57346b

File tree

10 files changed

+336
-2
lines changed

10 files changed

+336
-2
lines changed

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ terraform-provider-snowflake*
1717
.DS_Store
1818
*tfstate*
1919
.terraform
20+
.terraform.lock.hcl
2021
crash.log
2122
.envrc
2223
bin

‎pkg/manual_tests/README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,3 +7,4 @@ and should contain a file describing the manual steps to perform the test.
77
Here's the list of cases we currently cannot reproduce and write acceptance tests for:
88
- `user_default_database_and_role`: Setting up a user with default_namespace and default_role, then logging into that user to see what happens with those values in various scenarios (e.g. insufficient privileges on the role).
99
- `authentication_methods`: Some of the authentication methods require manual steps, like confirming MFA or setting more dependencies.
10+
- `benchmarks`: Performance benchmarks require manually running `terraform` command to imitate the user workflow.

‎pkg/manual_tests/benchmarks/README.md

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# Authentication methods manual tests
2+
3+
This directory is dedicated to hold steps for manual performance tests in the provider. These tests use simple Terraform files and are run with `terraform` CLI manually to imitate the user workflow and reduce bias with Terraform SDK testing libraries and the binary itself.
4+
The tests are organized by the resource type, e.g. schemas, tasks, and warehouses.
5+
6+
## Run tests
7+
8+
- Preferably use your secondary test account to avoid potential conflicts with our "main" environments.
9+
- Configure the needed modules in `main.tf`. The attribute `resource_count` is a count of the resources of a given type and configuration. Note that inside the modules there are resources with different configurations (i.e. only required fields set, all fields set). This means that the total number of resources may be bigger. For example, if `resource_count` is 100 and you are testing 2 different configurations using the `resource_count`, total number of resources is 200. If you do not want to uses resource from a module, simply set `resource_count` to 0.
10+
- If you want to test different resource configurations, adjust them in the relevant module.
11+
- Run `terraform init -upgrade` to enable the modules.
12+
- Run regular Terraform commands, like `terraform apply`.
13+
- The top-level objects names contain test ID and resource index, utilizing the format like `PERFORMANCE_TESTS_BED9310F_F8CE_D2CD_D4B6_B82F56D6FD42_BASIC_0`. The test ID is regenerated for every run.
14+
- Do not forget to remove the resources with `terraform destroy`.
15+
- To speed up the commands, you can use `-refresh=false` and `-parallelism=N` (default is 10).
16+
17+
## State size
18+
19+
After running the `terraform` commands, the state file should be saved at `terraform.tfstate`. This file can be analyzed in terms of file size.
20+
21+
Run the following command to capture state size:
22+
23+
```bash
24+
ls -lh terraform.tfstate
25+
```
26+
27+
To check potential size reduction with removed parameters, first remove parameters from the state (with using [jq](https://github.com/jqlang/jq)):
28+
29+
```bash
30+
jq 'del(.resources[].instances[].attributes.parameters)' terraform.tfstate > terraform_without_parameters.tfstate
31+
```
32+
33+
And capture the size of the new state.
34+
35+
```bash
36+
ls -lh terraform_without_parameters.tfstate
37+
```

‎pkg/manual_tests/benchmarks/id/id.tf

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# This module provides a test ID.
2+
terraform {
3+
required_providers {
4+
random = {
5+
source = "hashicorp/random"
6+
version = "~> 3.0"
7+
}
8+
}
9+
10+
}
11+
12+
resource "random_uuid" "test_id" {
13+
keepers = {
14+
first = "${timestamp()}"
15+
}
16+
}
17+
18+
output "test_id" {
19+
value = replace(upper(random_uuid.test_id.id), "-", "_")
20+
}

‎pkg/manual_tests/benchmarks/main.tf

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
module "schema" {
2+
source = "./schema"
3+
resource_count = 1
4+
}
5+
6+
module "warehouse" {
7+
source = "./warehouse"
8+
resource_count = 0
9+
}
10+
11+
module "task" {
12+
source = "./task"
13+
resource_count = 0
14+
}
15+
16+
provider "snowflake" {
17+
profile = "secondary_test_account"
18+
}
19+
20+
terraform {
21+
required_providers {
22+
snowflake = {
23+
source = "Snowflake-Labs/snowflake"
24+
version = "= 1.0.1"
25+
}
26+
}
27+
}

‎pkg/manual_tests/benchmarks/schema/schema.tf

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Test setup.
2+
module "id" {
3+
source = "../id"
4+
}
5+
6+
variable "resource_count" {
7+
type = number
8+
}
9+
10+
terraform {
11+
required_providers {
12+
snowflake = {
13+
source = "Snowflake-Labs/snowflake"
14+
version = "= 1.0.1"
15+
}
16+
}
17+
}
18+
19+
locals {
20+
id_number_list = {
21+
for index, val in range(0, var.resource_count) :
22+
val => tostring(val)
23+
}
24+
test_prefix = format("PERFORMANCE_TESTS_%s", module.id.test_id)
25+
}
26+
27+
resource "snowflake_database" "database" {
28+
count = var.resource_count > 0 ? 1 : 0
29+
name = local.test_prefix
30+
}
31+
32+
# basic resource
33+
resource "snowflake_schema" "all_schemas" {
34+
database = snowflake_database.database[0].name
35+
for_each = local.id_number_list
36+
name = format("perf_basic_%v", each.key)
37+
}
38+
39+
# resource with all fields set (without dependencies)
40+
resource "snowflake_schema" "schema" {
41+
database = snowflake_database.database[0].name
42+
for_each = local.id_number_list
43+
name = format("perf_complete_%v", each.key)
44+
45+
with_managed_access = true
46+
is_transient = true
47+
comment = "my schema"
48+
data_retention_time_in_days = 1
49+
max_data_extension_time_in_days = 20
50+
replace_invalid_characters = false
51+
default_ddl_collation = "en_US"
52+
storage_serialization_policy = "COMPATIBLE"
53+
log_level = "INFO"
54+
trace_level = "ALWAYS"
55+
suspend_task_after_num_failures = 10
56+
task_auto_retry_attempts = 10
57+
user_task_managed_initial_warehouse_size = "LARGE"
58+
user_task_timeout_ms = 3600000
59+
user_task_minimum_trigger_interval_in_seconds = 120
60+
quoted_identifiers_ignore_case = false
61+
enable_console_output = false
62+
pipe_execution_paused = false
63+
}

‎pkg/manual_tests/benchmarks/task/task.tf

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
# Test setup.
2+
module "id" {
3+
source = "../id"
4+
}
5+
6+
variable "resource_count" {
7+
type = number
8+
}
9+
10+
terraform {
11+
required_providers {
12+
snowflake = {
13+
source = "Snowflake-Labs/snowflake"
14+
version = "= 1.0.1"
15+
}
16+
}
17+
}
18+
19+
locals {
20+
id_number_list = {
21+
for index, val in range(0, var.resource_count) :
22+
val => tostring(val)
23+
}
24+
test_prefix = format("PERFORMANCE_TESTS_%s", module.id.test_id)
25+
}
26+
27+
28+
resource "snowflake_database" "database" {
29+
count = var.resource_count > 0 ? 1 : 0
30+
name = local.test_prefix
31+
}
32+
33+
resource "snowflake_schema" "schema" {
34+
database = snowflake_database.database[0].name
35+
count = var.resource_count > 0 ? 1 : 0
36+
name = "PERFORMANCE_TESTS"
37+
}
38+
39+
# Basic standalone task
40+
resource "snowflake_task" "task" {
41+
database = snowflake_database.database[0].name
42+
schema = snowflake_schema.schema[0].name
43+
for_each = local.id_number_list
44+
name = format("perf_basic_%v", each.key)
45+
warehouse = "SNOWFLAKE"
46+
started = true
47+
schedule {
48+
minutes = 5
49+
}
50+
sql_statement = "select 1"
51+
}
52+
53+
# Complete standalone task
54+
resource "snowflake_task" "test" {
55+
database = snowflake_database.database[0].name
56+
schema = snowflake_schema.schema[0].name
57+
for_each = local.id_number_list
58+
name = format("perf_complete_%v", each.key)
59+
warehouse = "SNOWFLAKE"
60+
started = true
61+
sql_statement = "select 1"
62+
63+
config = "{\"key\":\"value\"}"
64+
allow_overlapping_execution = true
65+
comment = "complete task"
66+
67+
schedule {
68+
minutes = 10
69+
}
70+
71+
# Session Parameters
72+
suspend_task_after_num_failures = 10
73+
task_auto_retry_attempts = 0
74+
user_task_minimum_trigger_interval_in_seconds = 30
75+
user_task_timeout_ms = 3600000
76+
abort_detached_query = false
77+
autocommit = true
78+
binary_input_format = "HEX"
79+
binary_output_format = "HEX"
80+
client_memory_limit = 1536
81+
client_metadata_request_use_connection_ctx = false
82+
client_prefetch_threads = 4
83+
client_result_chunk_size = 160
84+
client_result_column_case_insensitive = false
85+
client_session_keep_alive = false
86+
client_session_keep_alive_heartbeat_frequency = 3600
87+
client_timestamp_type_mapping = "TIMESTAMP_LTZ"
88+
date_input_format = "AUTO"
89+
date_output_format = "YYYY-MM-DD"
90+
enable_unload_physical_type_optimization = true
91+
error_on_nondeterministic_merge = true
92+
error_on_nondeterministic_update = false
93+
geography_output_format = "GeoJSON"
94+
geometry_output_format = "GeoJSON"
95+
jdbc_use_session_timezone = true
96+
json_indent = 2
97+
lock_timeout = 43200
98+
log_level = "OFF"
99+
multi_statement_count = 1
100+
noorder_sequence_as_default = true
101+
odbc_treat_decimal_as_int = false
102+
query_tag = ""
103+
quoted_identifiers_ignore_case = false
104+
rows_per_resultset = 0
105+
s3_stage_vpce_dns_name = ""
106+
search_path = "$current, $public"
107+
statement_queued_timeout_in_seconds = 0
108+
statement_timeout_in_seconds = 172800
109+
strict_json_output = false
110+
timestamp_day_is_always_24h = false
111+
timestamp_input_format = "AUTO"
112+
timestamp_ltz_output_format = ""
113+
timestamp_ntz_output_format = "YYYY-MM-DD HH24:MI:SS.FF3"
114+
timestamp_output_format = "YYYY-MM-DD HH24:MI:SS.FF3 TZHTZM"
115+
timestamp_type_mapping = "TIMESTAMP_NTZ"
116+
timestamp_tz_output_format = ""
117+
timezone = "America/Los_Angeles"
118+
time_input_format = "AUTO"
119+
time_output_format = "HH24:MI:SS"
120+
trace_level = "OFF"
121+
transaction_abort_on_error = false
122+
transaction_default_isolation_level = "READ COMMITTED"
123+
two_digit_century_start = 1970
124+
unsupported_ddl_action = "ignore"
125+
use_cached_result = true
126+
week_of_year_policy = 0
127+
week_start = 0
128+
}

‎pkg/manual_tests/benchmarks/warehouse/warehouse.tf

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# Test setup.
2+
module "id" {
3+
source = "../id"
4+
}
5+
6+
variable "resource_count" {
7+
type = number
8+
}
9+
10+
terraform {
11+
required_providers {
12+
snowflake = {
13+
source = "Snowflake-Labs/snowflake"
14+
version = "= 1.0.1"
15+
}
16+
}
17+
}
18+
19+
locals {
20+
id_number_list = {
21+
for index, val in range(0, var.resource_count) :
22+
val => tostring(val)
23+
}
24+
test_prefix = format("PERFORMANCE_TESTS_%s", module.id.test_id)
25+
}
26+
27+
resource "snowflake_resource_monitor" "monitor" {
28+
count = var.resource_count > 0 ? 1 : 0
29+
name = local.test_prefix
30+
}
31+
32+
# Resource with required fields
33+
resource "snowflake_warehouse" "basic" {
34+
for_each = local.id_number_list
35+
name = format("%s_BASIC_%v", local.test_prefix, each.key)
36+
}
37+
38+
# Resource with all fields
39+
resource "snowflake_warehouse" "complete" {
40+
for_each = local.id_number_list
41+
name = format("%s_COMPLETE_%v", local.test_prefix, each.key)
42+
warehouse_type = "SNOWPARK-OPTIMIZED"
43+
warehouse_size = "MEDIUM"
44+
max_cluster_count = 4
45+
min_cluster_count = 2
46+
scaling_policy = "ECONOMY"
47+
auto_suspend = 1200
48+
auto_resume = false
49+
initially_suspended = false
50+
resource_monitor = snowflake_resource_monitor.monitor[0].fully_qualified_name
51+
comment = "An example warehouse."
52+
enable_query_acceleration = true
53+
query_acceleration_max_scale_factor = 4
54+
max_concurrency_level = 4
55+
statement_queued_timeout_in_seconds = 5
56+
statement_timeout_in_seconds = 86400
57+
}

0 commit comments

Comments
 (0)