add normalize-input-file with sqlfluff

Author: RulerOf

.gitignore

@@ -27,3 +27,4 @@ override.tf.json
 
 # Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
 # example: *tfplan*
+.DS_Store

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2020 Andrew Bobulsky
+Copyright (c) 2022 Andrew Bobulsky
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -4,9 +4,13 @@ Documenting solutions to interesting Terraform problems.
 
 ## Requirements
 
-Consider installing [tfenv](https://github.com/tfutils/tfenv) if you want these code exmaples to "just work" for you.
+Just terraform, though consider installing [asdf-vm](https://asdf-vm.com/) if you want to be sure you're using the same version I used.
 
 # Examples
 
 * [Dynamic Templatefile Rendering](./dynamic-template-output/README.md)
-  * Keeping `templatefile()` function calls and the files themselves DRY — avoids writing variables twice, as input and output.
+  * Keeping `templatefile()` function calls and the files themselves DRY — avoids writing variables twice, as input and output.
+* [Traversing Nested Maps](traversing-nested-maps)
+  * Illustrating how to navigate nested map structures to facilitate iterative resource creation.
+* [Normalizing Input Files](normalize-input-file)
+  * Using the external data source to pass a file through a linter before consuming it to avoid unnecessary destroy/recreate cycles.`

dynamic-template-output/.terraform-version

@@ -0,0 +1 @@
+terraform 0.13.4

dynamic-template-output/.tool-versions

@@ -0,0 +1 @@
+terraform 1.3.3

normalize-input-file/.tool-versions

@@ -0,0 +1,49 @@
+# Formatting SQL in Terraform
+
+[This question](https://www.reddit.com/r/Terraform/comments/xvm6xc/terraform_pre_commit_hooks/) came up on reddit where a user wanted to use git commit hooks to format SQL code that terraform was consuming so that it would be consistent regardless of whitespace changes. My opinion was that the SQL code should be normalized by Terraform, not by git hooks, so I created this repo to illustrate the idea.
+
+----
+
+[unformatted.sql](unformatted.sql) shows a SQL query with terrible formatting, making it nearly unreadable.
+
+```sql
+SELECT  country.country_name_eng, SUM(CASE WHEN call.id IS NOT NULL THEN 1 ELSE 0 END) AS calls, AVG(ISNULL(DATEDIFF(SECOND, call.start_time, call.end_time),0)) AS avg_difference FROM country  LEFT JOIN city ON city.country_id = country.id LEFT JOIN customer ON city.id = customer.city_id LEFT JOIN call ON call.customer_id = customer.id GROUP BY  country.id, country.country_name_eng HAVING AVG(ISNULL(DATEDIFF(SECOND, call.start_time, call.end_time),0)) > (SELECT AVG(DATEDIFF(SECOND, call.start_time, call.end_time)) FROM call) ORDER BY calls DESC, country.id ASC;
+```
+
+Using the [external_data data source](https://registry.terraform.io/providers/hashicorp/external/latest/docs/data-sources/data_source) in Terraform, and a [small wrapper script](sqlfluff-wrapper.sh), we pipe the data to jq -> shell -> [SQLFluff](https://github.com/sqlfluff/sqlfluff) to make it readable, as shown here:
+
+```
+╭─ ~/Projects/personal/terraform/sqlformat main ?
+╰─❯ terraform apply
+
+Changes to Outputs:
+  + formatted_sql = <<-EOT
+        SELECT
+            country.country_name_eng,
+            SUM(CASE WHEN call.id IS NOT NULL THEN 1 ELSE 0 END) AS calls,
+            AVG(
+                ISNULL(DATEDIFF(second, call.start_time, call.end_time), 0)
+            ) AS avg_difference
+        FROM
+            country
+        LEFT JOIN
+            city ON city.country_id = country.id
+        LEFT JOIN
+            customer ON city.id = customer.city_id
+        LEFT JOIN
+            call ON call.customer_id = customer.id
+        GROUP BY
+            country.id, country.country_name_eng
+        HAVING
+            AVG(
+                ISNULL(DATEDIFF(second, call.start_time, call.end_time), 0)
+            ) > (SELECT AVG(DATEDIFF(second, call.start_time, call.end_time)) FROM call)
+        ORDER BY calls DESC, country.id ASC;
+    EOT
+```
+
+# Requires
+
+* jq
+* sqlfluff
+* bash shell

normalize-input-file/README.md

@@ -0,0 +1,11 @@
+data "external" "format_sql" {
+  program = ["bash", "sqlfluff-wrapper.sh"]
+  query = {
+    sql     = file("unformatted.sql")
+    dialect = "ansi"
+  }
+}
+
+output "formatted_sql" {
+  value = data.external.format_sql.result.formatted_sql
+}

normalize-input-file/main.tf

@@ -0,0 +1,14 @@
+#!/bin/bash -e
+
+# Extract arguments from the input into shell variables.
+# jq will ensure that the values are properly quoted
+# and escaped for consumption by the shell.
+eval "$(jq -r '@sh "SQL=\(.sql) DIALECT=\(.dialect)"')"
+
+# Placeholder for whatever data-fetching logic your script implements
+FORMATTED_SQL="$(echo $SQL | sqlfluff fix - --dialect $DIALECT)"
+
+# Safely produce a JSON object containing the result value.
+# jq will ensure that the value is properly quoted
+# and escaped to produce a valid JSON string.
+jq -n --arg formatted_sql "$FORMATTED_SQL" '{"formatted_sql":$formatted_sql}'

normalize-input-file/sqlfluff-wrapper.sh

@@ -0,0 +1 @@
+SELECT  country.country_name_eng, SUM(CASE WHEN call.id IS NOT NULL THEN 1 ELSE 0 END) AS calls, AVG(ISNULL(DATEDIFF(SECOND, call.start_time, call.end_time),0)) AS avg_difference FROM country  LEFT JOIN city ON city.country_id = country.id LEFT JOIN customer ON city.id = customer.city_id LEFT JOIN call ON call.customer_id = customer.id GROUP BY  country.id, country.country_name_eng HAVING AVG(ISNULL(DATEDIFF(SECOND, call.start_time, call.end_time),0)) > (SELECT AVG(DATEDIFF(SECOND, call.start_time, call.end_time)) FROM call) ORDER BY calls DESC, country.id ASC;