@@ -136,13 +136,16 @@ Then start up Spark SQL
136
136
_ note that we assume you already have your AWS CLI setup and can export credentials_
137
137
138
138
``` bash
139
+ export AWS_ACCOUNT_ID=$( aws sts get-caller-identity --query Account --output text)
140
+ export TABLE_BUCKET_NAME=dacort-berg
141
+
139
142
kubectl exec -it spark-shell-pod -- /bin/bash -c " export AWS_REGION=us-west-2;$( aws configure export-credentials --format env | tr ' \n' ' ;' ) \
140
143
/opt/spark/bin/spark-sql \
141
144
--packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.6.1,software.amazon.awssdk:s3tables:2.29.26,software.amazon.awssdk:s3:2.29.26,software.amazon.awssdk:sts:2.29.26,software.amazon.awssdk:kms:2.29.26,software.amazon.awssdk:glue:2.29.26,software.amazon.awssdk:dynamodb:2.29.26,software.amazon.s3tables:s3-tables-catalog-for-iceberg-runtime:0.1.3 \
142
145
--conf spark.jars.ivy=/opt/spark/work-dir/.ivy2 \
143
146
--conf spark.sql.catalog.s3tablesbucket=org.apache.iceberg.spark.SparkCatalog \
144
147
--conf spark.sql.catalog.s3tablesbucket.catalog-impl=software.amazon.s3tables.iceberg.S3TablesCatalog \
145
- --conf spark.sql.catalog.s3tablesbucket.warehouse=arn:aws:s3tables:us-west-2:<YOUR_AWS_ACCOUNT_ID> :bucket/dacort-berg \
148
+ --conf spark.sql.catalog.s3tablesbucket.warehouse=arn:aws:s3tables:us-west-2:${AWS_ACCOUNT_ID} :bucket/${TABLE_BUCKET_NAME} \
146
149
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"
147
150
```
148
151
@@ -174,7 +177,7 @@ The neat(?) thing about S3 Tables is that it's just Iceberg behind the scenes.
174
177
So if you use ` aws s3tables get-table ` , you can find the metadata location:
175
178
176
179
``` bash
177
- aws s3tables get-table --table-bucket-arn arn:aws:s3tables:us-west-2:< YOUR_AWS_ACCOUNT_ID > :bucket/dacort-berg --namespace default --name demo
180
+ aws s3tables get-table --table-bucket-arn arn:aws:s3tables:us-west-2:${AWS_ACCOUNT_ID} :bucket/${TABLE_BUCKET_NAME} --namespace default --name demo
178
181
```
179
182
180
183
``` json
@@ -241,4 +244,8 @@ SELECT * FROM iceberg_scan('s3://502d9-5de1-46a4-<SOME_OTHER_ID>--table-s3/metad
241
244
└───────┴─────────┴───────┘
242
245
```
243
246
244
- 🤯
247
+ 🤯
248
+
249
+ - What happens if I insert more data?
250
+
251
+ The ` metadataLocation ` gets updated and we can, of course, query each different version of the table. 🎉
0 commit comments