Skip to content

Commit 25e5d9f

Browse files
committed
Minor updates and test impact of inserting data
1 parent 1b2af7b commit 25e5d9f

File tree

1 file changed

+10
-3
lines changed

1 file changed

+10
-3
lines changed

spark/local-k8s/README.md

+10-3
Original file line numberDiff line numberDiff line change
@@ -136,13 +136,16 @@ Then start up Spark SQL
136136
_note that we assume you already have your AWS CLI setup and can export credentials_
137137

138138
```bash
139+
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
140+
export TABLE_BUCKET_NAME=dacort-berg
141+
139142
kubectl exec -it spark-shell-pod -- /bin/bash -c "export AWS_REGION=us-west-2;$(aws configure export-credentials --format env | tr '\n' ';') \
140143
/opt/spark/bin/spark-sql \
141144
--packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.6.1,software.amazon.awssdk:s3tables:2.29.26,software.amazon.awssdk:s3:2.29.26,software.amazon.awssdk:sts:2.29.26,software.amazon.awssdk:kms:2.29.26,software.amazon.awssdk:glue:2.29.26,software.amazon.awssdk:dynamodb:2.29.26,software.amazon.s3tables:s3-tables-catalog-for-iceberg-runtime:0.1.3 \
142145
--conf spark.jars.ivy=/opt/spark/work-dir/.ivy2 \
143146
--conf spark.sql.catalog.s3tablesbucket=org.apache.iceberg.spark.SparkCatalog \
144147
--conf spark.sql.catalog.s3tablesbucket.catalog-impl=software.amazon.s3tables.iceberg.S3TablesCatalog \
145-
--conf spark.sql.catalog.s3tablesbucket.warehouse=arn:aws:s3tables:us-west-2:<YOUR_AWS_ACCOUNT_ID>:bucket/dacort-berg \
148+
--conf spark.sql.catalog.s3tablesbucket.warehouse=arn:aws:s3tables:us-west-2:${AWS_ACCOUNT_ID}:bucket/${TABLE_BUCKET_NAME} \
146149
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"
147150
```
148151

@@ -174,7 +177,7 @@ The neat(?) thing about S3 Tables is that it's just Iceberg behind the scenes.
174177
So if you use `aws s3tables get-table`, you can find the metadata location:
175178

176179
```bash
177-
aws s3tables get-table --table-bucket-arn arn:aws:s3tables:us-west-2:<YOUR_AWS_ACCOUNT_ID>:bucket/dacort-berg --namespace default --name demo
180+
aws s3tables get-table --table-bucket-arn arn:aws:s3tables:us-west-2:${AWS_ACCOUNT_ID}:bucket/${TABLE_BUCKET_NAME} --namespace default --name demo
178181
```
179182

180183
```json
@@ -241,4 +244,8 @@ SELECT * FROM iceberg_scan('s3://502d9-5de1-46a4-<SOME_OTHER_ID>--table-s3/metad
241244
└───────┴─────────┴───────┘
242245
```
243246

244-
🤯
247+
🤯
248+
249+
- What happens if I insert more data?
250+
251+
The `metadataLocation` gets updated and we can, of course, query each different version of the table. 🎉

0 commit comments

Comments
 (0)