Skip to content

Commit c3e534e

Browse files
committed
[Iceberg] deprecate some table property names
This commit introduces the deprecation of a few tables property names in favor of the iceberg library's naming scheme. It is to help reduce confusion for users as to which table properties in Iceberg documentation map to presto. Users will get a warning on any queries which set table properties such as CREATE TABLE and ALTER TABLE .. SET PROPERTIES statements. Refer to IcebergTableProperties.java or iceberg.rst for the list of deprecated names and their corresponding mapping.
1 parent d7c0930 commit c3e534e

File tree

15 files changed

+397
-138
lines changed

15 files changed

+397
-138
lines changed

presto-docs/src/main/sphinx/connector/iceberg.rst

Lines changed: 57 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -357,42 +357,46 @@ connector using a WITH clause:
357357

358358
The following table properties are available, which are specific to the Presto Iceberg connector:
359359

360-
======================================= =============================================================== =========================
361-
Property Name Description Default
362-
======================================= =============================================================== =========================
363-
``format`` Optionally specifies the format of table data files, ``PARQUET``
364-
either ``PARQUET`` or ``ORC``.
360+
======================================================== =============================================================== =========================
361+
Property Name Description Default
362+
======================================================== =============================================================== =========================
363+
``commit.retry.num-retries`` Determines the number of attempts for committing the metadata ``4``
364+
in case of concurrent upsert requests, before failing.
365365

366-
``partitioning`` Optionally specifies table partitioning. If a table
367-
is partitioned by columns ``c1`` and ``c2``, the partitioning
368-
property is ``partitioning = ARRAY['c1', 'c2']``.
366+
``format-version`` Optionally specifies the format version of the Iceberg ``2``
367+
specification to use for new tables, either ``1`` or ``2``.
369368

370-
``location`` Optionally specifies the file system location URI for
371-
the table.
369+
``location`` Optionally specifies the file system location URI for
370+
the table.
372371

373-
``format_version`` Optionally specifies the format version of the Iceberg ``2``
374-
specification to use for new tables, either ``1`` or ``2``.
372+
``partitioning`` Optionally specifies table partitioning. If a table
373+
is partitioned by columns ``c1`` and ``c2``, the partitioning
374+
property is ``partitioning = ARRAY['c1', 'c2']``.
375375

376-
``commit_retries`` Determines the number of attempts for committing the metadata ``4``
377-
in case of concurrent upsert requests, before failing.
376+
``read.split.target-size`` The target size for an individual split when generating splits ``134217728`` (128MB)
377+
for a table scan. Generated splits may still be larger or
378+
smaller than this value. Must be specified in bytes.
378379

379-
``delete_mode`` Optionally specifies the write delete mode of the Iceberg ``merge-on-read``
380-
specification to use for new tables, either ``copy-on-write``
381-
or ``merge-on-read``.
380+
``write.delete.mode`` Optionally specifies the write delete mode of the Iceberg ``merge-on-read``
381+
specification to use for new tables, either ``copy-on-write``
382+
or ``merge-on-read``.
382383

383-
``metadata_previous_versions_max`` Optionally specifies the max number of old metadata files to ``100``
384-
keep in current metadata log.
384+
``write.format.default`` Optionally specifies the format of table data files, ``PARQUET``
385+
either ``PARQUET`` or ``ORC``.
385386

386-
``metadata_delete_after_commit`` Set to ``true`` to delete the oldest metadata file after ``false``
387-
each commit.
387+
``write.metadata.previous-versions-max`` Optionally specifies the max number of old metadata files to ``100``
388+
keep in current metadata log.
388389

389-
``metrics_max_inferred_column`` Optionally specifies the maximum number of columns for which ``100``
390-
metrics are collected.
390+
``write.metadata.delete-after-commit.enabled`` Set to ``true`` to delete the oldest metadata file after ``false``
391+
each commit.
391392

392-
``read.split.target-size`` The target size for an individual split when generating splits ``134217728`` (128MB)
393-
for a table scan. Generated splits may still be larger or
394-
smaller than this value. Must be specified in bytes.
395-
======================================= =============================================================== =========================
393+
``write.metadata.metrics.max-inferred-column-defaults`` Optionally specifies the maximum number of columns for which ``100``
394+
metrics are collected.
395+
396+
``write.update.mode`` Optionally specifies the write delete mode of the Iceberg ``merge-on-read``
397+
specification to use for new tables, either ``copy-on-write``
398+
or ``merge-on-read``.
399+
======================================================== =============================================================== =========================
396400

397401
The table definition below specifies format ``ORC``, partitioning by columns ``c1`` and ``c2``,
398402
and a file system location of ``s3://test_bucket/test_schema/test_table``:
@@ -410,6 +414,26 @@ and a file system location of ``s3://test_bucket/test_schema/test_table``:
410414
location = 's3://test_bucket/test_schema/test_table')
411415
)
412416

417+
Deprecated Table Properties
418+
^^^^^^^^^^^^^^^^^^^^^^^^^^^
419+
420+
Some table properties have been deprecated or removed. The following table lists the deprecated
421+
properties and their replacements. Update queries to use the new property names as soon as
422+
possible. They will be removed in a future version.
423+
424+
======================================= ===============================================================
425+
Deprecated Property Name New Property Name
426+
======================================= ===============================================================
427+
``format`` ``write.format.default``
428+
``format_version`` ``format-version``
429+
``commit_retries`` ``commit.retry.num-retries``
430+
``delete_mode`` ``write.delete.mode``
431+
``metadata_previous_versions_max`` ``write.metadata.previous-versions-max``
432+
``metadata_delete_after_commit`` ``write.metadata.delete-after-commit.enabled``
433+
``metrics_max_inferred_column`` ``write.metadata.metrics.max-inferred-column-defaults``
434+
======================================= ===============================================================
435+
436+
413437
Session Properties
414438
------------------
415439

@@ -754,9 +778,13 @@ already exists but does not known by the catalog.
754778

755779
The following arguments are available:
756780

781+
757782
===================== ========== =============== =======================================================================
783+
758784
Argument Name required type Description
785+
759786
===================== ========== =============== =======================================================================
787+
760788
``schema`` ✔️ string Schema of the table to register
761789

762790
``table_name`` ✔️ string Name of the table to register
@@ -1545,8 +1573,8 @@ identified by unique snapshot IDs. The snapshot IDs are stored in the ``$snapsho
15451573
metadata table. You can rollback the state of a table to a previous snapshot ID.
15461574
It also supports time travel query using SYSTEM_VERSION (VERSION) and SYSTEM_TIME (TIMESTAMP) options.
15471575

1548-
Example Queries
1549-
^^^^^^^^^^^^^^^
1576+
Example Time Travel Queries
1577+
^^^^^^^^^^^^^^^^^^^^^^^^^^^
15501578

15511579
Similar to the example queries in `SCHEMA EVOLUTION`_, create an Iceberg
15521580
table named `ctas_nation` from the TPCH `nation` table:

presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergAbstractMetadata.java

Lines changed: 37 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -135,14 +135,7 @@
135135
import static com.facebook.presto.iceberg.IcebergPartitionType.ALL;
136136
import static com.facebook.presto.iceberg.IcebergSessionProperties.getCompressionCodec;
137137
import static com.facebook.presto.iceberg.IcebergSessionProperties.isPushdownFilterEnabled;
138-
import static com.facebook.presto.iceberg.IcebergTableProperties.COMMIT_RETRIES;
139-
import static com.facebook.presto.iceberg.IcebergTableProperties.DELETE_MODE;
140-
import static com.facebook.presto.iceberg.IcebergTableProperties.FILE_FORMAT_PROPERTY;
141-
import static com.facebook.presto.iceberg.IcebergTableProperties.FORMAT_VERSION;
142138
import static com.facebook.presto.iceberg.IcebergTableProperties.LOCATION_PROPERTY;
143-
import static com.facebook.presto.iceberg.IcebergTableProperties.METADATA_DELETE_AFTER_COMMIT;
144-
import static com.facebook.presto.iceberg.IcebergTableProperties.METADATA_PREVIOUS_VERSIONS_MAX;
145-
import static com.facebook.presto.iceberg.IcebergTableProperties.METRICS_MAX_INFERRED_COLUMN;
146139
import static com.facebook.presto.iceberg.IcebergTableProperties.PARTITIONING_PROPERTY;
147140
import static com.facebook.presto.iceberg.IcebergTableProperties.SORTED_BY_PROPERTY;
148141
import static com.facebook.presto.iceberg.IcebergTableType.CHANGELOG;
@@ -166,6 +159,8 @@
166159
import static com.facebook.presto.iceberg.IcebergUtil.tryGetProperties;
167160
import static com.facebook.presto.iceberg.IcebergUtil.tryGetSchema;
168161
import static com.facebook.presto.iceberg.IcebergUtil.validateTableMode;
162+
import static com.facebook.presto.iceberg.IcebergWarningCode.SORT_COLUMN_TRANSFORM_NOT_SUPPORTED_WARNING;
163+
import static com.facebook.presto.iceberg.IcebergWarningCode.USE_OF_DEPRECATED_TABLE_PROPERTY;
169164
import static com.facebook.presto.iceberg.PartitionFields.getPartitionColumnName;
170165
import static com.facebook.presto.iceberg.PartitionFields.getTransformTerm;
171166
import static com.facebook.presto.iceberg.PartitionFields.toPartitionFields;
@@ -185,7 +180,6 @@
185180
import static com.facebook.presto.iceberg.util.StatisticsUtil.calculateBaseTableStatistics;
186181
import static com.facebook.presto.iceberg.util.StatisticsUtil.calculateStatisticsConsideringLayout;
187182
import static com.facebook.presto.spi.StandardErrorCode.NOT_SUPPORTED;
188-
import static com.facebook.presto.spi.StandardWarningCode.SORT_COLUMN_TRANSFORM_NOT_SUPPORTED_WARNING;
189183
import static com.facebook.presto.spi.statistics.TableStatisticType.ROW_COUNT;
190184
import static com.google.common.base.Verify.verify;
191185
import static com.google.common.collect.ImmutableList.toImmutableList;
@@ -201,8 +195,6 @@
201195
import static org.apache.iceberg.SnapshotSummary.REMOVED_POS_DELETES_PROP;
202196
import static org.apache.iceberg.TableProperties.DELETE_ISOLATION_LEVEL;
203197
import static org.apache.iceberg.TableProperties.DELETE_ISOLATION_LEVEL_DEFAULT;
204-
import static org.apache.iceberg.TableProperties.SPLIT_SIZE;
205-
import static org.apache.iceberg.TableProperties.UPDATE_MODE;
206198

207199
public abstract class IcebergAbstractMetadata
208200
implements ConnectorMetadata
@@ -217,6 +209,7 @@ public abstract class IcebergAbstractMetadata
217209
protected final FilterStatsCalculatorService filterStatsCalculatorService;
218210
protected Transaction transaction;
219211
protected final StatisticsFileCache statisticsFileCache;
212+
protected final IcebergTableProperties tableProperties;
220213

221214
private final StandardFunctionResolution functionResolution;
222215
private final ConcurrentMap<SchemaTableName, Table> icebergTables = new ConcurrentHashMap<>();
@@ -228,7 +221,8 @@ public IcebergAbstractMetadata(
228221
JsonCodec<CommitTaskData> commitTaskCodec,
229222
NodeVersion nodeVersion,
230223
FilterStatsCalculatorService filterStatsCalculatorService,
231-
StatisticsFileCache statisticsFileCache)
224+
StatisticsFileCache statisticsFileCache,
225+
IcebergTableProperties tableProperties)
232226
{
233227
this.typeManager = requireNonNull(typeManager, "typeManager is null");
234228
this.commitTaskCodec = requireNonNull(commitTaskCodec, "commitTaskCodec is null");
@@ -237,6 +231,7 @@ public IcebergAbstractMetadata(
237231
this.nodeVersion = requireNonNull(nodeVersion, "nodeVersion is null");
238232
this.filterStatsCalculatorService = requireNonNull(filterStatsCalculatorService, "filterStatsCalculatorService is null");
239233
this.statisticsFileCache = requireNonNull(statisticsFileCache, "statisticsFileCache is null");
234+
this.tableProperties = requireNonNull(tableProperties, "tableProperties is null");
240235
}
241236

242237
protected final Table getIcebergTable(ConnectorSession session, SchemaTableName schemaTableName)
@@ -702,10 +697,10 @@ private static String columnExtraInfo(List<String> partitionTransforms)
702697
protected ImmutableMap<String, Object> createMetadataProperties(Table icebergTable, ConnectorSession session)
703698
{
704699
ImmutableMap.Builder<String, Object> properties = ImmutableMap.builder();
705-
properties.put(FILE_FORMAT_PROPERTY, getFileFormat(icebergTable));
700+
properties.put(TableProperties.DEFAULT_FILE_FORMAT, getFileFormat(icebergTable));
706701

707702
int formatVersion = ((BaseTable) icebergTable).operations().current().formatVersion();
708-
properties.put(FORMAT_VERSION, String.valueOf(formatVersion));
703+
properties.put(TableProperties.FORMAT_VERSION, String.valueOf(formatVersion));
709704

710705
if (!icebergTable.spec().fields().isEmpty()) {
711706
properties.put(PARTITIONING_PROPERTY, toPartitionFields(icebergTable.spec()));
@@ -715,12 +710,12 @@ protected ImmutableMap<String, Object> createMetadataProperties(Table icebergTab
715710
properties.put(LOCATION_PROPERTY, icebergTable.location());
716711
}
717712

718-
properties.put(DELETE_MODE, IcebergUtil.getDeleteMode(icebergTable));
719-
properties.put(UPDATE_MODE, IcebergUtil.getUpdateMode(icebergTable));
720-
properties.put(METADATA_PREVIOUS_VERSIONS_MAX, IcebergUtil.getMetadataPreviousVersionsMax(icebergTable));
721-
properties.put(METADATA_DELETE_AFTER_COMMIT, IcebergUtil.isMetadataDeleteAfterCommit(icebergTable));
722-
properties.put(METRICS_MAX_INFERRED_COLUMN, IcebergUtil.getMetricsMaxInferredColumn(icebergTable));
723-
properties.put(SPLIT_SIZE, IcebergUtil.getSplitSize(icebergTable));
713+
properties.put(TableProperties.DELETE_MODE, IcebergUtil.getDeleteMode(icebergTable));
714+
properties.put(TableProperties.UPDATE_MODE, IcebergUtil.getUpdateMode(icebergTable));
715+
properties.put(TableProperties.METADATA_PREVIOUS_VERSIONS_MAX, IcebergUtil.getMetadataPreviousVersionsMax(icebergTable));
716+
properties.put(TableProperties.METADATA_DELETE_AFTER_COMMIT_ENABLED, IcebergUtil.isMetadataDeleteAfterCommit(icebergTable));
717+
properties.put(TableProperties.METRICS_MAX_INFERRED_COLUMN_DEFAULTS, IcebergUtil.getMetricsMaxInferredColumn(icebergTable));
718+
properties.put(TableProperties.SPLIT_SIZE, IcebergUtil.getSplitSize(icebergTable));
724719

725720
SortOrder sortOrder = icebergTable.sortOrder();
726721
// TODO: Support sort column transforms (https://github.com/prestodb/presto/issues/24250)
@@ -1125,22 +1120,36 @@ public void setTableProperties(ConnectorSession session, ConnectorTableHandle ta
11251120

11261121
UpdateProperties updateProperties = transaction.updateProperties();
11271122
for (Map.Entry<String, Object> entry : properties.entrySet()) {
1128-
switch (entry.getKey()) {
1129-
case COMMIT_RETRIES:
1130-
updateProperties.set(TableProperties.COMMIT_NUM_RETRIES, String.valueOf(entry.getValue()));
1131-
break;
1132-
case SPLIT_SIZE:
1133-
updateProperties.set(TableProperties.SPLIT_SIZE, entry.getValue().toString());
1134-
break;
1135-
default:
1136-
throw new PrestoException(NOT_SUPPORTED, "Updating property " + entry.getKey() + " is not supported currently");
1123+
if (!tableProperties.getUpdatableProperties()
1124+
.contains(entry.getKey())) {
1125+
throw new PrestoException(NOT_SUPPORTED, "Updating property " + entry.getKey() + " is not supported currently");
11371126
}
1127+
String propertyName = entry.getKey();
1128+
if (tableProperties.getDeprecatedProperties().containsKey(entry.getKey())) {
1129+
String newPropertyKey = tableProperties.getDeprecatedProperties().get(entry.getKey());
1130+
PrestoWarning warning = getPrestoWarning(newPropertyKey, propertyName);
1131+
session.getWarningCollector().add(warning);
1132+
propertyName = newPropertyKey;
1133+
}
1134+
updateProperties.set(propertyName, String.valueOf(entry.getValue()));
11381135
}
11391136

11401137
updateProperties.commit();
11411138
transaction.commitTransaction();
11421139
}
11431140

1141+
private static PrestoWarning getPrestoWarning(String newPropertyKey, String propertyName)
1142+
{
1143+
PrestoWarning warning;
1144+
if (newPropertyKey == null) {
1145+
warning = new PrestoWarning(USE_OF_DEPRECATED_TABLE_PROPERTY, format("Property \"%s\" is deprecated and will be completely removed in a future version. Avoid using immediately.", propertyName));
1146+
}
1147+
else {
1148+
warning = new PrestoWarning(USE_OF_DEPRECATED_TABLE_PROPERTY, format("Property \"%s\" has been renamed to \"%s\". This will become an error in future versions.", propertyName, newPropertyKey));
1149+
}
1150+
return warning;
1151+
}
1152+
11441153
/**
11451154
* Deletes all the files for a specific predicate
11461155
*

0 commit comments

Comments
 (0)