Skip to content

Commit 47c810f

Browse files
belieferHyukjinKwon
authored andcommitted
[SPARK-31279][SQL][DOC] Add version information to the configuration of Hive
### What changes were proposed in this pull request? Add version information to the configuration of `Hive`. I sorted out some information show below. Item name | Since version | JIRA ID | Commit ID | Note -- | -- | -- | -- | -- spark.sql.hive.metastore.version | 1.4.0 | SPARK-6908 | 05454fd#diff-ff50aea397a607b79df9bec6f2a841db |   spark.sql.hive.version | 1.1.1 | SPARK-3971 | 64945f8#diff-12fa2178364a810b3262b30d8d48aa2d |   spark.sql.hive.metastore.jars | 1.4.0 | SPARK-6908 | 05454fd#diff-ff50aea397a607b79df9bec6f2a841db |   spark.sql.hive.convertMetastoreParquet | 1.1.1 | SPARK-2406 | cc4015d#diff-ff50aea397a607b79df9bec6f2a841db |   spark.sql.hive.convertMetastoreParquet.mergeSchema | 1.3.1 | SPARK-6575 | 778c876#diff-ff50aea397a607b79df9bec6f2a841db |   spark.sql.hive.convertMetastoreOrc | 2.0.0 | SPARK-14070 | 1e88615#diff-ff50aea397a607b79df9bec6f2a841db |   spark.sql.hive.convertInsertingPartitionedTable | 3.0.0 | SPARK-28573 | d5688dc#diff-842e3447fc453de26c706db1cac8f2c4 |   spark.sql.hive.convertMetastoreCtas | 3.0.0 | SPARK-25271 | 5ad0360#diff-842e3447fc453de26c706db1cac8f2c4 |   spark.sql.hive.metastore.sharedPrefixes | 1.4.0 | SPARK-7491 | a855608#diff-ff50aea397a607b79df9bec6f2a841db |   spark.sql.hive.metastore.barrierPrefixes | 1.4.0 | SPARK-7491 | a855608#diff-ff50aea397a607b79df9bec6f2a841db |   spark.sql.hive.thriftServer.async | 1.5.0 | SPARK-6964 | eb19d3f#diff-ff50aea397a607b79df9bec6f2a841db |   ### Why are the changes needed? Supplemental configuration version information. ### Does this PR introduce any user-facing change? 'No'. ### How was this patch tested? Exists UT Closes apache#28042 from beliefer/add-version-to-hive-config. Authored-by: beliefer <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>
1 parent 4fc8ee7 commit 47c810f

File tree

2 files changed

+16
-1
lines changed

2 files changed

+16
-1
lines changed

docs/sql-data-sources-hive-tables.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -124,14 +124,15 @@ will compile against built-in Hive and use those classes for internal execution
124124
The following options can be used to configure the version of Hive that is used to retrieve metadata:
125125

126126
<table class="table">
127-
<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
127+
<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
128128
<tr>
129129
<td><code>spark.sql.hive.metastore.version</code></td>
130130
<td><code>2.3.6</code></td>
131131
<td>
132132
Version of the Hive metastore. Available
133133
options are <code>0.12.0</code> through <code>2.3.6</code> and <code>3.0.0</code> through <code>3.1.2</code>.
134134
</td>
135+
<td>1.4.0</td>
135136
</tr>
136137
<tr>
137138
<td><code>spark.sql.hive.metastore.jars</code></td>
@@ -153,6 +154,7 @@ The following options can be used to configure the version of Hive that is used
153154
they are packaged with your application.</li>
154155
</ol>
155156
</td>
157+
<td>1.4.0</td>
156158
</tr>
157159
<tr>
158160
<td><code>spark.sql.hive.metastore.sharedPrefixes</code></td>
@@ -166,6 +168,7 @@ The following options can be used to configure the version of Hive that is used
166168
custom appenders that are used by log4j.
167169
</p>
168170
</td>
171+
<td>1.4.0</td>
169172
</tr>
170173
<tr>
171174
<td><code>spark.sql.hive.metastore.barrierPrefixes</code></td>
@@ -177,5 +180,6 @@ The following options can be used to configure the version of Hive that is used
177180
prefix that typically would be shared (i.e. <code>org.apache.spark.*</code>).
178181
</p>
179182
</td>
183+
<td>1.4.0</td>
180184
</tr>
181185
</table>

sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,7 @@ private[spark] object HiveUtils extends Logging {
6565
.doc("Version of the Hive metastore. Available options are " +
6666
"<code>0.12.0</code> through <code>2.3.6</code> and " +
6767
"<code>3.0.0</code> through <code>3.1.2</code>.")
68+
.version("1.4.0")
6869
.stringConf
6970
.createWithDefault(builtinHiveVersion)
7071

@@ -73,6 +74,7 @@ private[spark] object HiveUtils extends Logging {
7374
// already rely on this config.
7475
val FAKE_HIVE_VERSION = buildConf("spark.sql.hive.version")
7576
.doc(s"deprecated, please use ${HIVE_METASTORE_VERSION.key} to get the Hive version in Spark.")
77+
.version("1.1.1")
7678
.stringConf
7779
.createWithDefault(builtinHiveVersion)
7880

@@ -89,12 +91,14 @@ private[spark] object HiveUtils extends Logging {
8991
| Use Hive jars of specified version downloaded from Maven repositories.
9092
| 3. A classpath in the standard format for both Hive and Hadoop.
9193
""".stripMargin)
94+
.version("1.4.0")
9295
.stringConf
9396
.createWithDefault("builtin")
9497

9598
val CONVERT_METASTORE_PARQUET = buildConf("spark.sql.hive.convertMetastoreParquet")
9699
.doc("When set to true, the built-in Parquet reader and writer are used to process " +
97100
"parquet tables created by using the HiveQL syntax, instead of Hive serde.")
101+
.version("1.1.1")
98102
.booleanConf
99103
.createWithDefault(true)
100104

@@ -103,12 +107,14 @@ private[spark] object HiveUtils extends Logging {
103107
.doc("When true, also tries to merge possibly different but compatible Parquet schemas in " +
104108
"different Parquet data files. This configuration is only effective " +
105109
"when \"spark.sql.hive.convertMetastoreParquet\" is true.")
110+
.version("1.3.1")
106111
.booleanConf
107112
.createWithDefault(false)
108113

109114
val CONVERT_METASTORE_ORC = buildConf("spark.sql.hive.convertMetastoreOrc")
110115
.doc("When set to true, the built-in ORC reader and writer are used to process " +
111116
"ORC tables created by using the HiveQL syntax, instead of Hive serde.")
117+
.version("2.0.0")
112118
.booleanConf
113119
.createWithDefault(true)
114120

@@ -118,6 +124,7 @@ private[spark] object HiveUtils extends Logging {
118124
"`spark.sql.hive.convertMetastoreOrc` is true, the built-in ORC/Parquet writer is used" +
119125
"to process inserting into partitioned ORC/Parquet tables created by using the HiveSQL " +
120126
"syntax.")
127+
.version("3.0.0")
121128
.booleanConf
122129
.createWithDefault(true)
123130

@@ -126,6 +133,7 @@ private[spark] object HiveUtils extends Logging {
126133
"instead of Hive serde in CTAS. This flag is effective only if " +
127134
"`spark.sql.hive.convertMetastoreParquet` or `spark.sql.hive.convertMetastoreOrc` is " +
128135
"enabled respectively for Parquet and ORC formats")
136+
.version("3.0.0")
129137
.booleanConf
130138
.createWithDefault(true)
131139

@@ -135,6 +143,7 @@ private[spark] object HiveUtils extends Logging {
135143
"that should be shared is JDBC drivers that are needed to talk to the metastore. Other " +
136144
"classes that need to be shared are those that interact with classes that are already " +
137145
"shared. For example, custom appenders that are used by log4j.")
146+
.version("1.4.0")
138147
.stringConf
139148
.toSequence
140149
.createWithDefault(jdbcPrefixes)
@@ -146,12 +155,14 @@ private[spark] object HiveUtils extends Logging {
146155
.doc("A comma separated list of class prefixes that should explicitly be reloaded for each " +
147156
"version of Hive that Spark SQL is communicating with. For example, Hive UDFs that are " +
148157
"declared in a prefix that typically would be shared (i.e. <code>org.apache.spark.*</code>).")
158+
.version("1.4.0")
149159
.stringConf
150160
.toSequence
151161
.createWithDefault(Nil)
152162

153163
val HIVE_THRIFT_SERVER_ASYNC = buildConf("spark.sql.hive.thriftServer.async")
154164
.doc("When set to true, Hive Thrift server executes SQL queries in an asynchronous way.")
165+
.version("1.5.0")
155166
.booleanConf
156167
.createWithDefault(true)
157168

0 commit comments

Comments
 (0)