Merge branch 'main' into refactor/untraitify-mergedconfig

Author: razvan

CHANGELOG.md

@@ -11,6 +11,7 @@ All notable changes to this project will be documented in this file.
 ### Changed
 
 - `operator-rs` `0.56.1` -> `0.57.0` ([#433]).
+- Change default value of `dfs.ha.nn.not-become-active-in-safemode` from `true` to `false` ([#458]).
 
 ### Fixed
 
@@ -19,6 +20,7 @@ All notable changes to this project will be documented in this file.
 
 [#433]: https://github.com/stackabletech/hdfs-operator/pull/433
 [#451]: https://github.com/stackabletech/hdfs-operator/pull/451
+[#458]: https://github.com/stackabletech/hdfs-operator/pull/458
 
 ## [23.11.0] - 2023-11-24
 

docs/modules/hdfs/pages/usage-guide/security.adoc

@@ -12,7 +12,7 @@ To configure HDFS to use Kerberos you first need to collect information about yo
 Additionally you need a service-user, which the secret-operator uses to create create principals for the HDFS services.
 
 === 2. Create Kerberos SecretClass
-Afterwards you need to enter all the needed information into a SecretClass, as described in  xref:home:secret-operator:secretclass.adoc#backend-kerberoskeytab[secret-operator documentation].
+Afterwards you need to enter all the needed information into a SecretClass, as described in xref:home:secret-operator:secretclass.adoc#backend-kerberoskeytab[secret-operator documentation].
 The following guide assumes you have named your SecretClass `kerberos-hdfs`.
 
 === 3. Configure HDFS to use SecretClass
@@ -33,7 +33,7 @@ The `kerberos.secretClass` is used to give HDFS the possibility to request keyta
 The `tlsSecretClass` is needed to request TLS certificates, used e.g. for the Web UIs.
 
 
-=== 4. Verify that Kerberos is used
+=== 4. Verify that Kerberos authentication is required
 Use `stackablectl stacklet list` to get the endpoints where the HDFS namenodes are reachable.
 Open the link (note that the namenode is now using https).
 You should see a Web UI similar to the following:
@@ -74,5 +74,5 @@ spec:
 ----
 
 == Wire encryption
-In case kerberos is enabled, `Privacy` mode is used for best security.
-Wire encryption without kerberos as well as https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SecureMode.html#Data_confidentiality[other wire encryption modes] are *not* supported.
+In case Kerberos is enabled, `Privacy` mode is used for best security.
+Wire encryption without Kerberos as well as https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SecureMode.html#Data_confidentiality[other wire encryption modes] are *not* supported.

rust/operator-binary/src/hdfs_controller.rs

@@ -494,6 +494,15 @@ fn rolegroup_config_map(
                 // IMPORTANT: these folders must be under the volume mount point, otherwise they will not
                 // be formatted by the namenode, or used by the other services.
                 // See also: https://github.com/apache-spark-on-k8s/kubernetes-HDFS/commit/aef9586ecc8551ca0f0a468c3b917d8c38f494a0
+                //
+                // Notes on configuration choices
+                // ===============================
+                // We used to set `dfs.ha.nn.not-become-active-in-safemode` to true here due to
+                // badly worded HDFS documentation:
+                // https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html
+                // This caused a deadlock with no namenode becoming active during a startup after
+                // HDFS was completely down for a while.
+
                 hdfs_site_xml = HdfsSiteConfigBuilder::new(hdfs_name.to_string())
                     .dfs_namenode_name_dir()
                     .dfs_datanode_data_dir(
@@ -512,7 +521,6 @@ fn rolegroup_config_map(
                     .dfs_client_failover_proxy_provider()
                     .security_config(hdfs)
                     .add("dfs.ha.fencing.methods", "shell(/bin/true)")
-                    .add("dfs.ha.nn.not-become-active-in-safemode", "true")
                     .add("dfs.ha.automatic-failover.enabled", "true")
                     .add("dfs.ha.namenode.id", "${env.POD_NAME}")
                     // the extend with config must come last in order to have overrides working!!!