Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: support for hbase 2.6.1 #627

Merged
merged 2 commits into from
Feb 12, 2025
Merged

chore: support for hbase 2.6.1 #627

merged 2 commits into from
Feb 12, 2025

Conversation

dervoeti
Copy link
Member

@dervoeti dervoeti commented Feb 11, 2025

Description

Part of stackabletech/docker-images#972

I had to add the --no-checksum-verify flag when exporting snapshots to/from S3, because our snapshot-export test was failing for HBase 2.6.1.
From what I see, checksum verification was merged and released with 2.6.1: https://issues.apache.org/jira/browse/HBASE-28625
However, it seems to have problems when comparing snapshots on different filesystems (like HDFS and S3).

https://issues.apache.org/jira/browse/HBASE-28702 was also merged into 2.6.1 and seems to introduce a flag to skip checksum-checks between snapshots when exporting tables. I'm not sure if this flag is supposed to work with snapshot exports as well, at least I could not make it work. Setting fs.checksum.combine.mode to COMPOSITE_CRC also does not seem to work.

https://issues.apache.org/jira/browse/HBASE-28998 (still open) also mentions that it does not work with COMPOSITE_CRC and it specificly mentions problems with HDFS in combination with S3.

So I decided to add the --no-checksum-verify flag in our tests when exporting snapshots. From what I understand snapshot checksums where not validated before 2.6.1 either. I don't have too much experience with HBase and thus am not sure if this is really the best solution. Could one of our HBase experts (@razvan @siegfriedweber @lfrancke ?) take a look at this as well? If this solution is fine I would adapt the documentation accordingly.

Definition of Done Checklist

Author

Preview Give feedback

Reviewer

Preview Give feedback

Acceptance

Preview Give feedback

@razvan razvan self-requested a review February 12, 2025 08:48
@razvan
Copy link
Member

razvan commented Feb 12, 2025

Indeed, it seems there is no workaround to the checksum problem currently.

The tests are green:

--- PASS: kuttl (4600.24s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/resources_hbase-latest-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (210.98s)
        --- PASS: kuttl/harness/shutdown_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (503.10s)
        --- PASS: kuttl/harness/smoke_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-3.4.0_zookeeper-3.9.2_listener-class-external-unstable_openshift-false (420.45s)
        --- PASS: kuttl/harness/smoke_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-3.4.0_zookeeper-3.9.2_listener-class-cluster-internal_openshift-false (299.91s)
        --- PASS: kuttl/harness/cluster-operation_hbase-latest-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (259.18s)
        --- PASS: kuttl/harness/omid_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false_omid-1.1.2 (262.05s)
        --- PASS: kuttl/harness/logging_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (229.79s)
        --- PASS: kuttl/harness/overrides_hbase-latest-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (213.66s)
        --- PASS: kuttl/harness/kerberos_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_krb5-1.21.1_listener-class-cluster-internal_kerberos-realm-PROD.MYCORP_kerberos-backend-mit_openshift-false (448.90s)
        --- PASS: kuttl/harness/orphaned_resources_hbase-latest-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (240.87s)
        --- PASS: kuttl/harness/kerberos_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_krb5-1.21.1_listener-class-external-unstable_kerberos-realm-PROD.MYCORP_kerberos-backend-mit_openshift-false (481.88s)
        --- PASS: kuttl/harness/profiling_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (226.43s)
        --- PASS: kuttl/harness/snapshot-export_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (288.24s)
        --- PASS: kuttl/harness/opa_hbase-opa-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_krb5-1.21.1_opa-1.0.0_openshift-false (514.78s)
PASS

Copy link
Member

@razvan razvan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm with a small fix

@dervoeti dervoeti added this pull request to the merge queue Feb 12, 2025
Merged via the queue into main with commit 63b9bb8 Feb 12, 2025
17 checks passed
@dervoeti dervoeti deleted the chore/add-hbase-2.6.1 branch February 12, 2025 14:13
@dervoeti dervoeti restored the chore/add-hbase-2.6.1 branch February 14, 2025 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants