Not able to deserialize ClickHouseBitmap for too many bitmaps(>4294967295)!

[wzqiang1332]

clickhouse-server version:22.2.2.1
clickhouse-jdbc version:0.3.2-patch8

an error occurs when I fetch the bitmap of the next.
even if I adjust the order of index_val, the second data will still report an error.

jdbc sql: select index_name, index_val, length(bitmapToArray(cst_bitmap)) as bitmap_count, cst_bitmap fromindex_bitmap_mergetree orderbyindex_name,index_val

jdbc sql: select index_name, index_val, length(bitmapToArray(cst_bitmap)) as bitmap_count, cst_bitmap fromindex_bitmap_mergetree orderbyindex_name,index_val desc

test code

  public static void main(String[] args) throws SQLException {
      String url = "jdbc:clickhouse://192.168.70.131:8123";
      Properties properties = new Properties();
      ClickHouseDataSource dataSource = new ClickHouseDataSource(url, properties);
//        insert(dataSource, 2000);
      searchBitmap(dataSource);
  }

  public static void searchBitmap(ClickHouseDataSource dataSource) {
      try (Connection connection = dataSource.getConnection("default", "");
           Statement statement = connection.createStatement();
           ResultSet resultSet = statement.executeQuery("" +
                   "select " +
                   "  index_name," +
                   "  index_val," +
                   "  length(bitmapToArray(cst_bitmap)) as bitmap_count, " +
                   "  cst_bitmap " +
                   "from index_bitmap_mergetree " +
                   "order by index_name, index_val desc")) {
          while (resultSet.next()) {
              String indexName = resultSet.getString(1);
              String indexVal = resultSet.getString(2);
              int bitmapCount = resultSet.getInt(3);
              Roaring64NavigableMap roaringBitmap = (Roaring64NavigableMap) resultSet.getObject(4, ClickHouseBitmap.class).unwrap();
              System.out.printf("indexName:%s, indexVal:%s, size:%d%n", indexName, indexVal, bitmapCount);
              roaringBitmap.stream().forEach(i -> System.out.print(i + ","));
              System.out.println();
          }
      } catch (SQLException e) {
          e.printStackTrace();
      }
  }

error

Exception in thread "main" java.lang.IllegalStateException: Not able to deserialize ClickHouseBitmap for too many bitmaps(>4294967295)!
	at com.clickhouse.client.data.ClickHouseBitmap.deserialize(ClickHouseBitmap.java:372)
	at com.clickhouse.client.data.ClickHouseBitmap.deserialize(ClickHouseBitmap.java:339)
	at com.clickhouse.client.data.BinaryStreamUtils.readBitmap(BinaryStreamUtils.java:210)
	at com.clickhouse.client.data.ClickHouseRowBinaryProcessor$MappedFunctions.lambda$buildMappingsForAggregateFunctions$0(ClickHouseRowBinaryProcessor.java:178)
	at com.clickhouse.client.data.ClickHouseRowBinaryProcessor$MappedFunctions.lambda$buildMappingsForAggregateFunctions$2(ClickHouseRowBinaryProcessor.java:185)
	at com.clickhouse.client.data.ClickHouseRowBinaryProcessor$MappedFunctions.deserialize(ClickHouseRowBinaryProcessor.java:464)
	at com.clickhouse.client.data.ClickHouseRowBinaryProcessor.readAndFill(ClickHouseRowBinaryProcessor.java:496)
	at com.clickhouse.client.ClickHouseDataProcessor$1.next(ClickHouseDataProcessor.java:122)
	at com.clickhouse.client.ClickHouseDataProcessor$1.next(ClickHouseDataProcessor.java:108)
	at com.clickhouse.jdbc.ClickHouseResultSet.next(ClickHouseResultSet.java:708)
	at com.yatop.clickhouse.JdbcDemo.searchBitmap(JdbcDemo.java:39)
	at com.yatop.clickhouse.JdbcDemo.main(JdbcDemo.java:25)

[zhicwu]

Hi @wzqiang1332, will UInt32 work for you? As far as I know there's discrepancy between Java and C++ implementation of RoaringBitmap64, but I'll check this one when I'm done with patch9 release.

[wzqiang1332]

@zhicwu UInt32 is ok

[wolfxyx]

was this bug solved?

[zhicwu]

was this bug solved?

You meant RoaringBitmap/RoaringBitmap#366? Perhaps you should consider multiple UInt32 bitmaps with or without flag instead. It would be great someone can help you address the issue. I'll revisit this when I'm done with new type system for Aggregatefunction - maybe a custom version of RoaringBitmap just for ClickHouse.

[wzqiang1332]

Hi @zhicwu I found that the old driver can deserialize 64-bit bitmaps, but this driver can not insert 64-bit bitmaps when datasize > 1000000. The new driver can be inserted normally

<dependency>
    <groupId>ru.yandex.clickhouse</groupId>
    <artifactId>clickhouse-jdbc</artifactId>
    <version>0.3.1-patch</version>
</dependency>

[zhicwu]

Hi @zhicwu I found that the old driver can deserialize 64-bit bitmaps, but this driver can not insert 64-bit bitmaps when datasize > 1000000. The new driver can be inserted normally

<dependency>
    <groupId>ru.yandex.clickhouse</groupId>
    <artifactId>clickhouse-jdbc</artifactId>
    <version>0.3.1-patch</version>
</dependency>

Thanks for the hint. Will cross check to understand the discrepancy after releasing patch10.

[panda0120]

Hi @zhicwu I found that the old driver can deserialize 64-bit bitmaps, but this driver can not insert 64-bit bitmaps when datasize > 1000000. The new driver can be inserted normally

<dependency>
    <groupId>ru.yandex.clickhouse</groupId>
    <artifactId>clickhouse-jdbc</artifactId>
    <version>0.3.1-patch</version>
</dependency>

Thanks for the hint. Will cross check to understand the discrepancy after releasing patch10.

This bug will be fixed in version patch10?
About when will it be released？
Now,when I use old jar with group id "ru.yandex.clickhouse",the query opertion is fine,but the insert operation will cause “Memory limit exceed” error on the ck server if the 64 bitmap capacity is more than 1 million.

[zhicwu]

@wzqiang1332, thanks for the pull request. Sorry I don't have much time to add more tests for the change. Since it's been merged into develop branch, would you mind to generate a build and test again in your environment? Also is it necessary to update RoaringBitmap dependency?

[wzqiang1332]

@zhicwu I have tested it in my local environment. The bug is caused by the bitmap reading 3 bytes too many, resulting in the following field reading misalignment

[zhicwu]

Thanks for confirming @wzqiang1332! It will be part of 0.3.2-patch11 which hopefully can be released by Friday, before I start to play Xenoblade 3 :)