Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support spatial extension types #168

Merged
merged 1 commit into from
Mar 25, 2025
Merged

Conversation

staticlibs
Copy link
Contributor

@staticlibs staticlibs commented Mar 20, 2025

This PR changes the handling of extension types. Instead of treating
all custom types (detected by having an alias) as VARCHAR, it ignores
type alias and uses underlying logical type.

This allows to work with the following types from the spatial
extension using default JDBC types java.sql.Struct, java.sql.Array,
and java.sql.Blob:

  • POINT_2D: as STRUCT{x: DOUBLE, y: DOUBLE}
  • POINT_3D: as STRUCT{x: DOUBLE, y: DOUBLE, z: DOUBLE}
  • POINT_4D: as STRUCT{x: DOUBLE, y: DOUBLE, z: DOUBLE, m: DOUBLE}
  • LINESTRING_2D: as LIST[STRUCT{x: DOUBLE, y: DOUBLE}]
  • POLYGON_2D: as LIST[LIST[STRUCT{x: DOUBLE, y: DOUBLE}]]
  • BOX_2D: as STRUCT{min_x: DOUBLE, max_x: DOUBLE, min_y: DOUBLE, max_y: DOUBLE}
  • BOX_2DF: as STRUCT{min_x: FLOAT, max_x: FLOAT, min_y: FLOAT, max_y: FLOAT}
  • GEOMETRY: as BLOB
  • WKB_BLOB: as BLOB

Special handling is left only for JSON type because it is mapped to
driver-specific JsonNode class.

Testing: existing test for extnsion types is adjusted to use
java.sql.Struct and java.sql.Blob. New test is added to cover all
spatial types in bind parameters and in result set, minor enhancements
are added to tests runner.

Fixes: #37

Edit: description updated

@staticlibs
Copy link
Contributor Author

Spatial extension installation fails on GitHub CI for some reason:

TestDuckDBJDBC#test_spatial failed with java.sql.SQLException: HTTP Error: Failed to download extension "spatial" at URL "http://extensions.duckdb.org/391593dc9c/linux_amd64/spatial.duckdb_extension.gz" (HTTP 403)
Extension "spatial" is an existing extension.
For more info, visit https://duckdb.org/docs/extensions/troubleshooting/?version=391593dc9c&platform=linux_amd64&extension=spatial
java.sql.SQLException: HTTP Error: Failed to download extension "spatial" at URL "http://extensions.duckdb.org/391593dc9c/linux_amd64/spatial.duckdb_extension.gz" (HTTP 403)
Extension "spatial" is an existing extension.

Temporary disabled spatial test, it's run is verified locally.

@Mytherin
Copy link
Contributor

Thanks for the PR!

This currently hard-codes the spatial extension types - but I wonder if that's a good idea for a client. Other extensions can add their own types, including community extensions, which indicates that it's not a scalable solution to custom types.

Given that nothing special is done for these types anyway - would it be possible to just ignore the alias and use the underlying type instead? (i.e. WKB_BLOB is stored as a BLOB - so just let it be operated on as a blob).

Spatial extension installation fails on GitHub CI for some reason:

Extensions are not distributed for all git hashes - but rather (1) for releases, and (2) (usually) for the hash that is used for the nightly tests, but then only after the nightly tests have finished running.

@staticlibs
Copy link
Contributor Author

Thanks for the review!

would it be possible to just ignore the alias and use the underlying type instead?

M, this should work I think. Though the way how the DB type is exposed to Java needs to be changed (for all types), because name matching is used now. I think we can expose the underlying type instead, and only match by it. It should be sufficient on Java side because only core types are mapped to Java classes anyway (and generic Array/Struct classes are used for composite types). Will experiment with it.

@staticlibs
Copy link
Contributor Author

Extensions are not distributed for all git hashes - but rather (1) for releases, and (2) (usually) for the hash that is used for the nightly tests, but then only after the nightly tests have finished running.

I believe Maia mentioned a solution to this in ODBC workflows (though it is not included in current ODBC) - will follow up on this in a separate PR.

This PR changes the handling of extension types. Instead of treating
all custom types (detected by having an alias) as `VARCHAR`, it ignores
type alias and uses underlying logical type.

This allows to work with the following types from the spatial
extension using default JDBC types `java.sql.Struct`, `java.sql.Array`,
and `java.sql.Blob`:

 - `POINT_2D`: as `STRUCT{x: DOUBLE, y: DOUBLE}`
 - `POINT_3D`: as `STRUCT{x: DOUBLE, y: DOUBLE, z: DOUBLE}`
 - `POINT_4D`: as `STRUCT{x: DOUBLE, y: DOUBLE, z: DOUBLE, m: DOUBLE}`
 - `LINESTRING_2D`: as `LIST[STRUCT{x: DOUBLE, y: DOUBLE}]`
 - `POLYGON_2D`: as `LIST[LIST[STRUCT{x: DOUBLE, y: DOUBLE}]]`
 - `BOX_2D`: as `STRUCT{min_x: DOUBLE, max_x: DOUBLE, min_y: DOUBLE, max_y: DOUBLE}`
 - `BOX_2DF`: as `STRUCT{min_x: FLOAT, max_x: FLOAT, min_y: FLOAT, max_y: FLOAT}`
 - `GEOMETRY`: as `BLOB`
 - `WKB_BLOB`: as `BLOB`

Special handling is left only for `JSON` type because it is mapped to
driver-specific `JsonNode` class.

Testing: existing test for extnsion types is adjusted to use
`java.sql.Struct` and `java.sql.Blob`. New test is added to cover all
spatial types in bind parameters and in result set, minor enhancements
are added to tests runner.

Fixes: duckdb#37
@staticlibs
Copy link
Contributor Author

@Mytherin

Just FYI, I've updated the PR ignoring aliases and using underlying types instead. Type handling looks cleaner now and there is actually no spatial specifics in this change - other extension that use composite types from DuckDB base types should all work the same way as spatial. Rebased the PR to latest main and updated the description above.

@Mytherin Mytherin merged commit 73139ae into duckdb:main Mar 25, 2025
7 checks passed
@Mytherin
Copy link
Contributor

Thanks!

@staticlibs staticlibs deleted the spatial_types branch March 27, 2025 01:03
@staticlibs
Copy link
Contributor Author

Extensions are not distributed for all git hashes - but rather (1) for releases, and (2) (usually) for the hash that is used for the nightly tests, but then only after the nightly tests have finished running.

I believe Maia mentioned a solution to this in ODBC workflows (though it is not included in current ODBC) - will follow up on this in a separate PR.

Just for the record, this PR was added to the main repo to trigger ODBC after the main tests passed and extensions are built. This way all versions ODBC will support installing extensions, including in tests in dev/CI. I intend to add the JDBC workflow triggering the same way, going to follow-up on this once workflow changes in #172 are ready.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for ResultSet.getBlob() calls when working with WKB_BLOB types
2 participants