[DRAFT] NoSQL database agnostic persistence

[snazy]

Mostly complete proposal for the database agnostic persistence layer to support NoSQL databases in Apache Polaris.

This PR is intended to illustrate the “whole spiel” and contains all necessities. Smaller, easier reviewable PRs will be opened soon for some of the areas covered in this PR to initially enable MongoDB persistence.

The overall approach borrows the architectural approach from Nessie to isolate database specifics from higher level concerns.

The most important objective for the implementation is correctness, especially in scenarios with high concurrent load. Explicit tests to verify the correctness are included, for the CI “use case” and for manual/special runs against a clustered database setup (which are just “too much” for the Github hosted runners).

The current integration point is MetaStoreManagerFactory/PolarisMetaStoreManager implemented in the “bridge” Gradle project.

See components/persistence/README.md in this PR for more technical information.

To run Polaris w/ the “dummy” in-memory “database”:

./gradlew quarkusBuild
java -jar -Djava.security.manager=allow \
  -Dpolaris.persistence.type=persistence \
  -jar quarkus/server/build/quarkus-app/quarkus-run.jar

To run Polaris against MongoDB:

docker run --publish 27017:27017 --rm -ti mongo:8.0.5
./gradlew quarkusBuild
java -jar -Djava.security.manager=allow \
  -Dpolaris.persistence.type=persistence \
  -Dpolaris.backend.name=MongoDb \
  -Dquarkus.mongodb.connection-string=mongodb://localhost:27017 \
  -jar quarkus/server/build/quarkus-app/quarkus-run.jar