Create a single binary distribution bundle #1589

flyrain · 2025-05-14T18:25:44Z

What’s included in the PR:

Introduced a new module that combines both Admin Tools and Server. Please refer to my original email for the motivation behind this change.
Removed the now-redundant run-scripts module.
Consolidated the README to reflect the unified binary distribution.
Standardized the binary distribution package naming to polaris-{version}.tgz and polaris-{version}.zip, following common conventions used by other projects (e.g., spark-3.5.5-bin-hadoop3.tgz).

TODOs:

Consolidate LICENSE and NOTICE files from both Admin Tools and Server.
Consolidate the shared libs between Admin Tool and Server

The PR is technically ready, but I plan to wait until the 0.10 release is finalized to avoid disrupting the release process.

jbonofre · 2025-05-14T19:16:58Z

@flyrain thanks for this draft PR ! Just a note: we should keep the changes proposed in #1588 and #1568 . I will double check here.

flyrain · 2025-05-14T20:14:34Z

@flyrain thanks for this draft PR ! Just a note: we should keep the changes proposed in #1588 and #1568 . I will double check here.

Yes, I added the disclaimer in the new commit.

quarkus/distribution/build.gradle.kts

dimas-b · 2025-05-14T21:18:05Z

quarkus/distribution/distribution/README.md

+├── README.md
+├── admin/           # Admin tool files
+├── server/          # Server files
+└── run.sh


Spark was mentioned as an example, but IIRC, Spark's distribution has a bin directory for runnable files... Why not put run.sh into bin in Polaris too?

I think one of reasons Spark has a dedicated bin directory is due to the number of items under the root directory like the following. Our distribution is different though, we don't have a lot of top level items, but I'm OK either way.

drwxr-xr-x@ 3 ygu staff 96B Aug 6 2024 yarn/ drwxr-xr-x@ 31 ygu staff 992B Aug 6 2024 sbin/ -rw-r--r--@ 1 ygu staff 166B Aug 6 2024 RELEASE -rw-r--r--@ 1 ygu staff 4.5K Aug 6 2024 README.md drwxr-xr-x@ 3 ygu staff 96B Aug 6 2024 R/ drwxr-xr-x@ 19 ygu staff 608B Aug 6 2024 python/ -rw-r--r--@ 1 ygu staff 56K Aug 6 2024 NOTICE drwxr-xr-x@ 58 ygu staff 1.8K Aug 6 2024 licenses/ -rw-r--r--@ 1 ygu staff 22K Aug 6 2024 LICENSE drwxr-xr-x@ 4 ygu staff 128B Aug 6 2024 kubernetes/ drwxr-xr-x@ 4 ygu staff 128B Aug 6 2024 examples/ drwxr-xr-x@ 6 ygu staff 192B Aug 6 2024 data/ drwxr-xr-x@ 8 ygu staff 256B Aug 6 2024 conf/ drwxr-xr-x@ 30 ygu staff 960B Aug 6 2024 bin/ drwxr-xr-x@ 255 ygu staff 8.0K Oct 14 2024 jars/

I'm not sure I agree with that rationale. I believe bin directories have roots in the old *nix tradition as the location of runnable program files.

If having a shared bin does not look convenient, how about admin/bin and server/bin?

If having a shared bin does not look convenient, how about admin/bin and server/bin?

Please disregard this proposal.

Since this PR provides a single distribution for all Polaris binaries, I think having one bin directory is preferable from the UX perspective.

Sounds good. Will add a bin directory under the root. I think the executables are buried too deep if we follow the proposal here, #1589 (comment).

This is fixed in the new commit

quarkus/distribution/scripts/run.sh

jbonofre · 2025-05-15T07:15:48Z

quarkus/distribution/build.gradle.kts

+
+description = "Apache Polaris Binary Distribution"
+
+val adminProject = project(":polaris-quarkus-admin")


As we do this change, I would suggest to rename polaris-quarkus-admin and polaris-quarkus-server as polaris-admin and polaris-server, and also maybe rename quarkus folder as dist folder or so.

Sounds good. I don't think we need quarkus within the module name. Quarkus is the way in Polaris, not a way. Adding Quarkus into name isn't necesary.

Yes, and I think it makes sense to do it as part of this PR (for consistency).

+1 to polaris-admin and polaris-server

Great suggestion! Let me do it as a follow-up.

snazy

This change uses a bunch of highly discouraged Gradle practices:

It's accessing another Gradle projects and those tasks - that's bad practice and highly discouraged.
This leads to task-dependency and in turn build and Gradle caching issues

Please take a look at Gradle artifacts and configurations and defining portable artifact dependencies including the proper task dependencies implied by these mechanisms.

adutra · 2025-05-15T20:49:48Z

quarkus/distribution/distribution/README.md

@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one


Nit: this file is placed under quarkus/distribution/distribution – is that intentional? I'd advocate for moving it one level up (it would become the README for the distribution module, but I think that's a good thing).

That's intentional, which follows the same structure in admin and server module.

quarkus/distribution/scripts/run.sh

adutra · 2025-05-15T21:06:18Z

@flyrain just to confirm, we are not planning to remove the individual distributions for server and tool, are we?

flyrain · 2025-05-15T21:27:11Z

@flyrain just to confirm, we are not planning to remove the individual distributions for server and tool, are we?

Good question. We will remove the individual distribution once the single binary distribution is ready. Their jar file will still be published individually though.

adutra · 2025-05-15T22:19:08Z

@flyrain just to confirm, we are not planning to remove the individual distributions for server and tool, are we?

Good question. We will remove the individual distribution once the single binary distribution is ready. Their jar file will still be published individually though.

Are we sure about this? I assume that most people downloading the distribution would be interested in the server, not the tool. Imposing the combined distribution would mean for them to download a bunch of jars they don't need. And the opposite is true as well: if I only need the tool, why would I need to download 200Mb of server jars?

dimas-b · 2025-05-16T19:04:46Z

With real (not in-memory) persistence the admin tool is pretty much always required for proper bootstrapping, I guess. So it makes sense to bundle it with the server.

As to whether to keep a separate downloadable archive just for the admin tool, I suppose it may be useful only if the server runs in docker. However, in that case, it might be preferable to have a separate docker image for the admin tool and have only the combined archive-based distribution. WDYT?

adutra · 2025-05-16T19:13:20Z

it might be preferable to have a separate docker image for the admin tool and have only the combined archive-based distribution. WDYT?

If that's the plan I can live with that 👍

snazy · 2025-05-19T10:45:51Z

@flyrain just to confirm, we are not planning to remove the individual distributions for server and tool, are we?

Good question. We will remove the individual distribution once the single binary distribution is ready. Their jar file will still be published individually though.

Are we sure about this? I assume that most people downloading the distribution would be interested in the server, not the tool. Imposing the combined distribution would mean for them to download a bunch of jars they don't need. And the opposite is true as well: if I only need the tool, why would I need to download 200Mb of server jars?

I second @adutra's concern. Polaris server and admin-tool are separate things. You deploy the server and let it run in k8s (or whatever). Bootstrapping happens rarely and is rather performed from an administrator's machine, whereas the servers run elsewhere. So server and admin-tool are run in very different ways.

The admin-tool is (at best) just overhead in a server and increases the size of the image + container. Playing devil's advocate here, having the admin-tool executable accessible from the server gives easy access to the purge functionality. So if you're (in whatever way) able to instruct the Polaris server container to start an arbitrary executable, it could nuke existing data.

This script combines the Polaris admin tool and server distributions into a single package: Maintains separate admin and server components, provides a unified run script to launch either component, preserves all necessary dependencies and configurations, and simplifies deployment by having both components in one distribution.

… run script - Remove repository configuration from plugins, restore run.sh script, fix build issues

…rojects.main.properties - Remove run-script dependency from admin and server modules - Fix code formatting in build files

flyrain · 2025-05-22T23:26:22Z

This change uses a bunch of highly discouraged Gradle practices:

It's accessing another Gradle projects and those tasks - that's bad practice and highly discouraged.

This leads to task-dependency and in turn build and Gradle caching issues

Please take a look at Gradle artifacts and configurations and defining portable artifact dependencies including the proper task dependencies implied by these mechanisms.

This is fixed in the new commit, cc @snazy.

flyrain · 2025-05-23T01:45:45Z

Local test passed. Here are the structure now in the binary bundle. All comments should be addressed. Please take a look @jbonofre @dimas-b @adutra @snazy @singhpk234. Consolidation of LICENSE and NOTICE files from both Admin Tools and Server will be done in a follow-up PR.

polaris-quarkus-distribution-@version@/
├── LICENSE
├── NOTICE
├── README.md
├── admin/           # Admin tool files
├── bin/             # Executable scripts
│   ├── admin
│   └── server
└── server/          # Server files

jbonofre · 2025-05-23T05:49:49Z

@flyrain I will do the LICENSE/NOTICE/DISCLAIMER work on another PR.

jbonofre · 2025-05-23T05:50:26Z

By the way, as we still have separate docker images, we also have to keep LICENSE/NOTICE/DISCLAIMER in both admin and server.

flyrain · 2025-05-23T06:01:23Z

By the way, as we still have separate docker images, we also have to keep LICENSE/NOTICE/DISCLAIMER in both admin and server.

Thanks @jbonofre ! I intentionally left them in these two modules in this PR, but I'd love to figure out a way to have a combined one later.

jbonofre · 2025-05-23T06:02:53Z

@flyrain you are good here. My previous comment is "for information" (to explain the reason why we have to keep those files :)).

jbonofre · 2025-05-23T06:03:28Z

Any artifact we distribute should have license/notice/disclaimer

singhpk234 · 2025-05-23T01:47:05Z

quarkus/distribution/README.md

+
+## Prerequisites
+
+- Java SE 21 or higher


I think we just say 21 here, please ref thread here : #1517 (comment)

You are right that Polaris target jdk version is 21,

polaris/build-logic/src/main/kotlin/polaris-server.gradle.kts

Line 24 in 9377aa7

tasks.withType(JavaCompile::class.java).configureEach { options.release = 21 }

. However, this is a binary distribution. Users should only care about the runtime jdk version here. I think it's the right call to say 21+ here.

I think 21+ is fine. We can always adjust docs if we hit incompatibilities.

Agree, 21+ is fine.

singhpk234 · 2025-05-23T01:48:18Z

quarkus/distribution/bin/admin

-exec "${JAVACMD}" -jar "${script_dir}/quarkus-run.jar" $@
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+cd "${SCRIPT_DIR}/../admin" || exit
+java -jar quarkus-run.jar "$@" 


[minor] new line

singhpk234 · 2025-05-23T01:48:35Z

quarkus/distribution/bin/server

+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+cd "${SCRIPT_DIR}/../server" || exit
+java -jar quarkus-run.jar "$@"


[minor] new line

singhpk234 · 2025-05-23T16:03:36Z

we still have separate docker images

I have a doubt in this area, and would like to understand this more, why we need two separate image if we agree on same binary distribution ?
My understanding is, When an admin tool is supposed to run, the DB user needs to have a different set of privileges (CREATE TABLE and DROP) rather than what DB user a server need to be functional ? Is the concern that it might be confusing for the user and they forget they need to do this and might end up giving the same ?

flyrain · 2025-05-23T16:43:56Z

we still have separate docker images

I have a doubt in this area, and would like to understand this more, why we need two separate image if we agree on same binary distribution ? My understanding is, When an admin tool is supposed to run, the DB user needs to have a different set of privileges (CREATE TABLE and DROP) rather than what DB user a server need to be functional ? Is the concern that it might be confusing for the user and they forget they need to do this and might end up giving the same ?

I also lean to a single docker image, which makes users and developers life much easier. We just need to a different entry for different functionality. As the comment pointed out, size shouldn't be a problem. But I'd suggest to move the discussion to the ML thread, https://lists.apache.org/thread/h2zxo9ho2xl6l2dpsyovllwjgv8hqp08

dimas-b · 2025-05-23T16:50:37Z

why we need two separate image if we agree on same binary distribution ?

Docker images define what runs in a container. Normally there's only one entry point. Therefore, it is more natural to have one image per tool / executable (rather than controlling what actually runs via args or cryptic env. vars, etc.).

That said, reducing image size is also valuable. A user running 100 server replicas and 1 admin tool probably would not want the extra burden of 100 admin tool binaries replicated wherever servers run.

dimas-b

LGTM overall, but it looks like README and scripts are not aligned 🤷

dimas-b · 2025-05-23T18:47:00Z

quarkus/server/build.gradle.kts

-      }
-    }
-  }
+  add("distributionElements", layout.buildDirectory.dir("quarkus-app")) { builtBy("quarkusBuild") }


After untar:

$ ls server/ app lib logs quarkus quarkus-app-dependencies.txt quarkus-run.jar

Is quarkus-app-dependencies.txt intended to be in the distro? (same for admin)

I'm OK with it. Any concern about the file?

It is noise and distraction from my POV. Not useful to end users.

Removed in the new commit.

Could we exclude quarkus-app-dependencies.txt here , rather than in the main dist module?

dimas-b · 2025-05-23T18:51:09Z

quarkus/distribution/README.md

+
+```bash
+# Configure server port
+JAVA_OPTS="-Dquarkus.http.port=8080" ./bin/server


I believe env JAVA_OPTS="-Dquarkus.http.port=8080" bin/server is more portable. IIRC, current syntax assumes running in sh or bash.

I'd also suggest using env. vars like POLARIS_JAVA_OPTS so that they could be saved in local env. without affecting other applications.

good catch, it doesn't work with the current script anymore. Removed this section.

Added support for POLARIS_JAVA_OPTS. All POSIX sh should work well without env, given this is a readme doc, I think it should be good.

dimas-b · 2025-05-23T18:52:01Z

quarkus/distribution/README.md

+JAVA_OPTS="-Dquarkus.http.port=8080" ./bin/server
+
+# Configure admin tool
+JAVA_OPTS="-Dpolaris.persistence.type=relational-jdbc" ./bin/admin


I do not think JAVA_OPTS work with the latest scripts 🤔

I see that this config example got removed from the README. How is the user supposed to pass config options to the server?

https://polaris.apache.org/in-dev/unreleased/configuration/ describes 5 different ways.

Only options 2 and 3 are applicable when using these distribution's scripts. It would be nice to mention that in the README and perhaps include an sample config/application.properties in the archive.

let me fix it, I think we should allow user to use option 1 as well.
option 4(The application.properties files packaged in Polaris.) should also work, right? User will need to put a application.properties file in the classpath.

I do not think users of the distribution bundle should put extra application.properties on the class path. The behaviour will be very uncertain in that case (with respect to the built-in application.properties in Polaris jars).

dimas-b · 2025-05-23T18:53:48Z

quarkus/distribution/README.md

+### Start the Server
+
+```bash
+./bin/server


nit: bin/server (./ is not necessary, PATH is not used anyway)

dimas-b · 2025-05-23T23:01:43Z

quarkus/distribution/build.gradle.kts

@@ -58,12 +58,16 @@ distributions {
        contents {
            // Copy admin distribution contents
            into("admin") {
-                from(adminDistribution)
+                from(adminDistribution) {
+                    exclude("quarkus-app-dependencies.txt")


It is probably preferable to do this excluding in the server and admin modules.

Tried to exclude it in server and admin module. Didn't find a clean solution. It may not be worth it. Let me know if you got better solutioin.

I still think this is sub-optimal from the gradle build perspective, but I do not have a solution ATM, so I'm fine merging this change "as is" unless someone has a solution handy.

dimas-b · 2025-05-23T23:02:54Z

@snazy : gradle script changes LGTM now... WDYT?

dimas-b · 2025-05-26T13:04:06Z

quarkus/distribution/README.md

+
+```bash
+# Configure server port
+POLARIS_JAVA_OPTS="-Dquarkus.http.port=8080" bin/server


nit: env POLARIS_JAVA_OPTS="-Dquarkus.http.port=8080" bin/server is more portable, AFAIK.

dimas-b · 2025-05-26T13:09:30Z

quarkus/server/build.gradle.kts

-      }
-    }
-  }
+  add("distributionElements", layout.buildDirectory.dir("quarkus-app")) { builtBy("quarkusBuild") }


Could we exclude quarkus-app-dependencies.txt here , rather than in the main dist module?

dimas-b · 2025-05-27T17:13:34Z

quarkus/distribution/build.gradle.kts

@@ -58,12 +58,16 @@ distributions {
        contents {
            // Copy admin distribution contents
            into("admin") {
-                from(adminDistribution)
+                from(adminDistribution) {
+                    exclude("quarkus-app-dependencies.txt")


I still think this is sub-optimal from the gradle build perspective, but I do not have a solution ATM, so I'm fine merging this change "as is" unless someone has a solution handy.

flyrain · 2025-05-27T17:17:22Z

The only solution I found for excluding quarkus-app-dependencies.txt from server and admin module is to zip it, in which case, the distribution module has to unzip it. I don't think it's worth to add additional zip and unzip process. My concern is that it is a complicated solution with worse build perf.

val quarkusAppZip by
  tasks.registering(Zip::class) {
    dependsOn(quarkusBuild)
    archiveClassifier.set("dist") 
    from(layout.buildDirectory.dir("quarkus-app")) {
      exclude("quarkus-app-dependencies.txt")
    }
  }

artifacts {
  add(quarkusRunner.name, provider { quarkusBuild.get().fastJar.resolve("quarkus-run.jar") }) {
    builtBy(quarkusBuild)
  }

  add("distributionElements", quarkusAppZip) {
    builtBy(quarkusAppZip) 
  }
}

Comments have been resolved. The module uses artifact dependency now.

flyrain · 2025-05-27T17:20:46Z

Thanks @jbonofre @adutra @snazy @singhpk234 @dimas-b for the review!

github-project-automation bot added this to Basic Kanban Board May 14, 2025

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board May 14, 2025

flyrain force-pushed the merge-distributions branch from 3d708aa to 30e9e7a Compare May 14, 2025 19:10

flyrain added the 1.0-blocker label May 14, 2025

flyrain mentioned this pull request May 14, 2025

Add doc for getting started with binary distribution #1590

Open

dimas-b reviewed May 14, 2025

View reviewed changes

jbonofre reviewed May 15, 2025

View reviewed changes

snazy previously requested changes May 15, 2025

View reviewed changes

adutra reviewed May 15, 2025

View reviewed changes

flyrain mentioned this pull request May 15, 2025

Remove verbose terminal output for admin tool #1598

Merged

Yufei Gu added 12 commits May 22, 2025 10:45

POLARIS-1: Fix repository configuration in custom plugins and restore…

9726cf7

… run script - Remove repository configuration from plugins, restore run.sh script, fix build issues

POLARIS-1: Remove run-script module - Remove run-script module from p…

60c3fb6

…rojects.main.properties - Remove run-script dependency from admin and server modules - Fix code formatting in build files

delete module run script

5409eb7

delete unused file

5c4ab8c

Add digest and fix the layout

3f0eddf

Remove unused build file

a7a73fb

Add signing

94624bb

Fixed README and moduel name

7280b9f

Change the distribution base name.

6fbbad8

Add ASF license header to the new files

06cd4de

Resolve comments

9e57ee8

Yufei Gu added 5 commits May 22, 2025 16:34

Remove the tarball and zip file build in admin and server module

a029404

Split run.sh to two commands server, admin.

62db142

Split run.sh to two commands server, admin.

be866e7

Update README.md

8744252

Add back DISCLAIMER

c62c61b

singhpk234 reviewed May 23, 2025

View reviewed changes

Resolve comments

76df496

dimas-b reviewed May 23, 2025

View reviewed changes

Yufei Gu added 2 commits May 23, 2025 13:28

Resolve comments

f9451c8

Resolve comments

8d0b09d

dimas-b reviewed May 23, 2025

View reviewed changes

Support command line java options

545d886

dimas-b reviewed May 26, 2025

View reviewed changes

dimas-b approved these changes May 27, 2025

View reviewed changes

flyrain merged commit 9001785 into apache:main May 27, 2025
9 checks passed

github-project-automation bot moved this from PRs In Progress to Done in Basic Kanban Board May 27, 2025


		description = "Apache Polaris Binary Distribution"

		val adminProject = project(":polaris-quarkus-admin")

		@@ -0,0 +1,69 @@
		# Licensed to the Apache Software Foundation (ASF) under one

Create a single binary distribution bundle #1589

Create a single binary distribution bundle #1589

Uh oh!

Conversation

flyrain commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jbonofre commented May 14, 2025

Uh oh!

flyrain commented May 14, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adutra commented May 15, 2025

Uh oh!

flyrain commented May 15, 2025

Uh oh!

adutra commented May 15, 2025

Uh oh!

dimas-b commented May 16, 2025

Uh oh!

adutra commented May 16, 2025

Uh oh!

snazy commented May 19, 2025

Uh oh!

flyrain commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

flyrain commented May 23, 2025

Uh oh!

jbonofre commented May 23, 2025

Uh oh!

jbonofre commented May 23, 2025

Uh oh!

flyrain commented May 23, 2025

Uh oh!

jbonofre commented May 23, 2025

Uh oh!

jbonofre commented May 23, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flyrain commented May 14, 2025 •

edited

Loading

dimas-b May 22, 2025 •

edited

Loading

flyrain commented May 22, 2025 •

edited

Loading

flyrain May 23, 2025 •

edited

Loading