Skip to content

Commit c7b03ac

Browse files
tomasstupkatomasstupka
committed
[GR-57985] Maven/Gradle support for freezing dependencies.
PullRequest: graalpython/3659
2 parents ef09009 + e12f6dd commit c7b03ac

File tree

31 files changed

+3105
-697
lines changed

31 files changed

+3105
-697
lines changed

‎ci.jsonnet

+1-1
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
{ "overlay": "ac2b03008a765064fba41da97cbcd096f6d19809" }
1+
{ "overlay": "9682e88c454813f57f9104b481421538895239b5" }

‎docs/user/Embedding-Build-Tools.md

+72-4
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,36 @@ Any manual change will be overridden by the plugin during the build.
7979

8080
The _src_ subdirectory is left to be manually populated by the user with custom Python scripts or modules.
8181

82-
## GraalPy Maven Plugin Configuration
82+
To manage third-party Python packages, a [Python virtual environment](https://docs.python.org/3.11/tutorial/venv.html) is used behind the scenes.
83+
Whether deployed in a virtual filesystem or an external directory, its contents are managed by the plugin based on the Python packages
84+
specified in the plugin configuration.
85+
86+
## Python Dependency Management
87+
The list of third-party Python packages to be downloaded and installed can be specified in Maven or Gradle plugin configuration. Unfortunately,
88+
Python does not enforce strict versioning of dependencies, which can result in problems if a third-party package or one of its transitive
89+
dependencies is unexpectedly updated to a newer version, leading to unforeseen behavior.
90+
91+
It is regarded as good practice to always specify a Python package with its exact version. In simpler scenarios, where only a few packages
92+
are required, specifying the exact version of each package in the plugin configuration,
93+
along with their transitive dependencies, might be sufficient. However, this method is often impractical,
94+
as manually managing the entire dependency tree can quickly become overwhelming.
95+
96+
### Locking Dependencies
97+
98+
For these cases, we **highly recommend locking** all Python dependencies whenever there is a change
99+
in the list of required packages for a project. The GraalPy plugins provide an action to do so,
100+
and as a result, a GraalPy lock file will be created, listing all required Python packages with their specific versions
101+
based on the packages defined in the plugin configuration and their dependencies. Subsequent GraalPy plugin executions
102+
will then use this file exclusively to install all packages with guaranteed versions.
103+
104+
The default location of the lock file is in the project root, and since it serves as input for generating resources,
105+
it should be stored alongside other project files in a version control system.
106+
107+
For information on the specific Maven or Gradle lock packages actions, please refer to the plugin descriptions below in this document.
108+
109+
## GraalPy Maven Plugin
110+
111+
### Maven Plugin Configuration
83112

84113
Add the plugin configuration in the `configuration` block of `graalpy-maven-plugin` in the _pom.xml_ file:
85114
```xml
@@ -104,7 +133,15 @@ The Python packages and their versions are specified as if used with `pip`:
104133
...
105134
</configuration>
106135
```
107-
136+
- The **graalPyLockFile** element can specify an alternative path to a GraalPy lock file.
137+
Default value is `${basedir}/graalpy.lock`.
138+
```xml
139+
<configuration>
140+
<graalPyLockFile>${basedir}/graalpy.lock</graalPyLockFile>
141+
...
142+
</configuration>
143+
```
144+
108145
- The **resourceDirectory** element can specify the relative [Java resource path](#java-resource-path).
109146
Remember to use `VirtualFileSystem$Builder#resourceDirectory` when configuring the `VirtualFileSystem` in Java.
110147
```xml
@@ -119,9 +156,21 @@ Remember to use the appropriate `GraalPyResources` API to create the Context. Th
119156
...
120157
</configuration>
121158
```
159+
160+
### Locking Python Packages
161+
To lock the dependency tree of the specified Python packages, execute the GraalPy plugin goal `org.graalvm.python:graalpy-maven-plugin:lock-packages`.
162+
```bash
163+
$ mvn org.graalvm.python:graalpy-maven-plugin:lock-packages
164+
```
165+
*Note that the action will override the existing lock file.*
122166

123-
## GraalPy Gradle Plugin Configuration
167+
For more information on managing Python packages, please refer to the descriptions of
168+
the `graalPyLockFile` and `packages` fields in the [plugin configuration](#maven-plugin-configuration), as well as the [Python Dependency Management](#python-dependency-management) section
169+
above in this document.
124170

171+
## GraalPy Gradle Plugin
172+
173+
### Gradle Plugin Configuration
125174
The plugin must be added to the plugins section in the _build.gradle_ file.
126175
The **version** property defines which version of GraalPy to use.
127176
```
@@ -146,6 +195,15 @@ The plugin can be configured in the `graalPy` block:
146195
}
147196
```
148197

198+
- The **graalPyLockFile** element can specify an alternative path to a GraalPy lock file.
199+
Default value is `$rootDir/graalpy.lock`.
200+
```
201+
graalPy {
202+
graalPyLockFile = file("$rootDir/graalpy.lock")
203+
...
204+
}
205+
```
206+
149207
- The **resourceDirectory** element can specify the relative [Java resource path](#java-resource-path).
150208
Remember to use `VirtualFileSystem$Builder#resourceDirectory` when configuring the `VirtualFileSystem` in Java.
151209
```
@@ -168,8 +226,18 @@ dependency `org.graalvm.python:python` to the community build: `org.graalvm.pyth
168226
...
169227
}
170228
```
229+
### Locking Python Packages
230+
To lock the dependency tree of the specified Python packages, execute the GraalPy plugin task `graalPyLockPackages`.
231+
```bash
232+
$ gradle graalPyLockPackages
233+
```
234+
*Note that the action will override the existing lock file.*
235+
236+
For more information on managing Python packages, please refer to the descriptions of
237+
the `graalPyLockFile` and `packages` fields in the [plugin configuration](#gradle-plugin-configuration), as well as the [Python Dependency Management](#python-dependency-management) sections
238+
in this document.
171239

172-
### Related Documentation
240+
## Related Documentation
173241

174242
* [Embedding Graal languages in Java](https://www.graalvm.org/reference-manual/embed-languages/)
175243
* [Permissions for Python Embeddings](Embedding-Permissions.md)

‎graalpython/com.oracle.graal.python.test.integration/src/org/graalvm/python/embedding/test/integration/GraalPyResourcesUtilsTests.java

+1-1
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@
4848
import java.nio.file.Path;
4949

5050
/**
51-
* Siple copy of GraalPyResourcesTests to test also the deprecated
51+
* Simple copy of GraalPyResourcesTests to test also the deprecated
5252
* org.graalvm.python.embedding.utils pkg
5353
*/
5454
@SuppressWarnings("deprecation")

‎graalpython/com.oracle.graal.python.test.integration/src/org/graalvm/python/embedding/test/integration/VirtualFileSystemIntegrationUtilsTest.java

+1-1
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,7 @@
7171
import static org.junit.Assert.assertTrue;
7272

7373
/**
74-
* Siple copy of VirtualFileSystemIntegrationTest to test also the deprecated
74+
* Simple copy of VirtualFileSystemIntegrationTest to test also the deprecated
7575
* org.graalvm.python.embedding.utils pkg
7676
*/
7777
@SuppressWarnings("deprecation")

‎graalpython/com.oracle.graal.python.test/src/org/graalvm/python/embedding/cext/test/MultiContextCExtTest.java

+78-68
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright (c) 2023, 2024, Oracle and/or its affiliates. All rights reserved.
2+
* Copyright (c) 2023, 2025, Oracle and/or its affiliates. All rights reserved.
33
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
44
*
55
* The Universal Permissive License (UPL), Version 1.0
@@ -46,33 +46,32 @@
4646
import static org.junit.Assert.assertTrue;
4747
import static org.junit.Assert.fail;
4848

49-
import java.io.File;
5049
import java.io.IOException;
5150
import java.nio.file.Files;
5251
import java.nio.file.Path;
5352
import java.util.ArrayList;
54-
import java.util.Arrays;
55-
import java.util.Comparator;
56-
import java.util.Set;
5753
import java.util.logging.Handler;
5854
import java.util.logging.LogRecord;
5955

6056
import org.graalvm.polyglot.Context;
6157
import org.graalvm.polyglot.Engine;
6258
import org.graalvm.polyglot.PolyglotException;
6359
import org.graalvm.polyglot.Source;
64-
import org.graalvm.python.embedding.tools.exec.SubprocessLog;
65-
import org.graalvm.python.embedding.tools.vfs.VFSUtils;
66-
import org.graalvm.python.embedding.tools.vfs.VFSUtils.Log;
60+
import org.graalvm.python.embedding.tools.exec.BuildToolLog;
6761
import org.junit.Test;
6862

63+
import static org.graalvm.python.embedding.test.EmbeddingTestUtils.deleteDirOnShutdown;
64+
import static org.graalvm.python.embedding.test.EmbeddingTestUtils.createVenv;
65+
6966
public class MultiContextCExtTest {
70-
static final class TestLog extends Handler implements SubprocessLog, Log {
67+
static final class TestLog extends Handler implements BuildToolLog {
7168
final StringBuilder logCharSequence = new StringBuilder();
7269
final StringBuilder logThrowable = new StringBuilder();
7370
final StringBuilder stderr = new StringBuilder();
7471
final StringBuilder stdout = new StringBuilder();
7572
final StringBuilder info = new StringBuilder();
73+
final StringBuilder warn = new StringBuilder();
74+
final StringBuilder debug = new StringBuilder();
7675
final StringBuilder truffleLog = new StringBuilder();
7776

7877
static void println(CharSequence... args) {
@@ -81,103 +80,114 @@ static void println(CharSequence... args) {
8180
}
8281
}
8382

84-
public void log(CharSequence txt) {
85-
println("[log]", txt);
86-
logCharSequence.append(txt);
87-
}
88-
89-
public void log(CharSequence txt, Throwable t) {
90-
println("[log]", txt);
83+
@Override
84+
public void warning(String txt, Throwable t) {
85+
println("[warning]", txt);
9186
println("[throwable]", t.getMessage());
9287
logThrowable.append(txt).append(t.getMessage());
9388
}
9489

95-
public void subProcessErr(CharSequence err) {
90+
@Override
91+
public void error(String s) {
92+
println("[err]", s);
93+
stderr.append(s);
94+
}
95+
96+
@Override
97+
public void debug(String s) {
98+
println("[debug]", s);
99+
debug.append(s);
100+
}
101+
102+
@Override
103+
public void subProcessErr(String err) {
96104
println("[err]", err);
97105
stderr.append(err);
98106
}
99107

100-
public void subProcessOut(CharSequence out) {
108+
@Override
109+
public void subProcessOut(String out) {
101110
println("[out]", out);
102111
stdout.append(out);
103112
}
104113

114+
@Override
105115
public void info(String s) {
106116
println("[info]", s);
107117
info.append(s);
108118
}
109119

110120
@Override
111-
public void publish(LogRecord record) {
112-
var msg = String.format("[%s] %s: %s", record.getLoggerName(), record.getLevel().getName(), String.format(record.getMessage(), record.getParameters()));
113-
println(msg);
114-
truffleLog.append(msg);
121+
public void warning(String s) {
122+
println("[warning]", s);
123+
warn.append(s);
115124
}
116125

117126
@Override
118-
public void flush() {
127+
public boolean isDebugEnabled() {
128+
return true;
119129
}
120130

121131
@Override
122-
public void close() {
132+
public boolean isWarningEnabled() {
133+
return true;
123134
}
124-
}
125135

126-
private static Path createVenv(TestLog log, String... packages) throws IOException {
127-
var tmpdir = Files.createTempDirectory("graalpytest");
128-
deleteDirOnShutdown(tmpdir);
129-
var venvdir = tmpdir.resolve("venv");
130-
try {
131-
VFSUtils.createVenv(venvdir, Arrays.asList(packages), tmpdir.resolve("graalpy.exe"), () -> getClasspath(), "", log, log);
132-
} catch (RuntimeException e) {
133-
System.err.println(getClasspath());
134-
throw e;
136+
@Override
137+
public boolean isInfoEnabled() {
138+
return true;
135139
}
136-
return venvdir;
137-
}
138140

139-
private static void deleteDirOnShutdown(Path tmpdir) {
140-
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
141-
try (var fs = Files.walk(tmpdir)) {
142-
fs.sorted(Comparator.reverseOrder()).map(Path::toFile).forEach(File::delete);
143-
} catch (IOException e) {
144-
}
145-
}));
146-
}
141+
@Override
142+
public boolean isErrorEnabled() {
143+
return true;
144+
}
147145

148-
private static Set<String> getClasspath() {
149-
var sb = new ArrayList<String>();
150-
var modPath = System.getProperty("jdk.module.path");
151-
if (modPath != null) {
152-
sb.add(modPath);
146+
@Override
147+
public boolean isSubprocessOutEnabled() {
148+
return true;
153149
}
154-
var classPath = System.getProperty("java.class.path");
155-
if (classPath != null) {
156-
sb.add(classPath);
150+
151+
@Override
152+
public void publish(LogRecord record) {
153+
var msg = String.format("[%s] %s: %s", record.getLoggerName(), record.getLevel().getName(), String.format(record.getMessage(), record.getParameters()));
154+
println(msg);
155+
truffleLog.append(msg);
156+
}
157+
158+
@Override
159+
public void flush() {
160+
}
161+
162+
@Override
163+
public void close() {
157164
}
158-
var cp = String.join(File.pathSeparator, sb);
159-
return Set.copyOf(Arrays.stream(cp.split(File.pathSeparator)).toList());
160165
}
161166

162167
@Test
163168
public void testCreatingVenvForMulticontext() throws IOException {
164169
var log = new TestLog();
165-
Path venv;
170+
Path tmpdir = Files.createTempDirectory("MultiContextCExtTest");
171+
Path venvDir = tmpdir.resolve("venv");
172+
deleteDirOnShutdown(tmpdir);
166173

167174
String pythonNative;
168175
String exe;
169176
if (System.getProperty("os.name").toLowerCase().contains("win")) {
170177
pythonNative = "python-native.dll";
171-
venv = createVenv(log, "delvewheel==1.9.0");
172-
exe = venv.resolve("Scripts").resolve("python.exe").toString().replace('\\', '/');
178+
createVenv(venvDir, "0.1", log, "delvewheel==1.9.0");
179+
180+
exe = venvDir.resolve("Scripts").resolve("python.exe").toString().replace('\\', '/');
173181
} else if (System.getProperty("os.name").toLowerCase().contains("mac")) {
174182
pythonNative = "libpython-native.dylib";
175-
venv = createVenv(log);
176-
exe = venv.resolve("bin").resolve("python").toString();
183+
createVenv(venvDir, "0.1", log);
184+
185+
exe = venvDir.resolve("bin").resolve("python").toString();
177186
} else {
178187
pythonNative = "libpython-native.so";
179-
venv = createVenv(log, "patchelf");
180-
exe = venv.resolve("bin").resolve("python").toString();
188+
createVenv(venvDir, "0.1", log, "patchelf");
189+
190+
exe = venvDir.resolve("bin").resolve("python").toString();
181191
}
182192

183193
var engine = Engine.newBuilder("python").logHandler(log).build();
@@ -188,11 +198,11 @@ public void testCreatingVenvForMulticontext() throws IOException {
188198
Context c0, c1, c2, c3, c4, c5;
189199
contexts.add(c0 = builder.build());
190200
c0.initialize("python");
191-
c0.eval("python", String.format("__graalpython__.replicate_extensions_in_venv('%s', 2)", venv.toString().replace('\\', '/')));
201+
c0.eval("python", String.format("__graalpython__.replicate_extensions_in_venv('%s', 2)", venvDir.toString().replace('\\', '/')));
192202

193-
assertTrue("created a copy of the capi", Files.list(venv).anyMatch((p) -> p.getFileName().toString().startsWith(pythonNative) && p.getFileName().toString().endsWith(".dup0")));
194-
assertTrue("created another copy of the capi", Files.list(venv).anyMatch((p) -> p.getFileName().toString().startsWith(pythonNative) && p.getFileName().toString().endsWith(".dup1")));
195-
assertFalse("created no more copies of the capi", Files.list(venv).anyMatch((p) -> p.getFileName().toString().startsWith(pythonNative) && p.getFileName().toString().endsWith(".dup2")));
203+
assertTrue("created a copy of the capi", Files.list(venvDir).anyMatch((p) -> p.getFileName().toString().startsWith(pythonNative) && p.getFileName().toString().endsWith(".dup0")));
204+
assertTrue("created another copy of the capi", Files.list(venvDir).anyMatch((p) -> p.getFileName().toString().startsWith(pythonNative) && p.getFileName().toString().endsWith(".dup1")));
205+
assertFalse("created no more copies of the capi", Files.list(venvDir).anyMatch((p) -> p.getFileName().toString().startsWith(pythonNative) && p.getFileName().toString().endsWith(".dup2")));
196206

197207
builder.option("python.IsolateNativeModules", "true");
198208
contexts.add(c1 = builder.build());
@@ -211,7 +221,7 @@ public void testCreatingVenvForMulticontext() throws IOException {
211221
// First one works
212222
var r1 = c1.eval(code);
213223
assertEquals("tiny_sha3", r1.asString());
214-
assertFalse("created no more copies of the capi", Files.list(venv).anyMatch((p) -> p.getFileName().toString().startsWith(pythonNative) && p.getFileName().toString().endsWith(".dup2")));
224+
assertFalse("created no more copies of the capi", Files.list(venvDir).anyMatch((p) -> p.getFileName().toString().startsWith(pythonNative) && p.getFileName().toString().endsWith(".dup2")));
215225
// Second one works because of isolation
216226
var r2 = c2.eval(code);
217227
assertEquals("tiny_sha3", r2.asString());
@@ -221,10 +231,10 @@ public void testCreatingVenvForMulticontext() throws IOException {
221231
// first context is unaffected
222232
r1 = c1.eval(code);
223233
assertEquals("tiny_sha3", r1.asString());
224-
assertFalse("created no more copies of the capi", Files.list(venv).anyMatch((p) -> p.getFileName().toString().startsWith(pythonNative) && p.getFileName().toString().endsWith(".dup2")));
234+
assertFalse("created no more copies of the capi", Files.list(venvDir).anyMatch((p) -> p.getFileName().toString().startsWith(pythonNative) && p.getFileName().toString().endsWith(".dup2")));
225235
// Third one works and triggers a dynamic relocation
226236
c3.eval(code);
227-
assertTrue("created another copy of the capi", Files.list(venv).anyMatch((p) -> p.getFileName().toString().startsWith(pythonNative) && p.getFileName().toString().endsWith(".dup2")));
237+
assertTrue("created another copy of the capi", Files.list(venvDir).anyMatch((p) -> p.getFileName().toString().startsWith(pythonNative) && p.getFileName().toString().endsWith(".dup2")));
228238
// Fourth one does not work because we changed the sys.prefix
229239
c4.eval("python", "import sys; sys.prefix = 12");
230240
try {

0 commit comments

Comments
 (0)