Skip to content

Commit 81f81c6

Browse files
bors[bot]Bromeon
andauthored
Merge #284
284: Compatibility with 4.1 GDExtension API r=Bromeon a=Bromeon This pull request migrates gdext to the **major GDExtension API change**, as introduced in godotengine/godot#76406. It also adopts to the new "uninitialized pointer" types, which were added in godotengine/godot#35813. Doing so renders the CI green again, unblocking other PRs that have been broken by the upstream changes. --- # Major GDExtension API update TLDR of Godot's changes: instead of one giant `GDExtensionInterface` struct with all the function pointers as data members, the C API now offers only a single function pointer `get_proc_address`. This one is passed to the entry point, in place of a pointer to the interface. With `get_proc_address`, it is possible to retrieve concrete API functions. In C++, this is done by looking up via name and casting the result to the correct function pointer type: ```cpp auto variant_call = (GDExtensionInterfaceVariantCall) get_proc_address("variant_call"); variant_call(...); ``` On the Rust side, I incorporated these changes by only modifying the FFI layer (`godot-ffi` crate), so the rest of the project is mostly unaffected. This is done by generating a `GDExtensionInterface` struct very similarly to how it existed before, in `godot-codegen`. As input, we parse the `gdextension_interface.h` header's documentation metadata. This works well so far, but we'll need to make sure with Godot devs that the doc metadata doesn't suddenly change format. --- # Uninitialized pointer types Certain FFI functions construct objects "into an uninitialized pointer" (placement new in C++). There used to be quite some confusion regarding _which_ functions do that. As a result, the C API introduced new types such as `GDExtensionUninitializedStringPtr`, which represent the uninitialized counterpart to `GDExtensionStringPtr`. These typedefs are declared as `void*` in the C API, but we turn them into strongly typed opaque pointers by post-processing the C header. As such, it is impossible to accidentally mix initialized and uninitialized pointer types. I also added a trait `AsUninit` which allows explicit conversion between the two. This may not be necessary in the future, but is helpful until we are sure about correct usage everywhere. I marked some places that may need another look as `// TODO(uninit)`. --- # Compatibility between Godot 4.0.x and 4.1+ Due to the API changes in GDExtension, extensions compiled under Godot 4.0.x are by default **not compatible** with Godot 4.1+ (which includes current `master` versions of Godot, so also our nightly CI builds). From now on, the `.gdextension` interface must contain a new property `compatibility_minimum`: ```toml [configuration] entry_symbol = "gdext_rust_init" compatibility_minimum = 4.0 [libraries] linux.debug.x86_64 = "res://../../../target/debug/libdodge_the_creeps.so" ... ``` This value must be set to `4.0` if you compile against the old GDExtension API (current gdext uses 4.0.3 by default). Set it to `4.1` if you use a Godot development version (Cargo feature `custom-godot`). If you do this wrong, gdext will abort initialization and print a descriptive error message. ## Compat bridge Since breaking changes can be very disruptive, I built a bridging layer that allows to use existing compiled extensions under Godot v4.1 (or current development versions). Because the Rust code relies on certain features only available in the newer C header (e.g. uninitialized pointers), such changes were backported via dynamic post-processing of `gdextension_interface.h`. Therefore, you don't need to mess around with `custom-godot`, just keep using gdext as-is and set `compatibility_minimum = 4.0` to run it under newer engine versions. Developing against a specific GDExtension API is still possible via patch, the prebuilt artifacts have been updated for all 4.0.x versions. For example, to use version `4.0`: ```toml [patch."https://github.com/godot-rust/godot4-prebuilt"] godot4-prebuilt = { git = "https://github.com//godot-rust/godot4-prebuilt", branch = "4.0"} ``` Once Godot 4.1.0 is released, we will likely make that the default version used by gdext, but I plan to maintain the compat layer as long as this is possible with reasonable effort. A new CI setup verifies that the integration tests run against 4.0, 4.0.1, 4.0.2, 4.0.3 and nightly Godot versions, at least for Linux. This will become a challenge as soon as there are breaking API changes (and there is already one upcoming in `Basis::looking_at()`). --- # Other changes There is now an experimental `godot::sys::GdextBuild` struct, which provides informations about static (compiled-against) and runtime (loaded-by) Godot versions. This may be extended by other build/runtime metadata and will likely be promoted to an official API in the `godot::init` module. Several memory leaks that came up during the change to uninitialized pointers were addressed. In many places, we now use `from_sys_init()` instead of `from_sys_init_default()`, avoiding the need to construct throwaway default instances. In addition to more CI jobs for integration tests, there are now also 2 more linux-memcheck ones, to cover both 4.0.3 and nightly. This is mostly to have extra confidence during the migration phase and may be removed later. Godot 4.0.3 is now the default version used by gdext. Co-authored-by: Jan Haller <[email protected]>
2 parents 8990464 + 974085c commit 81f81c6

37 files changed

+1075
-187
lines changed

.github/composite/godot-itest/action.yml

Lines changed: 49 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,11 @@ inputs:
2424
default: 'false'
2525
description: "Should the job check against latest gdextension_interface.h, and warn on difference"
2626

27+
godot-prebuilt-patch:
28+
required: false
29+
default: ''
30+
description: "If specified, sets the branch name of the godot4-prebuilt crate to this value"
31+
2732
rust-toolchain:
2833
required: false
2934
default: 'stable'
@@ -71,14 +76,56 @@ runs:
7176
rust: ${{ inputs.rust-toolchain }}
7277
with-llvm: ${{ inputs.with-llvm }}
7378

79+
- name: "Patch prebuilt version ({{ inputs.godot-prebuilt-patch }})"
80+
if: inputs.godot-prebuilt-patch != ''
81+
env:
82+
VERSION: ${{ inputs.godot-prebuilt-patch }}
83+
# sed -i'' needed for macOS compatibility, see https://stackoverflow.com/q/4247068
84+
run: |
85+
echo "Patch prebuilt version to $VERSION..."
86+
87+
# For newer versions, update the compatibility_minimum in .gdextension files to 4.1
88+
# Once a 4.1.0 is released, we can invert this and set compatibility_minimum to 4.0 for older versions.
89+
if [[ "$VERSION" == "4.1" ]]; then
90+
echo "Update compatibility_minimum in .gdextension files..."
91+
dirs=("itest" "examples")
92+
for dir in "${dirs[@]}"; do
93+
find "$dir" -type f -name "*.gdextension" -exec sed -i'.bak' 's/compatibility_minimum = 4\.0/compatibility_minimum = 4.1/' {} +
94+
done
95+
96+
# Versions 4.0.x
97+
else
98+
# Patch only needed if version is not already set
99+
if grep -E 'godot4-prebuilt = { .+ branch = "$VERSION" }' godot-bindings/Cargo.toml; then
100+
echo "Already has version $version; no need for patch."
101+
else
102+
cat << HEREDOC >> Cargo.toml
103+
[patch."https://github.com/godot-rust/godot4-prebuilt"]
104+
godot4-prebuilt = { git = "https://github.com//godot-rust/godot4-prebuilt", branch = "$VERSION" }
105+
HEREDOC
106+
echo "Patched Cargo.toml for version $version."
107+
fi
108+
fi
109+
110+
shell: bash
111+
112+
# else
113+
- name: "No patch selected"
114+
if: inputs.godot-prebuilt-patch == ''
115+
run: |
116+
echo "No patch selected; use default godot4-prebuilt version."
117+
shell: bash
118+
74119
- name: "Build gdext (itest)"
75120
run: |
76121
cargo build -p itest ${{ inputs.rust-extra-args }}
77122
shell: bash
78123
env:
79124
RUSTFLAGS: ${{ inputs.rust-env-rustflags }}
80125

81-
# Note: no longer fails, as we expect header to be forward-compatible; instead issues a warning
126+
# This step no longer fails if there's a diff, as we expect header to be forward-compatible; instead issues a warning
127+
# However, still fails if patch cannot be applied (conflict).
128+
# Step is only run in latest, not for compat 4.0.1 etc. -> no need to take into account different header versions.
82129
- name: "Copy and compare GDExtension header"
83130
if: inputs.godot-check-header == 'true'
84131
run: |
@@ -96,7 +143,7 @@ runs:
96143
echo "\`\`\`diff" >> $GITHUB_STEP_SUMMARY
97144
git diff --no-index gdextension_interface_prebuilt.h gdextension_interface.h >> $GITHUB_STEP_SUMMARY || true
98145
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
99-
echo "After manually updating file, run: \`git diff -R > tweak.patch\`." >> $GITHUB_STEP_SUMMARY
146+
echo "After manually updating file, run: \`git diff -R > tweak2.patch && mv tweak2.patch tweak.patch\`." >> $GITHUB_STEP_SUMMARY
100147
101148
# Undo modifications
102149
mv gdextension_interface_prebuilt.h gdextension_interface.h

.github/workflows/full-ci.yml

Lines changed: 62 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -127,9 +127,13 @@ jobs:
127127
# Additionally, the 'linux (msrv *)' special case will then be listed next to the other 'linux' jobs.
128128
# Note: Windows uses '--target x86_64-pc-windows-msvc' by default as Cargo argument.
129129
include:
130-
- name: macos
130+
# macOS
131+
132+
- name: macos-4.0.3
131133
os: macos-12
134+
artifact-name: macos
132135
godot-binary: godot.macos.editor.dev.x86_64
136+
godot-prebuilt-patch: '4.0.3'
133137

134138
- name: macos-double
135139
os: macos-12
@@ -142,10 +146,15 @@ jobs:
142146
godot-binary: godot.macos.editor.dev.x86_64
143147
rust-extra-args: --features godot/custom-godot
144148
with-llvm: true
149+
godot-prebuilt-patch: '4.1'
145150

146-
- name: windows
151+
# Windows
152+
153+
- name: windows-4.0.3
147154
os: windows-latest
155+
artifact-name: windows
148156
godot-binary: godot.windows.editor.dev.x86_64.exe
157+
godot-prebuilt-patch: '4.0.3'
149158

150159
- name: windows-double
151160
os: windows-latest
@@ -157,12 +166,35 @@ jobs:
157166
artifact-name: windows
158167
godot-binary: godot.windows.editor.dev.x86_64.exe
159168
rust-extra-args: --features godot/custom-godot
169+
godot-prebuilt-patch: '4.1'
170+
171+
# Linux
160172

161173
# Don't use latest Ubuntu (22.04) as it breaks lots of ecosystem compatibility.
162174
# If ever moving to ubuntu-latest, need to manually install libtinfo5 for LLVM.
163-
- name: linux
175+
- name: linux-4.0.3
176+
os: ubuntu-20.04
177+
artifact-name: linux
178+
godot-binary: godot.linuxbsd.editor.dev.x86_64
179+
godot-check-header: false # disabled for now
180+
181+
- name: linux-4.0.2
164182
os: ubuntu-20.04
183+
artifact-name: linux
165184
godot-binary: godot.linuxbsd.editor.dev.x86_64
185+
godot-prebuilt-patch: '4.0.2'
186+
187+
- name: linux-4.0.1
188+
os: ubuntu-20.04
189+
artifact-name: linux
190+
godot-binary: godot.linuxbsd.editor.dev.x86_64
191+
godot-prebuilt-patch: '4.0.1'
192+
193+
- name: linux-4.0
194+
os: ubuntu-20.04
195+
artifact-name: linux
196+
godot-binary: godot.linuxbsd.editor.dev.x86_64
197+
godot-prebuilt-patch: '4.0'
166198

167199
- name: linux-double
168200
os: ubuntu-20.04
@@ -180,27 +212,50 @@ jobs:
180212
artifact-name: linux
181213
godot-binary: godot.linuxbsd.editor.dev.x86_64
182214
rust-extra-args: --features godot/custom-godot
215+
godot-prebuilt-patch: '4.1'
183216

184217
# Special Godot binaries compiled with AddressSanitizer/LeakSanitizer to detect UB/leaks.
185218
# Additionally, the Godot source is patched to make dlclose() a no-op, as unloading dynamic libraries loses stacktrace and
186219
# cause false positives like println!. See https://github.com/google/sanitizers/issues/89.
187220
# The gcc version can possibly be removed later, as it is slower and needs a larger artifact than the clang one.
188221

189222
# --disallow-focus: fail if #[itest(focus)] is encountered, to prevent running only a few tests for full CI
190-
- name: linux-memcheck-gcc
223+
- name: linux-memcheck-gcc-4.0.3
191224
os: ubuntu-20.04
225+
artifact-name: linux-memcheck-gcc
192226
godot-binary: godot.linuxbsd.editor.dev.x86_64.san
193227
godot-args: -- --disallow-focus
194228
rust-toolchain: nightly
195229
rust-env-rustflags: -Zrandomize-layout
196230

197-
- name: linux-memcheck-clang
231+
- name: linux-memcheck-clang-4.0.3
198232
os: ubuntu-20.04
233+
artifact-name: linux-memcheck-clang
199234
godot-binary: godot.linuxbsd.editor.dev.x86_64.llvm.san
200235
godot-args: -- --disallow-focus
201236
rust-toolchain: nightly
202237
rust-env-rustflags: -Zrandomize-layout
203238

239+
- name: linux-memcheck-gcc-nightly
240+
os: ubuntu-20.04
241+
artifact-name: linux-memcheck-gcc
242+
godot-binary: godot.linuxbsd.editor.dev.x86_64.san
243+
godot-args: -- --disallow-focus
244+
rust-toolchain: nightly
245+
rust-env-rustflags: -Zrandomize-layout
246+
rust-extra-args: --features godot/custom-godot
247+
godot-prebuilt-patch: '4.1'
248+
249+
- name: linux-memcheck-clang-nightly
250+
os: ubuntu-20.04
251+
artifact-name: linux-memcheck-clang
252+
godot-binary: godot.linuxbsd.editor.dev.x86_64.llvm.san
253+
godot-args: -- --disallow-focus
254+
rust-toolchain: nightly
255+
rust-env-rustflags: -Zrandomize-layout
256+
rust-extra-args: --features godot/custom-godot
257+
godot-prebuilt-patch: '4.1'
258+
204259

205260
steps:
206261
- uses: actions/checkout@v3
@@ -211,11 +266,12 @@ jobs:
211266
artifact-name: godot-${{ matrix.artifact-name || matrix.name }}
212267
godot-binary: ${{ matrix.godot-binary }}
213268
godot-args: ${{ matrix.godot-args }}
269+
godot-prebuilt-patch: ${{ matrix.godot-prebuilt-patch }}
214270
rust-extra-args: ${{ matrix.rust-extra-args }}
215271
rust-toolchain: ${{ matrix.rust-toolchain || 'stable' }}
216272
rust-env-rustflags: ${{ matrix.rust-env-rustflags }}
217273
with-llvm: ${{ matrix.with-llvm }}
218-
godot-check-header: ${{ matrix.name == 'linux' }}
274+
godot-check-header: ${{ matrix.godot-check-header }}
219275

220276

221277
license-guard:

examples/dodge-the-creeps/godot/DodgeTheCreeps.gdextension

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
[configuration]
22
entry_symbol = "gdext_rust_init"
3+
compatibility_minimum = 4.0
34

45
[libraries]
56
linux.debug.x86_64 = "res://../../../target/debug/libdodge_the_creeps.so"

godot-bindings/Cargo.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ custom-godot = ["dep:bindgen", "dep:regex", "dep:which"]
1818
custom-godot-extheader = []
1919

2020
[dependencies]
21-
godot4-prebuilt = { optional = true, git = "https://github.com/godot-rust/godot4-prebuilt", branch = "4.0.1" }
21+
godot4-prebuilt = { optional = true, git = "https://github.com/godot-rust/godot4-prebuilt", branch = "4.0.3" }
2222

2323
# Version >= 1.5.5 for security: https://blog.rust-lang.org/2022/03/08/cve-2022-24713.html
2424
# 'unicode-gencat' needed for \d, see: https://docs.rs/regex/1.5.5/regex/#unicode-features

godot-bindings/res/tweak.patch

Lines changed: 33 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,32 +1,42 @@
11
diff --git b/godot-ffi/src/gen/gdextension_interface.h a/godot-ffi/src/gen/gdextension_interface.h
2-
index 0b7615f..6db266e 100644
2+
index 4e4f300..e1cd5fb 100644
33
--- b/godot-ffi/src/gen/gdextension_interface.h
44
+++ a/godot-ffi/src/gen/gdextension_interface.h
5-
@@ -139,22 +139,22 @@ typedef enum {
6-
7-
} GDExtensionVariantOperator;
5+
@@ -155,27 +155,27 @@ typedef enum {
6+
// - Some types have no destructor (see `extension_api.json`'s `has_destructor` field), for
7+
// them it is always safe to skip the constructor for the return value if you are in a hurry ;-)
88

99
-typedef void *GDExtensionVariantPtr;
1010
-typedef const void *GDExtensionConstVariantPtr;
11+
-typedef void *GDExtensionUninitializedVariantPtr;
1112
-typedef void *GDExtensionStringNamePtr;
1213
-typedef const void *GDExtensionConstStringNamePtr;
14+
-typedef void *GDExtensionUninitializedStringNamePtr;
1315
-typedef void *GDExtensionStringPtr;
1416
-typedef const void *GDExtensionConstStringPtr;
17+
-typedef void *GDExtensionUninitializedStringPtr;
1518
-typedef void *GDExtensionObjectPtr;
1619
-typedef const void *GDExtensionConstObjectPtr;
20+
-typedef void *GDExtensionUninitializedObjectPtr;
1721
-typedef void *GDExtensionTypePtr;
1822
-typedef const void *GDExtensionConstTypePtr;
23+
-typedef void *GDExtensionUninitializedTypePtr;
1924
-typedef const void *GDExtensionMethodBindPtr;
2025
+typedef struct __GdextVariant *GDExtensionVariantPtr;
2126
+typedef const struct __GdextVariant *GDExtensionConstVariantPtr;
27+
+typedef struct __GdextUninitializedVariant *GDExtensionUninitializedVariantPtr;
2228
+typedef struct __GdextStringName *GDExtensionStringNamePtr;
2329
+typedef const struct __GdextStringName *GDExtensionConstStringNamePtr;
30+
+typedef struct __GdextUninitializedStringName *GDExtensionUninitializedStringNamePtr;
2431
+typedef struct __GdextString *GDExtensionStringPtr;
2532
+typedef const struct __GdextString *GDExtensionConstStringPtr;
33+
+typedef struct __GdextUninitializedString *GDExtensionUninitializedStringPtr;
2634
+typedef struct __GdextObject *GDExtensionObjectPtr;
2735
+typedef const struct __GdextObject *GDExtensionConstObjectPtr;
36+
+typedef struct __GdextUninitializedObject *GDExtensionUninitializedObjectPtr;
2837
+typedef struct __GdextType *GDExtensionTypePtr;
2938
+typedef const struct __GdextType *GDExtensionConstTypePtr;
39+
+typedef struct __GdextUninitializedType *GDExtensionUninitializedTypePtr;
3040
+typedef const struct __GdextMethodBind *GDExtensionMethodBindPtr;
3141
typedef int64_t GDExtensionInt;
3242
typedef uint8_t GDExtensionBool;
@@ -38,7 +48,22 @@ index 0b7615f..6db266e 100644
3848

3949
/* VARIANT DATA I/O */
4050

41-
@@ -203,7 +203,7 @@ typedef struct {
51+
@@ -195,11 +195,11 @@ typedef struct {
52+
int32_t expected;
53+
} GDExtensionCallError;
54+
55+
-typedef void (*GDExtensionVariantFromTypeConstructorFunc)(GDExtensionVariantPtr, GDExtensionTypePtr);
56+
-typedef void (*GDExtensionTypeFromVariantConstructorFunc)(GDExtensionTypePtr, GDExtensionVariantPtr);
57+
+typedef void (*GDExtensionVariantFromTypeConstructorFunc)(GDExtensionUninitializedVariantPtr, GDExtensionTypePtr);
58+
+typedef void (*GDExtensionTypeFromVariantConstructorFunc)(GDExtensionUninitializedTypePtr, GDExtensionVariantPtr);
59+
typedef void (*GDExtensionPtrOperatorEvaluator)(GDExtensionConstTypePtr p_left, GDExtensionConstTypePtr p_right, GDExtensionTypePtr r_result);
60+
typedef void (*GDExtensionPtrBuiltInMethod)(GDExtensionTypePtr p_base, const GDExtensionConstTypePtr *p_args, GDExtensionTypePtr r_return, int p_argument_count);
61+
-typedef void (*GDExtensionPtrConstructor)(GDExtensionTypePtr p_base, const GDExtensionConstTypePtr *p_args);
62+
+typedef void (*GDExtensionPtrConstructor)(GDExtensionUninitializedTypePtr p_base, const GDExtensionConstTypePtr *p_args);
63+
typedef void (*GDExtensionPtrDestructor)(GDExtensionTypePtr p_base);
64+
typedef void (*GDExtensionPtrSetter)(GDExtensionTypePtr p_base, GDExtensionConstTypePtr p_value);
65+
typedef void (*GDExtensionPtrGetter)(GDExtensionConstTypePtr p_base, GDExtensionTypePtr r_value);
66+
@@ -224,7 +224,7 @@ typedef struct {
4267

4368
/* EXTENSION CLASSES */
4469

@@ -47,7 +72,7 @@ index 0b7615f..6db266e 100644
4772

4873
typedef GDExtensionBool (*GDExtensionClassSet)(GDExtensionClassInstancePtr p_instance, GDExtensionConstStringNamePtr p_name, GDExtensionConstVariantPtr p_value);
4974
typedef GDExtensionBool (*GDExtensionClassGet)(GDExtensionClassInstancePtr p_instance, GDExtensionConstStringNamePtr p_name, GDExtensionVariantPtr r_ret);
50-
@@ -266,7 +266,7 @@ typedef struct {
75+
@@ -287,7 +287,7 @@ typedef struct {
5176
void *class_userdata; // Per-class user data, later accessible in instance bindings.
5277
} GDExtensionClassCreationInfo;
5378

@@ -56,7 +81,7 @@ index 0b7615f..6db266e 100644
5681

5782
/* Method */
5883

59-
@@ -323,7 +323,7 @@ typedef struct {
84+
@@ -345,7 +345,7 @@ typedef struct {
6085

6186
/* SCRIPT INSTANCE EXTENSION */
6287

@@ -65,7 +90,7 @@ index 0b7615f..6db266e 100644
6590

6691
typedef GDExtensionBool (*GDExtensionScriptInstanceSet)(GDExtensionScriptInstanceDataPtr p_instance, GDExtensionConstStringNamePtr p_name, GDExtensionConstVariantPtr p_value);
6792
typedef GDExtensionBool (*GDExtensionScriptInstanceGet)(GDExtensionScriptInstanceDataPtr p_instance, GDExtensionConstStringNamePtr p_name, GDExtensionVariantPtr r_ret);
68-
@@ -353,13 +353,13 @@ typedef GDExtensionBool (*GDExtensionScriptInstanceRefCountDecremented)(GDExtens
93+
@@ -375,13 +375,13 @@ typedef GDExtensionBool (*GDExtensionScriptInstanceRefCountDecremented)(GDExtens
6994
typedef GDExtensionObjectPtr (*GDExtensionScriptInstanceGetScript)(GDExtensionScriptInstanceDataPtr p_instance);
7095
typedef GDExtensionBool (*GDExtensionScriptInstanceIsPlaceholder)(GDExtensionScriptInstanceDataPtr p_instance);
7196

0 commit comments

Comments
 (0)