Java 17, Records, Reflection: New Method or Constructor found as reachable after static analysis

[ipsi]

So I've got a relatively simple (I think) application that I'm trying to natively compile. It's using Jackson to parse some HTTP responses into records, and Graal is having some issues with this.

Code sample:

    public static PdfDocument downloadFile(String fileId, String token) throws Exception {
        ...
        var fileContentRequest = HttpRequest.newBuilder(URI.create(fileDownloadResponse.message().downloadUrl()))
                .GET()
                .header("Authorization", "Bearer " + accessToken)
                .build();
        ...
    }

    public record DtrpgFileMessage(
            @JsonProperty("file_tasks_id") String fileTasksId,
            String progress,
            @JsonProperty("download_url") String downloadUrl
    ) {
    }

    public record DtrpgFileResponse(String status, DtrpgFileMessage message) {
    }

Build output:

Warning: Could not resolve com.intellij.rt.execution.application.AppMainV2$Agent for reflection configuration. Reason: java.lang.ClassNotFoundException: com.intellij.rt.execution.application.AppMainV2$Agent.
[1/7] Initializing...                                                                                    (4.1s @ 0.18GB)
 Version info: 'GraalVM 22.1.0 Java 17 CE'
 C compiler: cc (apple, x86_64, 13.0.0)
 Garbage collector: Serial GC
[2/7] Performing analysis...  [**********]                                                              (24.6s @ 3.63GB)
   9,110 (89.39%) of 10,191 classes reachable
  16,739 (66.63%) of 25,123 fields reachable
  45,388 (58.96%) of 76,980 methods reachable
     382 classes, 1,039 fields, and 1,139 methods registered for reflection
      67 classes,    71 fields, and    64 methods registered for JNI access
[3/7] Building universe...                                                                               (4.0s @ 1.61GB)
[4/7] Parsing methods...      [*]                                                                        (1.5s @ 2.55GB)
[5/7] Inlining methods...     [****]                                                                     (2.2s @ 2.34GB)
[6/7] Compiling methods...    [****]                                                                    (20.3s @ 4.09GB)
[7/7] Creating image...
                                                                                  (0.0s @ 1.49GB)
Fatal error: com.oracle.svm.core.util.VMError$HostedError: com.oracle.svm.core.util.VMError$HostedError: New Method or Constructor found as reachable after static analysis: public name.ipsi.project.fwbp.dtrpg.Downloader$DtrpgFileMessage name.ipsi.project.fwbp.dtrpg.Downloader$DtrpgFileResponse.message()
        at com.oracle.svm.core.util.VMError.shouldNotReachHere(VMError.java:72)
        at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:678)
        at com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:515)
        at com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:407)
        at com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:585)
        at com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:128)
        at com.oracle.svm.hosted.NativeImageGeneratorRunner$JDK9Plus.main(NativeImageGeneratorRunner.java:615)
Caused by: com.oracle.svm.core.util.VMError$HostedError: New Method or Constructor found as reachable after static analysis: public name.ipsi.project.fwbp.dtrpg.Downloader$DtrpgFileMessage name.ipsi.project.fwbp.dtrpg.Downloader$DtrpgFileResponse.message()
        at com.oracle.svm.core.util.VMError.shouldNotReachHere(VMError.java:68)
        at com.oracle.svm.reflect.hosted.ReflectionFeature.getOrCreateAccessor(ReflectionFeature.java:96)
        at com.oracle.svm.reflect.hosted.ExecutableAccessorComputer.compute(ExecutableAccessorComputer.java:50)
        at com.oracle.svm.hosted.substitute.ComputedValueField.computeValue(ComputedValueField.java:383)
        at com.oracle.svm.hosted.substitute.ComputedValueField.readValue(ComputedValueField.java:352)
        at com.oracle.svm.core.meta.ReadableJavaField.readFieldValue(ReadableJavaField.java:38)
        at com.oracle.svm.hosted.ameta.AnalysisConstantReflectionProvider.readValue(AnalysisConstantReflectionProvider.java:102)
        at com.oracle.svm.hosted.meta.HostedField.readValue(HostedField.java:154)
        at com.oracle.svm.hosted.image.NativeImageHeap.addObjectToImageHeap(NativeImageHeap.java:439)
        at com.oracle.svm.hosted.image.NativeImageHeap.addObject(NativeImageHeap.java:295)
        at com.oracle.svm.hosted.image.NativeImageHeap.processAddObjectWorklist(NativeImageHeap.java:598)
        at com.oracle.svm.hosted.image.NativeImageHeap.addTrailingObjects(NativeImageHeap.java:198)
        at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:660)
        ... 5 more
------------------------------------------------------------------------------------------------------------------------
                        5.8s (9.3% of total time) in 29 GCs | Peak RSS: 7.03GB | CPU load: 8.21
========================================================================================================================

Prior to that, I ran the application (exercising this code path) with the following:

/Library/Java/JavaVirtualMachines/graalvm-ce-java17-22.1.0/Contents/Home/bin/java -agentlib:native-image-agent=config-output-dir=src/main/resources/META-INF/native-image -javaagent:/Users/ipsi/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-0/221.5787.30/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=51799:/Users/ipsi/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-0/221.5787.30/IntelliJ IDEA.app/Contents/bin -Dfile.encoding=UTF-8 -classpath /Users/ipsi/IdeaProjects/foundry-wod-book-parser/target/classes:/Users/ipsi/.m2/repository/com/itextpdf/barcodes/7.2.2/barcodes-7.2.2.jar:/Users/ipsi/.m2/repository/org/slf4j/slf4j-api/1.7.36/slf4j-api-1.7.36.jar:/Users/ipsi/.m2/repository/com/itextpdf/font-asian/7.2.2/font-asian-7.2.2.jar:/Users/ipsi/.m2/repository/com/itextpdf/forms/7.2.2/forms-7.2.2.jar:/Users/ipsi/.m2/repository/com/itextpdf/hyph/7.2.2/hyph-7.2.2.jar:/Users/ipsi/.m2/repository/com/itextpdf/io/7.2.2/io-7.2.2.jar:/Users/ipsi/.m2/repository/com/itextpdf/commons/7.2.2/commons-7.2.2.jar:/Users/ipsi/.m2/repository/com/itextpdf/kernel/7.2.2/kernel-7.2.2.jar:/Users/ipsi/.m2/repository/org/bouncycastle/bcpkix-jdk15on/1.70/bcpkix-jdk15on-1.70.jar:/Users/ipsi/.m2/repository/org/bouncycastle/bcutil-jdk15on/1.70/bcutil-jdk15on-1.70.jar:/Users/ipsi/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.70/bcprov-jdk15on-1.70.jar:/Users/ipsi/.m2/repository/com/itextpdf/layout/7.2.2/layout-7.2.2.jar:/Users/ipsi/.m2/repository/com/itextpdf/pdfa/7.2.2/pdfa-7.2.2.jar:/Users/ipsi/.m2/repository/com/itextpdf/sign/7.2.2/sign-7.2.2.jar:/Users/ipsi/.m2/repository/com/itextpdf/styled-xml-parser/7.2.2/styled-xml-parser-7.2.2.jar:/Users/ipsi/.m2/repository/com/itextpdf/svg/7.2.2/svg-7.2.2.jar:/Users/ipsi/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.13.3/jackson-core-2.13.3.jar:/Users/ipsi/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.13.3/jackson-annotations-2.13.3.jar:/Users/ipsi/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.13.3/jackson-databind-2.13.3.jar:/Users/ipsi/.m2/repository/com/fasterxml/jackson/module/jackson-module-parameter-names/2.13.3/jackson-module-parameter-names-2.13.3.jar:/Users/ipsi/.m2/repository/com/fasterxml/jackson/datatype/jackson-datatype-jsr310/2.13.3/jackson-datatype-jsr310-2.13.3.jar:/Users/ipsi/.m2/repository/com/fasterxml/jackson/datatype/jackson-datatype-jdk8/2.13.3/jackson-datatype-jdk8-2.13.3.jar name.ipsi.project.fwbp.Main

Any suggestions on what's going on here? Is this a bug with the agent not recording enough detail for Records? Converting them to Classes instead of Records fixes things - annoying, but not a big deal right now.

[cstancu]

That seems to be a bug in the heap scanning for analysis, it misses scanning the methodAccessor/constructorAccessor for a method/constructor. Then, the later scan from roots that builds the native image heap doesn't find the necessary metadata when reading this field. I don't think it is an agent bug, if the agent missed it would fail at run time with a different error. Can you share a complete reproducible?

[ipsi]

Note: added this as an answer, but moved it here.

Heya, apologies for the delay, but I think I've found a relatively simple test case - effectively, something like this. At the weekend I will try and build a complete project with it, but it fails in my project (where other, equally simple, test cases didn't):

The only dependency is Jackson, for the annotations and for the ObjectMapper.

public class Main {
    public static void main(String[] args) throws Exception {
        test();
    }
    public record Test(
            @JsonProperty("access_token") String accessToken,
            @JsonProperty("customers_id") String customerId,
            String expires,
            @JsonProperty("server_time") String serverTime,
            String jwt
    ) {
        @JsonCreator
        public Test(
                @JsonProperty("access_token") String accessToken,
                @JsonProperty("customers_id") String customerId,
                @JsonProperty("expires") String expires,
                @JsonProperty("server_time") String serverTime,
                @JsonProperty("jwt") String jwt
        ) {
            this.accessToken = accessToken;
            this.customerId = customerId;
            this.expires = expires;
            this.serverTime = serverTime;
            this.jwt = jwt;
        }
    }

    public static void test() throws JsonProcessingException {
        var json = "{\"access_token\": \"foo\"}";
        var om = new ObjectMapper();
        var test = om.readValue(json, new TypeReference<Test>() {});
        System.out.println(test);
    }
}

This can be worked around by invoking every method with reflection, like so

Test.class.getDeclaredMethod("accessToken").invoke(test);
Test.class.getDeclaredMethod("customerId").invoke(test);
Test.class.getDeclaredMethod("expires").invoke(test);
Test.class.getDeclaredMethod("serverTime").invoke(test);
Test.class.getDeclaredMethod("jwt").invoke(test);

[cstancu]

@ipsi please open an issue with your small reproducer. Please include a complete project definition with all config files and dependencies (maven or gradle) and complete instructions to build. Did you run the agent on the small repro?

[swaechter]

Dear @cstancu. I am quite new to GraalVM & native-image and not that experienced, but I was able to create a small example to reproduce the problem (https://github.com/swaechter/recordproblem). I had to use Micronaut to reproduce the problem, so I am not sure if the problem is GraalVM agent related or one in Micronaut (Maybe @graemerocher or another person of the MN team can take a look at it?).

[cstancu]

Welcome to GraalVM! Thank you for the reproducer and easy to follow instructions! I managed to reproduce also with 23.0.0-dev. Will look into it.

[swaechter]

@cstancu Glad to hear that :) Were you able to figure out the problem?

[morph027]

Coming over from AsamK/signal-cli#1016.

Does the answer by @ipsi mean the methods should be called this way or they can be added to the reflect config?

Sorry, no java knowledge here, just trying to maintain packages 🙈 😁

[swaechter]

I ran into the same problem with records and can't change the code (The code uses Jackson too in a Micronaut environment). Any ideas (GraalVM 22.2.0 Java 17 CE)?

[visruth]

I faced the same issue and fixed it!
This error happens when a record class is used for both reflection and normal way (new SomeRecord(...) to create an object in the code.
What I did is first I removed the entry from reflect-config.json which made the New Method or Constructor found as reachable after static analysis error and I implemented a custom feature and passed it as build arg (--features=com.example.GraalVMRuntimeReflectionRegistrationFeature) while executing the native image build command. The custom feature implementation class should contain the errored record classes, eg:

com.example.GraalVMRuntimeReflectionRegistrationFeature.java:

public class GraalVMRuntimeReflectionRegistrationFeature implements Feature {

    @Override
    public void beforeAnalysis(BeforeAnalysisAccess access) {
        try {

           // The classes which  make "New Method or Constructor found as reachable after static analysis" error
            var classes = List.of(
                    SomeClass.SampleRecord.class,
                    SomeRecord.class);

            for (var eachClass : classes) {
                for (var each : eachClass.getDeclaredFields()) {
                    RuntimeReflection.register(each);
                }
                for (var each : eachClass.getDeclaredMethods()) {
                    RuntimeReflection.register(each);
                }
                for (var each : eachClass.getDeclaredConstructors()) {
                    RuntimeReflection.register(each);
                }
            }

        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

[swaechter]

Thanks a lot for the trick. I didn't even had to remove the entries from the reflect-config.json :)

[visruth]

My pleasure @swaechter!

[xiaozaiyuji]

Hi ,I was trying to native-image with "-javaagent".I find a lot of docs,only find in this.I did not know what I should do to use -javaagent in native-image,

Does this is the key of how to use -javaagent in native-image?