Hot Code Replacement Question

[c-koell]

Java -version output

openjdk version "21.0.5" 2024-10-15 LTS
IBM Semeru Runtime Open Edition 21.0.5.11 (build 21.0.5+11-LTS)
Eclipse OpenJ9 VM 21.0.5.11 (build openj9-0.48.0, JRE 21 Windows 10 amd64-64-Bit Compressed References 20241015_305 (JIT enabled, AOT enabled)
OpenJ9 - 1d58314
OMR - d10a4d553
JCL - b1b311c53fe based on jdk-21.0.5+11)

Summary of problem

We are developing with eclipse on openliberty. During a debug session the Hot Code Replacement (HCR) worked fine in the past. For a while now we have more and more the feeling that the HCR does not work anymore. (we had upgraded to the latest release)
Now i have tried to find out the exact circumstances. I have tried with IBM Semeru jdk-21.0.3+9_openj9-0.44.0 and with that version the code changes will be picked up correctly.
With the current release jdk-21.0.5+11_openj9-0.48.0 a simple code change will not be picked up anymore.

We see also following log entry

[INFO] JVMVMUT015W Warnung: Erweiterte HCR ist deaktiviert, aktivieren Sie sie mit -XX:+EnableExtendedHCR. Diese Funktion ist veraltet. In Zukunft wird die erweiterte HCR-Funktion und die Option, sie zu aktivieren, aus allen Versionen vollständig entfernt.

translation:

Warning\: Extended HCR is disabled, enable with -XX\:+EnableExtendedHCR. This feature is deprecated. In the future, the extended HCR capability, and the option to enable it, will be completely removed from all versions.
If i set the System Property -XX:+EnableExtendedHCR the code changes work fine again.

So the question is, is it mandatory now to set the System Property ? But as the log entry says it is a old function that will be removed.

Thanks for any help

[pshipton]

@tajila @babsingh fyi

[babsingh]

is it mandatory now to set the System Property ?

Yes, the -XX:+EnableExtendedHCR option must be explicitly specified to use Extended HCR.

More background:
A discussion on this change can be found #17857 (comment). Since supporting Extended HCR has become infeasible, we announced its deprecation and planned removal in phases. The approach includes disabling it by default while allowing re-enablement, gathering feedback over multiple release cycles, and eventually removing it.

A detailed explanation is available in eclipse-openj9/openj9-docs#1352. Key points:

• Extended HCR is now disabled by default (except for IBM Java 8), which changes behavior from previous releases.
• Users encountering JVMTI failures due to this change will see the warning:
  "Warning: Extended HCR is disabled, enable with -XX:+EnableExtendedHCR."
• Deprecation has been announced, with removal planned for JDK 25. Extended HCR support will also be removed from legacy releases like JDK 8.

For migration details, refer to the Semeru Runtimes Migration Guide.

[c-koell]

@babsingh thank you for the explanation about the Extended HCR.

But for me there are still a few things that are unclear. If i want to have Hot Code Replacement during a Debug Session in Eclipse/VSCode, must i enable this feature from now on ? And if so will the hot code replacement will be removed in future releases at all ?

Thank you for any information !

[babsingh]

Basic HCR and Extended HCR are distinct features. If you want to use Basic HCR during a debug session in Eclipse/VSCode, no additional configuration is required. It remains enabled by default in OpenJ9 and supports method body modifications.

However, if you need Extended HCR (which allows structural changes like adding/removing methods or fields), you must explicitly enable it using -XX:+EnableExtendedHCR. This feature is deprecated and planned for removal, but Basic HCR will continue to be supported.

Adding members more experienced with HCR to validate my statements: @jdmpapin, @gacholio.

[jdmpapin]

That's right. I'll just add that the restrictions we impose on (basic) HCR are the same ones that OpenJDK imposes.

I'm curious, would you mind telling us what kind of simple code change is being rejected? The only case I know of where a change to a method body at the source level can add or remove methods at the bytecode level is when adding, removing, or reordering lambda expressions.

[gacholio]

@babsingh This is my understanding.

[c-koell]

@jdmpapin @gacholio thank you very much for the clarification! Now for me its more clear how HCR works and what the difference is between Basic and Extended.

I have one easy scenario and maybe you can explain me if it is a valid behaviour of Basic HCR.

I have following simple REST Endpoint

@Path("/sample")
public class SampleRestEndpoint {

	@GET
	@Path("sayHello")
	public String sayHello() {
		function1();
		function2();
		return "Hello World";
	}

	private void function1() {
		System.out.println("function 1");
	}

	private void function2() {
		System.out.println("function 2");
	}
}

If i change the Method Body of sayHello() the HCR works fine.

@Path("/sample")
public class SampleRestEndpoint {

	@GET
	@Path("sayHello")
	public String sayHello() {
                // This Change will be picked up by HCR
		//function1();
		function2();
		return "Hello World";
	}

	private void function1() {
		System.out.println("function 1");
	}

	private void function2() {
		System.out.println("function 2");
	}
}

But if i add a new Method to the Class

@Path("/sample")
public class SampleRestEndpoint {

	@GET
	@Path("sayHello")
	public String sayHello() {
		function1();
		function2();
		return "Hello World";
	}

	private void function1() {
		System.out.println("function 1");
	}

	private void function2() {
		System.out.println("function 2");
	}

	private void function3() {
		System.out.println("function 3");
	}
}

i see following log message

JVMVMUT015W Warnung: Erweiterte HCR ist deaktiviert, aktivieren Sie sie mit -XX:+EnableExtendedHCR. Diese Funktion ist veraltet. In Zukunft wird die erweiterte HCR-Funktion und die Option, sie zu aktivieren, aus allen Versionen vollständig entfernt.

Now its clear to me why this message will be displayed (structural change). But from this point a change to the Method Body of sayHello() will not be picked up.

@Path("/sample")
public class SampleRestEndpoint {

	@GET
	@Path("sayHello")
	public String sayHello() {
		function1();
                // This Change will not be picked up from HCR
		//function2();
		return "Hello World";
	}

	private void function1() {
		System.out.println("function 1");
	}

	private void function2() {
		System.out.println("function 2");
	}

	private void function3() {
		System.out.println("function 3");
	}

}

If i delete the new Method and bring the class file to its original structure the changes in the Method body will work again.

@Path("/sample")
public class SampleRestEndpoint {

	@GET
	@Path("sayHello")
	public String sayHello() {
		function1();
                // This Change works now
		//function2();
		return "Hello World";
	}

	private void function1() {
		System.out.println("function 1");
	}

	private void function2() {
		System.out.println("function 2");
	}
}

Is this a valid behaviour with Basic HCR ?

[gacholio]

The HCR operation is meant to be transactional, so either it completes successfully and makes all of the specified changes, or it fails and makes no changes (I'm assuming that you're adding the method and changing the other body in the same HCR request).

[jdmpapin]

To elaborate a little bit, @gacholio's point means that yes, that is the expected behaviour.

You start with the original class. When you add function3() and recompile, the new bytecode has function3() in it, which is a structural change to the class, so redefinition fails. That is, the running JVM continues using the original class. When you then change the body of sayHello() with function3() still in the source and recompile again, the new bytecode still has function3(), which is still a structural change compared to the bytecode that the JVM is using, so redefinition fails again. Finally, deleting function3() from the source and recompiling produces new bytecode that has the same structure as the original, so redefinition succeeds.

[c-koell]

@jdmpapin @gacholio thank you very much to clarify the HCM theme !

I haven't found any of this informations on https://eclipse.dev/openj9/docs, or i have not searched well ..

[pshipton]

@tajila are additional doc updates required to cover what was discussed here?

[gacholio]

We are intending to remove extended HCR, so it might be best not to discuss it too much,

[tajila]

All the details are in https://eclipse.dev/openj9/docs/xxenableextendedhcr/. But perhaps it is not well understood what extended HCR is in the first place.