ByteBuddy Agent Attach Failing with IllegalStateException (Error 200) on windows

[Jefiya-MJ]

Hi @raphw ,
I’m reaching out for some advice regarding an issue we're facing with ByteBuddy's agent attachment functionality in a customer environment.

Issue Summary:
The customer reports that their Java application runs fine when the Instana agent service is not running, but fails to start with an "Access is denied" error when the agent is active. No further details are shown beyond this message:

D:\ProconnectSBM>java ApiBanking.ManualAPIProcessing
Access is denied.

The error only appears when the Instana agent attempts to attach via ByteBuddy; both the agent and the application run independently without issue.
Environment Details:

Java Version: 1.8.0_451

java version used by the application :
C: \Users\eagle>java -version
java version "1.8.0_451"
Java(TM) SE Runtime Environment (build 1.8.0_451-b10)
Java HotSpot(TM) 64-Bit Server VM (build 25.451-b10, mixed mode)

OS: Windows

Instana is using ByteBuddy agent attach via:

C:\Windows\system32>"C:\Program Files\Java\jdk-1.8\bin\java.exe" -Xms16m -Xmx16m -Xint -XX:+PerfDisableSharedMem -cp C:\Users\eagle\AppData\Local\Temp\3\instana\attach-tools-1.17.2.jar;C:\Users\eagle\AppData\Local\Temp\3\instana\javaagent-loader-1.3.70.jar com.instana.agent.loader.AgentLoaderAttach agent=C:\Users\eagle\AppData\Local\Temp\3\instana\javaagent-loader-1.3.70.jar verify=false pid=18328 host=127.0.0.1 port=42699 boot=/com.instana.agent.artifact/com.instana/instrumentation-boot/1.1.31/jar shared=/com.instana.agent.artifact/com.instana/instrumentation-shared/1.0.146/jar inMemoryClassLoader=true agentSocket=true
Attaching using net.bytebuddy.agent.VirtualMachine$ForHotSpot
Attach using net.bytebuddy.agent.VirtualMachine$ForHotSpot
Attaching to 18328
Loading C:\Users\eagle\AppData\Local\Temp\3\instana\javaagent-loader-1.3.70.jar with bootjar=/com.instana.agent.artifact/com.instana/instrumentation-boot/1.1.31/jar,sharedjar=/com.instana.agent.artifact/com.instana/instrumentation-shared/1.0.146/jar,pid=18328,inMemoryClassLoader=true,agentSocket=true
java.lang.IllegalStateException: Target VM could not dispatch command successfully: 200
        at net.bytebuddy.agent.VirtualMachine$ForHotSpot$Connection$ForJnaWindowsNamedPipe.execute(VirtualMachine.java:1050)
        at net.bytebuddy.agent.VirtualMachine$ForHotSpot.load(VirtualMachine.java:365)
        at net.bytebuddy.agent.VirtualMachine$ForHotSpot.loadAgent(VirtualMachine.java:339)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.instana.agent.loader.AgentLoaderAttach.attach(AgentLoaderAttach.java:468)
        at com.instana.agent.loader.AgentLoaderAttach.run(AgentLoaderAttach.java:160)
        at com.instana.agent.loader.AgentLoaderAttach.parseArgsAndRun(AgentLoaderAttach.java:110)
        at com.instana.agent.loader.AgentLoaderAttach.main(AgentLoaderAttach.java:93)

below is the application log entries:

Application log entries:
Level	Date and Time	Source	Event ID	Task Category
Information	9/30/2025 5:07:32 PM	Microsoft-Windows-Security-SPP	16384	None	Successfully scheduled Software Protection service for re-start at 2125-09-06T11:37:32Z. Reason: RulesEngine.
Information	9/30/2025 5:06:52 PM	Microsoft-Windows-Security-SPP	16394	None	Offline downlevel migration succeeded.
Information	9/30/2025 4:38:49 PM	instana-agent-service	0	None	The operation completed successfully. 
Error	9/30/2025 4:38:06 PM	Application Error	1000	Application Crashing Events	"Faulting application name: java.exe, version: 8.0.4510.10, time stamp: 0x67f0a87a
Faulting module name: java.exe, version: 8.0.4510.10, time stamp: 0x67f0a87a
Exception code: 0xc0000409
Fault offset: 0x00000000000573f5
Faulting process id: 2b2c
Faulting application start time: 01dc31fa4cbc9e18
Faulting application path: C:\Program Files (x86)\Common Files\Oracle\Java\java8path\java.exe
Faulting module path: C:\Program Files (x86)\Common Files\Oracle\Java\java8path\java.exe
Report Id: 89acf2ae-8616-43e4-832d-4713d7a81e8a
Faulting package full name: 
Faulting package-relative application ID: "
Information	9/30/2025 4:32:33 PM	instana-agent-service	0	None	The operation completed successfully. 
Information	9/30/2025 4:32:33 PM	instana-agent-service	0	None	The operation completed successfully. 

My question:
Would you have any insights into what might cause this "Access is denied" error in java application in conjunction with the IllegalStateException from ByteBuddy, specifically at this line
? Could this be related to permission issues when using JNA on Windows, or possibly some internal OS policy preventing named pipe communication or the AttachVirtualMachine functionality? Is there any internal security mechanism of their code that blocks the whole java process when someone tries to do a late attachment? If that is the case how to check that

Any guidance or suggestions for debugging further would be greatly appreciated. TIA

[raphw]

It seems like you are using Byte Buddy's internal attaching mechanism that emulates the Attach API. Are you using some form of custom distribution of the JDK? Some of them are stripped such that the dynamic attachment mechanism is disabled. If so, you would need to add the agent upon startup.

I would also try the regular mechanism in the JVM, to see if that works around. This is to further localize the error.