[POC] [Security Manager Replacement] Native Java Agent (dynamic code rewriting, must be low overhead) #16731

reta · 2024-11-27T11:57:07Z

Description

Explore the the native Java Agent (dynamic code rewriting, must be low overhead).

How does it work:

the application (OpenSearch) and agent use common module bootstrap
the application (OpenSearch) is run with the agent
the application (OpenSearch) uses bootstrap module apply security policies

Example:

The sample security.policy (stays the same as before):

grant codeBase "${codebase.opensearch-core}" {
   permission  java.net.SocketPermission "localhost", "connect";
};

The application (OpenSearch) is run with the agent:

-javaagent:agent-3.0.0-SNAPSHOT.jar

The application (OpenSearch) is applies security policy to the agent:

final Policy policy =  new PolicyFile("/security.policy");
AgentPolicy.setPolicy(policy);

Running with 24-ea+31-3600:

[2025-01-22T11:58:11,913][INFO ][o.o.n.Node               ] [host] version[3.0.0-SNAPSHOT], pid[101497], build[tar/7cf6a66e74d8352cf42d60c50b97e46a2aa8866c/2025-01-21T18:11:00.731851515Z], OS[Linux/6.11.0-13-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/24-ea/24-ea+31-3600]                             
[2025-01-22T11:58:11,916][INFO ][o.o.n.Node               ] [host] JVM home [/home/user/jdk-24], using bundled JDK/JRE [false]
[2025-01-22T11:58:11,916][INFO ][o.o.n.Node               ] [host] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:
+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,CLDR, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:In
itiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-12632241661790883371, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, --add-modules=jdk.incubator.vector, -Djava.util.concurrent.ForkJoin
Pool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -javaagent:agent/opensearch-agent-3.0.0-SNAPSHOT.jar, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/home/user/opensearch-3.0.0-jdk24, -Dopensearch.path.conf=/home/user/opensearch-3.0.0-jdk24/config, -Do
pensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]                                                                                                                                                                                                                                                                                       
[2025-01-22T11:58:11,916][WARN ][o.o.n.Node               ] [host] version [3.0.0-SNAPSHOT] is a pre-release version of OpenSearch and is not suitable for production                              
[2025-01-22T11:58:11,967][WARN ][o.a.l.i.v.VectorizationProvider] [host] You are running with Java 23 or later. To make full use of the Vector API, please update Apache Lucene.                                                                                                                                                 
[2025-01-22T11:58:12,347][INFO ][o.o.i.r.ReindexModulePlugin] [host] ReindexPlugin reloadSPI called                                                                                                                                                                                                                              
[2025-01-22T11:58:12,348][INFO ][o.o.i.r.ReindexModulePlugin] [host] Unable to find any implementation for RemoteReindexExtension

Related Issues

Closes #16633

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-11-27T12:07:58Z

❌ Gradle check result for 6b73ddf: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

kumargu · 2024-11-27T13:59:05Z

thanks @reta this is really interesting and such a quick progress.

On a side note, it would be useful to add a small intro snippet how the agent would work overall.

reta · 2024-11-27T14:35:24Z

thanks @reta this is really interesting and such a quick progress.

Thanks @kumargu

On a side note, it would be useful to add a small intro snippet how the agent would work overall.

Absolutely, I have updated the description (but will push it a bit once we get JDK-21 baseline with #16366, it would simplify a lot the APIs usage)

reta · 2024-12-16T19:03:02Z

libs/agent-sm/agent/build.gradle

+      "Can-Retransform-Classes": "true",
+      "Agent-Class": "org.opensearch.javaagent.Agent",
+      "Premain-Class": "org.opensearch.javaagent.Agent",
+      "Boot-Class-Path": 'byte-buddy-1.15.10.jar opensearch-agent-bootstrap-3.0.0-SNAPSHOT.jar'


opensearch-agent-bootstrap is shared between the OpenSearch service and the agent (so the Policy instance could be propagated)

github-actions · 2024-12-16T19:04:16Z

❌ Gradle check result for ea045b0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2025-01-16T15:22:09Z

This PR is stalled because it has been open for 30 days with no activity.

github-actions · 2025-01-16T16:04:30Z

❌ Gradle check result for 58a227c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-01-21T18:49:20Z

❌ Gradle check result for 5e20fde: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-01-22T17:10:30Z

❌ Gradle check result for 4688fd1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-01-22T21:51:49Z

❌ Gradle check result for 930e6ef: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

kumargu · 2025-01-28T03:45:05Z

@reta is it feasible for the agent to coexist with SM enabled in 3.0, meaning both SM and Agent will enforce socket restrictions?

reta · 2025-01-28T13:45:39Z

@reta is it feasible for the agent to coexist with SM enabled in 3.0, meaning both SM and Agent will enforce socket restrictions?

@kumargu I think it is feasible in theory but should not be necessary in practice, could you share your thoughts why we may need that?

kumargu · 2025-01-28T15:45:51Z

@reta is it feasible for the agent to coexist with SM enabled in 3.0, meaning both SM and Agent will enforce socket restrictions?

@kumargu I think it is feasible in theory but should not be necessary in practice, could you share your thoughts why we may need that?

I was thinking we could bring in replacements of JSM in 3.0 while JSM remains enabled in 3.0 (because we'd be still on JDK-21 in 3.0). Having the alternatives coexist for sometime will give us confidence and enough community feedback before we decide to remove it in some 3.x or 4.0.

(note JDK-24 LTS will be available in Sep 2025)

reta · 2025-01-28T16:01:21Z

Having the alternatives coexist for sometime will give us confidence and enough community feedback before we decide to remove it in some 3.x or 4.0.

I think we would only target a most critical APIs by Java Agent (we just cannot much it to SM), however we should be able to run Java Agent on JDK-21 at least.

kumargu · 2025-01-28T16:47:07Z

Having the alternatives coexist for sometime will give us confidence and enough community feedback before we decide to remove it in some 3.x or 4.0.

I think we would only target a most critical APIs by Java Agent (we just cannot much it to SM), however we should be able to run Java Agent on JDK-21 at least.

100% agree. Maybe just the Socket interceptor for now since we see the problems with defining the port ranges in the PR #17107

kumargu · 2025-02-19T19:41:50Z

@reta, if you get a chance, an example of this working with the simplest Shiro Plugin would be very useful :)

I am willing to get this in 3.0 even in absence of File interceptor.

reta · 2025-02-19T22:13:07Z

@reta, if you get a chance, an example of this working with the simplest Shiro Plugin would be very useful :)

Sure @kumargu , I will try to wrap the POC up this week so we could make go / no-go decision (w/r Java Agent), thanks!

github-actions · 2025-02-20T02:23:57Z

❌ Gradle check result for 0059645: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-02-23T17:36:23Z

❌ Gradle check result for 1a72cde: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-02-23T18:23:17Z

❌ Gradle check result for ec22ea6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-02-23T18:38:02Z

❌ Gradle check result for ec22ea6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

kumargu · 2025-02-23T18:58:07Z

(probably not the right thread to ask this question, pardon me).

Given, you have already done the heavy-lifting here; do you think the FileInterceptor would look something as below?

A FileInterceptor Impl

import net.bytebuddy.asm.Advice;
import java.io.File;
import java.lang.reflect.Method;
import java.security.Permission;
import java.util.List;
import java.security.ProtectionDomain;
import java.io.FilePermission;


public class FileInterceptor {

    public static final String FILE_DELETE_ACTION = "delete";
    public static final String FILE_EXECUTE_ACTION = "execute";
    public static final String FILE_READ_ACTION = "read";
    public static final String FILE_WRITE_ACTION = "write";
    public static final String FILE_READLINK_ACTION = "readlink";

    @Advice.OnMethodEnter
    public static void intercept(@Advice.This File file, @Advice.Origin Method method) throws SecurityException {
        String action = getActionFromMethod(method.getName());
        
        if (action != null) {
            checkFilePermission(file, action);
        }
    }

    private static String getActionFromMethod(String methodName) {
        switch (methodName) {
            case "delete":
            case "deleteOnExit":
                return FILE_DELETE_ACTION;
            case "canExecute":
            case "setExecutable":
                return FILE_EXECUTE_ACTION;
            case "canRead":
            case "setReadable":
            case "list":
            case "listFiles":
                return FILE_READ_ACTION;
            case "canWrite":
            case "setWritable":
            case "createNewFile":
            case "mkdir":
            case "mkdirs":
                return FILE_WRITE_ACTION;
            case "readSymbolicLink":
                return FILE_READLINK_ACTION;
         // maybe more cases?
            default:
                return null;
        }
    }

    private static void checkFilePermission(File file, String action) throws SecurityException {
        Policy policy = AgentPolicy.getPolicy();
        if (policy == null) {
            return; // No policy set, allow all actions
        }

        String path = file.getAbsolutePath();
        FilePermission permission = new FilePermission(path, action);

        StackWalker walker = StackWalker.getInstance(StackWalker.Option.RETAIN_CLASS_REFERENCE);
        List<ProtectionDomain> callers = walker.walk(new StackCallerChainExtractor());

        for (ProtectionDomain domain : callers) {
            if (!policy.implies(domain, permission)) {
                throw new SecurityException("Access denied: " + action + " on " + path);
            }
        }
    }
}

High level changes in Agent

       AgentBuilder.Transformer fileOperationsTransformer = (b, typeDescription, classLoader, module, pd) -> b.visit(
            Advice.to(FileInterceptor.class)
                .on(ElementMatchers.named("delete")
                        .or(ElementMatchers.named("canExecute"))
                        .or(ElementMatchers.named("canRead"))
                        .or(ElementMatchers.named("canWrite"))
                        .or(ElementMatchers.named("readAllBytes"))
                        .or(ElementMatchers.named("newBufferedReader"))
                        .or(ElementMatchers.named("write"))
                        .or(ElementMatchers.named("isSymbolicLink"))
                        ...
                        ...
                )
        );


        final ByteBuddy byteBuddy = new ByteBuddy().with(Implementation.Context.Disabled.Factory.INSTANCE);
        return new AgentBuilder.Default(byteBuddy)
            .with(AgentBuilder.InitializationStrategy.NoOp.INSTANCE)
            .with(AgentBuilder.RedefinitionStrategy.REDEFINITION)
            .with(AgentBuilder.RedefinitionStrategy.Listener.StreamWriting.toSystemError())
            .with(AgentBuilder.TypeStrategy.Default.REDEFINE)
            .type(ElementMatchers.isSubTypeOf(SocketChannel.class))
            .transform(socketTransformer)
            .type(ElementMatchers.isSubTypeOf(File.class).or(ElementMatchers.isSubTypeOf(Files.class)))
            .transform(fileOperationsTransformer);
    }

reta · 2025-02-23T19:36:40Z

Given, you have already done the heavy-lifting here; do you think the FileInterceptor would look something as below?

@kumargu I haven't finished the heavy lifting part yet (working on changes in distribution etc), I would like to finish with these ones first before adding any more functionality into the agent, hope it makes sense. Thanks!

github-actions · 2025-02-23T20:13:13Z

❌ Gradle check result for 1881ad9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-02-23T21:20:00Z

❌ Gradle check result for bf36ceb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-02-24T02:05:58Z

❌ Gradle check result for bea80ca: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

kumargu · 2025-02-24T04:22:45Z

Given, you have already done the heavy-lifting here; do you think the FileInterceptor would look something as below?

@kumargu I haven't finished the heavy lifting part yet (working on changes in distribution etc), I would like to finish with these ones first before adding any more functionality into the agent, hope it makes sense. Thanks!

Sound good. I was not aware that major changes in distribution is needed. Thanks for the info.

github-actions · 2025-02-24T22:54:39Z

❌ Gradle check result for ff05507: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-02-25T02:09:05Z

❌ Gradle check result for b36a5ab: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-02-25T02:37:10Z

❌ Gradle check result for d4f505e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-02-27T02:00:54Z

❌ Gradle check result for c26b3d4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

…ing, must be low overhead) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]>

github-actions · 2025-02-27T02:39:45Z

❌ Gradle check result for 45bb092: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions bot added the enhancement Enhancement or improvement to existing feature or request label Nov 27, 2024

reta added the skip-changelog label Nov 27, 2024

reta force-pushed the issue-16633 branch from 7088bf8 to 6b73ddf Compare November 27, 2024 11:58

reta mentioned this pull request Nov 27, 2024

[RFC] Replace Java Security Manager (JSM) #1687

Closed

1 task

reta force-pushed the issue-16633 branch 2 times, most recently from 9858717 to ea045b0 Compare December 16, 2024 18:58

reta commented Dec 16, 2024

View reviewed changes

opensearch-trigger-bot bot added the stalled Issues that have stalled label Jan 16, 2025

reta force-pushed the issue-16633 branch from ea045b0 to 58a227c Compare January 16, 2025 15:59

opensearch-trigger-bot bot removed the stalled Issues that have stalled label Jan 17, 2025

reta force-pushed the issue-16633 branch from 58a227c to 5e20fde Compare January 21, 2025 18:43

reta force-pushed the issue-16633 branch from 5e20fde to 4688fd1 Compare January 22, 2025 16:57

reta force-pushed the issue-16633 branch from 4688fd1 to 930e6ef Compare January 22, 2025 21:42

kumargu mentioned this pull request Jan 26, 2025

Add systemd configurations to strengthen OS core security #17107

Merged

3 tasks

kumargu mentioned this pull request Jan 29, 2025

Permanently turn-off Security Manager (JSM) starting 3.0 #17181

Open

reta force-pushed the issue-16633 branch from 930e6ef to 82d4eb8 Compare February 4, 2025 18:00

reta force-pushed the issue-16633 branch from 1d426ae to 0059645 Compare February 20, 2025 02:02

reta force-pushed the issue-16633 branch from 0059645 to 1a72cde Compare February 23, 2025 17:13

reta mentioned this pull request Feb 23, 2025

[POC] [Security Manager Replacement] GraalVM sandboxing #16863

Closed

3 tasks

reta force-pushed the issue-16633 branch from 1a72cde to ec22ea6 Compare February 23, 2025 18:05

reta force-pushed the issue-16633 branch from ec22ea6 to 1881ad9 Compare February 23, 2025 20:00

reta force-pushed the issue-16633 branch from 1881ad9 to bf36ceb Compare February 23, 2025 20:59

reta force-pushed the issue-16633 branch from bf36ceb to bea80ca Compare February 24, 2025 01:51

reta force-pushed the issue-16633 branch from bea80ca to ff05507 Compare February 24, 2025 22:27

reta force-pushed the issue-16633 branch from ff05507 to b36a5ab Compare February 25, 2025 01:39

reta force-pushed the issue-16633 branch from b36a5ab to d4f505e Compare February 25, 2025 02:12

reta force-pushed the issue-16633 branch from d4f505e to c26b3d4 Compare February 27, 2025 01:55

[Security Manager Replacement] Native Java Agent (dynamic code rewrit…

45bb092

…ing, must be low overhead) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]>

reta force-pushed the issue-16633 branch from c26b3d4 to 45bb092 Compare February 27, 2025 02:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[POC] [Security Manager Replacement] Native Java Agent (dynamic code rewriting, must be low overhead) #16731

[POC] [Security Manager Replacement] Native Java Agent (dynamic code rewriting, must be low overhead) #16731

reta commented Nov 27, 2024 •

edited

Loading

github-actions bot commented Nov 27, 2024

kumargu commented Nov 27, 2024

reta commented Nov 27, 2024

reta Dec 16, 2024

github-actions bot commented Dec 16, 2024

opensearch-trigger-bot bot commented Jan 16, 2025

github-actions bot commented Jan 16, 2025

github-actions bot commented Jan 21, 2025

github-actions bot commented Jan 22, 2025

github-actions bot commented Jan 22, 2025

kumargu commented Jan 28, 2025 •

edited

Loading

reta commented Jan 28, 2025

kumargu commented Jan 28, 2025

reta commented Jan 28, 2025

kumargu commented Jan 28, 2025

kumargu commented Feb 19, 2025

reta commented Feb 19, 2025

github-actions bot commented Feb 20, 2025

github-actions bot commented Feb 23, 2025

github-actions bot commented Feb 23, 2025

github-actions bot commented Feb 23, 2025

kumargu commented Feb 23, 2025 •

edited

Loading

reta commented Feb 23, 2025

github-actions bot commented Feb 23, 2025

github-actions bot commented Feb 23, 2025

github-actions bot commented Feb 24, 2025

kumargu commented Feb 24, 2025

github-actions bot commented Feb 24, 2025

github-actions bot commented Feb 25, 2025

github-actions bot commented Feb 25, 2025

github-actions bot commented Feb 27, 2025

github-actions bot commented Feb 27, 2025

[POC] [Security Manager Replacement] Native Java Agent (dynamic code rewriting, must be low overhead) #16731

Are you sure you want to change the base?

[POC] [Security Manager Replacement] Native Java Agent (dynamic code rewriting, must be low overhead) #16731

Conversation

reta commented Nov 27, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Nov 27, 2024

kumargu commented Nov 27, 2024

reta commented Nov 27, 2024

reta Dec 16, 2024

Choose a reason for hiding this comment

github-actions bot commented Dec 16, 2024

opensearch-trigger-bot bot commented Jan 16, 2025

github-actions bot commented Jan 16, 2025

github-actions bot commented Jan 21, 2025

github-actions bot commented Jan 22, 2025

github-actions bot commented Jan 22, 2025

kumargu commented Jan 28, 2025 • edited Loading

reta commented Jan 28, 2025

kumargu commented Jan 28, 2025

reta commented Jan 28, 2025

kumargu commented Jan 28, 2025

kumargu commented Feb 19, 2025

reta commented Feb 19, 2025

github-actions bot commented Feb 20, 2025

github-actions bot commented Feb 23, 2025

github-actions bot commented Feb 23, 2025

github-actions bot commented Feb 23, 2025

kumargu commented Feb 23, 2025 • edited Loading

A FileInterceptor Impl

High level changes in Agent

reta commented Feb 23, 2025

github-actions bot commented Feb 23, 2025

github-actions bot commented Feb 23, 2025

github-actions bot commented Feb 24, 2025

kumargu commented Feb 24, 2025

github-actions bot commented Feb 24, 2025

github-actions bot commented Feb 25, 2025

github-actions bot commented Feb 25, 2025

github-actions bot commented Feb 27, 2025

github-actions bot commented Feb 27, 2025

reta commented Nov 27, 2024 •

edited

Loading

kumargu commented Jan 28, 2025 •

edited

Loading

kumargu commented Feb 23, 2025 •

edited

Loading