[BUG] OpenSearch security plugin unable to load after a pod is killed due to opensearch.yaml configuration. #5044
Description
What is the bug?
When a pod running OpenSearch is terminated by the operating system and restarted, the new pod is unable to load the OpenSearch Security plugin. We've found that this happens because OpenSearch uses the presence of plugins.security keys in the opensearch.yaml configuration file to determine whether the security plugin is configured. Which is still present after the process is killed.
When we delete the security.plugins entries in opensearch.yaml the pod is able to start up again with no errors.
In the logs we can see that opensearch.yaml is configured for Security.
Detected OpenSearch Version: 2.18.0 Detected OpenSearch Security Version: 2.18.0.0 /usr/share/opensearch/config/opensearch.yml seems to be already configured for Security. Quit.
And in the stacktraces we can see errors when loading the plugin:
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.18.0.jar:2.18.0]
...
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:805) ~[opensearch-2.18.0.jar:2.18.0]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
...
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
...
Caused by: org.opensearch.OpenSearchException: Unable to read the file root-ca.pem. Please make sure this files exists and is readable regarding to permissions
at org.opensearch.security.ssl.config.SslCertificatesLoader.resolvePath(SslCertificatesLoader.java:165) ~[?:?]
at org.opensearch.security.ssl.config.SslCertificatesLoader.loadConfiguration(SslCertificatesLoader.java:85) ~[?:?]
at org.opensearch.security.ssl.SslSettingsManager.loadConfigurations(SslSettingsManager.java:137) ~[?:?]
...
How can one reproduce the bug?
Steps to reproduce the behavior:
- Kill the process on the worker-node that is running the pod.
What is the expected behavior?
A pod should be able to load the security plugin even if its killed by the operating system.
What is your host/environment?
OpenSearch Version: 2.18.0
Do you have any additional context?
We are running a rke2 kubernetes cluster on ubuntu 24.04.
The issue was also present for us on version 2.15.0 of OpenSearch.