Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bootRun: Could not locate executable null\bin\winutils.exe in the Hadoop binaries And throwing NullPointerException when initLocalMetastore() #545

Open
fafacao86 opened this issue Jan 9, 2025 · 2 comments
Open

bootRun: Could not locate executable null\bin\winutils.exe in the Hadoop binaries And throwing NullPointerException when initLocalMetastore() #545

fafacao86 opened this issue Jan 9, 2025 · 2 comments

Comments

@fafacao86
Copy link

I encountered some errors getting started with Coral. I'm using jdk8 and Python3.8.
Under coral-service, I executed: ../gardlew bootRun --args='--spring.profiles.active=localMetastore'
then follows the logs:

Task :coral-service:bootRun

. ____ _ __ _ _
/\ / ' __ _ () __ __ _ \ \ \
( ( )__ | '_ | '| | ' / ` | \ \ \
\/ )| |)| | | | | || (| | ) ) ) )
' || .__|| ||| |_, | / / / /
=========||==============|/=////
:: Spring Boot :: (v2.4.0)

2025-01-09 18:34:04.871 INFO 3216 --- [ main] c.l.c.c.CoralServiceApplication : Starting CoralServiceApplication using Java 1.8.0_291 on LAPTOP-2S6ONLIK with PID 3216 (E:\project\coral\coral-service\build\classes\java\main started by xh254 in E:\project\coral\coral-service)
2025-01-09 18:34:04.873 INFO 3216 --- [ main] c.l.c.c.CoralServiceApplication : The following profiles are active: localMetastore
2025-01-09 18:34:06.194 INFO 3216 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2025-01-09 18:34:06.204 INFO 3216 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2025-01-09 18:34:06.204 INFO 3216 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.39]
2025-01-09 18:34:06.337 INFO 3216 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2025-01-09 18:34:06.337 INFO 3216 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1330 ms
2025-01-09 18:34:06.474 INFO 3216 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor'
2025-01-09 18:34:06.557 WARN 3216 --- [ main] ion$DefaultTemplateResolverConfiguration : Cannot find template location: classpath:/templates/ (please add some templates or check your Thymeleaf configuration)
2025-01-09 18:34:06.614 INFO 3216 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2025-01-09 18:34:06.631 ERROR 3216 --- [ main] org.apache.hadoop.util.Shell : Failed to locate the winutils binary in the hadoop binary path

java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:356) [hadoop-common-2.7.0.jar:na]
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:371) [hadoop-common-2.7.0.jar:na]
at org.apache.hadoop.util.Shell.(Shell.java:364) [hadoop-common-2.7.0.jar:na]
at org.apache.hadoop.hive.conf.HiveConf$ConfVars.findHadoopBinary(HiveConf.java:2333) [hive-common-1.2.2.jar:1.2.2]
at org.apache.hadoop.hive.conf.HiveConf$ConfVars.(HiveConf.java:368) [hive-common-1.2.2.jar:1.2.2]
at org.apache.hadoop.hive.conf.HiveConf.(HiveConf.java:108) [hive-common-1.2.2.jar:1.2.2]
at com.linkedin.coral.coralservice.utils.CoralProvider.loadResourceHiveConf(CoralProvider.java:86) [main/:na]
at com.linkedin.coral.coralservice.utils.CoralProvider.initLocalMetastore(CoralProvider.java:53) [main/:na]
at com.linkedin.coral.coralservice.controller.TranslationControllerLocal.onApplicationEvent(TranslationControllerLocal.java:36) [main/:na]
at com.linkedin.coral.coralservice.controller.TranslationControllerLocal.onApplicationEvent(TranslationControllerLocal.java:27) [main/:na]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:203) [spring-context-5.3.1.jar:5.3.1]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:196) [spring-context-5.3.1.jar:5.3.1]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:161) [spring-context-5.3.1.jar:5.3.1]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:426) [spring-context-5.3.1.jar:5.3.1]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:383) [spring-context-5.3.1.jar:5.3.1]
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:945) [spring-context-5.3.1.jar:5.3.1]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:591) [spring-context-5.3.1.jar:5.3.1]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:144) [spring-boot-2.4.0.jar:2.4.0]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:767) [spring-boot-2.4.0.jar:2.4.0]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759) [spring-boot-2.4.0.jar:2.4.0]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:426) [spring-boot-2.4.0.jar:2.4.0]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:326) [spring-boot-2.4.0.jar:2.4.0]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1309) [spring-boot-2.4.0.jar:2.4.0]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1298) [spring-boot-2.4.0.jar:2.4.0]
at com.linkedin.coral.coralservice.CoralServiceApplication.main(CoralServiceApplication.java:19) [main/:na]

2025-01-09 18:34:06.798 INFO 3216 --- [ main] c.l.c.c.CoralServiceApplication : Temp Workspace: C:\Users\xh254\AppData\Local\Temp/coral/service/d1ac1ebb-91cc-40b4-8614-928923d6c76a
2025-01-09 18:34:06.871 WARN 3216 --- [ main] org.apache.hadoop.util.NativeCodeLoader : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2025-01-09 18:34:06.897 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.HiveMetaStore : 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
2025-01-09 18:34:06.941 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.ObjectStore : ObjectStore, initialize called
log4j:WARN No appenders could be found for logger (DataNucleus.General).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2025-01-09 18:34:07.814 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.ObjectStore : Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
2025-01-09 18:34:08.896 INFO 3216 --- [ main] o.a.h.hive.metastore.MetaStoreDirectSql : Using direct SQL, underlying DB is DERBY
2025-01-09 18:34:08.897 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.ObjectStore : Initialized ObjectStore
2025-01-09 18:34:08.953 WARN 3216 --- [ main] o.a.hadoop.hive.metastore.ObjectStore : Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
2025-01-09 18:34:09.044 WARN 3216 --- [ main] o.a.hadoop.hive.metastore.ObjectStore : Failed to get database default, returning NoSuchObjectException
2025-01-09 18:34:09.138 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.HiveMetaStore : Added admin role in metastore
2025-01-09 18:34:09.141 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.HiveMetaStore : Added public role in metastore
2025-01-09 18:34:09.189 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.HiveMetaStore : No user is added in admin role, since config is empty
2025-01-09 18:34:09.247 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.HiveMetaStore : 0: get_all_databases
2025-01-09 18:34:09.247 INFO 3216 --- [ main] o.a.h.h.metastore.HiveMetaStore.audit : ugi=xh254 ip=unknown-ip-addr cmd=get_all_databases
2025-01-09 18:34:09.258 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.HiveMetaStore : 0: get_functions: db=default pat=*
2025-01-09 18:34:09.259 INFO 3216 --- [ main] o.a.h.h.metastore.HiveMetaStore.audit : ugi=xh254 ip=unknown-ip-addr cmd=get_functions: db=default pat=*
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:466)
at com.linkedin.coral.coralservice.utils.CoralProvider.initLocalMetastore(CoralProvider.java:64)
at com.linkedin.coral.coralservice.controller.TranslationControllerLocal.onApplicationEvent(TranslationControllerLocal.java:36)
at com.linkedin.coral.coralservice.controller.TranslationControllerLocal.onApplicationEvent(TranslationControllerLocal.java:27)
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:203)
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:196)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:161)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:426)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:383)
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:945)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:591)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:144)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:767)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:426)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:326)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1309)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1298)
at com.linkedin.coral.coralservice.CoralServiceApplication.main(CoralServiceApplication.java:19)
Caused by: java.lang.NullPointerException
Caused by: java.lang.NullPointerException

at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:483)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:815)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:798)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:728)
at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:486)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:527)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:504)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:305)
at org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3678)
at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:597)
at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
... 19 more

2025-01-09 18:34:09.341 INFO 3216 --- [ main] c.l.c.c.CoralServiceApplication : Started CoralServiceApplication in 4.765 seconds (JVM running for 5.318)
@fafacao86
Copy link
Author

Solved: install Hadoop first, download winutils.exe corresponds to the Hadoop version, then copy the winutils.exe to HADOOP_HOME/bin. https://stackoverflow.com/questions/35652665/java-io-ioexception-could-not-locate-executable-null-bin-winutils-exe-in-the-ha

@fafacao86
Copy link
Author

fafacao86 commented Jan 9, 2025 •
edited
Loading

Could anyone please add this precautions to readme file? Make sure hadoop is properly set up if one is on Windows Platform.

@fafacao86 fafacao86 reopened this Jan 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@fafacao86